Scalability Protocols

(nanomsg, nng, mangos, and other critters)
February 13, 2018

Garrett D'Amore, Staysail Systems, Inc.

About Me

  • Garrett D’Amore

  • CTO & Founder, Staysail Systems, Inc.

  • Founder of illumos

  • BDFL nanomsg & creator of nng, mangos

Scalability Protocols

  • Light-weight messaging layer

  • Solves recurring messaging problems

  • Liberally licensed

  • Embeddable & Portable (ANSI C)

  • Message-oriented

  • Brokerless


  • ZeroMQ

  • RabbitMQ

  • AMQP

  • MQTT


  • BSD sockets

  • Others…​


Much has been written here. Biggest turn offs to ZeroMQ are license and language, but some philosophical differences too.

See for some of Martin’s orgiinal thoughts.


  • RabbitMQ, AMQP require a broker, and a runtime (Java).


MQTT is newer than nanomsg or ZeroMQ, and is intended for IoT use cases. It lacks most of the flexibility and power, and is a very simple PUB/SUB model (generally over TCP).


REST (and HTTP) expreses the simple RPC style (REQ/REP) paradigm well. It does not support other patterns, and lacks any notion of automatic retry, reconnect, etc.

HTTP can be useful with proxies to load-balance, etc. however.

SP Implementations


  • Developer friendly

  • Administrator friendly

  • Business friendly


  • Brokerless

  • Lightweight

  • Securable

  • Reliable

  • Performant

  • Scalable

Other Buzzwords

  • Open Source

  • Observable (TBD)

  • Composable

  • Separation of Code & Config

Broker vs. Brokerless


Separate Daemon



Runtime Requirements



Persistent State



Database Required



Extra Administration




If you absolutely need persistent state, a broker is required. For all other cases, brokerless is better.


  • ANSI C

    • Except mangos (golang) and scaporust (rust)

    • nng uses C99

  • Minimal optional external dependencies

  • Zero required external dependencies

  • Minimizable (nng only)

Optional Dependencies

For nng only:


For TLS and wss transports. Apache licensed.


For zt transport; GPL, dev branch only


Every transport and every protocol can be disabled and omitted from the library. Furthermore, vanilla HTTP and TLS support may be removed.

Developer Friendly

  • Started with familiar POSIX API (send/recv, file descriptor based)

    • Even POSIX API is sometimes inconvenient.

    • Socket options style API for tunables.

  • Free developer from managing connections, reconnects, retries, etc.

  • Addresses are just URLs

Administrator Friendly

  • Brokerless!

  • No external runtime or other dependencies

    • Except for optional ZeroTier or mbedTLS (nng)

  • Device API supports proxies, concentrators, etc.

  • URL configuration for endpoint addresses

    • Some transports need extra config

  • Designed for scalability

Business Friendly

  • Liberal (MIT) license

  • mbedTLS dependency uses Apache license

  • No other dependencies

  • Commercial support available

  • Commercially sponsored

Securable (nng)

  • TLSv1.2 support availble (both TCP and websocket)

  • ZeroTier (!) for nng

  • Hardened against DoS and other protocol attacks

  • Different transport options provide flexibility

  • Device proxies

    • Hardened exterior

    • Inspectable interior


  • Never assert except on gross programmer error

  • All system call return codes checked

  • Built-in retries & reconnects

  • Brokerless (no broker to fail)

  • Multipath capable


  • Optimal use of OS facilities

  • Tunables (throughput vs latency)

  • Nagle (on or off)

  • Buffer loaning to reduce data copies

  • Minimal use of extra system calls

  • No premature optimization


  • Thread scaling to utilize multiple cores

  • Designed to support system aware pollers (kqueue, epoll, IOCP, etc.)

  • C10K capable (tested) (nng, mangos)

  • Device framework (horizontal scaling)

  • nng already outperforms libnanomsg (Win32)


  • TCP

  • IPC

  • inproc

  • websocket (with TLS on mangos and nng)

  • TLS (mangos and nng only)

  • ZeroTier (nng only)

  • QUIC (3rd party developed, mangos only)

  • UDP, KCP…​. others proposed

Patterns (Protocols)

  • Req/Rep

  • Pub/Sub

  • Bus

  • Pipeline (Push/Pull)

  • Survey

  • Pair

  • Polyamorous Pair (nng only)

  • Star (mangos only)

Composable Architecture

  • "Device" layer

  • Acts like a proxy

  • Can cross transport boundaries

  • Can be concentrator

  • Transparent to applications

History Lesson (101)

  • BSD sockets begat ZeroMQ

  • ZeroMQ begat nanomsg

  • nanomsg begat mangos

  • mangos begat nng

History (301)

  • ZeroMQ’s use of GPL and C++ deemed distasteful.

  • ZeroMQ’s lack of thread-safety an inhibitor.

  • Martin Sustrik and Peter Hintjens had a falling out.

PostModern History

  • Garrett wrote mangos

  • Martin lost interest

  • Garrett started maintaining

  • Garrett stepped away …​ for a while

  • Garrett took back over as BDFL

  • Garrett started the nng effort


  • mangos written to fill need for illumos/golang

  • project to learn nanomsg protocols and golang

  • over time became more featureful than nanomsg itself

Problems with nanomsg

  • FSMs run amok

  • Not "easily" extensible

  • Followed POSIX API "mistakes"

  • Suboptimal scaling

    • Single global processing thread

    • Hard-coded limits (e.g. on number of sockets)

  • Extra system calls

  • Not coded for production use

Enter nng

  • nanomsg-next-gen

  • Inspired by work on mangos

  • Started as fully thread-based design

  • Discovered that user threads scale poorly ~everywhere

  • Redesigned backend upon new asynchronous I/O framework modeled on Windows

  • Broke away from slavish adherence to POSIX API


  • All implementations are wire compatible (modulo specific features)

  • Protocols backed by "RFCs"

  • nng offers both legacy compat API & "modern" API

  • On some platforms, even ABI compatibility

  • Design goal is to support an entire ecosystem

Reliable By Design (nng)

  • Never assert on anything other than gross programmer error

  • Hardened against malformed wire packets

  • Protections against DoS (e.g. overallocation)

  • Tunable limits for "interior" use cases

Embeddability (nng)

  • Minimal dependencies

  • Clearly defined portability layer

  • Needs "thread" creation, locking, networking, and time APIs

  • Design supports coroutine based approach for "platforms"

  • Requires C99

  • VxWorks, FreeRTOS, etc. ports planned

ZeroTier Transport (nng)

  • Uses VL2 only (No IP addresses!)

  • Built on dev branch (libzerotiercore.a)

  • Supports persistent and ephemeral nodes.

  • Provides platform services (improved portability)

  • Bottom half uses UDPv4 and v6

  • Apps can resuse node multiple times

  • Apps can join multiple nodes & networks

  • Supports federations (moons)

ZeroTier Packet Format

Header for all ZeroTier transport frames:

zerotier0 header

ZeroTier Data Frames

This is the payload for DATA frames.

zerotier format

ZeroTier Connection Frames

User data for connetion commands:

zerotier0 conn

Sample ZeroTier Dialer

nng_zt_register(); // Register ZT transport
nng_dial(sock, "zt://fedcba9876.a09acf02337b057b:999"); (1)
1Port 999, node 0xfedcba9876, network a09acf02337b057b

WebSocket Support

  • Original implementation is toylike

    • No support for TLS

    • Only one server per TCP port (no port sharing)

    • Exclusive to SP use

WebSocket Support

  • New implementation is full featured

    • Inspired by mangos

    • TLS support

    • Full HTTP server and client

    • Multiple sockets can share a port (path discrimination)

    • Can support arbitrary headers, etc.

HTTP Server Support

  • Server supports pluggable handlers

    • Handlers for file, dir, and static content

    • Handler can "hijack" connection

  • HTTP/1.0 and HTTP/1.1 only

  • Pluggable transport (TCP, TLSv1.2, etc.)

  • Written for WebSocket, but "public" API

  • Probably only useful for C and C++ apps.

HTTP Client Support

  • Lower-level API support only at this point

  • HTTP/1.x only

  • No "convenience" methods (yet)

  • No auto support for proxies, redirects, etc.

  • And yet still can easily build all that on top

  • HTTP client auth (TLS) supported


  • RFCs

  • man pages

  • examples

New nng capabilities

  • TLS

  • rich WebSocket

  • Rich HTTP

  • ZeroTier

  • Binding to port 0

  • Peer/pipe properties

  • Polyamorous/PairV1

Test Suite

  • Uses CMake/CTest

  • Custom framwork inspired by GoConvey

  • Legacy tests imported for compat

  • Performance and scaling tests

  • Integrated with CI

Project Status

  • nanomsg - actively supported, sustaining

  • mangos - actively supported

  • nng - beta ready, actively developed

Future Directions

  • Docs & Examples

  • Statistics & Observability

  • Transports (UDP, QUIC, KCP, TIPC, RoCE, SSH?)

  • Pure JavaScript implementation

  • ZeroMQ interop?

  • HTTP enhancements

  • Other TLS backends? (libressl? bearssl?)

  • API enhancements

  • Performance

Thank You