Networking layers

After a couple of rewrites of the networking layers to support the distributed case, I finally have a setup that I can live with for some period of time.

There are three layers to the way we handle networking on the server side:

  • Logical layer: This is implemented with go channels, so when waiting for a call from the network, you use a channel to block on.
  • Netio layer: This layer implements a simple network protocol for sending and receiving bundles. A bundle is a protobuf-encoded blob of content. This protocol has a 8 byte preamble and a 4 byte crc at the end.
  • Quic layer: This is the networking that actually talks to the wire or its representatives. This is implemented primarily through goroutines that block on channels, the same ones that are visible in the logical layer.

As an example, consider an incoming request, an RPC, from some client across the network. This call is initially fielded by the quic layer as a connection, and a “stream” in quic parlance. These are accepted indepedently by the quic layer. Once a stream has been established to the calling program, the quic layer then uses the netio layer to read a bundle from the caller. Once the bundle has been read successfully, its contents are pushed through a channel to the logical layer. In addition to the content, also pushed to the logical layer is a response channel. The quic layer blocks on that channel waiting for the logical layer to send a response. In both directions, the data (content) is in the form of an anypb.Any that can hold any protobuf type. Once the quic layer has received the result through the response channel, it uses the netio layer to send a response over the wire.

To call a remote service, the entire process is the same but with the directional sense reversed. In the server case, the quic layer accepted the remote data then pushed that data to a blocked goroutine in the logical layer. In the client case, the quic layer blocks on a channel and the logical layer pushes an anypb.Any through the channel as well as a response channel. This is the “input” to a remote procedure call. The quic layer implements the call, and then responds through the response channel to the blocked logical layer.

At the moment we use channels to implement a serialization of the requests on user services, so there is no way for concurrent calls to user services. This is likely to be relaxed later, but I’m trying to get the simple case working first.

For now, we are being pretty aggressive about closing streams that are established to remote clients or servers. We are doing this now to try to insure “sync” across the connection, even though it implies that frequently clients must reconnect to the server to make progress. This, I hope, will be mitigated in the future by the use of the RTT0 reconnects that is offered by quic.