Proposal: Changing the Header

[jamesmunns]

Currently, the defined header in postcard-rpc is:

• a [u8; 8] representing the Key of the request/response/publish
• a varint(u32) representing the sequence number of the request/response/publish

This is generally fine for USB, as the header takes up 9-13 bytes, still often allowing a whole message to fit in a 64 byte frame. However, this amount of overhead is less desirable on low-bandwidth transports, like over a radio, where less is more.

I have been unsure about shortening the key, as it is procedurally generated, and the goal is to practically never require users to think about the chance of a collision. However for many users with a small number of endpoints/topics, one or two bytes is likely enough to be unique. Ideally we'd be able to have devices choose which size they'd like to use at compile time, however we should not feature-gate this, as a host may speak via multiple transports with different settings (e.g. "full fat" USB and "lite" I2C on the same machine).

Additionally for devices with low throughput, a smaller sequence number range is also likely acceptable.

I'd propose the following new format for postcard-rpc headers:

• One "settings" byte (described below), which controls the length of the following fields
• 1-8 bytes for Key (NOT a varint)
• 1-4 bytes for Sequence Number (NOT a varint)

The settings byte would act as the following bitfield values:

• 2b: lengths
  • 0b00: 1B Key, 1B SN (3 bytes total)
  • 0b01: 2B Key, 2B SN (5 bytes total)
  • 0b10: 4B Key, 4B SN (9 bytes total)
  • 0b11: 8B Key, 4B SN (13 bytes total)
• 6b: Protocol Version
  • 2b: Major Version
    • 0b00: 1.x
    • 0b01: 2.x
    • 0b10: 3.x
    • 0b11: 4.x
  • 4b: Minor Version
    • 0b0000: x.0
    • 0b0001: x.1
    • ...
    • 0b1110: x.14
    • 0b1111: RESERVED

Alternative uses for the 6b other that protocol version:

• feature/format flags?
  • "has CRC" - whether messages have a CRC footer
• Message kind?
  • Endpoint Req, Endpoint Resp, Topic Pub, Forwarding?
  • Special "protocol" kinds like for negotiating link?
• HTTP-method-alike?
  • GET, POST, DELETE, etc?
• Multipart number? For E+T streaming responses?
• Key algorithm kind?

How it works

Right now, I have the following thoughts:

• The "servers" are the ones that pick what size they prefer
• All devices must support all sizes, but can negotiate a lower size
• Communication defaults to the max size
• For shrinking Keys:
  • XOR folding should be used
  • For 8 bytes, all bytes ABCDEFGH sent
  • For 4 bytes, ABCD ^ EFGH sent
  • For 2 bytes, AB ^ CD ^ EF ^ GH sent
  • For 1 byte, A ^ B ^ C ^ D ^ E ^ F ^ G ^ H sent
• For shrinking seqno:
  • Wrapping truncation should be used, e.g.:
  • For 4 bytes, all bytes sent little endian
  • For 2 bytes, % 65536
  • For 1 byte, % 256
  • TODO: make host client avoid large seq_no when establishing connections?

This would make keys still natively eight bytes, but the end-devices could choose which key to use as part of their define_dispatch! invocation, e.g.:

define_dispatch! {
    // Dispatcher type
    dispatcher: Dispatcher<...>;
    // Size config             // Options:
    header_size: TwoKeyTwoSeq, // OneKeyOneSeq, TwoKeyTwoSeq,
                               // FourKeyFourSeq, EightKeyFourSeq
    // Endpoints
    PingEndpoint => blocking ping_handler,
    GetUniqueIdEndpoint => blocking unique_id,
}

This will configure the Dispatcher and Sender to use the appropriate types, particularly for the const-time check for collisions (here).

The dispatch machinery will still require accepting messages with a longer-than-preferred key/seq, by shrinking as defined above. Perhaps we also carry the config of requests and use the same size as response?

Needs definition

• Do we want to use the 6 header bits for something other than version?
• How to "negotiate" link size?
  • Accept all, prefer specific?
  • Special negotiation sequence?
  • Accept all, respond specific? (this would be tricky when listening for (key, seq) response

[CBJamo]

What all is the seqno used for? A system might have a small number of message types, but send them frequently. A use case we currently have is sending led commands, which are sometimes animations sent at 30hz and run for hours. That's on an i2c transport.

If the seqno is only used to disambiguate in-flight requests, then that application would only need a u8 (probably only a single bit, actually). It also suggests that it might be fine to always have a u8 seqno? Or perhaps each side could keep an internal seqno of usize but only send a u8 on the wire, handling rollovers so it appears as a usize to the user.

[jamesmunns]

Yes, sequence number is used to disambiguate in-flight requests. It is possible that you have multiple in flight requests at once. I would agree it is not very likely that you have >256 in flight.

[jamesmunns]

Merged in #53