Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Rework basically the entire protocol #53

Merged
merged 32 commits into from
Oct 27, 2024
Merged

Conversation

jamesmunns
Copy link
Owner

@jamesmunns jamesmunns commented Oct 22, 2024

This is a major change to postcard-rpc. This is a very breaking wire change!

In particular, this PR changes the following:

Server Rework

Signifcantly reworks how "servers" are implemented, removing the previous embassy-usb-specific one in favor of more reusable parts, that allow for implementing a server generically over different transports. This new version still has an implementation for embassy-usb 0.3, but ALSO provides a channel-based implementation for testing, and I am likely to port a TCP-based one I have in poststation as well. This unlocks the ability to reuse the bulk of the existing code for supporting other transports, like UART, SPI, I2C, Ethernet, or even over radio.

define_dispatch macro rework

As part of the Server Rework, I also mostly rewrote the define_dispatch! macro, which now can be used with ANY transport, not just embassy-usb.

This change also now allows servers to define topic handlers, so incoming published messages can be dispatched similar to endpoint dispatching. CC #15

Automatic Key Shrinking

Previously, we would always send the full 8-byte "hash of the path and schema" ID in every message, as well as checking at compile time whether there is a collision or not.

This PR takes that up a notch, now calculating the smallest hash key we can use (1, 2, 4, or 8 bytes) automatically at compile time, and uses that as our "native" mapping. This is similar to the concept of "Perfect Hash Functions".

Variable Sequence Number size

Additionally, the client can now send sequence numbers of 1, 2, or 4 bytes. Previously, sequence numbers were a varint(u32). Servers will always respond back with the same sequence number they received when replying to requests.

Completely redo message headers

As we now have variable sized keys and sequence numbers, headers can now scale dynamically to the necessary size. CC #51

Before, we had 8 byte keys (fixed) and 1-5 bytes (varint(u32)), meaning headers were between 9-13 bytes.

Now, we use one byte as a discriminant, containing the key and seqno len, as well as a 4-bit version field, as well as the variable key and variable sequence number. This now means that headers are 3-13 bytes (1B discriminant + 1/2/4/8B key + 1/2/4B sequence number), and will be 3 bytes in many common cases where it is not necessary to disambiguate more than 256 in-flight messages (via sequence numbers) or 256 endpoints (though the liklihood of having a collision at 8 bits is higher than that due to the birthday problem).

When a Client first connects to a Server, it will always start by sending an 8B key. If the Server replies with a shorter key, the Client will then switch to using keys of that size. It is not necessary to ever hardcode what size keys are necessary, as this is calculated when the Server is compiled, and is automatically detected by the Client.

In general:

  • The CLIENT (usually the PC) is in charge of picking the size of the sequence number
  • The SERVER (usually the MCU) is in charge of picking the size of the message keys

@jamesmunns jamesmunns merged commit 5d0ab79 into main Oct 27, 2024
1 check passed
@jamesmunns jamesmunns deleted the james/rework-server-apis branch October 27, 2024 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant