Want prefix margin support

[rzezeski]

At some point we will probably want the ability for an OPTE/xde client to negotiate a prefix margin for its outbound packets/mblks. Such a capability allows one to a) avoid additional allocation in the OPTE layer and b) OPTE could of course create its own mblk header pool; but by having the client create the margin we can keep everything in one mblk which may allow us to add additional optimizations down the stack.

For example, could we add an mblk flag to indicate to downstream consumers that all data is in one mblk and use optimized functions for any needed header parsing of their own? Specifically I'm thinking of things like mac_ether_offload_info() which have to greatly complicate their code to guard against chained packets. That said, why not go one step further and have OPTE provide this MEOI data by attaching it to the mblk somehow? That brings up a large discussion around mblk changes, but it's worth considering as it allows us to remove redundant parsing work.

[FelixMcFelix]

oxidecomputer/propolis#814 (and related illumos ticket) get us part of the way here, at least when delivering traffic from instances. We don't have a general mechanism to provide this for any upstream client, however.

#614 delivers on the idea of filling in MEOI and tunnel offsets as part of the dblk_t. This is very effective in buying us extra speed once we have send offloads (both csum AND LSO).