Auto shard

[mixmix]

This is WIP

Problem: the meta-feed spec describes where feeds should be located in nibble-shards. We want to save users from having to figure this out, so we need a nice methods for this

I wasn't sure what we want the final API of this module should be. I'd like to discuss this, but for the moment I've just put this method at a placeholder location ssb.metafeeds.v1.findOrCreate(details, cb)

I'll raise question I've encountered as comments in code

[mixmix]

TODO : export some test-vectors for sharding (to make it easier for Go / rust butts to align implementation)

[staltz]

I wasn't sure what we want the final API of this module should be. I'd like to discuss this

I also don't know what the final API should be. Might want to check @arj03 for ideas too.

One idea is to provide only a few APIs: findOrCreate (which is your new v1 findOrCreate) and getOrCreateRoot (name can be bikeshedded) and branchStream, and then tuck away all the other APIs under a lowlevel namespace or something like that. With the intention that the README only has the high level APIs, and we keep the low level APIs undocumented, sort of like a back stage access, that we can officialize if they seem important enough.

[staltz]

You know what, we would vastly simplify our lives if we dropped the possibility that a content leaf is a metafeed too. It would make implementing these changes in ssb-meta-feeds easier, and it would vastly simplify our upcoming changes to ssb-replication-scheduler.

To clarify, the metafeeds tree structure v1 spec allows this (focus on chess and tictactoe):

graph TB;
  root --> v1
  v1 --> 7 & b & f
  7 --> games
  games --> chess & tictactoe
  f --> posts
  b --> abouts

Loading

What if we would change the spec so that tree structure v1 is rigid, like this:

graph TB;
  root --> v1
  v1 --> 7 & b & f
  7 --> chess & tictactoe
  f --> posts
  b --> abouts

Loading

We can always just make tree structure v2 if we find a compelling case for that.

[staltz]

@mixmix Another thing: remove the package-lock.json that was added

[mixmix]

I like your suggestion about "flat" leaves. Then we can hide almost all the bullshit...... OH I remember I was keen to have the unique name !== purpose because I argued for "clustering" of feeds

e.g. I might want to have chess/games and chess/meetups be in the same shard, because:

• EITHER I'm a chess player, in which case i want
  • only the games feed
  • and MAYBE I want to upgrade to have the meetups for in person gathering of chess players
• OR I'm not a chess player in which case I want NEITHER feed.

So we could make unique name = purpose.split('/')[0]

I dunno....

[mixmix]

I like your suggestion about "flat" leaves. Then we can hide almost all the bullshit...... API which lets you make meta sub-feeds etc. and you really have to go looking to find those API

[staltz]

flat leaves

Yeah, this is going to require a PR to ssb-meta-feeds-spec.

[staltz]

I converted this PR to "draft" because it needs to be rebased on master.

[staltz]

Roger to go