Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto shard #69

Merged
merged 9 commits into from
Sep 27, 2022
Merged

Auto shard #69

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
dad1d53
add persistence tests
mixmix Sep 27, 2022
8a05f99
update the README
mixmix Sep 27, 2022
90f8183
Squashed commit of the following:
mixmix Sep 27, 2022
e1e33b4
make isFeed with deepEqual
mixmix Sep 27, 2022
dadb9b9
Add CC0 header to new README_ADVANCED.md
mixmix Sep 27, 2022
db369dd
Merge branch 'master' into auto-shard
staltz Sep 27, 2022
a7589f6
Update api.js
staltz Sep 27, 2022
3bab0d7
Update test/api.test.js
staltz Sep 27, 2022
49a96c0
Apply suggestions from code review
staltz Sep 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
# SPDX-License-Identifier: Unlicense

node_modules
package-lock.json
pnpm-lock.yaml
coverage
.nyc_output
.nyc_output
252 changes: 63 additions & 189 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ main

staltz marked this conversation as resolved.
Show resolved Hide resolved
classDef default fill:#3f506c,stroke:#3f506c,color:#fff;
```
How "classic" scuttlebutt worked - each device has one `main` feed with all messages
_How "classic" scuttlebutt worked - each device has one `main` feed with all messages_

```mermaid
graph TB
Expand All @@ -37,19 +37,20 @@ classDef root fill:#8338ec,stroke:#8338ec,color:#fff;
classDef default fill:#3a86ff,stroke:#3a86ff,color:#fff;
classDef legacy fill:#3f506c,stroke:#3f506c,color:#fff;
```
How scuttlebutt works with metafeeds - each device now has a `root` metafeed,
_How scuttlebutt works with metafeeds - each device now has a `root` metafeed,
whose sole responsibility is to announce (point to) subfeeds that you publish
content to. A subfeed can also be a metafeed, which then allows the existence of
"sub-subfeeds".
"sub-subfeeds"._

This means that when you first meet a peer you can replicate their `root` metafeed
and, having discovered their subfeeds, replicate just their `aboutMe` and `contacts` feeds
to get enough info to place them socially. Once you decide you want to follow them you may
replicate their other subfeeds.
This means that when you first meet a peer you can replicate their `root`
metafeed and, having discovered their subfeeds, replicate just their `aboutMe`
and `contacts` feeds to get enough info to place them socially. Once you decide
you want to follow them you may replicate their other subfeeds.

_NOTE: The ideal state is that all content is split out into subfeeds.
To add backwards compatability for devices that have already posted a lot of posts to their
classic `main` feed, this library will auto-link that main feed in as a "subfeed" of our root._
_NOTE: The ideal state is that all content is split out into subfeeds. To add
backwards compatability for devices that have already posted a lot of posts to
their classic `main` feed, this library will auto-link that main feed in as a
"subfeed" of our root._

## Installation

Expand All @@ -74,117 +75,59 @@ Add this plugin like this:

## Example usage

Let's start by creating a **root metafeed** with `findOrCreate(cb)`, necessary
for using this module. There can only be one _root_ metafeed, so even if you
call `findOrCreate(cb)` many times, it will not create duplicates, it will just
load the root metafeed.
We create a subfeed for `about` messages under our `root` feed using
`findOrCreate`. This will only create the subfeed if there is no existing
subfeed that matches the criteria.

```js
sbot.metafeeds.findOrCreate((err, root) => {
// ...
})
```
const details = { feedpurpose: 'aboutMe' }
sbot.metafeeds.findOrCreate(details, (err, aboutMeFeed) => {
console.log(aboutMeFeed)

Now we create a subfeed for `about` messages under our `root` feed using
`findOrCreate(parent, isFeed, details, cb)`. This will only create the subfeed
if there is no existing subfeed that matches the criteria`isFeed`:

```js
sbot.metafeeds.findOrCreate((err, root) => {
// Find an existing feed
const isFeed = (feed) => feed.feedpurpose === 'aboutMe'
// Details for creating a subfeed if it doesn't already exist:
const details = { feedpurpose: 'aboutMe', feedformat: 'classic' }

sbot.metafeeds.findOrCreate(root, isFeed, details, (err, aboutMeFeed) => {
// ...
})
//
})
```

The `details` argument is a *FeedDetails* object and it always requires
`feedpurpose` (any string) and `feedformat` (either `'classic'` for normal
ed25519 SSB feeds, or `'bendybutt-v1'` if you want a metafeed). The `isFeed`
function also takes a *FeedDetails* object as input.
The `details` argument is an object used to find (or create) a subfeed under
your "root feed". (It actually nests it under a couple of subfeeds, to handle
versioning, and sparse replication, but you generally don't need to know the
details).

Once you have a *FeedDetails* object, like `aboutMeFeed`, you can publish on
the new subfeed:

```js
sbot.metafeeds.findOrCreate((err, root) => {
const isFeed = (feed) => feed.feedpurpose === 'aboutMe'
const details = { feedpurpose: 'aboutMe', feedformat: 'classic' }

sbot.metafeeds.findOrCreate(root, isFeed, details, (err, aboutMeFeed) => {
const content = {
type: 'about',
name: 'baba yaga'
description: 'lives in a hutt in the forest, swing by sometime!'
}
sbot.db.publishAs(aboutMeFeed.keys, content, (err, msg) => {
// ...
})
const details = { feedpurpose: 'aboutMe' }
sbot.metafeeds.findOrCreate(details, (err, aboutMeFeed) => {
console.log(aboutMeFeed)

const content = {
type: 'about',
name: 'baba yaga'
description: 'lives in a hutt in the forest, swing by sometime!'
}
sbot.db.create({ keys: aboutMeFeed.keys, content }, (err, msg) => {
console.log(msg)
})
})
```

## API

### `sbot.metafeeds.findOrCreate(cb)`

Calls back with your `root` Metafeed object which has the form:

```js
{
metafeed: null,
subfeed: 'ssb:feed/bendybutt-v1/sxK3OnHxdo7yGZ-28HrgpVq8nRBFaOCEGjRE4nB7CO8=',
feedpurpose: 'root',
feedformat: 'bendybutt-v1',
seed: <Buffer 13 10 25 ab e3 37 20 57 19 0a 1d e4 64 13 e7 38 d2 23 11 48 7d 13 e6 3b 8f ef 72 92 7f db 96 64>,
keys: {
curve: 'ed25519',
public: 'sxK3OnHxdo7yGZ+28HrgpVq8nRBFaOCEGjRE4nB7CO8=.ed25519',
private: 'SOEx7hA9vRHrli0PZwNJ8jijH+PShmlrzz/JAKI7v6SzErc6cfF2jvIZn7bweuClWrydEEVo4IQaNETicHsI7w==.ed25519',
id: 'ssb:feed/bendybutt-v1/sxK3OnHxdo7yGZ-28HrgpVq8nRBFaOCEGjRE4nB7CO8='
},
metadata: {}
```

Meaning:
- `metafeed` - the id of the feed this is underneath. As this is the topmost feed, this is empty
- `subfeed` - the id of this feed, same as `keys.id`
- `feedpurpose` - a human readable ideally unique handle for this feed
- `feedformat` - the feed format ("classic" or "bendybutt-v1" are current options)
- `seed` - the data from which is use to derive the `keys` and `id` of this feed.
- `keys` - cryptographic keys used for signing messages published by this feed (see [ssb-keys])
- `metadata` - additional data

NOTES:
- the `root` metafeed is unique - you have only one, and it has no metafeed (it's at the top!)
- if you have a legacy `main` feed, this will also set that up as a subfeed of your `root` feed.

### `sbot.metafeeds.findOrCreate(metafeed, isFeed, details, cb)`

Looks for the first subfeed of `metafeed` that satisfies the condition in `isFeed`,
or creates it matching the properties in `details`.
### `sbot.metafeeds.findOrCreate(details, cb)`

This is strictly concerned with meta feeds and sub feeds that **you own**, not
with those that belong to other peers.
Looks for the first subfeed of `metafeed` that matches `details`, or creates
one which matches these. This creates feeds following the
[v1 tree structure](https://github.com/ssbc/ssb-meta-feeds-spec#v1).

Arguments:
- `metafeed` - the metafeed you are finding/ creating under, can be:
- *FeedDetails* object (as returned by `findOrCreate()` or `getRoot()`)
- *null* which is short-hand for the `rootFeed` (this will be created if doesn't exist)
- `isFeed` - method you use to find an existing *FeedDetails*, can be:
- *function* of shape `(FeedDetails) => boolean`
- *null* - this method will then return an arbitrary subfeed under provided `metafeed`
- `details` - used to create a new subfeed if a match for an existing one is not found, can be
- *Object*:
- `details.feedpurpose` *String* any string to characterize the purpose of this new subfeed
- `details.feedformat` *String* either `'classic'` or `'bendybutt-v1'`
- `details.metadata` *Object* (optional) - for containing other data
- if `details.metadata.recps` is used, the subfeed announcement will be encrypted
- *null* - only allowed if `metafeed` is null (i.e. the details of the `root` FeedDetails)
- `details` *Object* where
- `details.feedpurpose` *String* any string to characterize the purpose of this new subfeed
- `details.feedformat` *String* (optional)
- either `'classic'` or `'bendybutt-v1'`
- default: `'classic'`
mixmix marked this conversation as resolved.
Show resolved Hide resolved
- `details.metadata` *Object* (optional) - for containing other data
- if `details.metadata.recps` is used, the subfeed announcement will be encrypted
- `cb` *function* delivers the response, has signature `(err, FeedDetails)`, where FeedDetails is
```js
{
Expand All @@ -206,34 +149,29 @@ Arguments:
}
```

### `sbot.metafeeds.findById(feedId, cb)`
Meaning:
- `metafeed` - the id of the feed this is underneath
- `subfeed` - the id of this feed, same as `keys.id`
- `feedpurpose` - a human readable ideally unique handle for this feed
- `feedformat` - the feed format ("classic" or "bendybutt-v1" are current options)
- `seed` - the data from which is use to derive the `keys` and `id` of this feed.
- `keys` - cryptographic keys used for signing messages published by this feed (see [ssb-keys])
- `metadata` - object containing additional data

Given a `feedId` that is presumed to be a subfeed of some meta feed, this API
fetches the *Details* object describing that feed, which is of form:
NOTES:
- if you have a legacy `main` feed, this will also set that up as a subfeed of your `root` feed.

```js
{
metafeed,
feedpurpose,
feedformat,
id,
// seed
// keys
metadata
}
```

NOTE - may include `seed`, `keys` if this is one of your feeds.
### `sbot.metafeeds.findOrCreate(cb)`

### `sbot.metafeeds.findByIdSync(feedId)`
Fetches the **root metafeed** details. There can only be one _root_ metafeed,
so even if you call `findOrCreate(cb)` many times, it will not create duplicates,
it will just load the root metafeed.

Similar to `findById`, but returns synchronously. :warning: Note, in order to
use this API, you **must** call `sbot.metafeeds.loadState(cb)` first, and wait
for `cb` to be called.
Callsback with your `root` FeedDetails object (see `findOrCreate(details, cb)`)

You can also call `sbot.metafeeds.ensureLoaded(feedId, cb)` on an individual
basis to make sure that `findByIdSync` will operate at the correct time when the
`feedId`'s metadata has been processed in the local database.
NOTES:
- `metafeed = null` - the root metafeed is the topmost metafeed

### `sbot.metafeeds.branchStream(opts)`

Expand Down Expand Up @@ -296,73 +234,9 @@ The `opts` argument can have the following properties:
in the results; if `true`, only tombstoned branches are included; if `null`,
all branches are included regardless of tombstoning. (Default: `null`)

### Advanced API

### `sbot.metafeeds.findAndTombstone(metafeed, isFeed, reason, cb)`

_Looks for the first subfeed of `metafeed` that satisfies the condition in
`isFeed` and, if found, tombstones it with the string `reason`.

This is strictly concerned with meta feeds and sub feeds that **you own**, not
with those that belong to other peers.

Arguments:
- `metafeed` *FeedDetails* object (as returned by e.g. `findOrCreate()`, `getRoot()`).
- `isFeed` *function* of the shape `(FeedDetails) => Boolean`.
- `reason` *String* - describes why the found feed is being tombstoned.

The callback is called with `true` on the 2nd argument if tombstoning suceeded,
or called with an error object on the 1st argument if it failed.


### `sbot.metafeeds.getRoot(cb)`

Looks for the root meta feed declared by your main feed, and returns it (as
`{ seed, keys}`) via the callback `cb` if it exists.

If it does not exist, this API will **not** create the root meta feed.


## Validation

Exposed via the internal API.

### `isValid(msg, hmacKey)`

_Validate a single meta feed message._

Extracts the `contentSection` from the given `msg` object and calls
`validateSingle()` to perform validation checks.

If provided, the `hmacKey` is also given as input to the `validateSingle()`
function call. `hmacKey` may be `null` or a valid HMAC key supplied as a
`Buffer` or `string`.

The response is a boolean: `true` if validation is successful, `false` if
validation fails in any way. Note that this function does not return the
underlying cause of the validation failure.

### `validateSingle(contentSection, hmacKey)`

_Validate a single meta feed message `contentSection` according to the criteria
defined in the [specification](https://github.com/ssb-ngi-pointer/ssb-meta-feed-spec#usage-of-bendy-butt-feed-format)._

`contentSection` must be an array of `content` and `contentSignature`. If a
`string` is provided (representing an encrypted message, for instance) an error
will be returned; an encrypted `contentSection` cannot be validated.

`hmacKey` may be `null` or a valid HMAC key supplied as a `Buffer` or `string`.

The response will be `undefined` (for successful validation) or an `Error`
object with a `message` describing the error.

### `validateMetafeedAnnounce(msg)`

_Validates a `metafeed/announce` message expected to be published on "main"
feeds which are in the classic format, but are signed by a meta feed according
to the [ssb meta feed spec]._

The response will be `undefined` (for successful validation) or an `Error`
object with a `message` describing the error.
For lower level API docs, [see here](./README_ADVANCED.md).

## License

Expand Down
Loading