Skip to content

Commit

Permalink
Merge pull request #9 from arj03/ssb-db2
Browse files Browse the repository at this point in the history 
Ssb db2
  • Loading branch information
@arj03
arj03 authored Jan 5, 2021
2 parents bd4c7a9 + eefed65 commit 62373cd
Show file tree
Hide file tree
Showing 24 changed files with 2,003 additions and 2,537 deletions.
134 changes: 26 additions & 108 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@ not limited to, of course). The key of your feed is stored in the
browser together with the log, indexes and smaller images. To reduce
storage and network requirements, partial replication has been
implemented. Wasm is used for crypto and is around 90% the speed of
the C implementation. A WebSocket is used to connect to pubs. The
`bundle-core.js` file in dist/ is roughly 2mb.
the C implementation. A WebSocket is used to connect to pubs or
rooms. The `bundle-core.js` file in dist/ is roughly 2mb.

Replication in the browser is quite a bit slower than in node, around
4-5x. There doesn't seem to be a single cause, it appears to be all
the diferrent layers that are [slower]: end-to-end encryption,
database write etc.
2x. There doesn't seem to be a single cause, it appears to be all the
diferrent layers that are [slower]: end-to-end encryption, database
write etc.

SSB conn is used for connections and rooms are supported. Partial
replication is implemented which allows two connected browsers to do a
Expand All @@ -25,22 +25,25 @@ battery.

![Diagram](./diagram.svg)

Boxes represent modules, some internal to browser-core and some
external. Ellipses in gray represents overall areas and are thus not
modules.

<details>
3`graphviz
digraph hierarchy {
nodesep=0.6 node [shape=record];

nodesep=0.6
node [shape=record];

{ rank=same SSBBrowserCore Validate Keys }
{ rank=same SSBBrowserCore SecretStack MuxRPC }

SSBBrowserCore->{Network Connections Sync DB Feed}
Feed->{Validate Keys}
DB->{JITDB AsyncFlumelog Indexes}
Connections->{SSBconn Rooms}
Network->{SecretStack MuxRPC SHS}
Sync->{Partial EBT Blobs}
}
{ rank=same SSBBrowserCore Validate Keys }
{ rank=same SSBBrowserCore SecretStack MuxRPC }

Network [shape=ellipse style=filled]
Connections [shape=ellipse style=filled]
Sync [shape=ellipse style=filled]
Feed [shape=ellipse style=filled]

SSBBrowserCore->{Network Connections Sync SSBDB2 Feed} Feed->{Validate Keys} Connections->{SSBconn Rooms} Network->{SecretStack MuxRPC SecretHandshake} Sync->{FeedSyncer EBT Blobs} SSBDB2->{Indexes JITDB AsyncAppendOnlyLog } }
3`
</details>

# api
Expand All @@ -52,6 +55,9 @@ The api is not meant to be 100% compatible with regular
ssb-db. Overall there are two major parts: [`db`](#db) and
[`net`](#net).

I highly recommend looking at [ssb-browser-demo] for an example of how
this library can be used to build applications.

# config

Loading the bundle-core file as above will use `browser.js`, meaning
Expand All @@ -76,72 +82,7 @@ Default is 1.

## db

### get(id, cb)

Will get a message with `id` from the database. If the message is not
found an err will be returned.

### getSync(id, cb)

Same as `get` except this method will wait for the indexes to be in
sync with the main log.

### validateAndAdd(msg, cb)

Validate a raw message (without id and timestamp), meaning if its the
first message from the feed, validate it without the previous pointer
otherwise it has to be the next message for the feed. Callback is the
stored message (id, timestamp, value = original message) or err.

### validateAndAddOOO(msg, cb)

Works the same way as validateAndAdd, expect that it will always do
validate without the previous pointer, meaning it can be used to
insert out of order messages from the feed.

### add(msg, cb)

Add a raw message (without id and timestamp) to the database. Callback
is the stored message (id, timestamp, value = original message) or
err.

### get(key, cb)

Get a message based on the key. Callback is the stored message (id,
timestamp, value = original message) or err.

### del(key, cb)

Remove a message from the database. Please note that this can create
problems with replication, in that a remote peer that does not have
this message will not be able to get this messages and any message
that comes afterwards.

### deleteFeed(feedId, cb)

Delete all messages for a particular feed and removes any state
associated with the feed.

### getStatus()

Gets the current db status, same functionality as
[db.status](https://github.com/ssbc/ssb-db#dbstatus) in ssb-db.

### jitdb

Returns a [jitdb] instance of the database useful for queries.

### onDrain(cb)

Will cb when all outstanding writes for the log has been written to storage.

### getLatest(feedId, cb)

Returns the latest state ({ id (msg key), sequence, timestamp }) for a feedId.

### getDataFromAuthorSequence()

Internal method for EBT.
This is the [ssb-db2] module.

### contacts

Expand All @@ -154,16 +95,6 @@ object of: following, blocking and extended given the feed.

Returns the profiles index.

### getMessagesByRoot(key, cb)

Returns all the messages for a particular root in sorted order.

### getMessagesByMention(key, cb)

Returns a sorted array messages that has a particular key in the
mentions array. This is useful for notifications for a particular
feed.

## net

This is the [secret-stack] module with a few extra modules
Expand Down Expand Up @@ -262,19 +193,6 @@ Other things directly on the global SSB object

The path to where the database and blobs are stored.

### validate

The [ssb-validate] module.

### state

The current [state](https://github.com/ssbc/ssb-validate#state) of
known feeds.

### publish(msg, cb)

Validates a message and stores it in the database. See db.add for format.

### getPeer()

Gets one of the connected peers that is not a room server.
Expand Down Expand Up @@ -324,10 +242,10 @@ For a smaller bundle file you can apply (patch -p0 < x.patch):
[ssb-browser-demo]: https://github.com/arj03/ssb-browser-demo
[secret-stack]: https://github.com/ssbc/secret-stack
[ssb-ws]: https://github.com/ssbc/ssb-ws
[ssb-validate]: https://github.com/ssbc/ssb-validate
[ssb-blob-files]: https://github.com/ssbc/ssb-blob-files
[ssb-ooo]: https://github.com/ssbc/ssb-ooo
[ssb-conn]: https://github.com/staltz/ssb-conn
[ssb-db2]: https://github.com/ssb-ngi-pointer/ssb-db2

[ssb-get-thread]: https://github.com/arj03/ssb-get-thread
[ssb-partial-replication]: https://github.com/arj03/ssb-partial-replication
Expand Down
4 changes: 2 additions & 2 deletions core-helpers.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ exports.removeBlobs = function() {
exports.EBTSync = function(rpc)
{
console.log("doing ebt with", rpc.id)
SSB.db.contacts.getGraphForFeed(SSB.net.id, (err, graph) => {
SSB.db.getIndex('contacts').getGraphForFeed(SSB.net.id, (err, graph) => {
SSB.net.ebt.updateClock(() => {
SSB.net.ebt.request(SSB.net.id, true)
graph.following.forEach(feed => SSB.net.ebt.request(feed, true))
Expand All @@ -85,5 +85,5 @@ exports.EBTSync = function(rpc)

exports.fullSync = function(rpc)
{
SSB.db.feedSyncer.syncFeeds(rpc, exports.EBTSync)
SSB.feedSyncer.syncFeeds(rpc, exports.EBTSync)
}
56 changes: 10 additions & 46 deletions core.js
View file
Original file line number Diff line number Diff line change
@@ -1,79 +1,43 @@
exports.init = function (dir, config) {
const pull = require('pull-stream')
const FeedSyncer = require('./feed-syncer')

const EventEmitter = require('events')
SSB = {
events: new EventEmitter()
}

// outside browser
if (typeof localStorage === "undefined" || localStorage === null) {
const path = require('path')
const fs = require('fs')

if (!fs.existsSync(dir))
fs.mkdirSync(dir)

var LocalStorage = require('node-localstorage').LocalStorage
localStorage = new LocalStorage(path.join(dir, 'localstorage'))
events: new EventEmitter(),
dbOperators: require('ssb-db2/operators')
}
SSB.dbOperators.mentions = require('ssb-db2/operators/full-mentions')

const s = require('sodium-browserify')
s.events.on('sodium-browserify:wasm loaded', function() {

console.log("wasm loaded")

var net = require('./net').init(dir, config)
var db = require('./db').init(dir, config)

console.log("my id: ", net.id)

var helpers = require('./core-helpers')

var validate = require('ssb-validate')
var state = validate.initial()

// restore current state
db.getAllLatest((err, last) => {
// copy to so we avoid weirdness, because this object
// tracks the state coming in to the database.
for (var k in last) {
state.feeds[k] = {
id: last[k].id,
timestamp: last[k].timestamp,
sequence: last[k].sequence,
queue: []
}
}
})
const Partial = require('./partial')
const partial = Partial(dir)

SSB = Object.assign(SSB, {
db,
db: net.db,
net,
dir,
feedSyncer: FeedSyncer(net.id, partial, net.db),

getPeer: helpers.getPeer,

validate,
state,

removeDB: helpers.removeDB,
removeIndexes: helpers.removeIndexes,
removeBlobs: helpers.removeBlobs,

box: require('ssb-keys').box,
blobFiles: require('ssb-blob-files'),

// sbot convenience wrappers
publish: function(msg, cb) {
state.queue = []
state = validate.appendNew(state, null, net.config.keys, msg, Date.now())
//console.log(state.queue[0])
db.add(state.queue[0].value, (err, data) => {
net.ebt.onPost(data)
cb(err, data)
})
},
partial,

// config
hops: 1, // this means download full log for hops and partial logs for hops + 1
Expand All @@ -82,7 +46,7 @@ exports.init = function (dir, config) {
// helper for rooms to allow connecting to friends directly
SSB.net.friends = {
hops: function(cb) {
db.contacts.getGraphForFeed(SSB.net.id, (err, graph) => {
net.db.getIndex('contacts').getGraphForFeed(SSB.net.id, (err, graph) => {
let hops = {}
graph.following.forEach(f => hops[f] = 1)
graph.extended.forEach(f => hops[f] = 2)
Expand Down
Loading

0 comments on commit 62373cd

Please sign in to comment.