Update members tangle calculation for exclusion spec #75

Powersource · 2023-03-22T13:40:15Z

Fixes #69

Based on #65

Powersource · 2023-03-22T14:16:47Z

Was first thinking I could get the epoch by getting the latest epoch tangle and using the previous. But that might be an array. More accurate to look it up based on the current writeKey.

Powersource · 2023-03-22T14:44:27Z

Hmm I can't just get my own feed with the secret as the purpose though since someone else might've been the creator of the epoch init.

Powersource · 2023-03-22T15:17:48Z

Tested performance using the get-tangle-100-publishes test

With group, and epoch tangles:

getTangles: 17.845s

With group, epoch, and members tangles:

getTangles: 20.140s

Powersource · 2023-03-22T15:20:56Z

With only the group tangle it was

getTangles: 16.390s

mixmix

All looks good. All my comments are suggestions/ optimisations (no 🔥 i.e. must change)

Nice one

mixmix · 2023-03-28T01:49:29Z

lib/add-tangles.js

  function addSomeTangles(content, tangles, cb) {
    if (tangles.length === 0) return cb(null, content)

    const currTangle = tangles[0]

-    getTangle[currTangle](content.recps[0], (err, generatedTangle) => {
+    getTangle(server, currTangle, content.recps[0], (err, generatedTangle) => {
      // prettier-ignore
      if (err) return cb(clarify(err, 'Failed to get group tangle when adding group tangle to message'))


Suggested change

if (err) return cb(clarify(err, 'Failed to get group tangle when adding group tangle to message'))

if (err) return cb(clarify(err, `Failed to get ${currTangle} tangle when adding group tangle to message`))

mixmix · 2023-03-28T01:50:37Z

lib/add-tangles.js

-    addSomeTangles(content, tangles, (err, content) => {
-      // prettier-ignore
-      if (err) return cb(clarify(err, 'failed to add tangles to content'))
+  addSomeTangles(content, tangles, (err, content) => {


This is sufficient:

addSomeTangles(content, tangles, cb)

I know but I wanted more clarify since this gets run a lot

And new Error('CLARIFICATION', {cause: err}) is the new clarify(err, 'CLARIFICATION')

mixmix · 2023-03-28T02:08:09Z

lib/get-tangle.js

+        pull.map((feed) =>
+          pull(
+            server.db.query(where(author(feed.id)), toPullStream()),
+            // get all first messages, since that's where the init would be
+            pull.take(1)
+          )
+        ),
+        pull.flatten(),
+        // find the init
+        pull.filter((msg) => msg.value?.content?.type === 'group/init'),


Suggested change

pull.map((feed) =>

pull(

server.db.query(where(author(feed.id)), toPullStream()),

// get all first messages, since that's where the init would be

pull.take(1)

)

),

pull.flatten(),

// find the init

pull.filter((msg) => msg.value?.content?.type === 'group/init'),

pull.map((feed) =>

pull(

server.db.query(

where(and(author(feed.id), type('group/init'))),

toPullStream()

),

pull.filter(isInit)

pull.take(1)

)

),

pull.flatten(),

will require type and isInit to be imported

this is faster because we already have an index on type

mixmix · 2023-03-28T02:16:00Z

lib/get-tangle.js

+    if (tangle === 'members') {
+      // we find the id of the init msg for the current epoch
+      pull(
+        server.metafeeds.branchStream({ old: true, live: false }),


Question: does this return all feeds or just your meta-feed feeds? (I can't remember and the docs don't say!)

I think you have to specify opts.root to only get from one person

mixmix · 2023-03-28T02:59:36Z

lib/get-tangle.js

+    // prettier-ignore
+    return cb(new Error(`get-tangle expects valid groupId, got: ${groupId}`))
+  }
+


In ssb-tribes I put a cache over the groups tangle https://github.com/ssbc/ssb-tribes/blob/master/lib/get-group-tangle.js#L12

I will perhaps look at adding that in another PR in the future.

The jitdb cache seems really fast after the first query

mixmix · 2023-03-28T03:07:45Z

lib/get-tangle.js

+  const { where, and, toPullStream } = ssb.db.operators
+  pull(
+    ssb.db.query(where(and(tangleRoot(root))), toPullStream()),
+    pull.collect(cb)
+  )


Suggested change

const { where, and, toPullStream } = ssb.db.operators

pull(

ssb.db.query(where(and(tangleRoot(root))), toPullStream()),

pull.collect(cb)

)

const { where, toCallback } = ssb.db.operators

ssb.db.query(where(tangleRoot(root)), toCallback(cb))

mixmix · 2023-03-28T03:09:33Z

test/get-tangle.test.js

+
+    //console.time('getTangles')
+


Suggested change

//console.time('getTangles')

mixmix · 2023-03-28T03:09:51Z

test/get-tangle.test.js

@@ -129,6 +140,8 @@ test(`get-tangle-${n}-publishes`, (t) => {
        (err) => {
          t.error(err, 'no error publishing')

+          //console.timeEnd('getTangles')


Suggested change

//console.timeEnd('getTangles')

Make getUpdates in getTangle stateless

1844634

Powersource changed the base branch from master to exclude-member March 22, 2023 13:40

Powersource added 2 commits March 22, 2023 14:51

Refactor getTangle to be stateless

4f3dc4c

Make addTangles stateless

bf21ee4

Base automatically changed from exclude-member to master March 22, 2023 14:07

Error in addTangles if bad recps

74bb5b0

Powersource added 2 commits March 22, 2023 16:04

Adapt members tangle calculation to epochs

b839b1e

Add epoch init finding comments

b54b4e3

Powersource marked this pull request as ready for review March 22, 2023 15:09

Powersource requested review from staltz, mixmix and arj03 March 22, 2023 15:10

Add performance test for getTangles

398c264

mixmix approved these changes Mar 28, 2023

View reviewed changes

Powersource merged commit 9943ec7 into master Mar 30, 2023

Powersource deleted the members-epoch branch March 30, 2023 15:50

mixmix mentioned this pull request Feb 1, 2024

TRACKING: exclusion implementation #143

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update members tangle calculation for exclusion spec #75

Update members tangle calculation for exclusion spec #75

Powersource commented Mar 22, 2023 •

edited

Loading

Powersource commented Mar 22, 2023

Powersource commented Mar 22, 2023

Powersource commented Mar 22, 2023

Powersource commented Mar 22, 2023

mixmix left a comment

mixmix Mar 28, 2023

mixmix Mar 28, 2023

Powersource Mar 30, 2023

staltz Mar 31, 2023

mixmix Mar 28, 2023

mixmix Mar 28, 2023

mixmix Mar 28, 2023

Powersource Mar 30, 2023

mixmix Mar 28, 2023

Powersource Mar 30, 2023

mixmix Mar 28, 2023

mixmix Mar 28, 2023

mixmix Mar 28, 2023

	if (err) return cb(clarify(err, 'Failed to get group tangle when adding group tangle to message'))
	if (err) return cb(clarify(err, `Failed to get ${currTangle} tangle when adding group tangle to message`))

Update members tangle calculation for exclusion spec #75

Update members tangle calculation for exclusion spec #75

Conversation

Powersource commented Mar 22, 2023 • edited Loading

Powersource commented Mar 22, 2023

Powersource commented Mar 22, 2023

Powersource commented Mar 22, 2023

Powersource commented Mar 22, 2023

mixmix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Powersource commented Mar 22, 2023 •

edited

Loading