Blob work: Request blobs that have been mentioned in a post and store them

[nichoth]

Overall GOAL: We want to request any blobs that have been mentioned in a post and save it to our server.

choose a pub to request a blob from

• should iterate through our list of known pubs

stream errors

• listen for errors in the pull stream with each pub. (the muxrpc stream)
• if the stream doesn't work, should move on to the next one in our list

---

• should preemptively replicate blobs, not just when they are requested by a user. See this issue

https://github.com/ssbc/ssb-server/blob/99fad7c5f6e436cbd670346b4da20c57222a1419/plugins/blobs/replication.js#L190

// monitor the feed for new links to blobs
  pull(
    sbot.links({dest: '&', live: true}),
    pull.drain(function (data) {
      // do we have the referenced blob yet?
      sbot.blobs.has(data.dest, function (_, has) {
        if(!has) createJob(data.dest, data.source)
      })
    })
  )

You could use jitdb's predicate to check that a mention starts with the character & and run it with live(), or you could use the fullMentions leveldb in ssb-db2 but you'd have the add more code to the source code for doing this live logic.

[nichoth]

ssbc/ssb-db2#307

Blob replication is handled entirely by the plugin ssb-blobs

Why is this not my experience? We see broken links unless we explicitly replicate blobs.

---

See the push method

via %2s32sWdS14QUKXECTGmwSjTN9fost9eV6TdedsCcSvc=.sha256

We don’t do anything different in Manyverse in comparison to (say) Patchwork. The plugin we use is ssb-blobs, and I think one important part of that subprotocol is to “push” the blobs to peers connected, not just make it available for “pull”, which is the request thing you mentioned.

---

push though is only relevant if a client is posting a new message... for a viewer app we would need to get blobs as we notice them being referenced.

links to relevant posts

• db2 -- Example of getting messages with blobs
• historical example in flume

[nichoth]

How to test?

• need a second sbot running locally -- see https://github.com/ssb-ngi-pointer/netsim
• connect the second sbot to the first
• second sbot should have message that reference blobs that it has in its blobstore
• first sbot replicates messages from sbot2. I think this means adding sbot2 as a friend