Attachments design

[grep0]

Goal

Provide the way to add attachments (pictures, videos etc) to the posting.
Possible extension of adding attachments to comments is not discussed here.

Node interface

The node may reject attachment posting based on content type, length etc.

Data types

AttachmentMetadata {
  string name
  string owner
  int revision
  int moment
  string content_type
  string content_url  // the URL to access the attachment blob, something like `https://host.moera.blog/postings/{id}/attachment/{name}/{revision}/blob`
  optional byte[] thumbnail // for pictures
}

AttachmentList = AttachmentMetadata[]

REST API extension

PUT /postings/{id}/attachment - put new attachment and autogenerate name

Headers: Content-type, Content-encoding, Content-length
Body: the blob, optionally encoded with passed encoding
Returns: AttachmentMetadata

PUT /postings/{id}/attachment/{name} - put attachment with given name, or replace existing attachment.
{name} is expected to be filename with extension in usual filesystem style, case sensitive

Headers: Content-type, Content-encoding, Content-length
Body: the blob, optionally encoded with passed encoding
Returns: AttachmentMetadata

GET /postings/{id}/attachment_list - get list of attachments for the post

GET /postings/{id}/attachment/{name}[/{revision}] - get AttachmentMetadata. The latest revision by default

GET /postings/{id}/attachment/{name}[/{revision}]/blob - get attachment blob

Database

Do we want the blob in postgre or in filesystem?

UI

The Markdown image link !(foo)[filename.ext] is referring to the attachment to current post. The uninterpreted attachments are represented as download links below the posting text.

[smelamud]

First of all, thank you very much for starting this discussion! I appreciate a lot if anybody wants to take part in the development.

We definitely want to store the files in the filesystem, because it allows to utilize the web server directly for serving them. It also allows to upgrade to cloud object storage (using JuiceFS, for example) for large workloads.

I also think we should separate the files from their relationship with entries (posts and comments). Please note that one server may handle several nodes and these nodes may contain identical media files. We don't want to waste the disk space for keeping identical files. Think of userpics, popular memes, reposts and the newsfeed (that contains copies of posts from other nodes) - all these will create a lot of such duplicates.

So we store the media files in the filesystem under the names that are SHA-256 checksum of the file content - such an approach will help us to find duplicates easily. We also should store a table that contains the content-type of every file, its size and some metadata needed - for example, image dimensions.

A separate table should keep the list of attachments - id, name (optional), created_at, moment and media_id. The attachment should be linked to a revision of an entry. I don't think we should have attachment revisions separately from entry revisions - this will add unnecessary complexity. We also should store the type of attachment, because not only files may be attached, but also other posts, link previews etc. Owner name is not necessary for an attachment - its permissions are the same as permissions of the post/comment it is attached to.

Image thumbnails should be stored as separate files and a special table should link them to the full-sized image. Note that there may be several thumbnails for the same file - for different purposes. And also content-type of a thumbnail may differ from the content-type of the original files (think of static previews of GIFs or images previewing PDF files).

Thus, creation of an attachment should consist of two steps:

1. Uploading a file, getting its ID (checksum).
2. Creating/updating a post/comment referring this ID as an attachment.

So we don't need a separate API for managing attachments, but we do need an API for managing media files as a separate entity.

Feel free to make comments and update your document and we can talk further about smaller things.

[tigra]

We definitely want to store the files in the filesystem, because it allows to utilize the web server directly for serving them

General purpose filesystems are generally not good for storing lots of small files.

1. When you put a lot of files in directory (e.g. thousands), operations with this directory become quite slow
2. Some space is typically wasted with each file
3. Much more read operations is needed to serve a file is needed then possible

From article on Facebook's photo storage:
https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf

"We
engineered our own storage system for photos because
traditional filesystems perform poorly under our workload.
In our experience, we find that the disadvantages of
a traditional POSIX [21] based filesystem are directories and per file metadata. For the Photos application
most of this metadata, such as permissions, is unused
and thereby wastes storage capacity. Yet the more significant cost is that the file’s metadata must be read from
disk into memory in order to find the file itself. While
insignificant on a small scale, multiplied over billions
of photos and petabytes of data, accessing metadata is
the throughput bottleneck. We found this to be our key
problem in using a network attached storage (NAS) appliance mounted over NFS. Several disk operations were
necessary to read a single photo: one (or typically more)
to translate the filename to an inode number, another to
read the inode from disk, and a final one to read the
file itself. In short, using disk IOs for metadata was the
limiting factor for our read throughput. "

[grep0]

Good point with deduplication, the same attachment might be used for several posts/comments in the same node, and I like the idea of checksum as the canonical media id. It is still useful to have aliases associated with posts for three reasons: (1) it is easier to refer a short alias from markdown; (2) give a way to attach the files without mentioning them in the post, similar to mail attachments; (3) allow deleting the attachments with the post.

Owner's name is necessary for attachments as soon as we are going to eventually allow attachments to comments, as well as in communal nodes (groups).

[tigra]

Concerning duplicates, I think it may be useful to have everything allowing to generate a magnet link for a file, with one of way for hash calculation. This can have a nice side effect of ability to have a distributed hosting for attachments.
However BTIH (SHA-1 of BitTorrent metafile (list of SHA1 of each individual block)) seems a bit too complex if such a distributed hosting is not an explicit goal.

[tigra]

Do we want the blob in postgre or in filesystem?

Consider seeweedfs for storage. Especially good for images.
https://github.com/chrislusf/seaweedfs

[grep0]

Sounds like overengineering to me, at least at this stage. Storing blobs in the database is probably the easiest from the maintenance perspective, as you don't need to invent anything beyond the database backups. It's likely a suboptimal solution, but better than nothing

[tigra]

Agree that SeeweedFS based solution requires some additional effort to get up and running.

We need to:

1. Store attachments
2. Resize them
3. Serve them

Filesystem storage option is good in (3) - just have Apache serve the static content relatively cheaply, no extra work for JVM, but will require inventing directory structure system that will not create too much of entries in any of directories, and managing the housekeeping activities around that.

Blobs in Postgres miss the oportunity to delegate serving the content with optimized tool like Apache, so JVM will have to do some creation of Requests/Responses and some GC etc., i.e. with more overhead (which may be not important for a small node). However no filesystem problems and less housekeeping and pretty high limits of how much can be stored, and you have transactions without extra effort which would be needed to keep two systems consistent.

In SeeweedFS, additional effort to set the system up, relatively straightforward housekeeping activities using its API, and excellent optimized serving of the content + resizes out of box.

I'm leaning towards thinking DB storage is better solution from time-to-market perspective.

[tigra]

Another feature idea regarding attachments, would be Markdown support for referencing attachments using reference style links.

The "footnotes" part defining actual links may be implicitly added to each post - so [one][1], [two][2] etc become references to the attachments, ![alt text][2] becomes embedded image. Maybe even more meaningful names can be used instead of numbers.