Skip to content

CouchDB Architecture

swannodette edited this page Sep 13, 2010 · 35 revisions

ShiftSpace

Peers

This document describes ShiftSpace’s use of CouchDB in particular how peers interact and how data is replicated between them. There is no real difference between the ShiftServer that runs on a remote server and the ShiftServer that runs on the client. However, ShiftServer can be configured to behave differently in order to support the querying of shared user content as well as handling the distribution of such shared content to authenticated parties.

Shifts

The central document to ShiftSpace is the Shift. A Shift is simply a piece of JSON data that a Space (a ShiftSpace extension application) can use to recreate a modification to a website.

When a user makes a modification to a website with a Space, a shift document is saved locally on the user’s machine. It is not sent to the server.

Local Databases

When a user creates an account 5 databases (3 local, 2 remote) are created. The local ones are described here.

user/private

Holds all of the user’s unpublished shifts. This is replicated only to the local user/feed. user/private is never replicated to a peer or a publishing server. Thus there are at least 2 local copies of an private unpublished shift. There is an additional remote copy on the publishing server for each user or group that the shift get’s published to. Sending shifts directly to a user doesn’t scale well because multiple copies must exist in each remote user/inbox that the shift is published to. Publishing to a group is much more efficient. Like Twitter we imagine most people will make their shifts public or publish them to groups. In all likelihood only a small majority of power users will use the combination of sending to many individual users and groups – so we’re not too worried about the scalability of direct messages ;)

user/public

Whenever a user make a shift public, it is copied to user/public and deleted from user/private. This user/public is then replicated to user/feed. If the user is connected to a peer running as a publishing server, it is copied to the master public database on that server. A shift in user/public has at least 4 copies, 2 local and 2 remote (remote/master and remote/shared).

user/feed

This is the user’s view of the world. If the user is running ShiftSpace in full peer mode (replicating the public firehose onto their local machine) queries only happen against user/feed. If the user is not running in full peer mode, queries happen against remote/shared. the local databases user/public, user/private are replicated into user/feed. In addition the remote databases remote/master, any of the user’s groups, user/inbox, user/messages, system/messages are replicated into user/feed as well.

Remote Databases

user/inbox

Shifts published directly to a user are sent here. Upon user connection to the remote publishing server, user/inbox is replicated into the local user/feed.

user/messages

All messages to a user are sent here. This includes notifications about comments on a shift as well as system notices such as broken shift content and software updates.

system/messages

For efficiency system messages are stored in their own database and get replicated to client user/feed.

Group databases

When a group is created by a user, a group database is generated to hold all content shared by the group. When user joins a group, he/she is allowed to replicate the group database into their local user/feed. When a user publishes a shift to a group a copy is sent to the group so other can replicate that content into their own feeds.

remote/master

This database holds all public shifts. All of this content is replicated to remote/shared. This means each public shift has 2 copies on the server.

remote/shared

This database is analogous to the local user/feed. This database holds all shared content, whether public, in a group, or sent directly to a user. General queries are run against this database, as it is the merge of all of the social activity of the users registered on the publishing server. Again if a user is running as full peer, queries happen locally for that user not against remote/shared.

Local to Remote Replication (under construction)

Replication happens over https. The user’s local ShiftServer open’s up a connection to the remote ShiftServer, authentication is performed, and then bidirectional replication.