OpenStack Swift

The Human Brain Project has contributed support for OpenStack Swift:

The OpenStack Object Store project, known as Swift, offers cloud storage software so that you can store and retrieve lots of data with a simple API. It's built for scale and optimized for durability, availability, and concurrency across the entire data set. Swift is ideal for storing unstructured data that can grow without bound.

Installation

TODO

Configuring Swift

The Swift storage engine supports v1, v2, and v3 authentication. For v1 and v2 authentication, five parameters must be provided in DVID's TOML configuration file:

[store]
[store.default]
engine = "swift"
user = "test:tester" # The username.
key = "testing" # The key or password.
auth = "http://127.0.0.1:8080/auth/v1.0" # The Swift authentication URL.
container = "mycontainer" # The container name.

For v3 authentication, the parameters project and domain must also be provided:

[store]
[store.default]
engine = "swift"
user = "someuser" # The username.
key = "somepassword" # The key or password.
auth = "http://127.0.0.1:8080/v3" # The Swift authentication URL.
project = "someproject" # The project name.
domain = "somedomain" # The domain name.
container = "mycontainer" # The container name.

If the provided container does not exist, it will be automatically created (given the user has sufficient rights to create containers).

Performance and Reliability

Note that you may experience slower speeds with Swift as your storage backend than for example LevelDB. To mitigate this somewhat, the engine may open up to 10 concurrent connections to the Swift backend. (This is defined here).

If Swift becomes temporarily unavailable, the storage backend will try again with increasing pauses between attempts. If those pauses become longer than 20 minutes, it will finally give up with an error message.