Skip to content

Latest commit

 

History

History
1142 lines (972 loc) · 58.3 KB

CHANGELOG.md

File metadata and controls

1142 lines (972 loc) · 58.3 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

Breaking Changes

  • CollectionDocument::modify/CollectionDocument::modify_async now take an additional parameter: the return type of the callback function. This result from the call that succeeds in updating will be returned in Ok.

Added

  • bonsaidb::client::Error now implements From<bonsaidb::client::ApiError<Infallible>>.
  • KeyVisitor::visit_other is a new function that indicates the key encoded is a byte sequence of a known type.

Fixed

  • bonsaidb::client::Error::Core's Display no longer just prints "unexpected disconnection". Instead, the inner error's Display is now displayed.
  • bonsaidb::client::Async/Blocking are now exposed. These types are used when building a client.
  • bonsaidb::server::Backend::client_authenticated is now invoked.
  • bonsaidb::core::schema now exports these types already in-use by NamedCollection::entry:
    • AsyncEntry
    • Entry
    • EntryInsert
    • EntryUpdate
  • KeyDescriber no longer panics when a Key type calls no visitor methods in its KeyEncoding::describe implementation.

v0.5.0

Breaking Changes

  • The Minimum Supported Rust Version (MSRV) has been changed to 1.70.

  • All features in the bonsaidb crate have been updated to support the new features in Rust 1.60. Instead of needing separate flags for client-websockets and server-websockets, a single feature flag websockets is now able to work in conjunction with the client/server features to ensure everything works correctly.

    The net change is fewer feature flags. If you find that a feature flag is missing on upgrade, try removing "local-", "client-", or "server-" from the beginning of the feature and recompiling.

  • The Key implementation for Option<T> has changed. If you wish to preserve backwards compatibility, wrap the key type with OptionKeyV1.

    The old behavior was broken in two ways:

    • The length of the key was improperly always reported as a constant length. This only affects code that was using composite keys.
    • When using a type that can encode itself as 0 bytes, there was no way to distinguish between a 0-length contained value and None. Thus, Some("") would panic as a precaution. However, this type of lurking panic in an esoteric edge case is exactly the behavior a database should never exhibit.

  • The Key implementation for tuples (e.g., (T1, T2)) has changed. If you wish to preserve backwards compatibility, wrap the key type with TupleKeyV1.

    The old behavior had an incorrect sort order for the generated bytes. Instead of sorting only by the values themselves, the lengths of the variable-length Key types were taking precedence over the values. For more information, see issue #240.

    This change also deprecates encode_composite_key()/decode_composite_key() in favor of two new types: CompositeKeyEncoder and CompositeKeyDecoder.

  • Collection's get/list/list_with_prefix/insert/overwrite and View's with_key_range/with_key_prefix have all been updated to accept borrowed representations of keys. For example, when a Collection's PrimaryKey is String, &str can now be supplied to these functions.

    For most users, adding & in front of the argument will generally fix the compiler error upon upgrade.

  • DocumentId is no longer Copy, and now can support up to 64kb of data. Previously, the type was limited to 64 total bytes of stack space.

  • [Async]StorageConnection::schematic() have been moved to its own trait, HasSchema.

  • [Async]StorageConnection::authenticate() no longer takes a username or user id directly. This has been moved to the Authentication type.

  • AuthenticationMethod has moved from bonsaidb::core::permissions::bonsai to bonsaidb::core::connection

  • ApiName has been moved from bonsaidb::core::schema to bonsaidb::core::api.

  • Various error variants that were simply String representations have been consoldiated into bonsaidb::core::Error::Other.

  • bonsaidb::server::Backend now takes a &self parameter in all functions except configure(). Implementing Default for your Backend implementor will allow all existing code to continue working.

  • Client::effective_permissions() is no longer async.

  • CustomServer::connected_clients() is no longer async.

  • CustomServer::broadcast() is no longer async.

  • CustomServer::listen_on now takes impl Into<BonsaiListenConfig> instead of just a u16 parameter specifying the port. BonsaiListenConfig implements From<u16> to minimize code breakage.

  • The command-line Serve type has had its listen-on field changed to a SocketAddr type to reflect the support for BonsaiListenConfig. The full address and port must be specified when providing a listen-on argument now.

  • Client has been renamed to AsyncClient, and a new type, BlockingClient has been added. This splits the Client into two separate parts: an async version and a blocking version. This makes it less likely to accidentally call a blocking method in an async context.

    Client::send_api_request_async has been renamed to send_api_request.

    This change has also been introduced to RemoteDatabase and RemoteSubscriber: both async and blocking versions are available.

  • #254 Key::from_ord_bytes has had its &'k [u8] parameter changed to a new type with an additional lifetime: ByteSource<'k, 'e>. This new type allows from_ord_bytes to be called with an owned Vec<u8>, a Key-lifetime bound byte slice (&'k [u8]), or an ephemeral byte slice (&'e [u8]).

    This change allows code paths that can pass an owned Vec<u8> in for decoding to not require allocations in some cases -- for example, when using Vec<u8>::from_ord_bytes.

    This change also means that Key implementors should expect to be called with any of the three variants of ByteSource, because BonsaiDb's internal code only passes borrowed slices in some code paths.

    Thank you to @asonix for the request and help on implementation!

  • KeyEncoding::describe is a new function that allows a key encoder to document the data contained within the encoded representation. The goal of this is to allow tools to generically operate on key types used within BonsaiDb. This function is automatically implemented when using the Key derive.

    The new KeyDescription type uses this function to create a nested representation of the contained information.

    Types that utilize a custom encoding format can use KeyDescription::Other to uniquely identify the key.

  • Schematic has had several methods changed to impl Iterator of the original type being returned to avoid extra unnecessary allocations. These methods are:

    • Schematic::views_in_collection
    • Schematic::eager_views_in_collection
    • Schematic::collections

  • StorageConnection::list_available_schemas/AsyncStorageConnection::list_available_schemas now return a Vec<SchemaSummary>. The SchemaSummary type contains additional information such as what collections and views each schema contains.

  • bonsaidb::cli::Command now flattens the server command rather than in-lining the admin command. This introduces additional top-level commands that were previously hidden underneath the server command.

  • ConnectedClient::all_sessions() is a new function that returns all of the active sessions for the given client.

  • View schemas are now partially deriveble, and have had their map/reduce functionality split into their own traits. The following notes all apply to this refactoring:

    • ViewSchema::map() and ViewSchema::reduce() have been moved to a new trait: MapReduce.
    • CollectionViewSchema has been removed, and the map() and reduce() functions have been moved to a new trait: CollectionMapReduce.
    • ViewSchema::unique() and ViewSchema::lazy() have been replaced with ViewSchema::update_policy(), which returns a new enum: ViewUpdatePolicy. This enum contains three variants: Lazy, Eager, and Unique, allowing the same options that the previous two methods supported without any ambiguities.
    • ViewSchema is now implementable/derivable for all views, regardless of whether the map/reduce functionality utilizes SerializedCollections or not.

  • ViewSchema::MappedKey<'doc> is a new associated type with a generic associated lifetime that enables map()/reduce() to utilize borrowed data for the view's key type. This has caused the map() and reduce() functions to have new lifetime annotations added. These changes are part of an effort to support more flows where borrowing data is possible to minimize allocations while indexing and querying views.

    For all existing users, setting this associated type to the view's Key type will work, or pasting this associated type definition in:

    type MappedKey<'doc> = <Self::View as View>::Key

  • natural_id support in the Collection derive macro has been changed. The attribute now expects an expression rather than a closure. The expression can reference self. This was done to avoid the required type annotations that the closure approach required.

    Alternatively, #[natural_id] can be annotated directly on a field to have it become the natural id automatically.

  • bonsaidb::server::api::Handler has had its generic arguments order reversed, which allows the type to specify a default Backend of NoBackend.

  • View query APIs now return CollectionMap types instead of Map types. The change allows for the Collection::PrimaryKey type to be used instead of DocumentId. The affected APIs are:

    • bonsaidb::core::connection::View::query()
    • bonsaidb::core::connection::View::query_with_docs()
    • bonsaidb::core::connection::AsyncView::query()
    • bonsaidb::core::connection::AsyncView::query_with_docs()
    • bonsaidb::core::connection::LowLevelConnection::query()
    • bonsaidb::core::connection::LowLevelConnection::query_with_docs()
    • bonsaidb::core::connection::AsyncLowLevelConnection::query()
    • bonsaidb::core::connection::AsyncLowLevelConnection::query_with_docs()
    • MappedDocuments: Both mappings and documents have had their types updated.
    • MappedSerialiedSocuments::deserialized()

Deprecated

  • bonsaidb::core::connection::ViewMappings as been moved to bonsaidb::core::schema::view::ViewMappings. A deprecated re-export has been provided to minimize code breakage when upgrading.

Added

  • #239 Key can now be derived on enums and structs, allowing an easier way to use composite keys. The primary-keys example has been updated to use the derive macro instead of a tuple for the composite key.

  • Key is now implemented for Result<T,E>, where T and E both implement Key and KeyEncoding using the same error type.

  • Key is now implemented for Cow<'a, str>.

  • Key is now implemented for isize and usize. This is implemented using variable integer encoding, allowing for proper cross-architecture behavior. When trying to decode a value that is too large for the given target architecture, an error will be returned.

  • Key is now implemented for NonZeroU* and NonZeroI* types, wrapping the inner type's encoding functionality and adding extra checks for 0 values. For signed types, next_value() will skip 0.

  • bonsaidb::core::key::KeyFormat is a new transmog::Format that can be used as a SerializedCollection::Format associated type. This implements serialization using the Key trait rather than Serde.

    When deriving the Collection trait, specifying serialization = Key will use this format.

  • ViewSchema can now be derived.

  • bonsaidb_core::Error::is_unique_key_error() is a convenience function to quickly check if an error is a result of a unique key violation from a specific view.

  • bonsaidb_core::Error::conflicting_document() is a convenience function to return the conflicting document's header if the error is a conflict from a specific collection.

  • Operation::Check is a new operation that can be performed during a transaction. It checks whether a document exists in a given collection, and optionally can verify that a revision matches the currently stored revision.

    These constructor functions provide more ergonomic ways to create this variant:

    • Operation::check_document_id_exists(CollectionName,DocumentId)
    • Operation::check_document_exists<Collection>(Collection::PrimaryKey)
    • Operation::check_document_is_current(&impl HasHeader)

  • Storage now acquires an exclusive lock to ensure multiple instances are not able to be opened at the same time. This lock is held across all locations the database is accessed, including background threads.

  • bonsaidb-files is a new crate that enables storing large files in BonsaiDb, and includes abstractions for reading/writing files using common traits:

    • std::io::Read
    • std::io::Seek
    • std::io::Write
    • tokio::io::AsyncRead
    • tokio::io::AsyncSeek
    • tokio::io::AsyncWrite
    • Iterator<Item = std::io::Result<Vec<u8>>>
    • futures::Stream<Item = std::io::Result<Vec<u8>>>

    This crate can be added directly to your project, or if you're using the omnibus crate, feature files will enable this crate at bonsaidb::files.

  • SerializedCollection::push_all[_async]() is a new function that accepts an iterator of document contents to push into the database using a single transaction. It returns the created collection documents if successful. If any errors occur, no documents will be inserted.

  • ViewSchema::lazy()/CollectionViewSchema::lazy() are provided functions that return true by default. This preserves the existing behavior for map/reduce views -- they are only updated when queried, and as dictated by the AccessPolicy. By returning false, a view can become eagerly updated, which means that the views are fully updated by the time each transaction completes.

    This was how unique views already functioned, but now users who have workflows where an eagerly-updated view will be more efficient than a lazy view can opt-in to this behavior.

  • bonsaidb::server::cli::Command has a new function, execute_on which accepts an already constructed server instance.

  • ServerDatabase, AnyDatabase, and AnyConnection now all implement Clone and Debug.

  • DefaultPermissions now implements From<Vec<Statement>> and From<Statement>, enabling simpler usage when using default_permissions() and authenticated_permissions().

  • [Async]StorageConnection::authenticate_with_password are new functions providing a simpler interface for authenticating with a username/user id and password.

  • [Async]StorageConnection::authenticate_with_token is a new function that allows authenticating with a previously created AuthenticationToken.

    Token authentication is an optional feature, enabled with the token-authentication feature. This feature is currently powered by BLAKE3, but has been written to be able to support multiple algorithms.

    An AuthenticationToken is a randomly generated id, a private token, and an identity that are used to perform a cryptographically secure verification during authentication without transferring the private token to the server. Upon successful authentication, the identity is assumed on the newly returned connection. This supports both assuming users and roles.

  • The Api trait can now be derived.

  • SerializedView has two new methods: entries and entries_async, which enable a "Type-first" query pattern. These two statements produce the same results:

    let mappings = MyView::entries(&db).query()?;
let mappings = db.view::<MyView>().query()?;

    These new APIs are provided purely for style preference considerations.

  • LimitedResolutionDuration now has a function checked_add() supports iter::Sum into Option<LimitedResolutionDuration<T>> or LimitedResolutionDuration<T>. The option-wrapped version does a checked operation, and the other will panic if the result of the operation is not representable by the LimitedResolutionDuration.

  • BonsaiListenConfig is a new structure that controls the settings of the BonsaiDb network protocol server socket. This structure currently allows specifying the specific SocketAddr to listen on and whether the SO_REUSEADDR flag should be specified on the underlying socket.

  • CollectionDocument now implements Serialize and/or Deserialize if both PrimaryKey and Contents are Serialize and/or Deserialize, respectively.

  • Backend::client_session_ended is a new function that is invoked any time a connected client's session is ending.

  • VarInt<T> is a new type that implements Key using the ordered-varint crate. This allows using types such as VarInt<u64> instead of u64 to reduce the number of bytes encoded keys consume on average.

  • SerializedCollection now has insert_in_transaction(), push_in_transaction(), and overwrite_in_transaction() which are new helpers that help writing transactional code easier by creating and pushing the Operations in one step.

  • CollectionDocument<T> now has update_in_transaction() and delete_in_transaction() which are new helpers that help writing transactional code easier by creating and pushing the Operations in one step.

  • Client's builder now has two additional settings: request_timeout and connect_timeout. If not specified, both timeouts are 60 seconds. Thank you to @phantie for requesting these settings in #296.

  • CollectionDocument::refresh()/CollectionDocument::refresh_async() are new methods that reload the document from the database.

  • ConnectedClient::connected() is a new function that returns the current state of whether the client is still connected. Because ConnectedClient is able to be cloned and sent between threads, this allows other threads to notice when clients have disconnected without needing to be notified via the Backend.

Changed

  • bonsaidb::cli::Command::Server now calls CommandLine::open_server() rather than constructing the server directly. This allows CommandLine implementors to use open_server as a location to launch extra services upon server startup.
  • bonsaidb::cli::Args and bonsaidb::cli::Command::execute now accept a token id or a username, allowing for commands to be executed with authentication if the features are enabled.
  • bonsaidb::local::cli::Command now offers the ability to create a user and set a user's password.
  • bonsaidb::local::cli::admin is a newly exposed module that allows some basic user management. This set of commands is also available on bonsaidb::cli::Command through the Admin variant, allowing for both local and remote administration.
  • Header, CollectionHeader, and Revision now all implement Hash.
  • Database names are no longer case insensitive. This was implemented poorly, and in hindsight, the feature was removed in favor of strictness. Because the database name is used as a directory name on-disk, care must be taken on case-insensitive filesystems to not attempt to create two databases with different casing.
  • TimedArgonParameters now guarantees that the minimum parameters chosen will meet the OWASP recommendations. Manual configuration still is allowed to set exact parameters.

Fixed

  • Unique views/eager views now are properly updated when an overwrite operation is performed that resulted in a new document being created.
  • Argon2 ram configuration is now correctly applied. Previously, the memory cost was being supplied in bytes, but the underlying API was expecting kilobytes.
  • When a View's version has changed, the view is now fully cleaned before reindexing. Previously, BonsaiDb was lazily cleaning up the entries, which led to slower reindexing and disk bloat.
  • Fixed the source field of mappings returned from a View query when a document was updated but emitted the same key. Previously the value was correctly updated, but the source's revision was not updated.
  • When querying a View with AccessPolicy::NoUpdate, the integrity scanner is now checked and waited upon before allowing access to the view.
  • When querying a View with AccessPolicy::UpdateAfter, the update task is no longer blocked until complete.
  • CustomServer::listen_for_shutdown() now listens for CustomServer::shutdown() in addition to operating system signals.
  • Client will no longer return a Session from a previous connection. Because the Client automatically reconnects, the Sessions are no longer authenticated.
  • CustomServer::shutdown now fully shuts down the server by forcefully disconnecting clients after the optional grace period has elapsed. Additionally, QUIC-connected workers are sent the proper disconnection notification.
  • The Key generic parameter to Operation::overwrite_serialized is now used in the function definition. Because previously C::PrimaryKey was hard-coded, any existing code should work as long as the correct Key type was provided. This fix also allows for types like str to be used instead of String with this function.
  • Limited case insensitivity was implemented incorrectly yielding inconsistent results.
  • CustomServer::listen_on no longer will return an error if an incoming connection fails during the TLS or QUIC handshake. Thank you to @phantie for reporting this in #296.

v0.4.1

Fixed

  • The View indexing system had a bug when deleting the last view entries for a key while also inserting new entries for that key in the same mapping update operation. This prevented the recording of new entries being made during that mapping operation. This bug was introduced during the optimizations in v0.3.0.

    All views will be reindexed automatically on upgrade.

  • insert_bytes/push_bytes no longer require SerializedCollection to be implemented.

v0.4.0

Breaking Changes

  • BonsaiDb now has both an async interface as well as a blocking interface. This has caused significant changes, but they can be summarized simply:

    • Connection-related async-compatible traits have had the Async prefix added to them.

      Blocking Async
      Connection AsyncConnection
      StorageConnection AsyncStorageConnection
      PubSub AsyncPubSub
      Subscriber AsyncSubscriber
      KeyValue AsyncKeyValue
      LowLevelConnection AsyncLowLevelConnection

    • Functions that take parameters of the above traits now are offered in pairs: a blocking function and an async function with "_async" at the end of the name. For example, SerializedCollection::get() is the blocking version of SerializedCollection::get_async().

    • For bonsaidb-local, the Database and Storage types are now blocking implementations. Under the hood, BonsaiDb previously used tokio::task::spawn_blocking() to wrap calls to the database in an async API. New types, AsyncDatabase and AsyncStorage have been added that provide the previous behavior. The types can be converted between each other using helpers as/into/to_blocking/async available on each type.

      These changes allow bonsaidb-local to be compiled without Tokio. To enable tokio, enable feature async if using bonsaidb-local directly, or enable feature local-async when using the bonsaidb crate.

    • For bonsaidb-server, it still uses networking driven by Tokio. Server/CustomServer implement AsyncStorageConnection, and Server can convert to Storage via the From trait for synchronous database access.

    • For bonsaidb-client, Client implements both AsyncStorageConnection and StorageConnection and is safe for use in both synchronous and asynchronous contexts. In WASM, Client only implements AsyncStorageConnection. For all other platforms, the Client builder supports specifying the Tokio runtime handle if needed. Otherwise, the current runtime will be used or a default runtime will be created automatically if unavailable.

  • Connection::query_with_docs/Connection::query_with_connection_docs now verify the user has permission to DocumentAction::Get. This allows schema authors to allow querying views without allowing the documents themselves to be fetched.

  • ViewAction::DeleteDocs has been removed. Delete docs is now composed of two permissions checks: ViewAction::Query to retrieve the list of documents to delete, and DocumentAction::Delete for each document retrieved. This ensures if permission is denied to delete a specific document, it still cannot be deleted through delete_docs().

  • All APIs have had their limit parameters changed from usize to u32. Since usize is platform-dependent, picking a fixed-width type is more appropriate.

  • CustomApi has been renamed to Api and changed significantly.

    On the Server, Apis are registered on the ServerConfiguration. The Api implementor is treated as the "request" type and is what Clients send to the Server. The Api::Response type is what the Server sends to the Client. Out-of-band responses can still be delivered.

    On the Client, Apis can simply be used without any extra steps. If you expect out-of-band responses, callbacks can be registered when building the client.

    Internally, all BonsaiDb APIs have been refactored to use this -- there is no distinction.

  • The multiuser feature flag has been removed. In the end this caused a lot of needless conditional compilation for removing a single lightweight dependency.

  • User::assume_identity and Role::assume_identity allow assuming the identity of a user or role by their unique ID or name. The connection must be permitted with the newly added ServerAction::AssumeIdentity for the appropriate resource name (user_resource_name or role_resource_name).

  • StorageConnection::authenticate and StorageConnection::assume_identity both return a new instance with the new authentication. This enables authenticating as multiple roles with the same underlying storage connection.

    StorageConnection::session() is a new function that returns the current Session, if one exists. This new type contains information about any currently authenticated identity, the unique id of the session, and the current effective permissions.

    This release note applies equally to AsyncStorageConnection.

  • LowLevelConnection and AsyncLowLevelConnection have been added to group functionality that is not generally meant for the average user to utilize. The methods that were documented as low-level in Connection/AsyncConnection have been moved to this trait. Additionally, new methods that allow performing operations without the generic types have been added to this trait. This functionality is what will be useful in providing applications that can interact with BonsaiDb without having the Schema definitions available.

  • PubSub/AsyncPubSub now allows any Serialize implementation to be used as the topic parameter. New methods publish_bytes and publish_bytes_to_all have been added enabling publishing raw payloads.

  • CollectionName/SchemaName have had common methods extracted to a trait, Qualified. This was part of a refactoring to share code between these two types and the newly introduced ApiName type.

  • BackupLocation and VaultKeyStorage have been changed to blocking traits. bonsaidb-keystorage-s3 wraps a tokio Runtime handle as the AWS SDK requires Tokio.

  • ServerConfiguration now takes a Backend generic parameter, which must match the CustomServer being created. In general the Rust compiler should be able to infer this type based on usage, and therefore shouldn't be a breaking change to many people.

  • The Backend trait now has an associated Error type, which allows for custom error types to be used. When an error occurs during initialization, it is returned. Currently, errors that are returned during client connection handling are printed using log::error and ignored.

  • Key has had its encoding functionality moved to a new trait, KeyEncoding. KeyEncoding has been implemented for borrowed representations of Key types.

    This change allows all view query and collection access to utilize borrowed versions of their key types. For example, if a View's Key type is String, it is now possible to query the view using an &str parameter.

Added

  • Range::default() now returns an unbounded range, and Bound::default() returns Bound::Unbounded.

  • Range now has several builder-pattern style methods to help construct ranges. In general, users should simply use the built-in range operators (.., start.., start..end, start..=end), as they are able to represent nearly every range pattern. The built-in range operators do not support specifying an excluded start bound, while the new method Range::after allows setting an excluded start bound.

  • #215: StorageConnection::delete_user() is a new function that allows deleting a user by name or id. Deleting a user is permitted with the ServerAction::DeleteUser action.

  • bonsaidb_core::key::encode_composite_field and bonsaidb_core::key::decode_composite_field have been added which allow building more complex Key implementations that are composed of multiple fields. These functions are what the Key implementation for tuples is powered by.

  • Key is now implemented for [u8; N].

  • #221: headers() has been as a function to all collection list builders, enabling querying just the headers of a document.

  • Transaction now has apply() and apply_async(), which the higher-level API to LowLevelConnection::apply_transaction.

  • ArgonConfiguration can now be specified when building StorageConfiguration/ServerConfiguration using Builder::argon.

  • SystemTime and Duration now have Key implementations.

  • bonsaidb_core::key::time has been added which contains a wide array of types that enable storing timestamps and durations with limited resolution, powered by variable integer encoding to reduce the number of bytes needed to encode the values.

    These types are powered by two traits: TimeResolution and TimeEpoch. Using these traits, the LimitedResolutionDuration and LimitedResolutionTimestamp types can be used for completely custom resolutions (e.g., 15 minutes) and epochs (the base moment in time to store the limited resolution relative to).

    By constraining the resolution and using an epoch that is closer to the average timestamp being stored, we can reduce the number of bytes required to represent the values from 12 bytes to significantly fewer.

    These type aliases have been added in these three categories:

    • Durations: Weeks, Days, Hours, Minutes, Seconds, Milliseconds, Microseconds, and Nanoseconds.
    • Timestamps relative to the Unix Epoch (Jan 1, 1970 00:00:00 UTC): WeeksSinceUnixEpoch, DaysSinceUnixEpoch, ...
    • Timestamps relative to the Bonsai Epoch (Mar 20, 2031 04:31:47 UTC): TimestampAsWeeks, TimestampAsDays, ...

  • Backend::configure() is a new function that allows a Backend to set configuration options on ServerConfiguration before the server is initialized. This is a good location for Backends to define their Apis and Schemas.

Changed

  • Counting a list of documents now uses reduce() in Nebari, a new feature that allows aggregating the embedded statistics without traversing the entire tree. The net result is that retrieving a Collection's count should be near instant and returning the count of a range of keys should be very fast as well.
  • StorageConnection::create_database/AsyncStorageConnection::create_database now returns the newly created database.

Fixed

  • Defining multiple views with the same name for the same collection will now return an error.

v0.3.0

Breaking Changes

  • bonsaidb::local::jobs is now private. It used to be a separate, public crate in the PliantDb days. After thinking about the job scheduler more, this initial implementation is better suited for the internal task management than the higher-level jobs system. As such, it has been internalized.

  • bonsaidb::core::transaction::Changes::Documents has been changed to store the CollectionNames separately from the ChangedDocuments. This makes the transaction log entries smaller, as collection names aren't copied for each document.

    The storage layer is fully backwards compatible and will automatically convert existing transactions to the new format.

Fixed

  • Listing executed transactions that were written in v0.1 was broken in v0.2. Backwards compatibility is now automatically tested to help ensure this sort of issue won't happen in the future again.

Added

  • SerializedCollection::list_with_prefix, connection::Collection::list_with_prefix, and connection::View::with_key_prefix have been added as an easy way to filter results based on whether the key starts with the given prefix.

    This is supported by a new trait, IntoPrefixRange. This trait has been implemented for all byte-based key implementations as well as for String.

  • Operation::push_serialized has been added, which calls natural_id before creating an Operation::Insert variant.

  • Tasks::parallelization and Builder::workers_parallelization have been added as a way to control how many threads can be used by any given task/worker. This is automatically configured to be the number of cpu cores detected.

  • count() is a new function on the list builders, available via:

    • SerializedCollection::all(db).count().await
    • SerializedCollection::list(42.., db).count().await
    • db.collection::<Collection>().all().count().await
    • db.collection::<Collection>().list(42..).count().await

    The performance of this call is not as good as it will eventually be, as it is currently doing more work than strictly necessary.

  • #215: StorageConnection::delete_user has been added, enabling deletions of users without directly interacting with the admin database.

Changed

  • The view map/reduce system has been optimized to take advantage of some parallelism. The view system is still not hightly optimized, but this change makes a significant improvement on performance.

v0.2.0

Breaking Changes

  • bonsaidb::core::Error::DocumentConflict now contains a Header instead of just the document's ID. This allows an application to re-submit an update with the updated header without another request to the database.

  • StorageConfiguratation::vault_key_storage now uses an Arc instead of a Box. This change allows StorageConfiguration and ServerConfiguration to implement Clone.

  • Document::create_new_revision has been removed. It was meant to be an internal function.

  • Document now requires AsRef<Header> and AsMut<Header> instead of Deref<Header>/DerefMut. The publicly visible change is that the shortcut of accessing document.header.emit_* through deref by using document.emit_* will no longer work. This impacts CollectionDocument, OwnedDocument, and BorrowedDocument.

    This removes a little magic, but in some code flows, it was impossible to use Deref anyways due to Deref borrowing the entire document, not just the header.

  • Collection::PrimaryKey is a new associated type that allows customizing the type that uniquely identifies documents inside of a Collection. Users of the derive macro will be unaffected by this change. If you're upgrading existing collections and wish to maintain backwards compatibility, use u64 as the type.

    A natural_id() function can now be implemented on SerializedCollection or DefaultSerialization which allows extracting a primary key value from a new document being pushed

    A new example, primary-keys.rs, as been added showing basic usage of changing the primary key type. This change resulted in a sequnce of breaking changes that will be listed independently.

  • Key has been moved from bonsaidb::core::schema::view to bonsaidb::core::key.

  • Key::as/from_big_endian_bytes have been renamed to Key::as/from_ord_bytes.

  • Key::first_value() and Key::next_value() are new provided functions. By default, these functions return NextValueError::Unsupported.

    Key::first_value() allows a Key type to define the first value in its sequence. For example, 0_u64 is the result of u64::first_value().

    Key::next_value() allows a Key type to find the next value in sequence from the current value. Implementors should never wrap, and should instead return NextValueError::WouldWrap.

    Sensible defaults have been implemented for all numeric types.

  • Connection and its related types have had all previously hard-coded primary keys pf u64 changed to generic parameters that can accept either a DocumentId or Collection::PrimaryKey. The affected methods are:

    • Connection::insert
    • Connection::overwrite
    • Connection::get
    • Connection::get_multiple
    • Connection::list
    • connection::Collection::insert
    • connection::Collection::overwrite
    • connection::Collection::get
    • connection::Collection::get_multiple
    • connection::Collection::list
    • SerializedCollection::insert
    • SerializedCollection::insert_into
    • SerializedCollection::overwrite
    • SerializedCollection::overwrite_into
    • SerializedCollection::get
    • SerializedCollection::get_multiple
    • SerializedCollection::list
    • transaction::Operation::insert_serialized
    • transaction::Operation::overwrite_serialized

  • Header::id has changed from u64 to DocumentId, and CollectionHeader<PrimaryKey> has been added which contains Collection::PrimaryKey deserialized.

    These previous usages of Header have been changed to CollectionHeader:

    • Connection::insert result type
    • Connection::overwrite result type
    • connection::Collection::insert result type
    • connection::Collection::insert_bytes result type
    • connection::Collection::push result type
    • connection::Collection::push_bytes result type
    • CollectionDocument::header

    The Header::emit* functions have been moved to a new trait, Emit. This trait is implemented by both Header and CollectionHeader. The functions moved are:

    • emit()
    • emit_key()
    • emit_value()
    • emit_key_and_value()

    These functions now return a Result, as encoding a primary key value can fail if it is larger than DocumentId::MAX_LENGTH.

  • HasHeader is a new trait that allows accessing a Header generically from many types. This type is used in Connection::delete and connection::Collection::delete.

  • Types and functions that used u64 as a document ID have been replaced with DocumentIds. The number of locations are too many to list. If you need to convert from a u64 to a DocumentId, you can use DocumentId::from_u64().

  • Document::contents and Document::set_contents are now ore "painful" to access due to the generic parameter added to Document. SerializedCollection::document_contents(doc) and SerializedCollection::set_document_contents(doc, new_contents) have been provided as easier ways to invoke the same functionality. For example:

    let contents = doc.contents::<MyCollection>()?;

    Becomes:

    let contents = MyCollection::document_contents(&doc)?;

  • Backups made prior to 0.2.0 will not be able to be restored with this updated version. The document IDs are encoded differently than in prior versions.

Added

  • Optional compression is now available, using the LZ4 algorithm. StorageConfiguration::default_compression controls the setting. When using the bonsaidb crate, the feature can be made available using either local-compression or server-compression. When using bonsaidb-server or bonsaidb-local directly, the feature name is compression.

    This compression is currently applied on all chunks of data written to BonsaiDb that are larger than a hardcoded threshold. This includes the Key-Value store. The threshold may be configurable in the future.

    Some of the benchmark suite has been expanded to include comparisons between local storage with and without compression.

  • Added ability to "overwrite" documents without checking the stored revision information. Because this removes a layer of safety, it has its own permissible action: DocumentAction::Overwrite. The functions that have been added are:

    • connection::Connection::overwrite
    • connection::Collection::overwrite
    • schema::SerializedCollection::overwrite
    • schema::SerializedCollection::overwrite_into
    • document::CollectionDocument::overwrite
    • transaction::Transaction::overwrite
    • transaction::Operation::overwrite

Changed

  • Internal dependencies between crates are now pinned based on their needs. This means that bonsaidb-sever will require a matching verison of bonsaidb-local when compiling. A simple example of a change that is a breaking compilation change but is not breaking from a compatibility standpoint is a change to a structure where #[serde(rename)] is used to remap an old value to a new value.

    The only crate currently not pinning its dependencies is bonsaidb-keystorage-s3. This crate, and hopefully many crates to come, are only tying themselves to the public API of BonsaiDb.

    This may generate slightly more crate updates than absolutely necessary, but for a small team it seems like the most manageable approach.

Fixed

  • The view system now tracks an internal version number in addition to the version specified in the view definiton. This allows internal structures to be upgraded transparently.
  • Applying a transaction to a collection with a unique view now ensures a view mapping job has finished if the view's integrity check spawns one before allowing the transaction to begin.

v0.1.2

Fixed

  • #196: Fixed a panic a user reported relating to an undocumented panic within tokio::time::sleep().

v0.1.1

Added

  • SchemaName:private and CollectionName::private are two new constructors that allow defining names without specifying an authority. Developers creatingreusable collections and/or schemas should not use these methods as namespacing is meant to help prevent name collisions.

  • connection::Collection::all() and SchemaCollection::all() have been implemented as simple wrappers around list(..).

  • #146, #187: The Schema, Collection, and View traits can now be derived rather than manually implemented:

    #[derive(Debug, Schema)]
#[schema(name = "my-schema", collections = [Shape])]
struct MySchema;

#[derive(Debug, Serialize, Deserialize, Collection)]
#[collection(name = "shapes", views = [ShapesByNumberOfSides])]
struct Shape {
    pub sides: u32,
}

#[derive(Debug, Clone, View)]
#[view(collection = Shape, key = u32, value = usize, name = "by-number-of-sides")]
struct ShapesByNumberOfSides;

  • Memory-only instances of Storage can be created now. This is primarily intended for testing purposes.

Changed

  • Inline examples have been added for every connection::Collection and connection::View function.
  • All examples have been updated to the new derive macro syntax. Additionally, documentation and examples for deriving Schema, Collection, and View have been added to the respective traits.

v0.1.0

Added

  • bonsaidb::local::admin now exposes collections that are used to manage BonsaiDb.
  • Ability to add users, set a user's password, and log in as a user.
  • Each bonsaidb::local::Storage now has a unique ID. It will be randomly generated upon launch. If for some reason a random value isn't desired, it can be overridden in the Configuration.
  • Centralized secrets vault: Enables limited at-rest encryption. See bonsaidb::core::vault for more information.
  • For serializable types, Collection now defines easier methods for dealing with documents. NamedCollection allows collections that expose a unique name view to have easy ways to convert between IDs and names.
  • Server Backend trait now defines connection lifecycle functions that can be overridden to customize behavior when clients connect, disconnect, or authenticate.
  • Client now has a build() method to allow for customizing connections.
  • CustomApi responses can now be sent by the server via ConnectedClient::send(). The client can now register a callback to receive out-of-band responses.
  • Backend has a new associated type, CustomApiDispatcher which replaces Server::set_custom_api_dispatcher. This change allows custom api dispatchers to receive the server or client during initialization if needed. For example, if a custom API needs to know the caller's identity, you can store the client in your dispatcher and access it in your handlers.
  • Backend has a new associated type: ClientData. This associated type can be used to associate data on a per-ConnectedClient basis.
  • ServerConfiguration has a new setting: client_simultaneous_request_limit. It controls the amount of query pipelining a single connection can achieve. Submitting more queries on a single connection will block reading additional requests from the network connection until responses have been sent.
  • ServerConfiguration now supports authenticated_permissions, allowing a set of permissions to be defined that are applied to any user that has successfully authenticated.
  • CollectionView is a new trait that can be implemented instead of View. The map() function takes a CollectionDocument parameter that is already deserialized for you.
  • bonsaidb_core now has two new macros to ease some tedium of writing simple views: define_basic_mapped_view and define_basic_unique_mapped_view.
  • BonsaiDb now uses log internally to report errors that are being ignored or happened in a background task.
  • Multiple crates now offer an "instrument" feature which enables instrumentation using the tracing ecosystem.
  • Moved all database() functions to StorageConnection. This allows fully generic code to be written against a "server".
  • Added listen_for_shutdown() which listens for SIGINT and SIGQUIT and attemps to shut the server down gracefully.
  • Automatic TLS certificate acquisition can be performed using ACME on TCP port 443. This is automatically performed if the feature flag is enabled.
  • BonsaiDb server can now listen for generic TCP connections with and without TLS. This is primarily designed to allow extending the HTTP port to support additional HTTP endpoints than just websockets. However, because the TLS certificate aquired on port 443 can be used in other protocols such as SMTP, it can be useful to allow BonsaiDb to control the networking connections. listen_for_tcp_on and listen_for_secure_tcp_on accept a parameter that implements the TcpService trait. See the Axum example for how this integration can be used in conjunction with websockets.
  • Added convenience methods to Transaction and Operation to make it easier to build multi-operation transactions.
  • The Key trait is now automatically implemented for tuples of up to 8 entries in length.
  • CollectionDocument::modify() enables updating a document using a flow that will automatically fetch the document if a conflict is detected, re-invoking the callback to redo the modifications. This is useful for situations where you may have cached a document locally and wish to update it in the future, but don't want to refresh it before saving. It also will help, in general, with saving documents that might have some contention.
  • CustomServer::authenticate_client_as() allows setting a ConnectedClient's authenticated user, skipping BonsaiDb's authentication. This allows developers to still utilize the users and permissions within BonsaiDb while authenticating via some other mechanism.
  • SerializedCollection::list() and SerializedCollection::get_multiple() have been added to make it easier to retrieve CollectionDocument<T>s.

Changed

  • Configuration has been refactored to use a builder-style pattern. bonsaidb::local::config::Configuration has been renamed StorageConfiguration, and bonsaidb::server::ServerConfiguration has been renamed ServerConfiguration.

  • Database::open_local and Storage::open_local have been renamed to open.

  • Database::open, Storage::open, and Server::open no longer take a path argument. The path is provided from the configuration.

  • Listing all schemas and databases will now include the built-in admin database.

  • The underlying dependency on sled has been changed for an in-house storage implementation nebari.

  • The command-line interface has received an overhaul.

    • A new trait, CommandLine can be implemented on a type that implements Backend to utilize the built-in, extensible command line interface. An example of this is located at ./examples/basic-server/examples/cli.rs.
    • The parameter types to execute() functions have changed.
    • This interface will receive further refinement as part of switching to clap 3.0 once it is fully released.

  • View::map now returns a Mappings instead of an Option, allowing for emitting of multiple keys.

  • View mapping now stores the source document header, not just the ID.

  • ServerConfiguration::default_permissions has been changed into a DefaultPermissions enum.

  • Changed the default serialization format from CBOR to an in-house format, Pot.

  • Key now has a new associated constant: LENGTH. If a value is provided, the result of converting the value should always produce the length specified. This new information is used to automatically implement the Key trait for tuples.

  • The Key implementation for EnumKey has been updated to use ordered-varint to minimize the size of the indexes. Previously, each entry in the view was always 8 bytes.

  • Connection and SerializedCollection have had their insert/insert_into functions modified to include an id parameter. New functions named push and push_into have been added that do not accept an id parameter. This is in an effort to keep naming consistent with common collection names in Rust.

  • Operation::insert and Command::Insert now take an optional u64 parameter which can be used to insert a document with a specific ID rather than having one chosen. If an document exists already, a conflict error will be returned.

  • bonsaidb::local::backup has been replaced with Storage::backup and Storage::restore. This backup format is incompatible with the old format, but is built with proper support for restoring at-rest encrypted collections. Backups are not encrypted, but the old implementation could not be updated to support restoring the documents into an encrypted state.

    This new functionality exposes BackupLocation, an async_trait that enables arbitrary backup locations.

  • KeyValue storage has internally changed its format. Because this was pre-alpha, this data loss was premitted. If this is an issue for anyone, the data is still there, the format of the key has been changed. By editing any database files directly using Nebari, you can change the format from "namespace.key" to "namespace\0key", where \0 is a single null byte.

  • ExecutedTransactions::changes is now a Changes enum, which can be a list of ChangedDocuments or ChangedKeys.

  • The Key-Value store is now semi-transactional and more optimized. The behavior of persistence can be customized using the key_value_persistence option when opening a BonsaiDb instance. This can enable higher performace at the risk of data loss in the event of an unexpected hardware or power failure.

  • A new trait, SerializedCollection, now controls serialization within CollectionDocument, CollectionView, and other helper methods that serialized document contents. This allows any serialization format that implements transmog::Format can be used within BonsaiDb by setting the Format associated type within your SerializedCollection implementation.

    For users who just want the default serialization, a convenience trait DefaultSerialization has been added. All types that implement this trait will automatically implement SerializedCollection using BonsaiDb's preferred settings.

  • A new trait, SerializedView, now controls serialization of view values. This uses a similar approach to SerializedCollection. For users who just want the default serialization, a convenience trait DefaultViewSerialization has been added. All types that implement this trait will automatically implement SerializedView using BonsaiDb's preferred settings.

    The view-histogram example has been updated to define a custom transmog::Format implementation rather creating a Serde-based wrapper.

  • View has been split into two traits to allow separating client and server logic entirely if desired. The ViewSchema trait is where map(), reduce(), version(), and unique() have moved. If you're using a CollectionView, the implementation should now be a combination of View and CollectionViewSchema.

  • CollectionName, SchemaName, and Name all no longer generate errors if using invalid characters. When BonsaiDb needs to use these names in a context that must be able to be parsed, the names are encoded automatically into a safe format. This change also means that View::view_name(), Collection::collection_name(), and Schema::schema_name() have been updated to not return error types.

  • The Document type has been renamed to OwnedDocument. A trait named Document has been introduced that contains most of the functionality of Document, but is now implemented by OwnedDocument in addition to a new type: BorrowedDocument. Most APIs have been updated to return OwnedDocuments. The View mapping process receives a BorrowedDocument within the map() function.

    This refactor changes the signatures of ViewSchema::map(), ViewSchema::reduce(), CollectionViewSchema::map(), and CollectionViewSchema::reduce().

    The benefit of this breaking change is that the view mapping process now can happen with fewer copies of data.

  • A new function, query_with_collection_docs() is provided that is functionally identical to query_with_docs() except the return type contains already deserialized CollectionDocument<T>s.

  • Moved CollectionDocument from bonsaidb_core::schema to bonsaidb_core::document.

  • Due to issues with unmaintained crates, X25519 has been swapped for P256 in the vault implementation. This is an intra-alpha breaking change. Use the backup functionality with the existing version of BonsaiDb to export a decrypted version of your data that you can restore into the new version of BonsaiDb.

    If you have encryption enabled but aren't actually storing any encrypted data yet, you can remove these files from inside your database:

    • mydb.bonsaidb/master-keys
    • mydb.bonsaidb/vault-keys/ (or remove the keys from your S3 bucket)

  • query_with_docs() and query_with_collection_docs() now return a MappedDocuments structure, which contains a list of mappings and a BTreeMap containing the associated documents. Documents are allowed to emit more than one mapping, and due to that design, a single document can be returned multiple times.

Fixed

  • Adding two collections with the same name now throw an error.

v0.1.0-dev.4

Added

  • View::unique() has been added, allowing for a View to restrict saving documents when multiple documents would end up with the same key emitted for this view. For example, if you have a User collection and want to ensure each User has a unique email_address, you could create a View with the key of email_address and return true from unique(), and BonsaiDb will enforce that constraint.

  • Permissions support has been added across the platform with granular access. The bonsaidb::core::permissions module contains the data types involved. More documentation and examples are to-come -- users and roles haven't been added yet.

  • The initial underpinnings of customizing the BonsaiDb server have been added. First, there's the Backend trait. Right now, its only purpose is to allow defining a CustomApi. This allows applications built with BonsaiDb to extend the network protocol with Request and Response types that just need to support Serde. For a full example, check out this in-development Gooey example.

  • An initial version of a WebAssembly client is now supported. It only supports WebSockets. While there has been news of QUIC support in the browser, it's a limited implementation that only exposes an HTTP protocol. As such, it is incompatible with the BonsaiDb protocol. Eventually, we hope to support WebRTC as an alternative to TCP in WebAssembly. The example linked in the previous bullet point can be built and loaded in a browser.