From e8c8e964df065cb8b4fb91da620c618cc2e4ca18 Mon Sep 17 00:00:00 2001 From: Emilien Bevierre Date: Wed, 19 Jun 2024 14:47:07 +0100 Subject: [PATCH 1/4] Add .NET SDK Compression Docs --- modules/concept-docs/pages/compression.adoc | 117 ++++++++++++++++++-- 1 file changed, 108 insertions(+), 9 deletions(-) diff --git a/modules/concept-docs/pages/compression.adoc b/modules/concept-docs/pages/compression.adoc index 54989514..a4e161c2 100644 --- a/modules/concept-docs/pages/compression.adoc +++ b/modules/concept-docs/pages/compression.adoc @@ -1,5 +1,5 @@ = Compression -:description: Data compression to reduce traffic costs from app to Server. +:description: In response to increasing volumes of data being sent over the wire, Couchbase Data Platform provides data compression between the SDK and Couchbase Server. :page-topic-type: concept :page-edition: Enterprise Edition :page-aliases:ROOT:compression-intro,compression-intro @@ -7,21 +7,120 @@ include::project-docs:partial$attributes.adoc[] [abstract] -In response to increasing volumes of data being sent over the wire, Couchbase Data Platform now provides data compression between the SDK and Couchbase Server. -However, stable Snappy support is not yet available in Microsoft's https://dotnet.microsoft.com/en-us/[.NET platform]. +{description} +[Note] +==== +From version 3.5.2, the .NET SDK enables compression by default using https://www.nuget.org/packages/Snappier[Snappier], a performance-focused C# port of https://github.com/google/snappy[Snappy] provided by https://btburnett.com/[Brant Burnett]. This was previously opt-in only via installing the corresponding https://www.nuget.org/packages/Couchbase.Extensions.Compression.Snappier/1.0.0-beta.1[Couchbase.Extensions] package. +==== +== Overview -Couchbase Server (in the Enterprise Edition) stores documents in compressed form, xref:7.1@server:learn:buckets-memory-and-storage/compression.adoc#compression-modes[when it is enabled in the server for a particular bucket], using Snappy Compression. -As the Snappy compression library is not available for .NET, the server will automatically uncompress any compressed documents before sending them to the .NET client. +Documents may already be stored compressed with Snappy on the server. +New documents may be passed from client applications to the server in compressed form, saving around 40% in bandwidth (depending on the document content and size), and also transmission time. +These operations take place automatically, after the SDK negotiates the capability with the server, and do not require any changes on the client side. -If compression is set to _active_ on the server, documents will be stored there in compressed form, even though the .NET client has sent them uncompressed, thus saving storage space (but not network bandwidth). +For SDKs with Snappy compression enabled, documents will be automatically compressed if compression is not turned `Off` in xref:7.1@server:learn:buckets-memory-and-storage/compression.adoc#compression-modes[Couchbase Server] (see xref:#minimum-size[below]). +Instructions to disable compression can be found at xref:#threshold[the bottom of the page]. +== Usage and customization +Starting with .NET SDK v3.5.2, Snappier compression is enabled by default in the ClusterOptions. +You can disable compression or customize certain settings, both via ClusterOptions or Connection String parameters: -== Community-Supported Snappy Library +[source,dotnet] +---- +// ConnectionString parameters +var connectionString = "couchbase://localhost:8091?compression=true&compression_min_size=512&compression_min_ratio=0.82"; -Whilst there is no official Microsoft-supported Snappy compression in the .NET platform, the community has produced a promising option in https://github.com/brantburnett/Snappier[Snappier] -- -integrated with the the https://github.com/couchbaselabs/Couchbase.Extensions[.NET Couchbase.Extensions Library], +// ClusterOptions parameters +var clusterOptions = new ClusterOptions +{ + Compression = true, + CompressionMinRatio = 0.82f, // Float value between 1 and 0. + CompressionMinSize = 512 // Integer value represented in bytes. +}; + +// Connect +clusterOptions.WithConnectionString(connectionString); +clusterOptions.WithCredentials("Administrator", "password"); + +var cluster = await Cluster.ConnectAsync(clusterOptions).ConfigureAwait(false); +---- + +Please refer to the https://docs.couchbase.com/sdk-api/couchbase-net-client/api/Couchbase.ClusterOptions.html#Couchbase_ClusterOptions_Compression[API Documentation] for details on `CompressionMinRatio` and `CompressionMinSize`. + +== **Advanced**: Bring Your Own Compression +You can provide the SDK with your own implementation of Snappy compression/decompression by implementing the `ICompressionAlgorithm` interface, and passing it through your ClusterOptions with: +[source,dotnet] +---- +clusterOptions.WithCompressionAlgorithm(myCompressionImplementation); + +// See Couchbase.Compression.Snappier.Internal.SnappierCompression for details on how Snappier implements the interface. +---- + +[Note] +==== +Even when compression is turned `Off` server-side, Couchbase Server will decompress documents in memory, and store them recompressed on disk (using Snappy) (xref:7.1@server:learn:buckets-memory-and-storage/compression.adoc#compression-modes[see server compression docs]). + +It is therefore not recommended to implement a different compression algorithm than Snappy as this may very well cause data corruption. +==== + +== Limits + +The document must be below 20MiB in size in both compressed and uncompressed form. +Compression is only available in the Enterprise Edition of Couchbase Data Platform. + +[TIP] +==== +This size limit is enforced by Couchbase Server; in practice it will affect very few users, as most JSON documents are considerably smaller. +A compressed doument of just under 20MB, which is greater than 20,971,520 bytes (20 MiB) when uncompressed, will be rejected by the server as follows: + +* Couchbase Server decompresses the document to check that it is valid JSON, and is correctly compressed with _Snappy_, and at this point measures it against `max data size` (20 MiB). +* If the decompressed value's size exceeds this limit, the mutation is failed with a "too big" error code (E2BIG code 3). + +Therefore, where necessary, enforce document size limits in your application on _uncompressed_ documents. +==== + +== Operating Modes (Server-side) + +The three modes in which compression can be used, "off", "passive" and "active", are configured per bucket in the configuration settings on the cluster. + +.Compression Operating Modes +[#compression-operating-modes] +|=== +| | *OFF* | *PASSIVE* | *ACTIVE* + +| Negotiating SNAPPY +| ignore +| acknowledge +| acknowledge + +| Sending compressed data when SNAPPY feature *enabled* +| - +| accept +| accept + +| Sending compressed data when SNAPPY feature *disabled* +| reject as invalid +| reject as invalid +| reject as invalid + +| Receiving data which were previously compressed on the server +| server inflates and sends plain data +| server sends compressed data +| server sends compressed data + +| Receiving data which were *not* previously compressed on the server +| server sends plain data +| server sends plain data +| server might send compressed data (the compression is done in the background on the server) +|=== + + +== Compression for .NET SDK 3.5.1 and below + +https://github.com/brantburnett/Snappier[Snappier] is +integrated with the https://github.com/couchbaselabs/Couchbase.Extensions[.NET Couchbase.Extensions Library]. Since xref:project-docs:sdk-release-notes.adoc#version-3-4-10[.NET SDK 3.4.10], the .NET SDK has been able to work with external Snappy libraries as the setting for the Server flag `SnappyEverywhere` is now supported. Install Snappier with the https://github.com/couchbaselabs/Couchbase.Extensions[.NET Couchbase.Extensions Library], From b96f61d037a6888e4d77bf0ceed9392de45965ff Mon Sep 17 00:00:00 2001 From: Emilien Bevierre Date: Wed, 19 Jun 2024 15:05:55 +0100 Subject: [PATCH 2/4] Small fixes --- modules/concept-docs/pages/compression.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/concept-docs/pages/compression.adoc b/modules/concept-docs/pages/compression.adoc index a4e161c2..c57077d8 100644 --- a/modules/concept-docs/pages/compression.adoc +++ b/modules/concept-docs/pages/compression.adoc @@ -60,7 +60,7 @@ clusterOptions.WithCompressionAlgorithm(myCompressionImplementation); [Note] ==== -Even when compression is turned `Off` server-side, Couchbase Server will decompress documents in memory, and store them recompressed on disk (using Snappy) (xref:7.1@server:learn:buckets-memory-and-storage/compression.adoc#compression-modes[see server compression docs]). +Even when compression is turned `Off` server-side, Couchbase Server will decompress documents in memory, and store them recompressed on disk (using Snappy) xref:7.1@server:learn:buckets-memory-and-storage/compression.adoc#compression-modes[see server compression docs]. It is therefore not recommended to implement a different compression algorithm than Snappy as this may very well cause data corruption. ==== @@ -117,13 +117,13 @@ The three modes in which compression can be used, "off", "passive" and "active", |=== -== Compression for .NET SDK 3.5.1 and below +== Compression for .NET SDK 3.4.10 to 3.5.1 https://github.com/brantburnett/Snappier[Snappier] is integrated with the https://github.com/couchbaselabs/Couchbase.Extensions[.NET Couchbase.Extensions Library]. Since xref:project-docs:sdk-release-notes.adoc#version-3-4-10[.NET SDK 3.4.10], the .NET SDK has been able to work with external Snappy libraries as the setting for the Server flag `SnappyEverywhere` is now supported. -Install Snappier with the https://github.com/couchbaselabs/Couchbase.Extensions[.NET Couchbase.Extensions Library], +Install Snappier with the https://github.com/couchbaselabs/Couchbase.Extensions[.NET Couchbase.Extensions] Library, // See the https://github.com/brantburnett/Snappier[Snappier GitHub page] for information on installing the external Snappier library, or get it from https://www.nuget.org/packages/Snappier[NuGet]. From e8a3d3fdc12984c0e328df02848e058adcdfdaec Mon Sep 17 00:00:00 2001 From: Emilien Bevierre Date: Thu, 20 Jun 2024 10:33:19 +0100 Subject: [PATCH 3/4] Ammend introduction --- modules/concept-docs/pages/compression.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/concept-docs/pages/compression.adoc b/modules/concept-docs/pages/compression.adoc index c57077d8..9d8387bb 100644 --- a/modules/concept-docs/pages/compression.adoc +++ b/modules/concept-docs/pages/compression.adoc @@ -11,7 +11,7 @@ include::project-docs:partial$attributes.adoc[] [Note] ==== -From version 3.5.2, the .NET SDK enables compression by default using https://www.nuget.org/packages/Snappier[Snappier], a performance-focused C# port of https://github.com/google/snappy[Snappy] provided by https://btburnett.com/[Brant Burnett]. This was previously opt-in only via installing the corresponding https://www.nuget.org/packages/Couchbase.Extensions.Compression.Snappier/1.0.0-beta.1[Couchbase.Extensions] package. +From version 3.5.2, the .NET SDK enables compression by default using https://www.nuget.org/packages/Snappier[Snappier], a performance-focused C# port of https://github.com/google/snappy[Snappy]. This was previously opt-in only via installing the corresponding https://www.nuget.org/packages/Couchbase.Extensions.Compression.Snappier/1.0.0-beta.1[Couchbase.Extensions] package. ==== == Overview From 4ff339c324ac7defe9d832e2f3bc4eef0114e056 Mon Sep 17 00:00:00 2001 From: Emilien Bevierre Date: Mon, 24 Jun 2024 15:46:45 +0100 Subject: [PATCH 4/4] Update Couchbase .NET SDK in example code --- modules/howtos/examples/SubDocument.csx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/howtos/examples/SubDocument.csx b/modules/howtos/examples/SubDocument.csx index b3d04d84..4d95c7e4 100644 --- a/modules/howtos/examples/SubDocument.csx +++ b/modules/howtos/examples/SubDocument.csx @@ -3,7 +3,7 @@ // dotnet script SubDocument.csx // -#r "nuget: CouchbaseNetClient, 3.4.8" +#r "nuget: CouchbaseNetClient, 3.5.0" using System.Threading.Tasks; using Couchbase;