v4.4.7

What's Changed

• Backport list versions and delete_snapshot race 8104588520 by @poodlewars in #2102

Full Changelog: v4.4.6...v4.4.7

The wheels are on PyPI. Below are for debugging:

v5.1.2

What's Changed

• Finalize staged data memory use improvements in (#2013) 755bbda

Full Changelog: v5.1.1...v5.1.2

---

The wheels are on PyPI. Below are for debugging:

v5.1.1

🐛 Fixes

• Fix string reference count leak by @willdealtry in #1998
• Fix issues with backwards compatibility from 5.1 by @willdealtry in #2017
• Claim more disk space for wheel building by @phoebusm in #1994
• Make static analysis a cron job by @vasil-pashov in #2003
• more tests on read_batch() by @grusev in #1987
• fix handling of empty DF in pandas 1.0 by @grusev in #2010
• Fix static analysis cron workflow by @vasil-pashov in #2009
• Update BSL table with 5.1 by @IvoDD in #2007
• Test new manylinux by @G-D-Petrov in #2025
• Fix nfs test setup fail by @phoebusm in #2022
• Introduce the block version ref key by @poodlewars in #1969

Full Changelog: v5.1.0...v5.1.1

The wheels are on PyPI. Below are for debugging:

v5.1.0

🚀 Features

• Enhancement 1895: Fully parallelise processing in read_batch (#1950)

🐛 Bugfixes:

• Fix reference counting on python strings (#1999)
• Make snapshot names in run_scenario more unique (#1961)
• Fix Mac ARM Build (#1960)
• Fix delete snapshot so that it doesn't orphan data keys or delete the wrong key (#1973)
• Bugfix 1970: Give helpful error message if and/or/not operators are provided in QueryBuilder operations (#1976)
• Bugfix 127: Improve error message when recursively normalized metastruct is too big (#1981)
  Fix publish result curl fail (#1974)
• Bugfix/1841/maintain empty series names (#1983)
• Bugfix/1937/do not write append ref keys when staging incompletes (#1985)
• Bugfix 1306: Add nonreg test for update failing with 1-nanosecond different timestamps (#1978)
• Test meaningful error message when reading incompletes from non-existent symbol (#1991)
• Bugfix 1655: Disallow / character in mongo lib names (#1992)

Full Changelog: v5.0.0...v5.1.0

The wheels are on PyPI. Below are for debugging:

v5.0.0

⚠️ API Changes

• date_range returned by get_info[_batch] was a datetime.datetime now returns a pandas.Timestamp with nanosecond precision (#1461)
• staged argument removed from write_pickle_batch (#1642)
• Set row_count/rows to None in get_description/get_info and batch versions thereof if symbol is pickled (#1664)
• Do not fallback to iterating snapshots if if version as_of int or timestamp not found in version chain in V2 API (#1672)
• Use float64 as the result type for all division operations in the processing pipeline (#1794)
• Make SymbolDescription return type match type hints (#1877)

🚀 Features

• Lazy dataframe implementation (#1703)
• Arbitrary clause ordering (#1860)
• Sort staged data before writing (#1869)
• Make library manager a LRU cache (#1930)
• Preserve nanoseconds and timezones in date range from get info (#1461)
• Do not fallback to iterating snapshots if if version as_of int or timestamp not found in version chain in V2 API (#1672)
• Use float64 as the result type for all division operations in the processing pipeline (#1794)

🐛 Fixes

• Improve full build time by 30% by adding PCH (#1764)
• Fix typo in docs flow (#1952)
• Fix entt dep management for conda builds (#1955)
• Run the pull request on the PR sha with the pull_request_target event (#1954)
• Fix the timers macro, configure it with a cmake option (#1914)
• Small fixes for mac arm compilation and fix some tests (#1905)
• Fixes for sort and finalize (#1763)
• Bugfix 1552: Set row_count/rows to None in get_description/get_info and batch versions thereof if symbol is pickled (#1664)
• Bugfix 1641: Remove unused staged arg from write_pickle_batch (#1642)
• Refactor 1749: make processing tests great again (#1758)
• Always use strict weak ordering in std::sort comparators (#1747)
• Use EncodedFieldImpl.blocks() rather than casting member variable _blocks (#1745)
• Fix batch_restore_version failing to use correct next_version_id (#1823)
• Bugfix arcticdb-man 96: Improve error message with pd.Timedelta columns (#1874)
• Bugfix 1865: Make SymbolDescription return type match type hints (#1877)
• Bugfix 1652: Improve error message when object dtype columns contains Timestamps with mixed timezones (#1880)
• Bugfix 1841: Correctly roundtrip None and empty string pd.Series names (#1878)
• Fix library options compatibility across ArcticDB versions (#1862)
• Bugfix 1830: fix resampling with multiindex (#1873)

Uncategorized

• Extend tests for merge sort (#1708)
• Check if object is an instance of QueryBuilder when comparing for equality in QueryBuilder.eq (#1754)
• Add resampling offset (#1743)
• Refactor/1722/remove composite from processing pipeline (#1741)
• Fix Segment use-after-move when replicating to NFS (#1756)
• Throw exception if appending using sort_and_finalize will create unordered index (#1760)
• Improve logging for stress tests (#1759)
• Remove rocksdb (#1761)
• Update BSL table for v4.5.0 (#1768)
• Handle KeyNotFoundException in recurse_index_key (#1766)
• Add Storage API to check for existence of a key matching a predicate (#1762)
• Revert "Handle KeyNotFoundException in recurse_index_key" (#1776)
• Add 1 Billion Row Challenge Notebook (#1774)
• Remove KeyNotFoundException catch in CopyCompressedInterStoreTask (#1778)
• Limit 3 item ref key bypass to only when loading undeleted versions (#1775)
• append docstring improvement: added that append on a new symbol will create it (#1787)
• Parallelize string handling, enable python type handlers, prepare for Arrow (#1698)
• Add interface to pass a cached entry to tombstone all (#1793)
• Fix sort index (#1796)
• Fix windpws CI by adding the string header in decimal.cpp (#1801)
• Remove unused RuntimeConfig feature (#1803)
• Add unimplemented increment counter (#1804)
• Pre arrow refactor (#1805)
• Fix regression in bool pickling (#1810)
• Add API to remove Prometheus metrics (#1806)
• Fix symbol list bug on compaction (#1324) (#1798)
• Allow metrics to be registered twice (#1816)
• Docs 1809: lazy dataframe documentation and example notebook (#1815)
• Bugfix 1818: Fix QueryBuilder equality checks (#1819)
• Support reading from prefixes that include a dot (#1820)
• Fix compilation errors for clang 18 (#1813)
• Add StagedDataFinalizeMethod to docs (#1744)
• Add demo video link to README.md (#1837)
• More sort and finalize fixes (#1799)
• Fix conda build (#1854)
• Fix row slicing with sort_and_finalize throw on column slicing when the column group is larger than the segment column size (#1838)
• Don't override the https value for backwards compatibility (#1840)
• Support the ignore missing key read option in recurse_index_keys (#1844)
• Add flag to clear staged data on failure to both finalize functions (#1856)
• Change test for library emptiness to allow specifying key types to exclude (#1861)
• Fixing clean builds (#1868)
• Extend LMDB instance lifetime (#1879)
• Allow customizing the most useful AWS SDK settings. (#1875)
• Refactor 1833:use entt as ecs (#1834)
• Fix mono CI issue (#1917)
• Fix lib_tool.read_to_keys (#1832)
• Skip lmdb compat tests for versions <= 4.5.0 (#1923)
• Use relative paths for compat tests (#1924)
• ci: Use mamba 2.0 via mamba-org/[email protected] (#1855)
• Remove unneeded step (#1929)
• Adds a type change utility for append data keys in lib_tool (#1932)
• fix introduction help (#1933)
• Rewrite the mkdocs template so the links point to the correct version (#1942)
• Change the pull_request trigger to pull_request_target (#1947)
• Revert pull_request_target back to pull_request (#1949)

New Contributors

• @grusev made their first contribution in #1933
• @maxim-morozov made their first contribution in #1947

Full Changelog: v4.5.1...v5.0.0

---

The wheels are on PyPI. Below are for debugging:

v4.5.1

🐛 Fixes

• Fix bug where libraries created with 4.5.0 are not compatible with older versions by @G-D-Petrov in #1842
• Sort and finalize fixes by @vasil-pashov in #1859
• Extend LMDB instance lifetime (#1879) by @vasil-pashov in #1908

Uncategorized

• Compat tests 451rc by @IvoDD in #1912
• Compat test flakyness fixes 451rc by @IvoDD in #1925
• Bugfix/1785/backport to 4.5.1 by @alexowens90 in #1927

Full Changelog: v4.5.0...v4.5.1

---

The wheels are on PyPI. Below are for debugging:

v4.5.0

🚀 Resampling (#1495)

We are pleased to introduce a first version of resampling into ArcticDB.
Using the QueryBuilder you can easily resample timseries data as part of the read. The following code will resample a symbol into hourly buckets, for example. Please see the QueryBuilder docs for more info.

q = adb.QueryBuilder()
q = q.resample("h").agg({"to_sum": "sum"})
lib.read("symbol", query_builder=q).data

🚀 Features

Remove Composite from processing pipeline (#1722)
Refactor VersionMap to include all combinations of LoadTypes and whether to include_deleted (#1443)
Multiple aggregators per column are now possible in QueryBuilder (#1468)
Add s3 custom ca support (#1338)
Add version_chain ASV benchmarks (#1508)
Support mongodb+srv connection strings (#1513)
Enhancement/515/support metadata in finalize staged data (#1525)
list_snapshots: Add option to skip loading metadata (#1531)
LibraryTool add read_index (#1533)
Make service alive checking more universal (#1607)
Enhancement/1610/method to force compact symbol list (#1624)

🐛 Fixes

Add maybe_unused to fix MSVC compilation error by @vasil-pashov in #1739
Properly hash descriptor in v1 (master) by @willdealtry in #1682
Bugfix 1692: fix batch read with row range no query builder (#1710)
Bugfix 1720: Speedup processing pipeline with lots of columns (#1723)
Fix use after move in op_log.cpp (#1732)
Implement support for very old style symbol list entries. (#1630)
Fix ArcticDB reading streaming data (#1647)
Add validate_index arg to staged data finalization in both V1 and V2 APIs (#1694)
Fix sort_and_finalize_staged_data (#1617)
Bugfix 1609: Allow parallel appends when incomplete index values match last existing index value in the symbol (#1619)
Fix SYMBOL_LIST Segment index out of bounds bug (#1473)
Fix overflow bug on append and update (#1585)
Bugfix/1250/compact incompletes rationalisation (#1467)
Guard against numpy v2 to prevent breakage when it gets released (#1465)
Consistently handle timezones in QueryBuilder filtering (#1483)
Align azure timeout with s3 (#1487)
bugfix/1247: Don't create an lmdb library if the name is invalid. (#1481)
Cleanup arctic client after usage in local_quary_builder in ASV (#1522)
Bugfix/1518/get description date range incorrect after delete data in range (#1523)
Bugfix 1507+1509: dynamic schema different index append fixes (#1529)
Only write ref key once when writing with prune_previous_versions (#1560)
Fix #1148: Show clearer error message for mismatched types in QueryBuilder (#1557)
Bugfix 1615: Fix segfault in benchmark_clause.cpp (#1616)
Bugfix 1612: Raise a meaningful exception type/message when an empty string is provided as a symbol/snapshot name (#1618)
Adapt Raise to use format string as message directly if no args are given (#1606)
Restore mistake in #1315 (#1482)
Update docstring: added section to clarify that an empty df does not delete (#1592)

New Contributors

• @StageTired made their first contribution in #1485

Full Changelog: v4.4.3...v4.5.0

v4.4.3

🐛 Fixes

• Release GIL on update and append (#1559)
• Only write ref key once when writing with prune_previous_versions (#1571)
• Fix overflow bug on append and update (#1585)
• Require numpy < 2 (#1465)

Uncategorized

---

The wheels are on PyPI. Below are for debugging:

v4.4.2

🐛 Fixes

• Fix segmentation fault related to Ray package (by reverting to C++ 17 build) #1542
• Fix get_description date range after using delete_data_in_range #1526
• Fix tail segfault with empty data #1543

---

Please see v4.4.1 release notes for the previous changes

v4.4.1

🐛 Fixes

• Bug fix in Azure Storage, clear to_delete list to allow deletion of more than 256 keys in a batch (#1482)

---

Please see v4.4.0 release notes for the main changes from the last release.