Main Features

Optimized columnar storage on HDFS, S3, MinIO, local FS, GCS, and Redis. Significantly outperforms Parquet and ORC.
A distributed in-memory columnar cache that further improves I/O performance for data analytics.
External query engine integrations for Trino, Presto, Hive, and DuckDB. Significantly improves query performance in these query engines.
Metadata (database schema and data catalog) management for data lakes and warehouses.
Internal query accelerator (Pixels-Turbo) in serverless computing environments, including AWS Lambda and vHive (K8S+Knative+Firecracker).
REST query API that exposes Pixels as a serverless analytics service for external users.

Release Notes

What's Changed

debug hive by @bianhq in #1
make pixels-hive usable by @bianhq in #7
Finish pixels-hive by @bianhq in #8
refine docs of pixels-hive. by @bianhq in #9
refine pixels-hive doc and fix pixels-load. by @bianhq in #10
Refine code and docs. by @bianhq in #11
add comments by @bianhq in #18
Rename the prefix of packages to io.pixelsdb by @ray6080 in #22
[Issue 24]: add balancers for pixels cache. by @bianhq in #25
[Issue #24]: move file if it is not local by @bianhq in #28
HOTFIX: upgrade fastjson to version 1.2.58 for security by @bianhq in #29
[Issue #30]: make NUMA interleaved. by @bianhq in #31
[Issue #30]: add start-vmtouch and make it NUMA interleaved. by @bianhq in #32
[HOTFIX]: remove mysql-connector from dependencies due to license conflicts. by @bianhq in #33
[HOTFIX]: remove or upgrade insecure packages from dependencies. by @bianhq in #34
[HOTFIX]: rollback jackson to 2.8.1. by @bianhq in #35
[Issue #36]: add copyright and license notice. by @bianhq in #37
Revert "[Issue #36]: add copyright and licence notice." by @bianhq in #38
[Issue #36]: add copyright and license notice. by @bianhq in #39
[Issue #36]: fix copyright and licence notice. by @bianhq in #40
[Issue #36]: rename LICENCE file. by @bianhq in #41
[Issue #36]: update NOTICE. by @bianhq in #42
HOTFIX: import external jars when starting pixels. by @bianhq in #43
HOTFIX: hive docs. by @bianhq in #45
[Issue #47]: using different lock files for coordinator and datanode daemons. by @bianhq in #48
[Issue #49] fix bugs in VarcharArrayBlock by @bianhq in #50
[Issue #52]: implement direct shared memory read. by @bianhq in #53
[Issue #52]: reduce memory copy in BooleanColumnReader and IntegerColumnReader. by @bianhq in #54
[Issue #55]: reduce memory copy in column readers. by @bianhq in #56
[HOTFIX]: bug in DynamicIntArray.toArray() and redundant memory copy. by @bianhq in #57
[Issue #58]: add close method to resources. by @bianhq in #59
Issue #58: add gc threshold to optimize gc for small queries. by @bianhq in #62
[Issue #44]: implement Etcd metadata store. by @bianhq in #65
[Issue #67] implement cache read / write coordination. by @bianhq in #68
[Issue #67] implement three-phase cache update. by @bianhq in #69
Hotfix: fix bugs in CacheWriter initialization and cache update. by @bianhq in #70
HOTFIX: clean unused exceptions. by @bianhq in #71
[Issue #72]: optimize memory allocation and access in PixelsCacheReader. by @bianhq in #73
[Issue #72]: fix bugs in pixels-cache and implement loading radix from index file. by @bianhq in #77
[Issue #67]: Implement cache read lease and optimize read performance. by @bianhq in #79
[Issue #78]: avoid cache probing on uncached tables in Presto. by @bianhq in #80
[Issue #78]: avoid cache probing on uncached tables in Hive. by @bianhq in #81
[Issue #83]: implement JIT splitting for ordered path. by @bianhq in #84
[Issue #85]: fix list table error when schema is empty. by @bianhq in #86
[Issue #87]: remove explicit gc from PixelsReader. by @bianhq in #89
[Issue #88]: fix MAX_READER_COUNT. by @bianhq in #90
[Issue #91]: use three bytes for cache reader count. by @bianhq in #92
[Issue #94]: Support Date and Time types. by @bianhq in #95
[Issue #98]: fix insert/update related metadata service and getLayout. by @bianhq in #101
[Issue #99]: fix null value storage. by @bianhq in #102
[Issue #103]: fix and enhance predicate processing. by @bianhq in #104
[Issue #105]: fix endless execution for select count(*). by @bianhq in #106
[Issue #100]: refine type management and add varchar/char support. by @bianhq in #107
[Issue #108]: replace hdfs FileSystem api with the unified Storage api. by @bianhq in #110
[Issue #108]: implement LocalFS and global auto-increment id. by @bianhq in #112
[Issue #113]: collect the cumulative memory usage in pixels record reader. by @bianhq in #116
[Issue #115]: replace message queue implementation. by @bianhq in #117
[Issue #114]:support S3 storage and asynchronous I/O scheduling. by @bianhq in #118
[Issue #120]: support configurable S3 clients. by @bianhq in #122
[Issue #121]: fix listing objects. by @bianhq in #123
[Issue #124]: refine read path. by @bianhq in #125
[Issue #126]: add rate limit and request retry policy. by @bianhq in #127
[Issue #128]: implement request diversion and refine java package layout. by @bianhq in #130
[Issue #131]: implement projections for compact layout. by @bianhq in #134
[Issue #133]: fix and refine retry policy. by @bianhq in #135
[Issue #136]: refine thread factory for async read using sync client. by @bianhq in #137
[Issue #132]: upgrade supported Presto version from 0.192 to 0.215. by @bianhq in #138
[Issue #142]: fix mbps rate-limit. by @bianhq in #143
[Issue #145]: fix date type for Presto-0.215. by @bianhq in #148
[Issue #149]: fix configuration and dependency. by @bianhq in #150
[Issue #153]: add adaptive reading method. by @bianhq in #154
[Issue #144]: fix scripts and finish docs. by @bianhq in #157
[Issue #156]: fix empty schema. by @bianhq in #159
[Issue #158]: support loading data from and to arbitrary storage. by @bianhq in #161
[Issue #160]: compact from and to arbitrary storage, including tail files. by @bianhq in #162
[Issue #163]: fix bounded varchar/char type support. by @bianhq in #165
[Issue #164]: define storage scheme in CREATE statement. by @bianhq in #166
[Issue #167]: fix retained size calculation of VarcharArrayBlock. by @bianhq in #168
[Issue #169]: add session properties about layout-path enabling to pixels-presto. by @bianhq in #171
[Issue #172]: implement record cursor and enhance record reader. by @bianhq in #173
[Issue #174]: implement transaction server and pass query (trans) id into record reader. by @bianhq in #176
[Issue #175]: enhance transaction info of queries and pass query id into I/O schedulers. by @bianhq in #177
[Issue #1...