diff --git a/hybridse/using-zetasql.md b/hybridse/using-zetasql.md index cd11a22..3f7ad23 100644 --- a/hybridse/using-zetasql.md +++ b/hybridse/using-zetasql.md @@ -38,7 +38,7 @@ ZetaSQL is a SQL Analyzer Framework from Google. We are considering using it as We do some survey on ZetaSQL. ZetaSQL provides many APIs and services. -![zetasql analyzer workflow](./images/image-zetasql-workflow.png) +![zetasql analyzer workflow](../images/image-zetasql-workflow.png) The figure above demonstrate the zetasql's analyze workflow. But we **only** use its parser module, since It can fullfill our requirements. @@ -163,7 +163,7 @@ ZetaSQL supports standard ANSI SQL. We are going to integrate zetasql's parser 2. Transform `ASTStatement` into `LogicalPlan` . 3. Convert `ASTExpression` to `ExprNode` -![Figure2-use-zetasql-parser](./images/image-hybridse-new-parser.png) +![Figure2-use-zetasql-parser](../images/image-hybridse-new-parser.png) **Pros**: diff --git a/hybridse/images/image-hybridse-new-parser.png b/images/image-hybridse-new-parser.png similarity index 100% rename from hybridse/images/image-hybridse-new-parser.png rename to images/image-hybridse-new-parser.png diff --git a/hybridse/images/image-zetasql-workflow.png b/images/image-zetasql-workflow.png similarity index 100% rename from hybridse/images/image-zetasql-workflow.png rename to images/image-zetasql-workflow.png diff --git a/images/memory_pool.png b/images/memory_pool.png new file mode 100644 index 0000000..7403676 Binary files /dev/null and b/images/memory_pool.png differ diff --git a/fedb/api-server.md b/openmldb/api-server.md similarity index 100% rename from fedb/api-server.md rename to openmldb/api-server.md diff --git a/openmldb/memory-pool.md b/openmldb/memory-pool.md new file mode 100644 index 0000000..7aa4644 --- /dev/null +++ b/openmldb/memory-pool.md @@ -0,0 +1,84 @@ +- Start Date: 2021-06-24 +- Target Major Version: (2.4.0) +- Reference Issues: +- Implementation PR: + +# Memory Pool + +# Summary +OpenMLDB use [TCMalloc](https://github.com/gperftools/gperftools) as memory alloction lib currently. Like other mallocs, the heap managed by TCMalloc consists of a set of pages. A run of contiguous pages is represented by a Span object. A span can either be allocated, or free. But if only a small buffer in a span is not freed, this span can not release to os which is called memory fragment. The memory is already located in central free list when a partition in OpenMLDB is dropped. + +So We try to design a novel memory pool to avoid this problem. + +# Detailed Design + +When insert one record to OpenMLDB, it will call multiple alloctions and construct structs/classes as follows: +| data structure | size | description +| --- | --- | --- | +| key | key length | +| value | value length | +| DataBlock | 16 | +| KeyEntry | 40 | +| Node| 40 | +| Node| 32 | +| std::atomic*> node0[12] | 96 | 8 * height + +By analyzing the above data structure, we found that the memory alloction has the following situations and we can use different alloction strategy. + +| - | - | struct / class | alloction strategy | +| --- | --- | --- | --- | +| Normal | Fixed length | KeyEntry | boost::pool +| Normal | Variable length| Key | Tcmalloc/Var-length pool +| Timeserise | Fixed length | Node | TimeSerisePool +| Timeserise | Variable length | Value | TimeSerisePool + + +TimeSerisePool is composed of multiple TimeBuckets as followers. + +
+ +```C++ +class TimeBucket { + public: + struct Block { + Block* next; + char data[]; + }; + Clear(); // free the block list + void* Alloc(uint32_t size) { + object_num_++; + if (current_offset_ + size <= block_size_ - sizeof(Block)) { + void* addr = head_->data + current_offset_; + current_offset_ += size; + return addr; + } else { + Block* block = (Block*)malloc(block_size_); + current_offset_ = size; + block->next = head_->next; + head_ = block; + return head_->data; + } + } + private: + uint32_t block_size_; + uint32_t current_offset_; + uint32_t object_num_; + Block* head_; +}; +``` + +```C++ +class TimeSerisePool { + public: + Alloc(uint32_t size, uint64_t time); + Free(void*, uint64_t time); + private: + // key is the time / (60 * 60 * 1000) + std::map> pool_; +}; +``` + +Note: TimeSerisePool does not guarantee thread safety. + +# Adoption stratege +If we implement this proposal, there is no impact on existing interfaces \ No newline at end of file