fix: Deallocate row addresses and size arrays after exporting #246
Conversation
| def getRowAddresses: Array[Long] = { | ||
| val array = rowAddresses.toArray | ||
| rowAddresses = null | ||
| array | ||
| } | ||
|
|
||
| def getRowSizes: Array[Int] = { | ||
| val array = rowSizes.toArray | ||
| rowSizes = null | ||
| array | ||
| } |
There was a problem hiding this comment.
hmmm, these are writing operations, I don't think it's suitable to put them in the getters? Besides, the getNumRows may still access rowAddresses.
How about adding a new method?
def destructiveRowAddressAndSizes: (Array[Long], Array[Int])There was a problem hiding this comment.
They won't be accessed again after getRowAddresses and getRowSizes get called. Once they are called, they are exported to native side for spilling. After that, RowPartition will be reset.
There was a problem hiding this comment.
i.e.,
long addrs[] = getRowAddresses();
long sizes[] = getRowSizes;
nativeSpilling(adds, sizes);
rowPartition.reset();
There was a problem hiding this comment.
They won't be accessed again after getRowAddresses and getRowSizes get called
I see. Maybe I'm being nitpicking. I think caller would be surprise to see the state change of RowPartition after calling a getter method. A more appropriate method name or some comments would help.
This reverts commit 4cf7a3d.
|
Merged. Thanks. |
…#246) When we export row addresses and sizes to native, they are copied to separate arrays. We should deallocate array buffers after exporting them to reduce memory allocation.
Which issue does this PR close?
Closes #.
Rationale for this change
When we export row addresses and sizes to native, they are copied to separate arrays. We should deallocate array buffers after exporting them to reduce memory allocation.
What changes are included in this PR?
How are these changes tested?