Explore lightweight compression of intermediate state

[pmenon]

State needed during execution (i.e., hash-tables, sorters etc.) can potentially be large. Any structures that exceed cache can negatively impact performance. Moreover, it limits the number of concurrent queries the system can support. Memory is not free, despite what the in-memory literature claims (me included).

This issue is to explore lightweight mechanisms we can deploy to compress such intermediate state on-line during execution. Anything we do must balance the cost of compressing data with the potential performance benefits.

Ideas: here