Optimise Postgres post processing to speedup process #75
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
anilthanki
force-pushed
the
feature/optimise_post_processing
branch
from
fab259e to
b8e10b0
Compare
pmb59
requested changes
Oct 23, 2024
There was a problem hiding this comment.
I've proposed maintenance_work_mem Increase (to be confirmed by DBA).
PostgreSQL supports parallel query execution, e.g.
SET max_parallel_workers_per_gather = 4;
current value in SCXA db is max_parallel_workers_per_gather 2. Increasing could help too with long data-intensive db operations.
| - re-enable autovacuum (mandatory) | ||
| - add a check constraint on the partition value (mandatory) | ||
| */ | ||
|
|
There was a problem hiding this comment.
Increase memory for maintenance, current one used is 256MB
Suggested change
| SET maintenance_work_mem = '16GB'; |
| - cluster the partition tables on primary key (optional) -> blocking op | ||
| - collect statistics on the partition tables (mandatory) | ||
| */ | ||
|
|
There was a problem hiding this comment.
Suggested change
| SET maintenance_work_mem = '16GB'; |
There was a problem hiding this comment.
DBA confirmed The memory allocated to the VM is 16GB. So using this with SET max_parallel_workers_per_gather = 4; not possible.
| */ | ||
|
|
||
| CLUSTER scxa_analytics_<EXP-ACCESSION> USING scxa_analytics_<EXP-ACCESSION>_pk; | ||
| ANALYZE scxa_analytics_<EXP-ACCESSION>; |
There was a problem hiding this comment.
Suggested change
| ANALYZE scxa_analytics_<EXP-ACCESSION>; | |
| ANALYZE scxa_analytics_<EXP-ACCESSION>; | |
| RESET maintenance_work_mem ; |
|
@irisdianauy @anilthanki Summary of sequential actions from DBA, after VM mem increase:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Splitting up
05-post_processing.sql.templateinto05-01-post_processing.sql.templateand05-02-post_processing.sql.templatewith advise from DBA team to speedup process