-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Clickhouse import process #59
base: main
Are you sure you want to change the base?
Clickhouse import process #59
Conversation
6648c06
to
901afad
Compare
8966e72
to
aa61d3b
Compare
scripts/clickhouse_import_support/copy_mysql_database_tables_to_clickhouse.sh
Show resolved
Hide resolved
get_genetic_profile_id_list_query="SELECT genetic_profile_id FROM genetic_profile WHERE genetic_alteration_type NOT IN ('GENERIC_ASSAY', 'MUTATION_EXTENDED', 'STRUCTURAL_VARIANT')" | ||
query_argument_template="--query={0}" | ||
query_argument = query_argument_template.format(get_genetic_profile_id_list_query) | ||
clickhouse_client_obtain_genetic_profile_id_list = ["clickhouse", "client", "--config-file=clickhouse_client_config_2024-10-14-09-03-02.yaml", query_argument] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should the config-file always hardcoded to this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see below you have a TODO to deal with that.
@@ -0,0 +1,36 @@ | |||
DROP TABLE IF EXISTS sample_to_gene_panel_derived; | |||
DROP TABLE IF EXISTS gene_panel_to_gene_derived; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Aren't these commands stored somewhere else?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. This file was a placeholder. I need to add calls to the script which processes the .sql files from the github repo (clickhouse.sql and materialized_views.sql) and bursts them into individual sql statements (1 per file). Also, we need to detect and skip over the special cases (genetic_alteration_derived, generic_assay_data_derived).
So this file will be going away
return 1 | ||
fi | ||
update_management_database_name="${my_properties['mysql_update_management_database']}" | ||
### TODO : fix this |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this needs adding back.
return 1 | ||
fi | ||
update_management_database_name="${my_properties['mysql_update_management_database']}" | ||
### TODO : fix this |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add back?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes
also bugfixes added and TODO notes
…rloaded functions that clashed in mysql/sling/clickhouse shell scripts
…les_in_clickhouse_database_by_profile
2512f43
to
0affe74
Compare
- focus on derived table construction and wrap up steps Co-authored-by: Manda Wilson <[email protected]> Co-authored-by: Robert Sheridan <[email protected]>
c84d0a0
to
4713301
Compare
Scripts, property files, and documentation for the blue-green deployment strategy to enable MySQL/Clickhouse database updates during cancer study import.