Clickhouse mode of study view #11224

* Add Columnar SQL file to init Clickhouse DB * Refactored Mapper xml to extract StudyViewFilterMapper

* ✅ Add Unit test for StudyViewMapper Clickhouse * ✅ Update db props to include mysql and clickhouse datasources to fix tests * Address comments * Rename package to clickhouse * Update to static final * Use bean name instead of qualifier

* Create new wide table sql file and rename package * Remove genomic_event view * Add AlterationFilter to mutated_genes endpoint * Add AlterationFilter to mutated-genes endpoint * Fix unit test * Fix sonar issues * Add test for mutation types and status * remove unused imports

* add missing poc clinical data binning function

* Add sample_mv materialized view and use it in mappers

* Add Support for TotalProfiledCase Counts for Mutated-genes endpoint. * Create sql files to create new tables * Add unit test for totalProfiledCount * Add matching gene panel ids * Add TotalProfiledCountsWithoutPanelData * Add profileCount for genes without gene panel data * Add Comments for SQL * Update matching Gene Panel Ids * Clean up code * Fix test * Add query to get correct Gene Panels * Fix unit test * Add comments

* working poc * refactor logic into service, so clean * refactor for parameters builder, simplify min max logic, streamline service call * remove unused services and imports * remove more unused imports

* Implement molecular profile count endpoint using Clickhouse * Cleanup

* ✨ Add CNA Gene Endpoint * 🐛 Fix StudyViewFilterMapper.xml to allow ability to filter on gene and alteration * Fix merge conflict * Address comments * Fix unit tests * Fix sonar issues

* ✨ Add StructuralVariant-genes endpoint * Fix sonar issues * Update MatchingGenePanel request to return list * Create and use sample_derive * Update where sample_derived is stored to fix unit test

Co-authored-by: Bryan Lai <[email protected]>

* use clinical_data_derived instead of sample_clinical_attribute_numeric_mv and patient_clinical_attribute_numeric_mv * use clinical_attribute_meta instead of sample_clinical_attribute_numeric_mv and patient_clinical_attribute_numeric_mv * remove unused clinical data count methods and SQL * fix numericalClinicalDataCountFilter * Move CategoricalClinicalAttributeFilter to repository * remove unused columns * Add override to methods --------- Co-authored-by: haynescd <[email protected]>

…0857) * Add patient_id column to genomic_event_derived * Update sql to convert list of patients to list of samples

* refactor to use clickhouse * filter out empty attr values * edit comment * fix sonarcloud issues * use parallel stream, shaves off 5s * use newer mapping annotation

* simplify clinical data binning related SQL * fix numericalClinicalDataCountFilter

…ase64) (#10898)

* Add Clinical Events Timeline counts endpoint * Add comments for StudyViewFilter

* Fix alteration count services * Only add genes that are protein coding to gene_panel_to_gene table for WES * fix test * fix sonar issues * Add documentation for filtering out WES

* Create new endpoint treatment/patient-counts/fetch * Create StudyViewFilterHelper class * ⬆️ Upgrade CH client Version * Fix Merge Conflict issue * Update Treatment event name * Update to handle patient treatment filtering * Fix sonar issues * Fix sonar comments 2

…ent) (#10919)

* Add Sample Count Report Endpoint * Push counting logic to sql * Remove unused methods * Update PatientTreatments to pass int from db for totalPatientTreatmentCount * Fix tests * Add unit test for sample Treatments * Remove unused import * Fix indentation

* Fix Sample Treatment Endpoint * Fix Sample ID only StudyView Filter

* add unit tests for molecular-profile-counts-service * fix multiple-study scenario Update method name getGenomicDataCounts -> getMolecularProfileSampleCounts

Clickhouse implementations of mutation data counts & genomic data counts services --------- Co-authored-by: Qi-Xuan Lu <[email protected]>

* Add tests for case list sample counts endpoint and fix merge of lists across studies

* Implement custom-data-bin-counts associated filter * Create StudyViewFilterContext and Helper --------- Co-authored-by: Bryan Lai <[email protected]> Co-authored-by: Charles Haynes <[email protected]>

#10959) * Update Treatment Endpoint to handle event_type TREATMENT, previously only searched on Treatment * Handle all variations of Treatment

…#10974)

* Genomic and Generic Assay bin counts endpoints

fix clinical data NA counts for multiple studies --------- Co-authored-by: lismana <[email protected]>

* Fix CNA table definition

…10986) * 🐛 Fix Mutated Genes Filter bug with Or logic * ✅ Add Tests for Mutated-genes,Cna-genes, and structural-variant filters

…ompare them properly (#11021)

#11026) * Merge Alteration Counts that have conflicting HugoSymbols for Study View Page * Add comments

* add clickhouse custom data endpoints * move custom data binning logic to binner and util class * address review comments * remove wildcards * use basicDataBinner for custom data binning * split custom and clinical data binning case * make custom and clinical binning switch case more readable --------- Co-authored-by: Bryan Lai <[email protected]>

… session (#10998) * add samples field and handling to custom filters --------- Co-authored-by: Bryan Lai <[email protected]> Co-authored-by: Zhaoyuan (Ryan) Fu <[email protected]>

* Add NA for genomic-data-bin-counts * Filtering with NA for genomic-data-bin-counts * Fix when subqueries return empty for NA counts * Refinement and comments * Address comments * Fix intersection with parens (#11032) * Fix intersection with parens * Demo rfc80 poc genomic data filter tests (#11036) * add genomic data filter tests testing for missing NAs * add NA genomic data filter tests --------- Co-authored-by: Zhaoyuan (Ryan) Fu <[email protected]>

* Update to add QValue for Alteration Count Services * Update to return List instead of Set * Update to only add mut_sig to mutated_genes * Remove unneeded Exception * Add Qvalue for Structural Variant and CNA * Update logic to add gistic to map

* Add NA for generic-assay-bin-counts * Filtering with NA for generic-assay-data-bin-counts * Parens precautions * unskip generic assay api tests --------- Co-authored-by: alisman <[email protected]>

* Add Categorical Clinical Attribute NA filtering * add api tests --------- Co-authored-by: alisman <[email protected]>

* Add generic-assay-data-counts endpoint * Update GenericAssayFilter to support NA's * Add tests for generic assay data counts endpoint (categorical) --------- Co-authored-by: alisman <[email protected]>

…11071)

…ngle study (#11093)

* add separate property for clickhouse cache enabling --------- Co-authored-by: Bryan Lai <[email protected]> Co-authored-by: alisman <[email protected]>

* Fix trim trailing error * reduce substr logic using negative end index * Update clickhouse.sql --------- Co-authored-by: Robert Sheridan <[email protected]>

* add singleStudyUnfiltered condition to cacheable clickhouse methods * add condition for density and violin plot data methods * change singleStudyUnfiltered to just unfiltered --------- Co-authored-by: Bryan Lai <[email protected]>

* use samples filtered without numerical clinical data filter when converting patient clinical data to sample clinical data * make clinical data sample/patient ids match legacy ids, and use stable ids to count samples when internal id is null

* Fix errant genetic profile matches due to overlapping type suffix * fix legacy broken profile counts * Fix clinical event availability grouping (broken for multi study where stable sampleids are repeated)

…g for sample acquisition and specimen (#11143)

…ceptions (#11135) Co-authored-by: alisman <[email protected]>

* ♻️ Refactor NewStudyViewFilterUtil * Update SQL to return ClinicalDataCountItem.. NA's not implemented * Add NA counts to SQL for clinical data counts * Remove unused methods * Fix unit tests for clinical data counts * Remove unused import * Fix Review Comments --------- Co-authored-by: alisman <[email protected]>

…n assume total Profile Count

…tion

…dy view (#11185) * Fix counting of clinical data na when filtered out * Add caching and cleanup * Fix Sonar issues

* add api tests for circleci

This reverts commit 80ef8c9.

* add tests for profiled counts * fix profiled counts assertions --------- Co-authored-by: Bryan Lai <[email protected]>

#11209) * separate clickhouse tests into different files * change name of cna counts test file --------- Co-authored-by: Bryan Lai <[email protected]>

…inical data counts endpoint (#11192) Co-authored-by: alisman <[email protected]>

Use PR-specific image for circleci api tests

Co-authored-by: Bryan Lai <[email protected]>

* Hide select properties of ClinicalDataFilter from frontend * Update swagger decorators on clickhouse controller

… obj)

…#11155) * Add patient level filtering for aggregation * Patient level filtering works for non-NA * Categorical patient level filtering & clean up * Use new generic assay table schema

…y clickhouse_enabled is set (#11256) * Update cBioPortal to dynamically load ch Components only when property clickhouse_enabled is set * Update env var for circleCi

* Merge genomic data bins working * Workaround for clickhouse bug in numerical data parsing --------- Co-authored-by: alisman <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clickhouse mode of study view #11224

Clickhouse mode of study view #11224

Commits on Nov 24, 2024

Commits on Nov 25, 2024

Commits on Dec 2, 2024

Commits on Dec 3, 2024

Commits on Dec 4, 2024

Commits on Dec 5, 2024

Commits on Dec 6, 2024

Commits on Dec 7, 2024

Clickhouse mode of study view #11224

Are you sure you want to change the base?

Clickhouse mode of study view #11224

Commits on Nov 24, 2024

Commits on Nov 25, 2024

Commits on Dec 2, 2024

Commits on Dec 3, 2024

Commits on Dec 4, 2024

Commits on Dec 5, 2024

Commits on Dec 6, 2024

Commits on Dec 7, 2024