[O2B-1365] Implement GAQ periods views #1808
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| 'use strict'; | ||
|
|
||
| const CREATE_GAQ_PERIODS_TIMESTAMPS_VIEW = ` | ||
| CREATE OR REPLACE VIEW gaq_periods_timestamps AS | ||
| SELECT | ||
| data_pass_id, | ||
| run_number, | ||
| timestamp AS \`from\`, | ||
| NTH_VALUE(timestamp, 2) OVER ( | ||
| PARTITION BY data_pass_id, | ||
| run_number | ||
| ORDER BY ap.timestamp | ||
| ROWS BETWEEN CURRENT ROW AND 1 FOLLOWING | ||
| ) AS \`to\` | ||
| FROM ( | ||
| -- Two selects for runs' timestamps (in case QC flag's eff. period doesn't start at run's start or end at run's end ) | ||
| ( | ||
| SELECT gaqd.data_pass_id, | ||
| gaqd.run_number, | ||
| COALESCE(UNIX_TIMESTAMP(first_tf_timestamp), UNIX_TIMESTAMP(time_start), 0) AS timestamp | ||
| FROM global_aggregated_quality_detectors AS gaqd | ||
| INNER JOIN runs as r | ||
| ON gaqd.run_number = r.run_number | ||
| ) | ||
| UNION | ||
| ( | ||
| SELECT gaqd.data_pass_id, | ||
| gaqd.run_number, | ||
| UNIX_TIMESTAMP(COALESCE(last_tf_timestamp, time_end, NOW(3))) AS timestamp | ||
| FROM global_aggregated_quality_detectors AS gaqd | ||
| INNER JOIN runs as r | ||
| ON gaqd.run_number = r.run_number | ||
| ) | ||
| UNION | ||
| -- Two selectes for timestamps of QC flags' effective periods | ||
| ( | ||
| SELECT gaqd.data_pass_id, | ||
| gaqd.run_number, | ||
| COALESCE(UNIX_TIMESTAMP(qcfep.\`from\`), 0) AS timestamp | ||
| FROM quality_control_flag_effective_periods AS qcfep | ||
| INNER JOIN quality_control_flags AS qcf ON qcf.id = qcfep.flag_id | ||
| INNER JOIN data_pass_quality_control_flag AS dpqcf ON dpqcf.quality_control_flag_id = qcf.id | ||
| -- Only flags of detectors which are defined in global_aggregated_quality_detectors | ||
| -- should be taken into account for calculation of gaq_effective_periods | ||
| INNER JOIN global_aggregated_quality_detectors AS gaqd | ||
| ON gaqd.data_pass_id = dpqcf.data_pass_id | ||
| AND gaqd.run_number = qcf.run_number | ||
| AND gaqd.detector_id = qcf.detector_id | ||
| ) | ||
| UNION | ||
| ( | ||
| SELECT gaqd.data_pass_id, | ||
| gaqd.run_number, | ||
| UNIX_TIMESTAMP(COALESCE(qcfep.\`to\`, NOW(3))) AS timestamp | ||
| FROM quality_control_flag_effective_periods AS qcfep | ||
| INNER JOIN quality_control_flags AS qcf ON qcf.id = qcfep.flag_id | ||
| INNER JOIN data_pass_quality_control_flag AS dpqcf ON dpqcf.quality_control_flag_id = qcf.id | ||
| -- Only flags of detectors which are defined in global_aggregated_quality_detectors | ||
| -- should be taken into account for calculation of gaq_effective_periods | ||
| INNER JOIN global_aggregated_quality_detectors AS gaqd | ||
| ON gaqd.data_pass_id = dpqcf.data_pass_id | ||
| AND gaqd.run_number = qcf.run_number | ||
| AND gaqd.detector_id = qcf.detector_id | ||
| ) | ||
| ORDER BY timestamp | ||
| ) AS ap | ||
| `; | ||
|
|
||
| const DROP_GAQ_PERIODS_TIMESTAMPS_VIEW = 'DROP VIEW gaq_periods_timestamps'; | ||
|
|
||
| const CREATE_GAQ_PERIODS_VIEW = ` | ||
| CREATE OR REPLACE VIEW gaq_periods AS | ||
| SELECT | ||
| gaq_periods_timestamps.data_pass_id AS dataPassId, | ||
| gaq_periods_timestamps.run_number AS runNumber, | ||
| IF(gaq_periods_timestamps.\`from\` = 0, null, gaq_periods_timestamps.\`from\`) AS \`from\`, | ||
| IF(gaq_periods_timestamps.\`to\` = UNIX_TIMESTAMP(NOW(3)), null, gaq_periods_timestamps.\`to\`) AS \`to\`, | ||
| IF(COUNT( DISTINCT gaqd.detector_id ) > COUNT( DISTINCT qcfep.flag_id ), | ||
| null, | ||
| SUM(qcft.bad) >= 1 | ||
| ) AS bad, | ||
| IF(COUNT( DISTINCT gaqd.detector_id ) > COUNT( DISTINCT qcfep.flag_id ), | ||
| null, | ||
| SUM(IF(qcft.monte_carlo_reproducible, false, qcft.bad)) >= 1 | ||
| ) AS badWhenMcReproducibleAsNotBad, | ||
| SUM(qcft.bad) = SUM(qcft.monte_carlo_reproducible) AND SUM(qcft.monte_carlo_reproducible) AS mcReproducible, | ||
| GROUP_CONCAT( DISTINCT qcfv.flag_id ) AS verifiedFlagsList, | ||
| GROUP_CONCAT( DISTINCT qcfep.flag_id ) AS flagsList | ||
|
|
||
| FROM gaq_periods_timestamps | ||
| INNER JOIN global_aggregated_quality_detectors AS gaqd | ||
| ON gaqd.data_pass_id = gaq_periods_timestamps.data_pass_id | ||
| AND gaqd.run_number = gaq_periods_timestamps.run_number | ||
|
|
||
| LEFT JOIN ( | ||
| data_pass_quality_control_flag AS dpqcf | ||
| INNER JOIN quality_control_flags AS qcf | ||
| ON dpqcf.quality_control_flag_id = qcf.id | ||
| INNER JOIN quality_control_flag_types AS qcft | ||
| ON qcft.id = qcf.flag_type_id | ||
| INNER JOIN quality_control_flag_effective_periods AS qcfep | ||
| ON qcf.id = qcfep.flag_id | ||
| LEFT JOIN quality_control_flag_verifications AS qcfv | ||
| ON qcfv.flag_id = qcf.id | ||
| ) | ||
| ON gaq_periods_timestamps.data_pass_id = dpqcf.data_pass_id | ||
| AND qcf.run_number = gaq_periods_timestamps.run_number | ||
| AND gaqd.detector_id = qcf.detector_id | ||
| AND gaq_periods_timestamps.run_number = qcf.run_number | ||
| AND (qcfep.\`from\` IS NULL OR UNIX_TIMESTAMP(qcfep.\`from\`) <= gaq_periods_timestamps.\`from\`) | ||
| AND (qcfep.\`to\` IS NULL OR gaq_periods_timestamps.\`to\` <= UNIX_TIMESTAMP(qcfep.\`to\`)) | ||
|
|
||
| WHERE gaq_periods_timestamps.\`to\` IS NOT null | ||
|
|
||
| GROUP BY | ||
| gaq_periods_timestamps.data_pass_id, | ||
| gaq_periods_timestamps.run_number, | ||
| IF(gaq_periods_timestamps.\`from\` = 0, null, gaq_periods_timestamps.\`from\`), | ||
| IF(gaq_periods_timestamps.\`to\` = UNIX_TIMESTAMP(NOW(3)), null, gaq_periods_timestamps.\`to\`) | ||
| `; | ||
|
|
||
| const DROP_GAQ_PERIODS_VIEW = 'DROP VIEW gaq_periods'; | ||
|
|
||
| /** @type {import('sequelize-cli').Migration} */ | ||
| module.exports = { | ||
| up: async (queryInterface) => queryInterface.sequelize.transaction(async (transaction) => { | ||
| await queryInterface.sequelize.query(CREATE_GAQ_PERIODS_TIMESTAMPS_VIEW, { transaction }); | ||
| await queryInterface.sequelize.query(CREATE_GAQ_PERIODS_VIEW, { transaction }); | ||
| }), | ||
|
|
||
| down: async (queryInterface) => queryInterface.sequelize.transaction(async (transaction) => { | ||
| await queryInterface.sequelize.query(DROP_GAQ_PERIODS_VIEW, { transaction }); | ||
| await queryInterface.sequelize.query(DROP_GAQ_PERIODS_TIMESTAMPS_VIEW, { transaction }); | ||
| }), | ||
| }; | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -15,51 +15,6 @@ const { Op } = require('sequelize'); | |
| const { models: { QcFlag } } = require('..'); | ||
| const Repository = require('./Repository'); | ||
|
|
||
| /** | ||
| * @typedef GaqPeriod | ||
| * | ||
|
|
@@ -101,33 +56,10 @@ class QcFlagRepository extends Repository { | |
| */ | ||
| async findGaqPeriods(dataPassId, runNumber) { | ||
| const query = ` | ||
| SELECT * FROM gaq_periods | ||
| WHERE IF(gaq_periods.\`to\` = UNIX_TIMESTAMP(NOW(3)), null, gaq_periods.\`to\`) IS NOT NULL | ||
| AND gaq_periods.dataPassId = ${dataPassId} | ||
| ${runNumber ? `AND gaq_periods.runNumber = ${runNumber}` : ''} | ||
| `; | ||
|
|
||
| const [rows] = await this.model.sequelize.query(query); | ||
|
|
@@ -136,14 +68,14 @@ class QcFlagRepository extends Repository { | |
| runNumber, | ||
| from, | ||
| to, | ||
| flagsList, | ||
| }) => ({ | ||
| dataPassId, | ||
| runNumber, | ||
| from: from * 1000, // Change unix seconds to miliseconds | ||
| to: to * 1000, | ||
| contributingFlagIds: flagsList ? flagsList.split(',').map((id) => parseInt(id, 10)) : [], | ||
| })).filter(({ contributingFlagIds }) => contributingFlagIds.length > 0); | ||
| } | ||
|
|
||
| /** | ||
|
|
@@ -156,107 +88,70 @@ class QcFlagRepository extends Repository { | |
| * @return {Promise<RunGaqSubSummary[]>} Resolves with the GAQ sub-summaries | ||
| */ | ||
| async getRunGaqSubSummaries(dataPassId, { mcReproducibleAsNotBad = false } = {}) { | ||
| const query = ` | ||
| SELECT | ||
| gaq_periods.runNumber, | ||
| gaq_periods.dataPassId, | ||
| gaq_periods.bad, | ||
| gaq_periods.badWhenMcReproducibleAsNotBad, | ||
| SUM(gaq_periods.mcReproducible) > 0 AS mcReproducible, | ||
| GROUP_CONCAT(gaq_periods.verifiedFlagsList) AS verifiedFlagsList, | ||
| GROUP_CONCAT(gaq_periods.flagsList) AS flagsList, | ||
|
|
||
| IF( | ||
| ( | ||
| run.time_end IS NULL | ||
| OR run.time_start IS NULL | ||
| ), | ||
| IF( | ||
| SUM( | ||
| COALESCE(gaq_periods.\`to\` , 0) | ||
| + COALESCE(gaq_periods.\`from\`, 0) | ||
|
Comment on lines
+108
to
+109
There was a problem hiding this comment. I created a PR that aims to simplify a bit the readability of these, if you can have a look and bring here what can be (for example the changes in this if): https://github.com/AliceO2Group/Bookkeeping/pull/1802/files#diff-a63ab748af8aa9d059b804a24f2e444657dbaf1848679c1a5eb11f4bccac752aR269 There was a problem hiding this comment. No, you need to have some grouping function there. This |
||
| ) = 0, | ||
| 1, | ||
| null | ||
| ), | ||
| SUM( | ||
| COALESCE( | ||
| gaq_periods.\`to\`, | ||
| UNIX_TIMESTAMP(run.time_end) | ||
| ) | ||
| - COALESCE( | ||
| gaq_periods.\`from\`, | ||
| UNIX_TIMESTAMP(run.time_start) | ||
| ) | ||
| ) / ( | ||
| UNIX_TIMESTAMP(run.time_end) | ||
| - UNIX_TIMESTAMP(run.time_start) | ||
| ) | ||
| ) AS effectiveRunCoverage | ||
|
|
||
| FROM gaq_periods | ||
| INNER JOIN runs AS run ON run.run_number = gaq_periods.runNumber | ||
|
|
||
| WHERE gaq_periods.dataPassId = :dataPassId | ||
|
|
||
| GROUP BY | ||
| gaq_periods.dataPassId, | ||
| gaq_periods.runNumber, | ||
| gaq_periods.bad | ||
| `; | ||
|
|
||
| const [rows] = await this.model.sequelize.query(query, { replacements: { dataPassId } }); | ||
| return rows.map(({ | ||
| runNumber, | ||
| bad, | ||
| badWhenMcReproducibleAsNotBad, | ||
| effectiveRunCoverage, | ||
| mcReproducible, | ||
| flagsList, | ||
| verifiedFlagsList, | ||
| }) => ({ | ||
| runNumber, | ||
| bad: mcReproducibleAsNotBad ? badWhenMcReproducibleAsNotBad : bad, | ||
|
There was a problem hiding this comment. Why is that change here? There was a problem hiding this comment. in the stored view the particular column cannot be parametrised with There was a problem hiding this comment. Ok got it now make sense 👍 |
||
| effectiveRunCoverage: parseFloat(effectiveRunCoverage) || null, | ||
| mcReproducible: Boolean(mcReproducible), | ||
| flagsIds: flagsList ? [...new Set(flagsList.split(','))] : [], | ||
|
There was a problem hiding this comment. Why is that change here? There was a problem hiding this comment. now the view my return |
||
| verifiedFlagsIds: verifiedFlagsList ? [...new Set(verifiedFlagsList.split(','))] : [], | ||
| })); | ||
| } | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From what I see as usage, you almost always re-convert the 0 and NOW to null.
Moreover, I am not sure when using the view that
NOW(3)will always return the same value as.tothat has been coalesced toNOW(3)inside the view, because of timing issue.Don't you think we can do something like select
COALESCE(UNIX_TIMESTAMP(first_tf_timestamp), UNIX_TIMESTAMP(time_start)) AS timestamp,COALESCE(UNIX_TIMESTAMP(first_tf_timestamp), UNIX_TIMESTAMP(time_start), 0) AS order` and sort by order?Then you can directly use timestamp without further if to make it null again
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am pretty sure that all
NOW(3)calls are evaluated to same value when called in one query.In the view, I used, indeed,
COALESCE(UNIX_TIMESTAMP(first_tf_timestamp), UNIX_TIMESTAMP(time_start), 0),and analogously for
end timestamps. The issue here is, to handle properly situation when all start or end timestamps of given run are not defined, then we have nullish boundary for QC flag, but in the view for GAQ timestamps I want to have properly ordered timestamps, in particular, when we have nullish end-boundary I want to have thenullvalue at the end of list, and when nullish start-boundary then at beginning, therefore there is coalesce toNOW(3)and0respectively, because it allows such orderingThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes I totally understand the ordering. What I proposed is to split in two and select 2 columns, one for ordering and one that you will actually return