Salinity data columns to check

[jds485]

I went through the list of columns in the harmonized WQP salinity dataset. I looked at the unique entered values for columns that sounded like useful metadata to screen out sites/samples that we use. The list of columns and attributes are below. Some of these columns may have already been processed out of the final dataset, as completed by the harmonized WQP processing scripts.

1. MonitoringLocationTypeName – remove tidal and ditch samples? We can save tidal samples in a separate spreadsheet. For now we're focusing on non-tidal areas, but maybe we would add these in the future.

2. ResultSampleFractionText – remove filtered samples? I'm thinking filtering would remove/alter salt concentrations.

3. ResultDetectionConditionText – remove systematic contamination, flag the non-detects (looks like this flagging is already done, but how are the detection limit values reported in the final dataset?)

4. ActivityMediaName – remove Sediment

5. HydrologicCondition – could be useful to retain if enough samples have this info.

6. HydrologicEvent – remove “Spill” entries. What is the “Volcanic action” in DRB?! Maybe that should also be removed?

[jds485]

Other thoughts:
Some of the CharacteristicName values are Conductivity even though the param is Specific Conductivity. It might be safest to drop the few Conductivity values for now because they might not be temperature corrected.

Does specific conductance, lab indicate that the temperature correction was applied in the lab instead of when sampled? Does it matter where the correction was applied to get the correct specific conductance value?

[lekoenig]

Yeah, I noticed this as well. CharacteristicName comes from the WQP, whereas param is meant to be a generic parameter name assigned in the multisource data product. I don't see any documentation or comments in the code that clarify why samples coded under Conductivity were assigned to param = "Specific Conductance". These may have been assumed equivalent because nearly all (recent) conductance probes have built-in temperature compensation, or even because conductivity is sometimes used interchangeably to report specific conductance even though this is incorrect. Without additional information we can't know that these samples represent temperature-corrected values. I've created an issue to clarify this decision step in the harmonized data with Meg, but because these Conductivity samples represent such a small number of samples (n ~ 50), we could also just omit them.

Regarding Specific conductance, lab samples, benchtop meters typically have an integrated temperature probe so conductance is measured and corrected to 25°C in the lab. This protocol is consistent with USGS National Field Manual methods (https://pubs.usgs.gov/tm/09/a6.3/tm9-a6_3.pdf) and so I'm comfortable leaving the lab samples in (n >25,000 measurements).

[jds485]

I agree with dropping the Conductivity (C) samples for now because there are so few of them. If needed, we can change after hearing back from Meg.

conductivity is sometimes used interchangeably to report specific conductance even though this is incorrect

That is troubling. Do you know of EDA methods we could use to screen our SC dataset for possible C values? Maybe that was already done in the harmonization process?

Thanks for the info about lab measured SC. So we wouldn't expect these to have the "C reported as SC" problem.

[lekoenig]

Sounds good, I will drop the Conductivity samples from the processed data for now.

conductivity is sometimes used interchangeably to report specific conductance even though this is incorrect

This is mostly a shorthand/language issue, I think, and I would edit the above to say "talk about" rather than "report." I would be more concerned about assuming reported C samples represent SC rather than vice versa. For our data, the vast majority of samples were originally tagged in WQP as CharacteristicName = Specific conductance, so with the "specific" qualifier I trust that the measurements do actually represent SC with temperature correction. To give us further confidence, the dataset contains a column called ResultTemperatureBasisText which is equal to 25°C in nearly 70% of our filtered SC samples (otherwise NA). We could use this column as a flag to filter the data down further, but I don't feel that's necessary in this case.

[jds485]

Okay, thanks for clarifying. I agree that there's no need to filter out samples now. We can compare the model errors for the SC samples with and without a temperature reference to evaluate possibility of a data reporting problem. I'll make note of that in our modeling notes.

[jds485]

Documenting a thought from #5:

Assign a "practical detection limit" when no detection limit is reported (e.g., an arbitrary value that would definitely be detected by any instrument)

[jds485]

Migrating discussion from #5, specifically this

So in the harmonization process they just replaced that NA with a quantitation limit (typically a lower reporting limit or an analytical method detection limit) if one was provided

I'm a little confused. I'm interpreting that as "detection limits in the harmonized dataset are a mix of being reported in WQP and added as a best estimated detection limit." Is that right? Or are there some samples that are marked as below detection and still do not have a listed detection limit, even after filling in NAs.

[lekoenig]

Thanks for asking for clarification! For samples marked by the original user as 'below detection', there were two cases:

1. A detection limit was also reported by the original user. In that case, the sample is retained and the harmonized value is assumed equal to the originally-reported detection limit.
2. No detection limit was reported by the original user and so the censored sample gets dropped from the harmonized dataset.

Does that make sense? The harmonized dataset creators never generated their own best estimated detection limit, which would be challenging to do without more knowledge about the original methods.

[jds485]

Thanks, that makes sense now! I'm fine with not assigning a detection limit to samples without them reported. I think that's the safest approach. If we could come up with a maximum lower detection limit, I think it would be useful to have when coupled with mass balance term and a downstream gauge that constrains the mass.