NaN Values in rzChiSquared Variables

[GNiendorf]

On an 1000 event sample from the PU200RelVal sample I noticed that there were some NaN values for the t5_rzChiSquared and pt5_rzChiSquared variables in the output ntuple. @YonsiG or anyone else, is this expected?

[YonsiG]

Yes, I think it is known because if a track lays exactly on one line, we cannot build circles and calculate radius according to it, so the helix r-z chi2 returns NaN. There is some special treatment for the NaN tracks if pT>100. There could be still remaining tiny portion of tracks having NaN, but won't affect much I think.

cmssw/RecoTracker/LSTCore/src/alpaka/Quintuplet.h

Line 607 in ff8446c

if (inner_pt > 100 || alpaka::math::isnan(acc, rzChiSquared)) {

[GNiendorf]

Yes, I think it is known because if a track lays exactly on one line, we cannot build circles and calculate radius according to it, so the helix r-z chi2 returns NaN. There is some special treatment for the NaN tracks if pT>100. There could be still remaining tiny portion of tracks having NaN, but won't affect much I think.

cmssw/RecoTracker/LSTCore/src/alpaka/Quintuplet.h

Line 607 in ff8446c

if (inner_pt > 100 || alpaka::math::isnan(acc, rzChiSquared)) {

The code that you linked though is "or" though right, not "and"? So even if the inner_pt is under 100 but the rzChisquared is NaN then it would still trigger that if statement? So the NaN tracks must be dealt with somewhere else if some are getting through?

[GNiendorf]

I guess if it falls into one of these if statements afterwards then the track is not rejected? Since the if statement with a NaN will return false always and that's probably how these NaN tracks are getting through? This doesn't seem like well defined behavior though.

Edit: Or I guess the tightCutFlag would be false still, so these tracks would get rejected still? If that's the case, I'm confused how any NaN values could get through. Looking at the code you linked, I see any NaN's prior to that if statement would return false so it must occur after?

cmssw/RecoTracker/LSTCore/src/alpaka/Quintuplet.h

Lines 662 to 666 in ff8446c

	if (layer4 == 8 and layer5 == 9) //5
	{
	if (rzChiSquared < 116.148f)
	TightCutFlag = true;
	return true;

[GNiendorf]

@slava77 Does it make sense to allow numbers to go NaN in our code and handle it through the is_nan check? Or should we always check before relevant division etc. that it won't produce nan?

[slava77]

@slava77 Does it make sense to allow numbers to go NaN in our code and handle it through the is_nan check? Or should we always check before relevant division etc. that it won't produce nan?

checking everywhere will have a cost.
It's better to write the math code in a way that NaNs are avoided wherever possible.
Check and fix or reject the candidate at some point where it matters, hopefully just once.
Specific to this chi2, it's an auxiliary/internal value that's used in a cut but does not affect properties (list of hits or kinematics) of a final candidate.
As you pointed out, there are categories where a cut on a value isn't even applied.

For DNN purposes it makes sense to go upstream and fix NaNs if possible. If NaNs are problematic for training, it's more practical to clamp all values to something sensible.

[GNiendorf]

I just checked and there are no NaN's being passed to addQuintupletToMemory. It's only when the results are being read in write_lst_ntuple.cc that the NaN's show up, so something must be going wrong...:

cmssw/RecoTracker/LSTCore/standalone/code/core/write_lst_ntuple.cc

Line 487 in ff8446c

ana.tx->pushbackToBranch<float>("t5_rzChiSquared", quintuplets->rzChiSquared[quintupletIndex]);

[GNiendorf]

As a follow-up on the skype conversation, the reason this occurs is that when fast math is enabled the is_nan checks in Quintuplet.h are disabled, leading to NaN's in the output ntuple.