SNN classifiers failing on Pippin analysis run

[paulshah]

Run :
Adapted PIPPIN analysis.yml to process simulated DES data.

Pippin Output dir :
/scratch/midway2/rkessler/PIPPIN_OUTPUT/PS_ANALYSIS_DUST

Error :
FAILED: SuperNNovaClassifier SNNTESTV19_z_data task (1 jobs, deps ['DATADESSIM'])
Other classifiers SNIRF, SCONE etc run OK.

Example output log error file :
/scratch/midway2/rkessler/PIPPIN_OUTPUT/PS_ANALYSIS_DUST/3_CLAS/SNNTESTV19_z_data_1_DATADESSIM_SNNTRAINV19_z_TRAINDES_V19

May be connected to recent updates, as same run had previously worked. Extract from logfile :
/project2/rkessler/PRODUCTS/miniconda/envs/snn_gpu/lib/python3.6/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
^[[32m[Loading FITRES file] ^[[0m
^[[33m[Warning: No FITRES file to load] ^[[0m
^[[33m[All lcs used for salt and photometry samples] ^[[0m
^[[32m[Computing saltfit splits for 2-way classification] ^[[0m

[RickKessler]

There is indeed a recent sim update that broke scone, and may have broke SNN as well. I updated the sim to write full filter strings (e.g., 'DES-g') instead of the single-char representation (e.g., 'g'); this update is needed for SALT3training code. I fixed scone to store only the last character of the string, and SNN may need a similar fix.

See #1446

[anaismoller]

Thanks for raising the issue and pinging me @paulshah and @RickKessler

The issue is indeed the filter naming. No need to change SNN as we support changing the filter names. The solution can be implmented in PIPPIN by using --list_filters DES-g DES-i DES-r DES-z instead of --list_filters g i r z
@OmegaLambda1998 is this something we can change easily in PIPPIN?

As a test @paulshah I ran the --data for this job /scratch/midway2/rkessler/PIPPIN_OUTPUT/PS_ANALYSIS_DUST/3_CLAS/SNNTESTV19_z_data_1_DATADESSIM_SNNTRAINV19_z_TRAINDES_V19 with the change and worked.

[OmegaLambda1998]

Resolved in dessn/Pippin@e4e25a1

[paulshah]

Thank you @RickKessler , @anaismoller and @OmegaLambda1998 !

A different error is now raised in SNN :

Extract from logfile : /scratch/midway2/rkessler/PIPPIN_OUTPUT/PS_ANALYSIS_DUST/3_CLAS/SNNTESTV19_z_data_1_DATADESSIM_SNNTRAINV19_z_TRAINDES_V19/output.log

Traceback (most recent call last):
File “run.py”, line 213, in
raise e
File “run.py”, line 99, in
model_settings, model_file=model_file
File “/project2/rkessler/PRODUCTS/classifiers/supernnova/supernnova/validation/validate_rnn.py”, line 132, in get_predictions
list_data_test = tu.load_HDF5(settings, test=True)
File “/project2/rkessler/PRODUCTS/classifiers/supernnova/supernnova/utils/training_utils.py”, line 272, in load_HDF5
test,
File “/project2/rkessler/PRODUCTS/classifiers/supernnova/supernnova/utils/training_utils.py”, line 172, in fill_data_list
X_all[:, settings.idx_features_to_normalize] = X_clip
ValueError: shape mismatch: value array of shape (26,9) could not be broadcast to indexing result of shape (1,26)

[anaismoller]

@paulshah i couldn't find the folder, can you let me know if it is somewhere else? thanks!

[paulshah]

@anaismoller Unfortunately I've accidentally deleted it! I'll recreate it, but there's a bit of a log-jam on Midway at the moment which is stopping runs from completing. I'll post here when re-done.

[anaismoller]

thanks @paulshah don't hesitate to ping me by email as sometimes I don't get the github notifications. Given the holidays I may need to get to this on january, hope it is not time sensitive!

[paulshah]

@anaismoller I've recreated the error : same logfile as above!