CANDLE tests
These should be runnable interactively or via Jenkins.
The Jenkins project is here: https://jenkins-ci.cels.anl.gov/job/CANDLE-CR
First, make sure your desired Python installation is in PATH and PYTHONPATH as needed.
Second, simply run the test you want, e.g.:
$ ./cr-nt3-1.sh
or to run all tests:
$ ./cr-list.sh
Each test has a separate run directory (RUNDIR), but this is overwritten from run to run. We assume the numbered/dated output logs in Jenkins will capture everything needed to debug a historical test run.
Each test is associated with a TOKEN based on its directory name ("NT3", "P1B1", etc.). A corresponding setup-TOKEN.sh may be created that will automatically be sourced by the scripts.
check-count.sh-
Check counts of things like files
check-epoch.sh-
Output checker for epochs. Knows how to find epoch output for Keras and PyTorch
check-output.sh-
Output checker
check_tf.py-
Keras/TensorFlow settings reporter
checkout-benchmarks.sh-
Checks out or updates Benchmarks into a subdirectory
find-benchmarks.sh-
Find the directory
Benchmarks py-settings.sh-
Report Python settings
setup-tests.sh-
Sets up Python,
PATH, etc. Needed only once for a series of tests. setup-test.sh-
Sets up a particular Benchmark
utils.sh-
Generic shell scripting tools
cr-nt3-1.sh-
Test checkpoint/restart for NT3
cr-p1b1-torch-1.sh-
Test checkpoint/restart for P1B1/PyTorch
cr-1.sh-
Internal test pattern script. Runs any Benchmark and applies a sequence of CANDLE checkpoint/restart arguments.