Skip to content

Latest commit

 

History

History
1226 lines (985 loc) · 108 KB

CHANGELOG.rst

File metadata and controls

1226 lines (985 loc) · 108 KB
Raw

CHANGELOG

v2.1 (Sep 5th, 2024)

Changes

  • Migrating from RefResolver to Referencing library to address API change in jsonschema #1802
  • Remove cobalt scheduler support #1773
  • Acquire system details and topology information using archspec, psutil #1772
  • Add support for specifying paths for docker, singularity, and podman in configuration file #1775
  • Suppress output of curl and installation of pip in setup.sh script. Buildtest will automatically install pip when sourcing setup script #1806, #1792

Buildspec Changes

  • Add support for post_run keyword to run commands after test is complete which does not impact test behavior #1778
  • Add support for paths keyword in configuration file to search for scheduler binaries in alternative location #1757
  • Add option to disable execute check using disable_check field #1786

Command Line Changes

  • Add support for specifying a file path when loading buildspec into cache via buildtest buildspec find -f #1808
  • Display output and error file for each test via buildtest build --display command which can be used toggle output and error #1784, #1779
  • Add support for buildtest build --strict to toggle set -eo pipefail into the test which will stop execution on first error. Also improve logic for container support #1777
  • Add option buildtest build --write-config-file to allow buildtest to write configuration file to alternate location when saving profile using --save-profile #1762
  • Validate test by name using buildtest buildspec validate --name #1785

Code Refactor

  • Major refactor to buildtest codebase #1763
  • Add utility modules to print rich tables #1780
  • Refactor logic for job dependency #1798

Documentation

  • Documentation for extracting line numbers in metrics fields #1781

v2.0 (Apr 16th, 2024)

Major Changes

This release comes with several improvements to batch scheduler integration including code refactoring of all schedulers. We have added supported for Torque scheduler. Furthermore, we have changed logic for PBS scheduler for polling jobs to be consistent with Torque since both schedulers are very similar. We have improved logging for batch jobs and scheduler detection. Furthermore we have added validation for queues to ensure executor validation takes place to invalidate an executor if queue is not found.

  • Add support for Torque batch scheduler #1734
  • Improve scheduler detection logic #1738, #1739
  • Change poll command to qstat -xf for PBS scheduler and qstat -f for Torque #1746
  • Improvement to LSF executors such as queue validation and testing LSF executors on OLCF system on Summit #1748, #1750

Buildspecs Changes

  • Add support for regular expression types such as re.search, re.match, and re.fullmatch using the re property for status check and metrics #1730
  • Add support for line count (linecount) and file line count (file_line_count) status check used for counting lines in a file or stdout/stderr. #1722, #1723
  • Remove burst buffer and data warp directives from spack schema #1717
  • Extract line number from stdout/stderr via linenum field used in status check particularly useful with regular expression search #1735

Command Line Changes

  • Add support for buildtest build --validate option that will be used for validating buildspecs and stop after parse stage. This feature will deprecate feature buildtest build --stage=parse #1729
  • Add dry run mode for buildtest via buildtest build --dry-run option. This feature will build the test but not run it. This feature will deprecate buildtest build --stage=build #1727
  • Add dryrun (--dryrun), write (--write) and failfast (--failfast) options to buildtest tutorial-examples command #1720
  • List all buildtest options via command line via buildtest --listopts #1715

Documentation Changes

  • Build prototype for AWS tutorial that includes several example buildspecs, configuration file and improvement to buildtest tutorial-examples command to help auto-generate examples. For instance we added positional argument to specify either buildtest tutorial-examples {aws,spack} which will be used to generate example docs for each section #1719
  • Rearrange documentation pages to move all status checks into single page #1733
  • Add documentation for Torque Executors #1743
  • Add documentation for LSF Executors #1745

Project Improvements

  • Add github workflow to run regression test examples with spack using github action spack/setup-spack to help increase code coverage for spack schema #1716
  • Change singularity invocation from singularity exec to singularity run when running singularity containers #1711
  • Fix a bug in argparser invocation when running buildtest --help-all that caused an error with showing all help options. #1708

v1.8 (Feb 14th, 2024)

Major Changes

  • Add support for container executors in buildtest configuration to allow one to run tests in a container. #1695
  • Build new spack container for spack tutorial #1700
  • Remove support for compiler schema from buildtest and all references from documentation and example tests #1686

General Improvements

  • Add command aliases for commands buildtest config executors list, buildtest config profiles list, buildtest config profiles, buildtest inspect list, buildtest report list. The command aliases are: buildtest config executors ls, buildtest config profiles ls, buildtest inspect ls, buildtest report ls, buildtest config prof. #1705
  • The command buildtest buildspec find --root will rebuild buildspec cache, when --root option is specified. Previously --rebuild option was required to rebuild cache which didn't make sense when --root option was specified. When root option is specified in configuration file and --root option is used, buildtest will use the command line option. Previously, buildtest would append both values into list. #1706
  • Remove latest and oldest keywords from buildtest configuration #1673
  • Add GitHub workflow for command line tutorial #1663
  • Trigger regression test when pyproject.toml or requirements.txt file is changed #1687
  • Rewrite documentation for Configuring Buildtest #1696
  • Remove jsonschema documentation workflow used for publishing schemas to github pages posted on branch gh-pages #1682. This change meant we removed all content from gh-pages branch in #1683.
  • Add compilation examples using script schema #1692

v1.7 (Nov 12th, 2023)

Buildspec Changes

  • Add support for containers via container property. Initial support includes specifying container runtime, container options, and run command to invoke container. Current platform support includes docker, podman, singularity. #1642.
buildspecs:
  container_commands_ubuntu:
    type: script
    executor: generic.local.bash
    description: run arbitrary linux commands in ubuntu container
    container:
      platform: "docker"
      image: ubuntu:latest
      command: bash -c "cat /etc/os-release"
    run: |
      ls -l /etc/os-release || true
  • Update schema for Performance Checks (assert_ge, assert_ne, assert_gt, assert_ge, assert_le, assert_lt, assert_range, contains, not_contains) to support for logical AND/OR. The mode key can be defined in each performance check, and list of assertions are defined under comparisons. #1648
status:
  assert_ge:
    mode: and
    comparisons:
    - name: copy
      ref: 5000
    - name: scale
      ref: 5500
    - name: add
      ref: 6000
    - name: triad
      ref: 6500
  • Change mode property under status check (status) to use values (AND|and|OR|or), this is to be consistent with use of mode under each performance check such as assert_ge. #1656

Command Line Changes

  • Remove executors via command line buildtest config executors remove. #1636
  • Extend support for --count for other queries in buildtest buildspec find command. Rewrite regression test for buildtest buildspec find #1638

Project Improvements

  • Buildtest will gracefully terminate running jobs when user hits CTRL-C. For batch jobs, the jobIDs will be cancelled. Add trap command in generated script to allow users to run script and catch signal interrupt. #1644
  • Improvement to polling implementation for batch jobs by showing number of jobs by type (PENDING, RUNNING, COMPLETED) in the table. Fix issue where FAILED slurm jobs were not reported. #1650
  • Add error message of tests in console output of buildtest build. #1643
  • Change re.match -> re.fullmatch when hostname with ones specified in configuration file. #1651
  • Rewrite NERSC CI regression tests, update buildtest configuration and example buildspecs to be run. #1639
  • Slurm jobs TIMEOUT and OUT_OF_MEMORY will be reported as completed jobs. #1653
  • Rewrite documentation for Batch Scheduler, Troubleshooting, Buildtest Features, and improving bash completion script. #1637

v1.6 (Sep 28th, 2023)

Command Line Changes

  • The buildtest config profiles remove command will remove a given profile from configuration file #1585
  • The compiler listing is now done via buildtest config compilers list command, previously it was via buildtest config compilers. #1588
  • Add short option -t for buildtest inspect query --testpath. #1599
  • Change behavior of listing executors from command buildtest config executors to command name buildtest config executors list. #1610
  • Add support for building tests by names via buildtest build --name. #1616
  • Add support for limiting number of jobs that can run concurrently via max_jobs property in configuration file and option buildtest build --max-jobs to override configuration on command line. #1629, #1630

Buildspec/Test Changes

  • Add support for deactivating spack environments via deactivate property. #1587
  • Add set -eo pipefail command in generated test script to ensure test are run in strict mode and errors are captured properly. #1628

Project Improvements

  • Rewrite logic for building the command line arguments for buildtest. #1612
  • Update version of sphinx-autoapi to fix documentation build error with generating API. #1632
  • Ensure every buildtest CI run is executed in a unique directory. The environment variable BUILDTEST_CI_DIR will be root to where tests will be written in a unique directory. This can be set in shell environment prior to running tests. #1601
  • The buildtest commands run as part of documentation builds also performed in unique directory. #1603
  • Move pytest configuration to pyproject.toml #1596
  • Upgrade dependencies for docutils, sphinx, sphinx-rtd-theme, sphinx-design to be compatible with python 3.8 #1606
  • Rewrite bash completion script. #1611

Bug Fixes

  • Fix time calculation for batch jobs to ensure runtime is calculated correctly from when job runs to when job ends. Previously the test runtime took into account pending time. #1618
  • Fix issue with buildtest report only showing latest test records. #1620
  • Fix issue with CDASH upload (buildtest cdash upload) with uniccode characters in test output. #1627

v1.5 (Aug 4th, 2023)

Command Line Changes

  • Add support to exclude tests by tagnames via buildtest build --exclude-tags #1467
  • Change behavior of buildtest report --count to allow a negative value which will show all test records. #1468
  • Change behavior of buildtest build --filter with multiple filters from comma to semi-colon #1471
  • Remove option buildtest buildspec maintainers --list #1474
  • Add support for profiles which allows one to run a set of buildtest build options given a profile name via command buildtest build --profile=<NAME>. Add support for saving profile via buildtest build --save-profile which will record command and save into buildtest configuration file. #1489
  • Rename buildtest help to buildtest show #1530
  • Remove option buildtest build --disable-executor-check #1541
  • Add command buildtest --help-all to show all commands and options for buildtest and change behavior of buildtest --help to only show a subset of commands. #1568
  • Add buildtest report --detailed which will show detailed information about test record. #1567
  • Add alias buildtest config val for buildtest config validate command #1559

Misc Updates

  • buildtest now uses pyproject.toml to specify project dependencies, coverage, black and isort configuration. The installation script will no longer install from requirements.txt, instead it will use content from pyproject.toml. #1545, #1547
  • Deprecate support for Python 3.7 #1514
  • Add a security page to report vulnerability #1473
  • Reorganize example buildspecs for script.schema.json, compilers.schema.json and spack.schema.json into separate buildspecs used for regression tests. This was addressed in #1496, #1498, #1499
  • Improve regression test coverage #1501, #1503, #1507, #1511, #1515
  • Fix bug with running buildtest using multiprocessing library that led to error mostly caused on MacOS. #1522
  • Fix bug with buildtest report --pass and buildtest report --fail where test records were not filtered properly #1524
  • Fix error in buildtest configuration file used for tutorials #1513
  • Hardcode version dependency for jsonschema package to 4.18 since RefResolver is deprecated #1520. This will be addressed in #1521
  • Remove short option -t for long option --theme #1576

CI checks

  • Update regression test and installation workflow to use python 3.8-3.11. Also run tests on macos-13 #1512
  • Update installation workflow to add support for zsh #1548

v1.4 (May 4th, 2023)

Buildspec Changes

  • Add support for logical AND in status check by introducing a new property mode #1429
  • Remove suite property in spack schema to ensure suitenames are auto-generated when running spack test results 1451

Command Line Changes

  • Add support for building multiple tags by comma separated list buildtest build --tags tag1,tag2,tag3. 1419
  • Add option buildtest config compilers find --file to save configuration file in alternative location. #1420
  • Add pagination support for buildtest history and buildtest inspect. #1424, #1428
  • Add global option --row-count that can be used with buildtest inspect, buildtest buildspec find, buildtest buildspec invalid, buildtest buildspec maintainers, buildtest report and buildtest history command. #1463

Configuration Changes

  • Add support for buildspecs property in configuration file to configure behavior of buildtest buildspec find command. #1455
  • Add support for report section in configuration file to control behavior of buildtest report command 1459
  • Rename buildspecs_root to root under the buildspecs section. #1457
  • Change pattern property for executor names to be alphanumeric #1440

Misc Updates

  • Update github issue templates #1430, #1431
  • Fix issue with tab-completion for subcommands at top-level options #1432
  • Remove codecov dependency #1436
  • Add documentation on Assert Less Than #1439
  • Remove CI job for Cori and refactor tests #1442

v1.3 (Mar 7th, 2023)

Command Line Updates:

  • Add option buildtest inspect query --theme to select the color theme. #1383
  • Add option buildtest buildspec find --row-count to report number of rows for for a buildspec query. #1377
  • Add option buildtest build --limit to limit the number of tests to run. #1381

Buildspec Updates:

  • Add Assert Less Than performance check via property assert_lt. #1382
  • Add Symbolic Link check via property is_symlink. #1394
  • Add support for running regular expression on arbitrary files via property file_regex. #1396
  • Add support for file count via property file_count that can perform directory traversal. This also includes traversal by file extension, regular expression. We can filter by file type (file, dir, symlink) and control traversal depth. #1405

Misc Updates:

  • Added precommit ci for auto-updating .pre-commit-config.yaml #1389
  • Add a nightly regression test github workflow. #1384, #1378
  • Add yamllint and shellcheck check for lint check on YAML and shell scripts. Update contributing guide and fix all yamllint issues #1385, #1390, #1404
  • Fix issue where coverage was not tracked with with multiprocessing library that caused lack of test coverage. #1408

v1.2 (Feb 6th, 2023)

Documentation Updates

  • Buildtest Command Line Tutorial #1353
  • Buildtest Tutorial on Perlmutter #1338, #1357
  • Troubleshooting guide for buildtest #1336
  • Add Quick Start Guide #1335
  • Rewrite page for Building and Test Process #1345

Buildspec Changes

  • Add performance check for assert_le (#1339), assert_ne (#1361), assert_ge (#1354)
  • Add support for contains and not_contains for status check #1364
  • Change executor match in parse stage from equality check to re.fullmatch #1350

CI changes

  • Run Gitlab CI on Perlmutter #1359
  • Trigger Gitlab pipelines based on code changes to framework or regression test. This avoids having facility pipeline run on every commit #1358.
  • Add test coverage by running Perlmutter specific tests #1360
  • Run Github regression test on file changes #1356

v1.1 (Jan 6th, 2023)

Buildspec Updates:

  • Add support for Performance checks: Assert Equality (assert_eq), Assert Range (assert_range), Assert Greater Equal (assert_ge) #1313, #1320, #1326
  • Add support for File checks such as file existence, is file or is directory via properties exists, is_file, is_dir #1329
  • Change behavior of buildspec parser to check for executor check and raise different exceptions based on the validation error #1309
  • Increase test length for buildspecs from 32 to 48 characters #1286

Command Line Updates:

  • Add new command buildtest tutorial-examples that will auto-generate tutorial examples #1307
  • Add top-level argument buildtest --color to control coloring option #1280
  • Add color support for buildtest buildspec maintainers, buildtest inspect list command #1284, #1287
  • Add color support for terse output formats. This includes coloring for buildtest report --terse, buildtest buildspec maintainers --list, buildtest buildspec maintainers --breakdown, buildtest buildspec find --terse, buildtest history list --terse #1292
  • Add color support for buildtest buildspec summary #1298
  • Add support for filtering logs by loglevel via buildtest --loglevel #1312

Documentation Updates:

  • Add documentation on how to generate buildtest tutorial #1289
  • Redesign documentation for buildtest into based on divio #1285
  • Rearrange content of Buildspec Overview into sub-pages #1323
  • Move APIs under Reference #1304
  • Add link to SC22, PEARC22 talk and paper submission #1294 , #1291
  • Add section Useful Links on left navigation side-bar #1305
  • Change slack invitation link to communityiniviter instead of heroku. #1306

v1.0 (Oct 14th 2022)

  • Add support for configuring programming environment in buildtest configuration #1270
  • Remove Spell Check CI #1267
  • Add support to skip buildspec via skip property to skip all test in a buildspec instead of updating every test with skip property #1265
  • Add subcommand buildtest report path to print path to the report file being used #1260
  • Add option buildtest history list --color to colorize output of history table #1258
  • Add option to limit row count using buildtest report --row-count and limit output of buildspec cache using buildtest buildspec find --count #1249, #1250
  • Add options buildtest report --filterfields and buildtest report --formatfields to print filter and format fileds as raw text #1246
  • Use sphinx_designs to leverage dropdown feature in documentation #1243
  • Add positional arguments buildtest config compilers test to test specific compilers #1239
  • Add options buildtest buildspec find --filterfields and buildtest buildspec find --formatfields to print filter and format fileds as raw text #1238
  • Rename option buildtest build --keep-stage-dir to buildtest build --remove-stagedir #1237
  • Rename option buildtest rt --failure to buildtest rt --fail and buildtest rt --passed to buildtest rt --pass for retrieving pass and failed test from report #1234
  • Add command aliases for several buildtest commands #1233
  • Add set -e when running Cori CI pipeline #1223
  • Add pagination support for buildtest config view #1218
  • Add support for color themes via buildtest buildspec show --theme and buildtest buildspec show-fail --theme that changes output based in input theme. #1215
  • Add color theme option buildtest config view --theme to change output for buildtest configuration #1216
  • Add option buildtest report --passed to show only pass test results from report file #1140
  • Add new command buildtest info to print details on buildtest #1155
  • Add pager support for buildtest buildspec summary #1163
  • Add new command buildtest config compilers test to automatically test compiler instances via module load test #1175
  • Rename option --debug to --detailed in command buildtest report summary to avoid argparse from confusing with buildtest --debug option #1176
  • Add option buildtest --print-lastlog, buildtest --lastlog and buildtest --lastlog-path #1167 #1168, #1169, #1174
  • Make buildtest report --passed and buildtest report --failure mutually exclusive option on command line #1180
  • Change output of buildtest report summary such that success are printed in green and failures in red #1170
  • Add a Spelling check CI workflow to detect errors in user documentation #1166
  • Fix issue with uploading results from buildtest regression test to CodeCov in Github CI checks. #1149. Thanks to @Xiangs18 for the fix.
  • Fix a bug during slurm job submission when detecting slurm job state #1141
  • buildtest will return a non-zero returncode when running buildtest buildspec validate with invalid buildspecs #1220
  • Remove github-gitlab mirroring workflow for ascent pipeline now we use Gitlab builtin feature for CI/CD #1229
  • Trigger Cori and Ascent Regression test using external pull request event from Gitlab #1161, #1162

v0.15.0 (June 30th 2022)

New Options

  • Add option --buildenv to buildtest report, buildtest path and buildtest inspect query and add entry buildenv to capture build environment from test #1062
  • Add support to timeout test via buildtest build --timeout which is calculated in number of seconds #1052, #1057
  • Add option buildtest --editor to specify your preferred editor when opening files #1070
  • Rename buildtest buildspec edit to buildtest buildspec edit-test used for editing files by testname which is to be consistent with buildtest buildspec edit-file which edits by filename #1081
  • Add option buildtest report --failure to report all failure tests #1076
  • Add new command buildtest stats to show test stats which can be useful for analysis purpose. This command is subject to change #1074
  • Add option buildtest cdash upload --open to CDASH result that was recently pushed in webbrowser which can be useful if one wants to run this command as pose to clicking link that is shown in output #1083
  • Add option buildtest report --start and buildtest report --end to filter test records by starttime and endtime #1082
  • Add new command buildtest config path to show path to configuration file #1100
  • Add new command buildtest buildspec maintainers to show maintainer details from buildspec cache. The command buildtest buildspec maintainers --list will report list of maintainers. The --terse and ---no-header can be used for parseable format. The buildtest buildspec maintainers find <name> can be used to search for buildspecs given a maintainer name #1093.
  • Add option buildtest buildspec find --quiet to suppress output which can be useful when rebuilding cache #1135
  • Add option buildtest report --count to limit number of entries that is displayed in output #1130

Changes to buildspec

  • Rename property numprocs to poolsize in buildtest configuration for configuring number of worker process when using multiprocessing library. #1064
  • Add summary property to provide extended description of test that is not limited to 80 character unlike the description field. #1069, #1073
  • The dash - character is now supported character in the test name #1097
  • Remove executor check during Parse stage this avoids having an invalid buildspec when one specifies a buildspec using regular expression where test can run across multiple system #1098

Other Changes

  • Add coverage for ubuntu-22.04 and macos-12 in regression test and test all python version from 3.7-3.10 #1120, #1118, #1115
  • Remove urlchecker cron workflow #1110
  • Fix regression test for buildtest --editor when specifying invalid editor name. Now editor will resort to vi if invalid editor is specified #1124
  • Fix bug in invocation for buildtest buildspec edit-test #1133
  • Add coverage for ubuntu-22.04 and macos-12 in installation workflow #1129
  • Fix version of black in .pre-commit-config.yaml #1131

v0.14.0 (Mar 18th 2022)

Changes buildspec structure

  • Add support for test dependencies in buildspec using the needs property. #1000, #1003, #1015
  • Add compilers property in script schema for defining multiple compilers when using the type: script in buildspec. This can be used in the run section for selecting multiple compilers #999
  • Add field nvhpc to settings schema for compiler declaration and under compiler schema for using nvhpc compiler #996
  • Remove the run_only property from schema, remove buildspec examples and update user documentation #994
  • Remove version numbers from all schemas, example buildspec, refactor codebase and update user documentation #988

New command line options:

  • Add PAGING output using --pager option which is available for buildtest buildspec find, buildtest report, buildtest report summary, and buildtest history list command #1012 #1016, #1017
  • Add new option buildtest build --modules, buildtest build --module-purge and buildtest build --unload-modules which allows one to specify modules to load/unload on command line and purging modules command before running any test. #1004, #1020
  • Add new option buildtest build --rerun which will rerun last successful buildtest build command which can be used to repeat a command without having to remember what you typed. #1023
  • Add new option buildtest build --executor-type to filter test by executor types which can be local or batch. #1026
  • Remove buildtest edit command and now this is buildtest buildspec edit-file. Add new command buildtest buildspec edit which will edit buildspecs based on testname while buildtest buildspec edit-file uses filepath #1006, #1007
  • Add new option buildtest --lastlog to see output of last build log. Enable RichHandler for logging #993

Miscelleaneous Updates

  • buildtest has a logo! #989
  • Enable bash completion support for zsh #987
  • Refactor Polling Implementation for Batch Job Submission #986
  • Add documentation on starting PBS container #985

v0.13.0 (Jan 20th, 2022)

  • Add option for scaling batch jobs with buildtest build --procs and buildtest build --nodes which allows one to specify processor and node values to run test with different process and node configuration. See #957, #977
  • Rename command line options --max-pend-time -> --maxpendtime and --poll-interval -> --pollinterval #959
  • Change behavior of buildtest inspect query command to support regular expression to query builders. We remove buildtest inspect query -d all which retrieved all records now this can be done with regex. We print additional metadata for each builder and color code via rich.
  • Add new command buildtest unittests to run regression test via buildtest. Added several options including --coverage, --pytestopts and --sourcefiles #962, #966
  • Add new command buildtest stylecheck to run style checks such as isort, black and pyflakes. We have added options --no-black, --no-isort, --no-pyflakes to disable a particular check and --apply to apply changes to buildtest codebase. See #964, #965, #969
  • We can configure buildtest to use a certain python wrapper using environment BUILDTEST_PYTHON #963
  • Add argparse options to run buildtest unittest via script python $BUILDTEST_ROOT/buildtest/tools/unittests.py which is equivalent to running buildtest unittests. #968
  • Add sphinx-copybutton extension in sphinx project to enable copy button in user docs #973
  • We have changed behavior of how builders are printed during buildtest build command now one will see breakdown of builders by each type along with batch builders and builders created via --procs or --nodes option. #978
  • Overall improvement in user documentation which included the following:
    • Documentation for buildtest unittests and buildtest stylecheck #967.
    • Add Facility test page #976
    • Regenerate tutorial examples #979

v0.12.0 (Dec 17th, 2021)

  • Make use of rich python library for printing output and utilized this throughout codebase. We have removed termcolor and tabulate library used for printing. #896
  • Slight change to spack test behavior when generating script. #893
  • Add state property in status section to allow one to explictly mark status of test. This can be useful to explicitly fail test. #892
  • buildtest will automatically build buildspec cache if file not found when running buildtest build command. This was an issue when building with tags, executors. #898
  • Add new option buildtest build --account to specify account name for batch job submissions. #899
  • Add new command to show debug report buildtest debugreport - #902
  • Change behavior of buildtest inspect name now one can specify a unique record by hash via / followed by test ID. Remove buildtest inspect name --all option. - #904
  • Remove buildtest inspect id command this can now be done via buildtest inspect name #906
  • Remove property load_default_buildspecs from configuration file and default specs are loaded if no root trees are specified. #909
  • Introduce buildtest tutorial that can be run in docker container - #917, #918, #942
  • Add workflow to build tutorial container to ghcr. #926
  • Add command buildtest config edit to edit configuration file #920
  • Add support for declaring module commands in executor settings via module property. #925
  • Remove buildtest config summary command - #922
  • Change behavior of buildtest config compilers find to display content of configuration without updating configuration file. One can automatically update config file via buildtest config compilers find --update. #924
  • Change behavior of how test are launched using bash, csh, sh shell when invoking build script. #929
  • Fix CDASH bug when multiple runs of same test were lost during upload. #891
  • Fix issue where buildtest incorrectly detected Torque as PBS Scheduler even though we don't support Torque at the moment #936

v0.11.0 (Sep 9th, 2021)

In this release we tweaked the core implementation of buildtest for running test. Now we make use of asynchronous test execution and job submission to get parallelization of jobs. Previously we were running jobs serially which was quite slow. We are using python multiprocessing library to leverage asynchronous job execution. This work was done in #858. In this release we have reimplemented the poll implementation for batch job submission which was done in #863

  • Add new command buildtest cd to change directory to root of test - #871 and #877
  • Add new command buildtest path to see path details for a test - #874, #876
  • Add new command buildtest clean to clean up buildtest generated data - #869
  • Add option to retry failed tests via buildtest build --retry - #864
  • Added alias for subcommands - #860
  • Remove batch property from buildtest - #868
  • Add support to disable executors in configuration file via disable property - #870
  • Add napolean and intersphinx extension and rewrite docstring for entire codebase in Google Style see #885

v0.10.2 (Aug 16, 2021)

  • Add support for defining multiple executors via executors property see #842
  • Add option buildtest --color to toggle color mode on command line. The environment BUILDTEST_COLOR will also work, if environment is set we will use this value. See #839
  • Add option buildtest build --helpfilter to show filter fields using with buildtest build --filter command #838
  • Add new command buildtest edit to edit buildspecs interactively see #852
  • Add new command buildtest report summary to show summary of report file #848
  • Add new command buildtest inspect buildspec to inspect tests by buildspec #843
  • Add new command buildtest buildspec show to view content of buildspec based on test name #853

v0.10.1 (July 30th, 2021)

  • Add command buildtest buildspec summary to show summary of buildspec cache #831
  • Add command buildtest buildspec invalid to show invalid buildspecs from cache #830
  • Add option buildtest build --filter for filtering buildspecs and remove option --filter-tags #828
  • Add metrics property in buildspecs to define arbitrary metrics name that can store values based on variables, environment variables or regular expression on stdout/stderr. See #819 #824 #832.
  • Add option --no-header to toggle output header with terse format in buildtest history list, buildtest report, buildtest buildspec find see #815 #816 #818
  • Add terse output --terse for command buildtest history list #811, buildtest report command #814 and buildtest buildspec find command #809
  • Change output of buildtest help #808
  • Add support for buildtest buildspec find --filter buildspec to filter cache by buildspec file see #807
  • Add new command buildtest inspect query for querying test records see #806

v0.10.0 (July 13th, 2021)

In this release we added support for spack through a new schema spack-v1.0.schema.json which is used by type: spack to write buildspecs using the spack schema. Initial support includes installing specs, managing spack environment, adding mirror, running tests via spack test. The relevant PRs for spack support are #774, #778, #786, #788.

In this release we added bash completion for buildtest commands this is done in #773. There were few additional features added in this release including a new command buildtest buildspec validate to validate buildspecs on command line see #785. The buildtest help command is another new command that provides an output summary of buildtest commands that one can read as a reference manual see #787.

  • Add command buildtest report list to list all report files #799
  • Using isort utility to sort import statements in python code see #798.
  • Fix issue with sphinx documentation with malformed bullet point rendering. Required to downgrade docutils package see #802
  • Use sphinx-argparse to generate documentation for buildtest commands see #796
  • Rename global option --config-file to --config for specifying alternate configuration file see #791
  • Rename long option --report_file to --report across all subcommands to be consistent. This option is used for specifying alternate report file #775

v0.9.6 (June 11th, 2021)

  • Add command buildtest history to keep track of builds with command names buildtest history list and buildtest history query <id> to retrieve specific builds. buildtest will keep track of every buildtest build command for retrieval. See 767
  • Change behavior of how test are generated by introducing a build script that ends in _build.sh script which is used for running the script. In #751 we are able to properly handle returncode of generated script. We updated the buildtest installation by installing dependencies in $BUILDTEST_ROOT/.packages using pip install --target which addressed issue in buildtest installation if user was to install outside of virtual environment.
  • Add option --keep-stage-dir in buildtest build for preserving stage directory. With this change, buildtest will remove stage directory by default unless this option is specified. See #758
  • Reimplement Slurm, LSF, Cobalt and PBS executor implementation by adding a python class to represent a SlurmJob, LSFJob, CobaltJob, PBSJob. See #750
  • Add global option buildtest -c <config> for specifying alternate configuration file. See #738
  • Add option --poll-interval in buildtest build command for overriding poll interval for batch job submission. See #745
  • Add option --max-pend-time in buildtest build for overriding configuration max_pend_time. See #741
  • Add runtime property in the status section for marking test state based on runtime of code. See #736
  • Add support for buildtest cdash integration for uploading test results using buildtest cdash command. The buildtest cdash upload command can be used to upload test results to CDASH server. See #721, #733, #734
  • Change behavior of buildtest to write files to $HOME/.buildtest instead of $BUILDTEST_ROOT. See #719.
  • Add command buildtest report clear to clear report file. In addition we add option to specify report file on command line. This can be specified using buildtest report --report_file, buildtest build --report_file, and buildtest inspect --report_file see #727, #730.
  • Change behavior of buildtest config executors to print a list of executors, previously it was showing content of YAML. Now we have option buildtest config executors --yaml for YAML format.

v0.9.5 (Mar 31, 2021)

This release added support for PBS Scheduler see #691 and #693 We changed the starttime and endtime format for batch job submission when reported to report.json see #700. We added color output for buildtest commands using termcolor library. We can disable color using BUILDTEST_COLOR environment variable. See #703. We re-implemented command buildtest inspect command by introducing commands buildtest inspect name, buildtest inspect list, buildtest inspect id see #704.

  • Add wrapper script to run regression test via sh $BUILDTEST_ROOT/scripts/regtest.py. This script is wrapper to pytest and coverage. See #710.
  • Add description field for system description in configuration file #694
  • Add command buildtest config system see #697
  • Increase test coverage and add few more facility tests for Cori and Ascent see #712
  • The buildtest config compilers implements the --list option which is removed see #708
  • Add upcxx compiler support in compiler schema and settings schema see #709

v0.9.4 (Mar 14, 2021)

The major change in this version is configuration template for buildtest is organized into system groups by top-level key system. This allows facility to deploy a single configuration used for multiple HPC clusters. See #668 for more details. We provide a generic system name that can be used to run buildtest. We reorganized the getting started page into sub-pages so it's easier to navigate to each section, for more details see #659. The executors are referenced in <system>.<executor-type>.<executor-name>. Previously we would reference executor like local.bash now it would be referenced as generic.local.bash which is a local executor of name bash tied to system name generic. buildtest will detect the system based on hostnames field which is a list of regular expression to search where buildtest can run. If no system is found it will raise an error.

  • Add option to override configuration file via buildtest build -c see #684
  • Changed the output of how jobs are printed during poll stage by showing a table of jobIDs see #664
  • Add regression test for Ascent system at OLCF for LSF #663
  • Replace black workflow to use the registered black action psf/black@stable see #662
  • We removed documentation and scripting with buildtest and CI checks see #674
  • Update diagram for buildspec structure and parser stage because we changed the executor name format #678
  • Add JLSE gitlab CI file that can be found at .gitlab/jlse.yml which runs tests buildtest regression test at https://gitlab.jlse.anl.gov
  • The configuring buildtest page was redesigned into sub-pages and new configuration format was documented. For more details see commit cdd56db4c15dc68031e8162cdd1b34cfafe2e5d3

v0.9.3 (Feb 22, 2021)

In this version we updated the copyright details to include UC - LBNL see COPYRIGHT and LICENSE. We added dependabot for tracking out-of-date dependencies for github action and python dependencies. In addition, we made minor tweaks to buildtest buildspec find command in its output. We added regression test pipeline for OLCF - Ascent system and moved gitlab pipelines in top-level folder .gitlab.

  • Add command buildtest config executors to view executors - #648
  • Add --oldest and --latest option in buildtest report command for retrieving oldest and latest record see #634
  • Remove --validate option from buildtest schema for validating JSON schema, this option mimics the regression test. See commit 5825637
  • Change Copyright and License to include UC - LBNL - #611
  • Add dependabot for github actions and pypi packages - #615
  • Add gitlab CI file for running regression test on OLCF #603
  • Now when you install buildtest via setup.[sh|csh], we set BUILDTEST_ROOT variable so user can navigate to root of buildtest. We fixed how to detect root of buildtest on Linux and Mac system see #646. We also add installation workflow to test buildtest on different platform and python distro.
  • Add option to query buildspecs by maintainers and breakdown by buildspecs using --maintainers and --maintainers-by-buildspecs see #599
  • Add option to filter tests by tags using buildtest build --filter-tags, the behavior of buildtest build --tags is used to for discovery of buildspecs #587
  • Add singularity tests see #645

v0.9.2 (Jan 12th, 2021)

In this version, we added significant changes to compiler-v1.0-schema.json to support compiler test. This includes ability for building a single test across multiple compiler instance and across compiler groups (gcc, intel, cray, etc...) User can search compilers via regular expression when building test, and specify setting common to compiler group or shared across all compilers. In addition, one can override properties at the compiler level.

This version introduced significant refactor in codebase responsible for building, running and buildspec operation. We introduce classes when appropriate. We added a Gitlab CI job at Cori to run regression test and report coverage report to codecov. This pipeline is run manually and functionality will change until we have stable environment for running PR pipelines.

We changed the behavior of buildtest build --tags previously it was used for discovering buildspecs and filtering test, now it only discovers test. A new option was added --filter-tags which is used for filtering tests by tagname. Previously if one used --tags with --buildspec or --executor would result in filtered tests by tags and buildtest may ignore some tests that were expected to run. With this change we can better support both use-case where one wants to discover tests by tag and filter them. This was implemented in #587.

  • Refactor implementation for buildtest report into class see #555
  • The module property is changed from array to object type which allows one to specify module load, module swap, module purge. See #556
  • Fix bug in slurm job when executor was indefinitely polling jobs in TIMEOUT and OUT_OF_MEMORY job state. See #561.
  • Increase test coverage for buildtest inspect and searching compilers see #575.

v0.9.1 (Nov 24th, 2020)

In this version, we added support for Cobalt scheduler provided by Argonne National Laboratory. We can define cobalt executors in buildtest settings which can be mapped to cobalt queues. There is a cobalt property for adding #COBALT directives into test script. Some of the cobalt options are mapped to batch field for scheduler agnostic configuration.

In this version we added support for compiler query and detection using buildtest config compilers find. We make use of lmodule API for querying modules if system is using Lmod.

We made significant changes to buildspec cache file (var/buildspec-cache.json) that allowed us to add several options to buildtest buildspec find including: --group-by-tags, --group-by-executor, --paths, --helpformat, --format, --helpfilter, --filter, --root.

There was significant code refactor to several class and issues reported by CodeFactor. In addition we added CI checks such as Daily Check URL see eb601b, gh-pages for master branch see 267f7f. The gh-pages for JSON schema push documentation for devel and master in separate sub-directories. This allows user to view schema examples and markdown pages for schema for devel and master branch.

  • Add new maintainers checklist guide see #529
  • Rename --clear --> --rebuild, --list-executors --> --executors in buildtest buildspec find see e7ec37
  • Added property moduletool in settings schema for configuring module system
  • Add property load_default_buildspecs in settings schema for configuring buildtest to load default buildspecs in buildspec cache. See commit dac444
  • Remove property editor from settings schema and buildtest buildspec view and buildtest buildspec edit were deprecated see b8479b
  • Fix bug during job timeout in poll stage. Buildtest will ignore cancelled jobs, but there no check if no builders were returned after poll stage. See #532
  • Add Burst Buffer (BB) and Data Warp (DW) directives for Cray support. See #525 and #526
  • Add csh, tcsh, zsh shell support in script-v1.0.schema.json #523

v0.9.0 (Oct 21st, 2020)

The major changes in v0.9.0 are the following

First we moved schema development from https://github.com/buildtesters/schemas into buildtest and add custom RefResolver for validating schemas on local filesystem as pose to fully qualified URI.

We host schema, examples, and schema docs on Github pages at https://buildtesters.github.io/buildtest/ by adding a jsonschemadocs workflow. We moved JSON definitions to separate file called definitions.schema.json.

We added setup.sh, setup.csh script to install buildtest for bash/csh shells, this now changes the way we install buildtest as pose to using pip. We introduced scheduler agnostic configuration using batch field. This property currently translates a subset of options for Slurm and LSF. We have added generic tests to buildtest in top-level folder generic-tests which is an attempt to provide buildspecs that anyone can use. Currently, these tests are run using Local Executors. We added the properties account and max_pend_time in executor configuration. The account field is used for sites to specify a project account to charge resource, this can be set default on all executors or defined per executor setting. The max_pend_time is maximum time limit job can stay pending in executor queue, this was an enhancement from previous model where jobs can run indefinitely without any cancellation option.

  • Add new command buildtest inspect to view test details see #516
  • Disable Travis and enable codecov comments see #519
  • Add account field in buildtest setting to specify job account, this can be set default on all batch executors or set within executor scope which overrides default. See #514
  • Add max_pend_time in buildtest settings to cancel job if its in pending state. This was tested for Slurm and LSF scheduler. See #509, #510
  • Add option buildtest schema --validate to validate example schemas. The option buildtest schema --example shows content of schema examples see #502
  • Deprecate command buildtest config edit see #512
  • Fix bug when retrieving tags with command buildtest buildspec find --tags see #501
  • Add scheduler agnostic configuration via batch field see #493 and #494
  • Add a setup.sh, setup.csh script to install buildtest. This changes the way buildtest is installed as pose to using pip see #491 #503
  • Add a custom RefResolver for resolving JSON schemas in filesystem as pose to using public URL, this was important for testing schema changes locally which was not present before. See #487
  • The returncode field can be a string or a list for matching returncode status. The tags field can be a string or list of strings, before it could only be a list. See #486
  • Migrate schema development from https://github.com/buildtesters/schemas into main project. see #480
  • Fix bug when when writing python scripts in run section, we add stage/run directory in test destination directory see #477.

v0.8.1 (Sep 14th, 2020)

  • We now running regression test in github action see #455
  • Add command to filter by executor names using buildtest build --executor. #463
  • Add option for filtering buildspec cache using buildtest buildspec find --filter and see list of available filter option using buildtest buildspec find --helpfiler see #464
  • Support for building with multiple tags #462
  • Add option for filtering test report using buildtest report --filter option and buildtest report --helpfilter with list of filter fields. See #449
  • Add option for buildtest --docs and buildtest --schemadocs to access documentation through CLI. See #452
  • Retrieve a list of unique executors (buildtest buildspec find --list-executors) from buildspec cache see #448
  • Query buildspec tags and buildspec files using buildtest buildspec find --tags and buildtest buildspec find --buildspec-files option see #445

v0.8.0 (Sep 3rd, 2020)

This release includes major changes to framework, in particular we use jsonschema to validate schemas and add separate repository: https://github.com/buildtesters/schemas for development of schemas. The schemas are hosted in Github pages at https://buildtesters.github.io/schemas/schemadocs/. There are four main schemas: global.schema.json, script-v1.0.schema.json, compiler-v1.0.schema.json, and settings.schema.json. The settings.schema.json is used for configuring buildtest. The global.schema.json is used for validating global section of buildspec and sub-schema script-v1.0.schema.json and compiler-v1.0.schema.json are used for validating test section. These are used when type: script or type: compiler is set.

All tests are run via executors defined in buildtest configuration, currently we support LocalExecutor, LSFExecutor, and SlurmExecutor for submitting jobs to local host, LSF and Slurm scheduler. As part of this release, we removed all features related to buildtest modules and they are now part of a Python API called lmodule which is a separate project.

At high level the following commands were introduced: buildtest build, buildtest buildspec, buildtest schema, buildtest config, and buildtest report. To build any buildspecs use the buildtest build command, main options are buildtest build --buildspec which takes input file or directory. You can use buildtest build --exclude to exclude buildspec files. Both options can be specified multiple times. buildtest can search buildspecs by tags when building them using buildtest build --tags <TAGNAME>. This feature assumes you a buildspec cache which can be populated using buildtest buildspec find. This command discovers and validates all buildspecs and invalid buildspecs are reported in file. The buildtest buildspec view and buildtest buildspec edit can view or edit a buildspec file provided you specify name of buildspec.

The buildtest schema command provides access to schemas and examples, if you run buildtest schema it will display all schema names, you can select a schema using buildtest schema -n <schema> with option --examples or --json to view schema examples or json file. The buildtest config command is used showing buildtest configuration, you can view buildtest configuration using buildtest config view and validate the configuration with schema using buildtest config validate. The buildtest config edit can be used to open configuration using an editor and validate configuration upon closing file. If file is not valid, buildtest will print message exception from jsonschema.validate to stdout and open file again. This process happens in a while loop until user has validated the configuration. The buildtest report command is used for showing test reports. The output can be filtered using buildtest report --format to select fields which alter the column outputs. The available fields can be retrieved using buildtest report --helpformat.

In this release, we added significant coverage to regression tests and organize tests such that source directory (buildtest) mirrors to test directory (tests) for instance testing module buildtest.menu.build will have a test in tests/menu/test_build.py. buildtest comes with a set of example tests meant to serve as a tutorial for buildtest. These tests are toy examples meant to augment documentation examples and serve as means to automate documentation examples or used in regression tests.

  • Add Github Issue Templates
  • Remove workflow Issue Label Bot
  • Add pyflakes check in black workflow
  • Add TutorialsValidation workflow for validating buildspecs
  • Change First Issue Greeting workflow to run only on first issue and not for pull request
  • Upgrade version of urlcheck workflow changed from SuperKogito/[email protected] --> urlstechie/[email protected]
  • Add pre-commit hook to automate python format via black. Add black --check as automated check see #172, #179
  • Remove black pre-commit file .github/hooks/pre-commit in replacement for .pre-commit-config.yaml that installs the pre-commit file
  • Remove Lmod installation from Travis since buildtest doesn't depend on Lmod anymore
  • Rename GitHub Organization from HPC-buildtest to buildtesters and update links throughout documentation
  • Update License Copyright from 2017-2019 to 2017-2020 and add Vanessa Sochat
  • Add more badges in README.rst and updates to file
  • We can retrieve tags and buildspec files from cache using buildtest buildspec find --tags and buildtest buildspec find --buildspec-files see
  • Add logging support via python logging library. Logs are written to file and they can be streamed to stdout using buildtest -d <DEBUGLEVEL>
  • Use sphinx-autoapi to automate api docs instead of using sphinx.ext.autodoc
  • Add documentation for Contributing Guide, Maintainer guide, Github Integration, and Regression Testing
  • Add tox.ini file for automating python tests using tox
  • Remove CLI option buildtest build [run|log|test] see #163
  • Remove all module operations and cli menu buildtest module. This is now moved to an API lmodule at https://github.com/buildtesters/lmodule
  • removing extra dependencies argcomplete and termcolor
  • removing bash script and sourcing in favor of Python module install

v0.7.6 (Feb 4th, 2020)

  • Add GitHub actions: greetings, trafico, URLs-checker, pull-request-size
  • Add coveralls for coverage report
  • Use Imgbot bot to convert all images via lossless compression to reduce image size
  • Update .gitignore file to reflect file extension relevant to buildtest
  • Remove command option buildtest testconfigs maintainer and benchmark feature buildtest benchmark
  • Rename output style when showing buildtest configuration (buildtest show --config)
  • Add option to list all parent modules buildtest module --list-all-parents
  • Move code base from src/buildtest --> buildtest and move buildtest script --> bin/buildtest
  • Update contributing docs, and upload slides from 5th Easybuild User Meeting and FOSDEM20

v0.7.5 (Dec 31st, 2019)

  • Major improvement to Travis build. buildtest will now test for python 3.6, 3.7, 3.8 for Lmod version 6.6.2 and 7.8.2
  • Travis will install easybuild and setup a mini software stack that is used for by regression test
  • Port the regression test to comply with Travis build environment and coverage report automatically get pushed to CodeCov
  • Removing subcommand buildtest benchmark [hpl | hpcg]
  • Add options to buildtest module loadtest to control behavior on module loadtest.
  • buildtest can run module loadtest in a login shell via buildtest module loadtest --login and restrict number of test using --numtest flag. buildtest will automatically purge modules before loading test but this can be tweaked using --purge-modules flag
  • Remove command buildtest list and remove implementation for retrieving easyconfigs buildtest list --easyconfigs
  • Option buildtest list --software is now buildtest module --software and buildtest list --modules is now buildtest module list
  • Add the following flags: --exclude-version-files, --filter-include, --querylimit to buildtest module list to tweak behavior on module list
  • Update buildtest configuration (settings.yml) with equivalent key/value to control behavior of buildtest module [list | loadtest]. The configuration values are overridden by command line flags
  • buildtest will ignore .version, .modulerc and .modulerc.lua files when reporting modules in buildtest module list. This is controlled by exclude-version-files in configuration or flag --exclude-version-files
  • Remove sanity check feature buildtest build --package and buildtest build --binary and remove configuration BUILDTEST_BINARY from configuration file
  • Remove option buildtest build --parent-module-search and remove BUILDTEST_PARENT_MODULE_SEARCH from configuration file
  • Update documentation procedure regarding installation of buildtest and remove Concepts page

v0.7.4 (Dec 11th, 2019)

  • update documentation section Background, Motivation, Inception, and Description
  • make use of $SRCDIR when setting variable SRCFILE in test script.
  • add documentation issue template page
  • add clang compiler support via compiler:clang
  • add contributing pages to buildtest documentation and add further clarification on release process, buildtest regression testing, and GitHub app integration
  • add EDITOR key in buildtest configuration (settings.yml) to tweak editor when editing files
  • change path to output/error files in buildtest module loadtest and print actual module load command
  • adding github stalebot configuration see .github/stale.yml
  • adding github sponsor page .github/FUNDING.yml
  • add stream benchmark test see d2a2a4
  • adding github workflow black to format all python code base see .github/workflow/black.yml
  • install lmod and its dependency in travis build

v0.7.3 (Nov 25th, 2019)

  • enable cuda, intel, pgi compilation, this can be set via compilers key
  • Define shell variables CC, FC, CXX to be used to reference builds
  • Define shell variable EXECUTABLE to reference generated executable
  • Fix Code Style issues reported by CodeFactor (https://www.codefactor.io/repository/github/buildtesters/buildtest)
  • Add , hust-19 slides, buildtest architecture and workflow diagram in documentation
  • Simplify output of buildtest module --easybuild and buildtest module --spack
  • Add module purge or module --force purge in test (#122)
  • automate documentation examples for building test examples
  • move all documentation examples to toolkit/suite/tutorial
  • update CONTRIBUTING.rst guide to include section on building buildtest API docs, automating documentation examples and running regression test via pytest

v0.7.2 (Nov 8th, 2019)

  • automate documentation test generation using python script
  • add support for coverage see https://codecov.io/gh/buildtesters/buildtest
  • adding dry option when building tests (short: -d or long option:--dry)
  • automate buildtest testing process via pytest. Add initial support with 25+ regression tests
  • adding directory expansion support when files or directory are references such as $HOME or tilde (~) operation
  • adding several badges to README.rst

v0.7.1 (Oct 30th, 2019)

  • Re-implement core mechanics of the build framework by using new YAML schema.
  • Release buildtest under MIT license
  • Yaml schema can be printed via buildtest show -k singlesource. The schema provides building C, C++, Fortran code along with MPI test. Provides keys such as cflags, cxxflags, fflags cppflags, ldflags for passing compiler options. The schema provides a dictionary to insert #BSUB and #SBATCH directives into job scripts via bsub: and sbatch: keys.
  • Add documentation example on C, C++, Fortran, MPI, and OpenACC code.
  • Add options buildtest build bsub (bsub wrapper) such as -n, -W, -M, -J, --dry-run.
  • Add key TESTDIR in build.json to identify test directory, this makes it easier when running test

v0.7.0 (Oct 16th, 2019)

  • autodetect slurm configuration from system and write to json file
  • add option buildtest module --module-deps that prints modules dependent on parent modules
  • add subparser buildtest module tree that provides operation for managing module trees (BUILDTEST_MODULEPATH)
  • remove subparser buildtest find
  • add option buildtest build --collection for building test with Lmod user collection
  • remove option buildtest build --software
  • add option buildtest build --modules which allows test to be build with multiple module versions
  • add option buildtest module collection for managing module collection using buildtest. Alternative to Lmod user collection
  • remove option buildtest --clean-logs
  • Color output of Lua and non-lua modules in buildtest list --modules.
  • Remove option --python-package, --perl-package, --ruby-package, --r-package from build menu. Also delete all reference in documentation and delete repository
  • --buildtest-software option is removed
  • --format option in list submenu only supports json. Previously it also supported csv
  • Rename all test scripts for documentation and rst files to be lower case
  • Convert CONTRIBUTING guide from Markdown to Restructured Text (RsT) and add Contributing section in documentation
  • Change buildtest config file path to be $HOME/.buildtest/settings.yml
  • Use sphinx-argparse to automate argparse documentation
  • Rename main program _buildtest to buildtest and changed source code directory layout
  • Add option -b or --binary for native support for sanity check on binary commands in framework without using yaml files
  • Update requirements.txt
  • Migrate documentation to buildtest
  • Create subcommand find and move option -ft and -fc to this menu
  • Add logo for license, version, download, status to README.rst
  • Type checking support for buildtest configuration file
  • Remove option --output from run submenu
  • Add support for OSU Benchmark and add this to benchmark submenu and document this page
  • Add threshold value for running test. This can be configured using BUILDTEST_SUCCESS_THRESHOLD
  • Create submenu module and move option --diff-trees and --module-load-test to this menu

v0.6.3 (Oct 26th, 2018)

  • OpenHPC yaml files are moved from $BUILDTEST_CONFIGS_REPO/ohpc to $BUILDTEST_CONFIGS_REPO/buildtest/ohpc
  • This led to minor fix on how buildtest will write yaml files via buildtest yaml --ohpc and build tests via buildtest build --ohpc
  • Add OpenHPC integration with buildtest with option --ophc. This is available for build and yaml subcommand
  • Rename option --ignore-easybuild to --easybuild. When this is set, buildtest will check if software is easybuild software.
  • BUILDTEST_EASYBUILD and BUILDTEST_OHPC can be defined in configuration file or environment variable
  • Fix sorting issue with output for buildtest list -svr and buildtest list -bs
  • Add option --prepend-modules that can prepend modules to test script before loading application module.
  • buildtest will now ignore all .version* files as pose to .version file, this is due to Lmod 7 and how OpenHPC module files have hidden modules with format .versionX.Y.Z

v0.6.1 (Oct 18th, 2018)

  • Fix issue with pypi package dependency in version 0.6.0

v0.6.0 (Oct 17th, 2018)

  • New Feature: option to build all software and system packages using --all-software and --all-package
  • New Feature: option to build all yaml configuration for software and system package using --all-software and --all-package
  • New Feature: option to run all tests for software and system package using --all-software and --all-package
  • New Feature: add option --output to control output for test execution. Output can be redirected to /dev/null or /dev/stdout
  • rename option --system to --package
  • option --software and --package is consistent across build, yaml, and run subcommand
  • Add test count, passed and failed test after each test run when using buildtest run.
  • option --rebuild and --overwrite will work with --all-software and --all-package in yaml subcommand to automate rebuilding of yaml files
  • Move option --module-naming-scheme to build subcommand
  • bug fix: directory issue for running buildtest first time #81
  • bug fix: print error #80

v0.5.0 (Oct 8th, 2018)

  • New Feature: Add new sub-commands build list run to buildtest
  • Move the following options to build sub command
    • -s
    • -t
    • --enable-job
    • --job-template
    • --system
    • --r-package-test
    • --python-package-test
    • --perl-package-test
    • --ruby-package-test
    • --shell
    • --ignore-easybuild
    • --clean-tests
    • --testdir
    • --clean-build
  • Move the following option to list sub command - -ls - -lt - -svr
  • Add option --format in list sub command to view output in csv, json. Default is stdout
  • Add the following option to run sub command
    • --app
    • --systempkg
    • --interactive (originally --runtest)
    • --testname
  • Added basic error handling support
  • Add description key in all yaml files
  • Tests have permission 755 so they can run automatically as any user see 6a2570
  • Options for --ebyaml now generates date-time stamp for command.yaml see a59682
  • Add script check.sh to automate testing of buildtest features and package building for verification

v0.4.0 (Sep 11th, 2018)

  • Must use Python 3.6 or higher to use this version. All versions < 0.4.0 are supported by Python 2.6 or higher

v0.3.0 (Aug 7th, 2018)

  • Package buildtest as pypi package, now it can be installed via pip install buildtest
  • Rename buildtest to _buildtest and all code is now under buildtest
  • All buildtest repos are now packaged as pypi package and test are moved under buildtest directory
  • The option --ebyaml is now working with auto-complete feature and ability to create yaml files for software packages
  • Binary test are now created based on unique sha256sum see 92c012
  • Output of --scantest has been fixed and added to documentation
  • Add singularity CDASH script, need some more work on getting server setup properly

New options

  • --r-package: build test for r packages
  • --python-package: build test for python packages
  • --perl-package: build test for perl packages
  • --ruby-package: build test for ruby packages
  • --show-keys : Display description of yaml keys
  • The option --testset is removed and will be replaced by individual option for r, perl, python, ruby package options

Bug Fixes

  • Fix issue with --runtest option, it was broken at some point now it is working as expected
  • Add extra configuration option in config_opts to reuse variable that were needed throughout code and fix bug with --sysyaml see 493b53.
  • Fix bug with directory not found in menu.py by moving function check_configuration and override_configuration from main.py to menu.py see d2c780

v0.2.0 (May 18th, 2018)

This is a major release update on buildtest with additional options and most importantly ability to test software stack without easybuild. buildtest can be used to test multiple software trees, with ability to disable easybuild check for software stack built without easybuild. The easybuild verification in buildtest has been simplified and it can easily report which software is built by easybuild.

buildtest can report difference between 2 module trees and multiple module trees can be specified at same time for building test, and listing software, and software-version. There has been some improvement on how buildtest operates with Flat-Naming-Scheme (FNS) module naming scheme for module tree. Basically you don't need to use --toolchain option with buildtest if you are using FNS naming scheme but for HMNS module tree you will need to use --toolchain option

  • Add short option -mns for --module-naming-scheme and report total count for software, toolchain and software-version for options -ls, -lt, -svr
  • Adding options --clean-logs, --clean-tests for removing directories via command line
  • The file config.yaml is used to modify buildtest configuration and users can modify this to get buildtest working.
  • Environment variables can override configuration in config.yaml to allow further flexibility
  • add option --logdir to specify alternate path from the command line
  • remove option --check-setup
  • buildtest can operate on multiple module trees for option -ls and -svr
  • rename option --modules-to-easyconfigs to --easyconfigs-to-moduletrees with a short option -ecmt
  • add option to show difference between module trees using --diff-tree
  • Fixed a bug where .version files were reported in method get_module_list
  • Add option --ignore-easybuild to disable easybuild check for a module tree
  • rename buildtest variables in source code
  • add option --show to display buildtest configuration
  • add option --enable-job to enable Job integration with buildtest this is used with options --job-template
  • rename all sub-directories in repo BUILDTEST_CONFIGS_REPO to lowercase to allow buildtest to generate tests if software is lower case such as gcc and GCC in the module file. This enables buildtest to operate with module trees that dont follow easybuild convention
  • buildtest will only generate tests for packages in python, R, ruby, perl when using --testset option if software has these packages installed. This avoids having to create excess test when they are bound to fail
  • --testset option now works properly for both HMNS and FNS module naming scheme and is able to operate on modules that don't follow easybuild module naming convention

v0.1.8 (Jan 8th, 2018)

  • Automate batch job submission from buildtest via --submitjob
  • Fix shell magic (#!/bin/sh, #!/bin/bash, #!/bin/csh) for binary test
  • Tab completion for buildtest argument using argcomplete module. See ddb9e4
  • autopopulate choice for --system, --sysyaml, and --software
  • Fix output of -svr and resolve bug when 2 modules with same app/version found in different trees. Only in HMNS. See 7ddf91 for more details
  • Group buildtest commands for ease of use.
  • Support for yaml keys scheduler and jobslot to enable jobscript creation from yaml files. See 0fe418
  • Further support for scheduler and automatic detection. Currently supports LSF and SLURM.

v0.1.7 (Nob 28th, 2017)

  • Add support for creating LSF Job scripts via templates. Use buildtest --job-template see 927dc0
  • Add support for creating YAML config for system package binary testing via buildtest --sysyaml see 4ab887
  • adding keys envvar, procrange, threadrange in YAML 9a2152 152423 3d43b8
  • Add support for multiple shell (csh, bash, sh) see aea9d6 c154db
  • remove verbose option from buildtest
  • major code refactor see fd8d46 9d112c
  • Fixed a bug during compiler detection when building GCC see f1397
  • create a pretty menu for Interactive Testing via buildtest --runtest see 231cfe
  • replace shell commands subprocess.Popen() with python library equivalents
  • Add support for --testset Tcl see 373cc1
  • Add support for --testset Ruby see c6b7133
  • Print software in alphabetical order for -svr option see fcf610

v0.1.6 (Sep 15th, 2017)

  • Add support for FlatNamingScheme in buildtest, added flag --module-naming-scheme to control setting
  • Add prototype functions
    • get_appname()
    • get_appversion()
    • get_toolchain_name()
    • get_toolchain_version()
  • Add support for logging via Python Logger module
  • Fix buildtest version, in 0.1.5 release buildtest was reporting version 1.0.1
  • Provide clean termination when no easyconfig is found
  • Fix issue when no toolchain is provided in CMakeList.txt
  • Optimize nested loop when performing --software-version-relationship

v0.1.5 (Aug 30th, 2017)

The buildtest repo has been moved from http://github.com/shahzebsiddiqui to http://github.com/buildtesters

  • Report what tests can be generated from buildtest through YAML files by using --scantest
  • Fixed a bug with flag -svr that was related to structure of easybuild repo, now no dependency on easybuild repo. Also added pretty output
  • Adding CONTRIBUTION page
  • Fix out software, toolchain, and easyconfig check is done. Arguments to --software and --toolchain must go through module check, then toolchain check, and then finally easyconfig check
  • Add support for --check-setup which can be used to determine if buildtest framework is setup properly
  • Add interactive testing via buildtest --runtest which is menu-driven with ability to run all tests, or run individual test directory in menu and see output
  • Fix some issues with --testset and now buildtest reports number of tests generated not the path for each test to limit output. For --testset like R, Python, Perl buildtest will report generated test for each package
  • buildtest will now use eb --list-toolchains to get list of all toolchains for toolchain check
  • Can properly generate tests via --testset when R, Python, and Perl repos were created and moved out of buildtest-configs
  • Add buildtest -V for version display

There has been lots of restructuring of code. There still needs some improvement for organizing scripts by functions

v0.1.4 (May 23th, 2017)

  • Major code restructure around processing binary test and support for logging environment variable
    • BUILDTEST_LOGCONTENT
    • BUILDTEST_LOGDIR
    • BUILDTEST_LOGFILE
  • Provide get functions to retrieve value from arg.parser
  • Add support for Perl with --testset
  • Add for more logging support in module and eb verification

v0.1.3 (May 17th, 2017)

There have been several changes in the buildtest framework to allow for more capabilities.

The following changes have been done in this release - buildtest can generate binary test for same executable with multiple parameters. See - Adding support for R, Perl and Python with more tests. - R, Python, Perl (soon to come), and MPI tests are organized in testset using --testset flag this allows for multiple packages to reuse tests across different apps. For instance OpenMPI, MPICH, MVAPICH and intel can now reference the mpi testset.

  • Add support for inputfile YAML key to allow input redirection into program.
  • Add support for outputfile YAML key to allow output redirection.
  • Add support for argument passing using arg key word
  • Add support for iter YAML key to allow N tests to be created.
  • Switching BUILDTEST_MODULEROOT to BUILDTEST_MODULE_EBROOT to emphasize module tree should be coming from what easybuild generates.
  • Fixed some bugs pertaining to CMakeLists.txt

v0.1.2 (May 9th, 2017)

The current release add supports for logging by default.

buildtest will now report useful operations for each function call that can be used for troubleshooting. The logs work with options like --verbose to report extra details in log file.

  • The logs display output on the following
    • Verification of software and toolchain with module file and easyconfig parameters
    • Display output of each test generated
    • Display changes to any CMakeLists.txt
    • Output key values from YAML configs
    • Output log from ancillary features like (--list-toolchain, --list-unique-software, --software-version-relation)
  • buildtest can now search YAML configs and buildtest generated test scripts using the command -fc and -ft
  • Now all buildtest-config files are removed and migrated to

v0.1.1 (May 1, 2017)

In this release, we have restructured the source directory. Now there are two sub directories
  • ebapps
  • system

buildtest can now support binary tests for system packages. There is a command.yaml file for each system package in its own directory. Each system package is in its own subdirectory where the name of the directory is the name of the system package. buildtest is using RHEL7 package names as reference.

The following system package tests have been added

  • binutils
  • chrony
  • git
  • hwloc
  • ncurses
  • pinfo
  • procps-ng
  • sed
  • time
  • wget

Compile from source YAML scripts can now be stored in subdirectories. buildtest can now generate tests in sub directory, this would be essential for building tests for R, Python, Ruby, Perl, etc...

Tests for the following python packages:
  • blist
  • cryptography
  • Cython
  • dateutil
  • deap
  • funcsigs
  • mpi4py
  • netaddr
  • netifaces
  • nose
  • numpy
  • os
  • paramiko
  • paycheck
  • pytz
  • scipy
  • setuptools

Added python documentation header for each function and GPL license section in all the files

v0.1.0 (Feb 26th, 2017)

buildtest generates test scripts from YAML files. The following apps have tests:

EasyBuild Applications

  • Anaconda2
  • binutils
  • Bowtie
  • Bowtie2
  • CMake
  • CUDA
  • GCC
  • git
  • HDF5
  • hwloc
  • intel
  • Java
  • netCDF
  • numactl
  • OpenMPI
  • Python

System Packages

  • acl
  • coreutils
  • curl
  • diffstat
  • gcc
  • gcc-c++
  • gcc-gfortran
  • iptables
  • ltrace
  • perl
  • powertop
  • python
  • ruby