Skip to content

Development How To Add Frameworks or Tests

Jump to bottom
Nate edited this page Jul 26, 2019 · 7 revisions

Adding a New Test

To create a new framework you can start the new test initialization wizard with the command:

tfb --new

This will walk you through the scaffolding process. You will still need to follow the steps in Additional Information below.

Adding a New Test Without the Installation Wizard

When adding a new framework or new test to an existing framework, please follow these steps:

  • Add/Update a test docker file.
  • Add/Update a benchmark_config.json.
  • Implement at least one test (we'd be even happier with six completed tests). When creating a database test, use the table/collection hello_world. Our database setup scripts are stored inside the config/ folder if you need to see the database schema.
  • Add/Update a README for your framework.
  • Add/Update a README for your language.

Additional Information

  • Make sure that any packages/dependencies you are using or any git source repositories you are pulling in are locked down to specific version numbers or releases.
  • Add an entry for the framework test directory in .travis.yml if the framework is new to the benchmarks.
  • Test your framework appropriately.
    • Ensure the framework tests implemented pass in your local environment.
    • Travis CI will test the framework when a pull request is opened. Tip: Setup your own Travis CI account and know the outcome of the tests before you submit a merge request. See Travis CI Tips and Tricks for more details.
  • When all tests are passing, submit a pull request following the pull request procedure.

Tuning By Environment

Our official round results come from two different environments. If you'd like to tune your tests based on those environments, you can make use of the BENCHMARK_ENV environment variable. To do that, add the following line to your test's dockerfile:

ARG BENCHMARK_ENV

Then you'll have access to the $BENCHMARK_ENV variable throughout your dockerfile and scripts. See the environments section for the values of that variable on each environment.

To change this value during local testing, use the --results-environment flag.

Example:

tfb --test gemini --results-environment Local
Clone this wiki locally