diff --git a/documentation/AddingLitmusTests.md b/documentation/AddingLitmusTests.md deleted file mode 100644 index 93ea20c7..00000000 --- a/documentation/AddingLitmusTests.md +++ /dev/null @@ -1,98 +0,0 @@ -#Adding Your Own Litmus Tests to FLiT - -All litmus tests that are included in general execution are located in the `QFP/qfpc/tests` directory. For a test to be included in a run it only need be present in this directory. Tests must be in a templated C++ class which extends `flit::TestBase`. - -##getInputsPerRun -This function returns the number of arguments the code under tests uses. - -##getDefaultInput -This function returns a `flit::TestInput`, which is a vector like object whose `.vals` field holds a list of inputs. Note that more than one set of inputs may be placed in this field. - -##run_impl -This function is the actual code under test. The argument is a `flit::TestInput` with length of the `.vals` field equal to the number of inputs requested per run. The return is a vector of two values representing the score of the test. This does not need to represent 'good' or 'bad' results, as the score is used to classify different compilations. - -##REGISTER_TEST -Just as it sounds, this function is used to register the test in the framework. It's argument is the class created for the test. - -#CUDA Tests -The main setup for CUDA tests is the same, with the exception that the `run_impl` function is a CUDA kernel which is placed in the framework though `getKernel()` and it has a second argument which is treated as an inout variable with fields `.s1` and `.s2` to hold the scores. - -#Integrating into the test framework -The test cpp file must be placed in `QFP/qfpc/tests` for it to be included in subsequent runs. No other changes are required. - -#Starting code -This is a simple source file to start your own tests. - -``` -#include "test_base.hpp"
 -#include "QFPHelpers.hpp" -#include "CUHelpers.hpp" - -// These functions are not needed, but it is nice to break out the core computation being done -template -DEVICE -T -my_cuda_test_core(const T input_1) // Number of arguments can be changed -{ - return input_1; // Cuda test code -} - -template -T -my_cpp_test_core(const T input_1) // Number of arguments can be changed -{ - return input_1; // Cpp test code -} - - -template -GLOBAL -void -my_cuda_kern(const flit::CuTestInput* tiList, flit::CudaResultElement* results){ -#ifdef __CUDA__ - auto idx = blockIdx.x * blockDim.x + threadIdx.x; -#else - auto idx = 0; -#endif - T input_1 = tiList[idx].vals[0]; // Extract arguments from vals - - // Loops or other structures can be used here such as in TrianglePSylv.cpp - double score = my_cuda_test_core(input); - - results[idx].s1 = score; - results[idx].s2 = 0.0; -} - -template -class MyTest: public flit::TestBase { -public: - MyTest(std::string id) : flit::TestBase(std::move(id)) {} - - // This must be changed to match the number of arguments used in the test code - virtual size_t getInputsPerRun() { return 1; } - - virtual flit::TestInput getDefaultInput() { - flit::TestInput ti; - ti.vals = { 6.0 }; - return ti; - } - -protected: - virtual - flit::KernelFunction* getKernel() {return my_cuda_kern; } - virtual - flit::ResultType::mapped_type run_impl(const flit::TestInput& ti) { - T input_1 = ti.vals[0]; // Extract arguments from vals - - // Loops or other structures can be used here such as in TrianglePSylv.cpp - double score = my_cpp_kern(input_1); - - return {score, 0.0}; - } - -protected: - using flit::TestBase::id; -}; - -REGISTER_TYPE(MyTest) -``` diff --git a/documentation/DisableOrRemoveTests.md b/documentation/DisableOrRemoveTests.md deleted file mode 100644 index 969f3c9d..00000000 --- a/documentation/DisableOrRemoveTests.md +++ /dev/null @@ -1,12 +0,0 @@ -#Disabling or Removing Litmus Tests -The simplest way to remove tests from the next run is to (re)move the source code from the directory `QFP/qfpc/tests` - - 1. cd QFP/qfpc - 2. mkdir disabled_tests - 3. mv disabled_tests - -If you want to compile a single test on the current machine for debug purposes then run: - - 1. cd QFP/qfpc - 2. make -f MakefileDev TESTS= - diff --git a/documentation/InputGenerator.md b/documentation/InputGenerator.md deleted file mode 100644 index 8bba18bd..00000000 --- a/documentation/InputGenerator.md +++ /dev/null @@ -1,69 +0,0 @@ -#Automatic Test Input Generation -The input generator is a script which can produce random input for a test, and given two sets of compile flags, find divergence points between two compiles. This is useful for creating default test input. - -To build the tool simply run: - - 1. cd QFP/inputGen - 2. make - -This will make the `inputGen` executable as well as all the tests in the `QFP/qfpc/tests` directory, this creates the list of tests for the tool. In the `makefile` is listed the compiler and flags used, which can be changed and recompiled to test different optimizations. The executable is well documented. - -``` ->./inputGen --help - -Usage: - inputGen [-h] - inputGen --list-tests - inputGen [-f|-d|-e|-a] [-m MAX_TRIES] [-i NUM_DIVERGENT_INPUTS] [-o FILENAME] - [-r RAND_TYPE] (--test TEST_NAME ... | --all-tests) - -Description: - Runs the particular tests under the optimization given in compilation. The - test input is randomly generated floating point numbers. It runs until the - number of desired divergences are discovered between the unoptimized code - and the optimized code. - -Required Arguments: - One of the following must be used since we need at least one test to run. - The --test option can be specified more than once to specify more than one - test. - - --test TEST_NAME - The test to run. For a list of choices, see --list-tests. - - --all-tests - Run all tests that are returned from --list-tests. - -Optional Arguments: - - -f, --float - Run each test only with float precision. This is the default. - - -d, --double - Run each test only with double precision. - - -e, --long-double - Run each test only with extended precision, also called long double. - - -a, --all-types - Run each test with all precisions. - - -i NUM_DIVERGENT_INPUTS, --divergent-input-count NUM_DIVERGEN_INPUTS - How many divergent inputs to attempt to identify and report. The - default value is 10. - - -m MAX_TRIES, --max-tries MAX_TRIES - How many tries before giving up on search for the specified numer of - divergent inputs. The default value is 1,000. - - -o FILENAME, --output FILENAME - Where to output the report. By default it outputs to stdout. - - -r RAND_TYPE, --rand-type RAND_TYPE - The type of random number distribution to use. Choices are: - - fp: Uniform over the floating-point number space - - real: Uniform over the real number line, then projected to - floating-point space - The default is "fp". - -``` \ No newline at end of file diff --git a/documentation/README.md b/documentation/README.md new file mode 100644 index 00000000..8b873905 --- /dev/null +++ b/documentation/README.md @@ -0,0 +1,16 @@ +# Documentation Table Of Contents + +[Return to top-level README file](../README.md) + +* [Installation](installation.md) +* [Litmus Tests](litmus-tests.md) +* [FLiT Command-Line](flit-command-line.md) +* [FLiT Configuration File](flit-configuration-file.md) + * [Available Compiler Flags](available-compiler-flags.md) +* [Writing Test Cases](writing-test-cases.md) +* [Database Structure](database-structure.md) +* [Analyze Results](analyze-results.md) +* **Extra Tools** + * [Autogenerated Tests](autogenerated-tests.md) + * [Test Input Generator](test-input-generator.md) + diff --git a/documentation/analyze-results.md b/documentation/analyze-results.md index d0444f39..6c514f8f 100644 --- a/documentation/analyze-results.md +++ b/documentation/analyze-results.md @@ -1,6 +1,18 @@ # Analyze Results +[Prev](database-structure.md) +| +[Table of Contents](README.md) +| +[Next](autogenerated-tests.md) + Since the database functionality is a little broken right now, the analysis is also not available. When the database functionality is fixed, you can expect to be here some instructions on how to generate useful charts from the gathered data. + +[Prev](database-structure.md) +| +[Table of Contents](README.md) +| +[Next](autogenerated-tests.md) diff --git a/documentation/autogenerated-tests.md b/documentation/autogenerated-tests.md index aaa67558..3522d1fd 100644 --- a/documentation/autogenerated-tests.md +++ b/documentation/autogenerated-tests.md @@ -1,8 +1,14 @@ # Autogenerated Tests +[Prev](analyze-results.md) +| +[Table of Contents](README.md) +| +[Next](test-input-generator.md) + There is some code to randomly generate test cases. For now, these autogenerated test cases are limited to a small number of arithmetic expressions, almost all of which demonstrate variability on at least one flag. -This tool requires an additional package dependency, `numpy`. +This tool requires an additional python package dependency, `numpy`. ```bash pip install numpy @@ -14,3 +20,9 @@ To run this generator, you will probably want an already initialized flit test d FLiT/gensrc/gensrc.py --outdir project/flit-tests/tests 1 ``` +[Prev](analyze-results.md) +| +[Table of Contents](README.md) +| +[Next](test-input-generator.md) + diff --git a/documentation/available-compiler-flags.md b/documentation/available-compiler-flags.md index 39c36f54..00783836 100644 --- a/documentation/available-compiler-flags.md +++ b/documentation/available-compiler-flags.md @@ -1,5 +1,11 @@ # Available Compiler Flags +[Prev](flit-configuration-file.md) +| +[Table of Contents](README.md) +| +[Next](writing-test-cases.md) + Convenient TOML lists: * [GCC](#gcc) @@ -162,3 +168,9 @@ switches = [ '--prec-sqrt=true', ] ``` + +[Prev](flit-configuration-file.md) +| +[Table of Contents](README.md) +| +[Next](writing-test-cases.md) diff --git a/documentation/d_all_bihexal.pdf b/documentation/d_all_bihexal.pdf deleted file mode 100755 index 2e77b30e..00000000 Binary files a/documentation/d_all_bihexal.pdf and /dev/null differ diff --git a/documentation/database-structure.md b/documentation/database-structure.md index 0f96a039..0b61153b 100644 --- a/documentation/database-structure.md +++ b/documentation/database-structure.md @@ -1,6 +1,18 @@ # Database Structure -The database functionality is a little broken. When the database is working -again, you can expect to be here the fields and tables (i.e. the database -specifications) here with a little explanation of each so that you can browse -the test results easily. +[Prev](writing-test-cases.md) +| +[Table of Contents](README.md) +| +[Next](analyze-results.md) + +The database functionality is a little broken. When the database is +working again, you can expect the fields and tables (i.e. the database +specifications) to be here with a little explanation of each so that +you can browse the test results easily. + +[Prev](writing-test-cases.md) +| +[Table of Contents](README.md) +| +[Next](analyze-results.md) diff --git a/documentation/flit-command-line.md b/documentation/flit-command-line.md index add6c93c..f454863f 100644 --- a/documentation/flit-command-line.md +++ b/documentation/flit-command-line.md @@ -1,19 +1,23 @@ # FLiT Command-Line +[Prev](litmus-tests.md) +| +[Table of Contents](README.md) +| +[Next](flit-configuration-file.md) + FLiT comes with a command-line tool called `flit`. This command-line tool is simply a symbolic link to `flit.py`. In the repository, it is located in `scripts/flitcli/flit.py`. When installed, it is installed in `/share/flit/scripts/flit.py`. This command is split up into many subcommands. Most of it is self documented. -Simply call +For more information, simply call: ```bash flit --help ``` -for more information. - Possible subcommands: * [flit help](#flit-help): display help for a specific subcommand @@ -21,7 +25,7 @@ Possible subcommands: * [flit update](#flit-update): Updates the Makefile based on `flit-config.toml` * [flit check](#flit-check): Verifies the correctness of a config file * [flit run](#flit-run): Run flit on the configured remote machine(s) -* [flit analyze](#flit-analyze): Runs analysis on a previous flit run +* [flit analyze](#flit-analyze): Performs analysis on a previous flit run ## flit help @@ -45,8 +49,7 @@ be up to date than this markdown description. ## flit init Initializes a flit test directory for use. It will initialize the directory by -copying the default configuration file into the given directory. If a -configuration file already exists, this command does nothing. The config file +copying the default configuration file into the given directory. The config file is called `flit-config.toml`. To see how to modify `flit-config.toml`, see the documentation for [FLiT @@ -84,10 +87,10 @@ _not yet implemented_ This command only verifies the correctness of the configurations you have for your flit tests. As part of this verification, this command checks to see if -the remote connections are capable of being done, such as the connection to +it is possible to establish the remote connections, such as the connection to the machines to run the software, the connection to the database machine, and the connection to the database machine from the run machine. You may need to -provide a few SSH passwords to do this check. +provide a few SSH passwords to perform this check. Since this subcommand is not yet implemented, it may change in nature when it does finally get implemented. @@ -98,7 +101,7 @@ _not yet implemented_ Run flit on the configured remote machine(s). Note that you may need to provide a password for SSH, but that should be taken care of pretty early on -in the process. The results should be sent to the database computer for later +in the process. The results will be sent to the database computer for later analysis. ## flit analyze @@ -109,3 +112,9 @@ Runs analysis on a previous flit run. The analysis will be of the current flit repository and will create a directory called analysis inside of the flit directory. +[Prev](litmus-tests.md) +| +[Table of Contents](README.md) +| +[Next](flit-configuration-file.md) + diff --git a/documentation/flit-configuration-file.md b/documentation/flit-configuration-file.md index fcaf6b43..8ca2fa00 100644 --- a/documentation/flit-configuration-file.md +++ b/documentation/flit-configuration-file.md @@ -1,9 +1,22 @@ # FLiT Configuration File +[Prev](flit-command-line.md) +| +[Table of Contents](README.md) +| +[Next](available-compiler-flags.md) + The FLiT configuration file is called `flit-config.toml`. It is written in the [TOML](https://github.com/toml-lang/toml) specification. -Here is a full example configuration file: +Here I have an example configuration file. I will go through it section by +section. If you want to see the full configuration file, it is at the end of +this page. + +_Note: there are no default values for these fields. You must specify all +fields for an entry to be valid. If in doubt, you can use [flit +check](flit-command-line.md#flit-check) to verify your configuration file is +valid._ ```toml [database] @@ -12,19 +25,47 @@ username = 'mbentley' address = 'localhost' type = 'postgres' port = 5432 +``` +Above we specify the information for the database connection. Since +it is not very secure to store passwords in the configuration file, +you will be prompted for the password at the time of execution. Right +now, `postgres` is the only database type that is supported, but more +are to come. + +```toml [[hosts]] name = 'my.hostname.com' flit_path = '/usr/bin/flit' config_dir = 'project/flit-tests' +``` + +Now we start specifying information of the first host called `my.hostname.com`. +The `flit_path` is the path of the `flit` command-line tool at this host. The +`config_dir` is the directory where this configuration file is located on the +host. For now, it is expected that each machine already has a copy of the +tests with the same configuration file and that their location is documented +here. If needed, this may be revisited. + +For any variable that specifies a relative path, the relative path is always +with respect to the current user's home directory. +```toml [hosts.ground_truth] compiler = 'g++' optimization_level = '-O0' switch = '' +``` + +Each host has a ground truth run to use as comparisons when doing analysis. +The `compiler` field here needs to match the `name` of the compiler specified +later in the `[[hosts.compilers]]`. Also, the `optimization_level` and +`switch` need to be available in the `optimization_levels` and `switches` for +the compiler with the matching name. +```toml [[hosts.compilers]] binary = 'g++' @@ -39,7 +80,26 @@ switch = '' '-fassociative-math', '-mavx2 -mfma', ] +``` + +Here we specify the first compiler for the first host. Since binary here is a +simple name, it will get the executable `g++` from the system `PATH`. If you +really mean you want to have a compiler that is located in your home directory, +you can do `./g++`. + +The `type` parameter can be one of + +* `gcc` +* `clang` +* `intel` +* `cuda` + +The `optimization_levels` and `switches` will be combined as a cartesian +product and each possible pairing will become a compilation performed by FLiT. +For a list of all possible flags for each compiler type, see [Available +Compiler Flags](available-compiler-flags.md). +```toml [[hosts.compilers]] binary = 'my-installs/g++-7.0/bin/g++' @@ -54,7 +114,14 @@ switch = '' '-mavx2 -mfma', '-funsafe-math-optimizations', ] +``` + +Here it is demonstrated that you can specify a second version of `g++` +with different optimization levels and switches from the first +version. It is simply required that the compiler name be unique for +this host. +```toml [[hosts.compilers]] binary = 'clang++' @@ -75,7 +142,13 @@ switch = '' '-mavx2 -mfma', '-march=core-avx2', ] +``` + +We also specify a third compiler `clang++`, again with different flags. So for +the host `my.hostname.com`, we have three compilers configured: `g++`, +`g++-7.0`, and `clang`. +```toml [[hosts]] name = 'other.hostname.com' @@ -101,13 +174,12 @@ switch = '' ] ``` -We will now go through the elements of this configuration file piece by piece -and explain what they represent. +Here it is demonstrated that you can specify another host. This one is called +`other.hostname.com` with a single compiler named `intel`. + +## Full Configuration File -_Note: there are no default values for these fields. You must specify all -fields for an entry to be valid. If in doubt, you can use [flit -check](flit-command-line.md#flit-check) to verify your configuration file is -valid._ +Combining all of the above sections together, here is the full example configuration file: ```toml [database] @@ -116,46 +188,19 @@ username = 'mbentley' address = 'localhost' type = 'postgres' port = 5432 -``` - -Above we specify the information for the database connection. Since it is not -very secure to store passwords in the configuration file, the password will be -asked at the time of execution. Right now, `postgres` is the only database -type that is supported, but more are to come. -```toml [[hosts]] name = 'my.hostname.com' flit_path = '/usr/bin/flit' config_dir = 'project/flit-tests' -``` - -Now we start specifying information of the first host called `my.hostname.com`. -The `flit_path` is the path of the `flit` command-line tool at this host. The -`config_dir` is the directory where this configuration file is located on the -host. For now, it is expected that each machine already has a copy of the -tests with the same configuration file and that their location is documented -here. If needed, this may be revisited. -For any variable that specifies a relative path, the relative path is always -with respect to the current user's home directory. - -```toml [hosts.ground_truth] compiler = 'g++' optimization_level = '-O0' switch = '' -``` - -Each host has a ground truth run to use as comparisons when doing analysis. -The `compiler` field here needs to match the `name` of the compiler specified -later in the `[[hosts.compilers]]`. Also, the `optimization_level` and -`switch` need to be available in the `optimization_levels` and `switches` for -the compiler with the matching name. -```toml [[hosts.compilers]] binary = 'g++' @@ -170,23 +215,7 @@ the compiler with the matching name. '-fassociative-math', '-mavx2 -mfma', ] -``` - -Here we specify the first compiler for the first host. Since binary here is a simple name, it will get the executable `g++` from the system `PATH`. If you really mean you want to have a compiler that is located in your home directory, you can do `./g++`. - -The `type` parameter can be one of - -* `gcc` -* `clang` -* `intel` -* `cuda` - -The `optimization_levels` and `switches` will be combined as a cartesian -product and each possible pairing will become a compilation performed by FLiT. -For a list of all possible flags for each compiler type, see [Available -Compiler Flags](available-compiler-flags.md). -```toml [[hosts.compilers]] binary = 'my-installs/g++-7.0/bin/g++' @@ -201,13 +230,7 @@ Compiler Flags](available-compiler-flags.md). '-mavx2 -mfma', '-funsafe-math-optimizations', ] -``` - -Here is demonstrated that you can specify another version of `g++` with even -different optimization levels and switches. It is simply required that the -compiler name be unique for this host. -```toml [[hosts.compilers]] binary = 'clang++' @@ -228,13 +251,7 @@ compiler name be unique for this host. '-mavx2 -mfma', '-march=core-avx2', ] -``` - -We also specify a third compiler `clang++` with again different flags. So for -this host of `my.hostname.com` we have three compilers configured: `g++`, -`g++-7.0`, and `clang`. -```toml [[hosts]] name = 'other.hostname.com' @@ -260,5 +277,9 @@ switch = '' ] ``` -Here is demonstrated that you can specify another host. This one is called -`other.hostname.com` with a single compiler named `intel`. + +[Prev](flit-command-line.md) +| +[Table of Contents](README.md) +| +[Next](available-compiler-flags.md) diff --git a/documentation/installation.md b/documentation/installation.md index 1adbfef5..c7090249 100644 --- a/documentation/installation.md +++ b/documentation/installation.md @@ -1,6 +1,10 @@ # FLiT Installation -Table of Contents: +[Table of Contents](README.md) +| +[Next](litmus-tests.md) + +Instruction Contents: * [Roles](#roles) * [Prerequisites](#prerequisites) @@ -24,8 +28,7 @@ compilers. There are 3 roles used in a FLiT architecture: Each role requires slightly different setup. A particular computer may only satisfy one role, or it may satisfy multiple roles. You can potentially have -all three roles on a single machine, which is the simplest use case. Each -setup step has associated with it which roles the instructions apply toward. +all three roles on a single machine, which is the simplest use case. ## Prerequisites @@ -39,7 +42,7 @@ The following prerequisites are required for all three roles: * The [toml](https://github.com/uiri/toml) module (for [TOML](https://github.com/toml-lang/toml) configuration files) * [make](https://www.gnu.org/software/make) -* [gcc](https://gcc.gnu.org) version 4.9 or later +* [gcc](https://gcc.gnu.org) version 4.9 or higher For Debian-based systems: @@ -54,16 +57,16 @@ brew install make python3 gcc git pip3 install toml ``` -If you install python version 3.0 or more, then you will need to have a -symbolic link called `python3` in the path pointing to that python executable. +If you install python version 3.0 or later, then you will need to have a +symbolic link called `python3` in your `PATH` pointing to that python executable. ### Runner Prerequisites The test runner can run multiple compilers. For now, only one compiler is supported from each of the types: GCC, Clang, Intel's icpc, and NVIDIA's nvcc. -Simply have the one you want used as the first on the system PATH. You do not +Simply have the one you want used as the first in your system PATH. You do not need all four of those, only those ones installed will be used. But all of -them need to be able to support C++11. +them need to support C++11. If this is not on the same machine as the Launcher, then the Database machine will need an SSH server running. @@ -157,7 +160,11 @@ to make the following connections: * **Launcher** -> **Database** * **Runner** -> **Database** -so these are the connections you may want to have an SSH connection. See +so these are the connections you may want to setup SSH keys for. See [Ubuntu's help documentation](https://help.ubuntu.com/community/SSH/OpenSSH/Keys) for setting up SSH keys. + +[Table of Contents](README.md) +| +[Next](litmus-tests.md) diff --git a/documentation/litmus-tests.md b/documentation/litmus-tests.md index 71b0cd82..182c308e 100644 --- a/documentation/litmus-tests.md +++ b/documentation/litmus-tests.md @@ -1,5 +1,11 @@ # Litmus Tests +[Prev](installation.md) +| +[Table of Contents](README.md) +| +[Next](flit-command-line.md) + FLiT comes with a large number of _litmus tests_ each that demonstrate some simple aspect of compiler variability (or lack thereof) due to different compilations. @@ -11,6 +17,11 @@ copied to your test directory using the `flit init` command-line tool (see Most litmus tests have a CUDA version as well. You may use these tests as examples or play with them to gain insights into how the compiler alters simple examples. The community is encouraged to contribute back interesting test -cases that can be added to these litmus tests. Simply do a **Pull Request** on +cases that can be added to these litmus tests. Simply create a **Pull Request** on GitHub. +[Prev](installation.md) +| +[Table of Contents](README.md) +| +[Next](flit-command-line.md) diff --git a/documentation/test-input-generator.md b/documentation/test-input-generator.md index 63d6f3e9..472fae8a 100644 --- a/documentation/test-input-generator.md +++ b/documentation/test-input-generator.md @@ -1,3 +1,11 @@ # Test Input Generator +[Prev](autogenerated-tests.md) +| +[Table of Contents](README.md) + The input generator is currently broken. Ideally, I would like to add input generation to the makefile of user-written code bases. + +[Prev](autogenerated-tests.md) +| +[Table of Contents](README.md) diff --git a/documentation/writing-test-cases.md b/documentation/writing-test-cases.md index 4445f668..a96f3b47 100644 --- a/documentation/writing-test-cases.md +++ b/documentation/writing-test-cases.md @@ -1,5 +1,11 @@ # Writing Test Cases +[Prev](available-compiler-flags.md) +| +[Table of Contents](README.md) +| +[Next](database-structure.md) + When you initialize a test directory using `flit init`, a test file called `Empty.cpp` will be placed in the `tests` directory. This test is as the name implies - empty. It basically shows you how to create a new test and is @@ -21,3 +27,9 @@ mv tests/BrokenTest.cpp disabled ``` That's it. + +[Prev](available-compiler-flags.md) +| +[Table of Contents](README.md) +| +[Next](database-structure.md)