Merge pull request #66 from PRUNERS/issue_53-return-string

Issue 53 return string - documentation update

Makefile

@@ -100,6 +100,7 @@ install: $(TARGET)
 	install -m 0755 $(SCRIPT_DIR)/flit_*.py $(PREFIX)/share/flit/scripts/
 	install -m 0644 $(SCRIPT_DIR)/flitutil.py $(PREFIX)/share/flit/scripts/
 	install -m 0644 $(SCRIPT_DIR)/README.md $(PREFIX)/share/flit/scripts/
+	install -m 0644 $(DOC_DIR)/*.md $(PREFIX)/share/flit/doc/
 	install -m 0644 $(DATA_DIR)/Makefile.in $(PREFIX)/share/flit/data/
 	install -m 0644 $(DATA_DIR)/custom.mk $(PREFIX)/share/flit/data/
 	install -m 0644 $(DATA_DIR)/main.cpp $(PREFIX)/share/flit/data/

README.md

@@ -2,37 +2,46 @@
 
 [![FLiT Bird](/images/flit-small.png)](https://github.com/PRUNERS/FLiT "FLiT")
 
-Floating-point Litmus Tests (FLiT) is a test infrastructure for detecting varibility
-in floating-point code caused by variations in compiler code generation,
-hardware and execution environments.
-
-FLiT works by compiling many versions of user-provided test cases, using multiple c++
-compilers, floating-point related compiler options (i.e. flags) and optimization
-levels.  These tests are then executed on target platforms, where a
-representative "score" is collected into a database, along with the other
-parameters relevant to the execution, such as host, compiler configuration and
-compiler vendor.  In addition to the user-configured test output, FLiT can collect
-counts of each assembly opcode executed (currently, this works with Intel
-architectures only, using their PIN dynamic binary instrumentation tool).
-
-After executing the test cases and collecting the data, the test results can be
-mined to find and demonstrate divergent behavior due to factors other than the
-source code.  This information can be used by the developer to understand and
-configure their software compilations so that they can obtain consistent and
-reproducible floating-point results.
-
-Additionally, we've provided the capability to __time__ the test suite, generating
-a time plot per test, comparing both the result divergence and the timing for
-each compiler flag.
+Floating-point Litmus Tests (FLiT) is a C++ test infrastructure for detecting
+variability in floating-point code caused by variations in compiler code
+generation, hardware and execution environments.
+
+Compilers are primarily focused on optimizing the speed of your code.  However,
+when it comes to floating-point, compilers go a little further than some might
+want, to the point that you may not get the same result from your
+floating-point algorithms.  For many applications, this may not matter as the
+difference is typically very small.  But there are situations where
+
+1. The difference is not so small
+2. Your application cares even about small differences, or
+3. Your application is so large (such as a weather simulation) that a small
+   change may propagate into very large result variability.
+
+FLiT helps developers determine where reproducibility problems may occur due to
+compilers.  The developer creates reproducibility tests with their code using
+the FLiT testing framework.  Then FLiT takes those reproducibility tests and
+compiles them under a set of configured compilers and a large range of compiler
+flags.  The results from the tests under different compilations are then compared
+against the results from a "ground truth" compilation (e.g. a completely
+unoptimized compilation).
+
+More than simply comparing against a "ground truth" test result, the FLiT
+framework also measures runtime of your tests.  Using this information, you can
+not only determine which compilations of your code are safe for your specific
+application, but you can also determine the fastest safe compilation.  This
+ability really helps the developer navigate the tradeoff between
+reproducibility and performance.
 
 It consists of the following components:
 
 * a c++ reproducibility test infrastructure
 * a dynamic make system to generate diverse compilations
 * _(currently broken)_ an execution disbursement system 
-* _(currently broken)_ a SQL database for collecting results
-  * a collection of queries to help the user understand results
-  * some data analysis tools, providing visualization of results
+* an SQL database for collecting results
+  * _(currently broken)_ a collection of queries to help the user understand
+    results
+  * _(currently broken)_ some data analysis tools, providing visualization of
+    results
 
 Contents:
 

documentation/analyze-results.md

@@ -6,10 +6,8 @@
 |
 [Next](autogenerated-tests.md)
 
-Since the database functionality is a little broken right now, the analysis is
-also not available.  When the database functionality is fixed, you can expect
-to be here some instructions on how to generate useful charts from the gathered
-data.
+The analysis functionality is broken at the moment.  You can expect to be here
+some instructions on how to generate useful charts from the gathered data.
 
 [Prev](database-structure.md)
 |

documentation/autogenerated-tests.md

@@ -6,15 +6,18 @@
 |
 [Next](test-input-generator.md)
 
-There is some code to randomly generate test cases.  For now, these autogenerated test cases are limited to a small number of arithmetic expressions, almost all of which demonstrate variability on at least one flag.
+There is some code to randomly generate test cases.  For now, these
+autogenerated test cases are limited to a small number of arithmetic
+expressions, almost all of which demonstrate variability on at least one flag.
 
 This tool requires an additional python package dependency, `numpy`.
 
 ```bash
 pip install numpy
 ```
 
-To run this generator, you will probably want an already initialized flit test directory (let's say it is located at `project/flit-tests`).
+To run this generator, you will probably want an already initialized flit test
+directory (let's say it is located at `project/flit-tests`).
 
 ```bash
 FLiT/gensrc/gensrc.py --outdir project/flit-tests/tests 1

documentation/database-structure.md

@@ -6,10 +6,72 @@
 |
 [Next](analyze-results.md)
 
-The database functionality is a little broken.  When the database is
-working again, you can expect the fields and tables (i.e. the database
-specifications) to be here with a little explanation of each so that
-you can browse the test results easily.
+There are currently only two tables in the created SQLite3 database (although
+there may be more added later).  The schema for this database can be found in
+`data/db/tables-sqlite.sql`, or in the install located at
+`<PREFIX>/share/flit/data/db/tables-sqlite.sql`.  Alternatively, you can see
+the schema of the SQLite3 database by querying for it:
+
+```sqlite3
+$ sqlite3 results.sqlite
+SQLite version 3.13.0 2016-05-18 10:57:30
+Enter ".help" for usage hints.
+sqlite> .tables
+runs   tests
+sqlite> .schema
+CREATE TABLE runs (
+  -- The run id used in tests table
+  id             integer    primary key autoincrement     not null,
+
+  -- timestamp is supported in python if you do the following:
+  --   conn = sqlite3.connect("flit.sqlite",
+  --                          detect_types=sqlite3.PARSE_DECLTYPES)
+  -- The secret sauce is in the "detect_types" that allows python to intercept
+  -- it and convert it to a sqlite3 basic type and back.
+  rdate          timestamp,
+
+  -- The message describing what this run is all about
+  notes          text
+  );
+CREATE TABLE tests (
+  id             integer    primary key autoincrement     not null,
+  run            integer,   -- run index from runs table
+  name           varchar,   -- name of the test case
+  host           varchar,   -- name of computer that ran the test
+  compiler       varchar,   -- compiler name
+  optl           varchar,   -- optimization level (e.g. "-O2")
+  switches       varchar,   -- compiler flag(s) (e.g. "-ffast-math")
+  precision      varchar,   -- precision (f = float, d = double, e = extended)
+  comparison     varchar,   -- metric of comparison - hex value
+  comparison_d   real,      -- metric of comparison of result vs ground truth
+  file           varchar,   -- filename of test executable
+  nanosec        integer    check(nanosec >= 0),  -- timing for the function
+
+  foreign key(run) references runs(id)
+  );
+sqlite> .quit
+.quit
+```
+
+This output is as of this writing.  You can execute those same commands to see
+the exact schema used in your version of FLiT.
+
+The `runs` table only stores information about each executed full run, the id,
+datetime and user-specified label for the run (called `notes`).
+
+The `tests` table contains the actual test results.  Each row has a run number
+that matches the `id` field of the `runs` table, so you can do things like:
+
+```sqlite3
+select * from tests where run = 3;
+```
+
+Information about each compilation is stored there as well as the comparison
+value.  The comparison value is what is returned from the `compare()` method in
+your test class that compares a test result against the ground truth result.
+Only this value is stored in the database for storage space reasons.  It also
+contains timing information in the `nanosec` column which can be used to find
+the fastest runtime execution.
 
 [Prev](writing-test-cases.md)
 |

documentation/flit-command-line.md

@@ -9,7 +9,8 @@
 FLiT comes with a command-line tool called `flit`.  This command-line tool is
 simply a symbolic link to `flit.py`.  In the repository, it is located in
 `scripts/flitcli/flit.py`.  When installed, it is installed in
-`<PREFIX>/share/flit/scripts/flit.py`.
+`<PREFIX>/share/flit/scripts/flit.py` with the symbolic link at
+`<PREFIX>/bin/flit`.
 
 This command is split up into many subcommands.  Most of it is self documented.
 For more information, simply call:
@@ -25,6 +26,7 @@ Possible subcommands:
 * [flit update](#flit-update): Updates the Makefile based on `flit-config.toml`
 * [flit check](#flit-check): Verifies the correctness of a config file
 * [flit run](#flit-run): Run flit on the configured remote machine(s)
+* [flit import](#flit-import): Imports test results into an SQLite3 database
 * [flit analyze](#flit-analyze): Performs analysis on a previous flit run
 
 ## flit help
@@ -66,7 +68,7 @@ There are a few other files copied over:
   framework
 
 If you want to play with the litmus tests in this directory, you can pass the
-`--litmus-tests` which will copy over the litmus tests into the `tests`
+`--litmus-tests` flag which will copy over the litmus tests into the `tests`
 directory for use.
 
 ## flit update
@@ -104,6 +106,29 @@ provide a password for SSH, but that should be taken care of pretty early on
 in the process. The results will be sent to the database computer for later
 analysis.
 
+## flit import
+
+Imports flit results into the database configured in `flit-config.toml`.  It
+can either import a CSV file created from a FLiT test executable (using the
+`--output` flag) or it can import another SQLite3 database file created from
+another call to `flit import`.
+
+The following is an example of how to do a full run of the FLiT litmus tests
+and import them into a database file.
+
+```bash
+flit init --directory litmus-test-run --litmus-tests
+cd litmus-test-run
+rm tests/Empty.cpp
+make run -j8
+flit import --new-run results/*.csv
+```
+
+These commands will populate by default a file called `results.sqlite` within
+the created `litmus-test-run` directory.  You are free to examine the results
+using the `sqlite3` interactive tool or any other method you have for running
+queries on an SQLite3 database.
+
 ## flit analyze
 
 _not yet implemented_