Home

lisp-unit is a Common Lisp library that supports unit testing. It is an extension of the library written by Chris Riesbeck. There is a long history of testing packages in Lisp, usually called "regression" testers. More recent packages in Lisp and other languages have been inspired by JUnit for Java.

Overview

The main goal for lisp-unit was to make it simple to use. The advantages of lisp-unit are:

Written in portable Common Lisp
Loadable as a single file
Loadable with ASDF or Quicklisp
Simple to define and run tests
Redfine functions and macros without reloading tests
Test return values, printed output, macro expansions, and conditions
Fined grained control over the testing output
Store all test results in a database object that can be examined
Group tests by package for modularity
Group tests using tags
Signal test completion and return results with the condtion.

Extensions

Floating point predicates
Test Anything Protocol output

How to use lisp-unit

The core definitions of lisp-unit may be used by loading the single file 'lisp-unit.lisp'. To use the extensions, lisp-unit must be loaded using either Quicklisp or ASDF.

Load (or compile and load) as a single file : (load "lisp-unit").
Load using Quicklisp : (ql:quickload :lisp-unit).
Load using ASDF : (asdf:load-system :lisp-unit).

Once lisp-unit has been loaded, load the file of unit-tests and run the tests with run-tests. A summary of how many assertions were run, how many passed, how many failed, how many generated execution errors, and how many tests were missing will be printed. A test results object will be returned containing the tests results. It may be examined to generate more detailed reports and refine the testing to focus on failed, erroneous, and missing tests. Individual test results can be printed as the tests are run by setting lisp-unit print parameters.

You define a test with define-test:

  (define-test name exp1 exp2 ...)

This defines a test called name. The expressions can be anything, but typically most will be assertion forms. Tests can be defined before the code they test, even if they're testing macros. This is to support test-first programming.

After defining your tests and the code they test, run the tests with

  (run-tests :all)

This runs every test defined in the current package. To run just certain specific tests, use

  (run-tests '(name1 name2 ...))

e.g., (run-tests '(greater summit)).

The following example

defines some tests to see if my-max returns the larger of two arguments
defines a deliberately broken version of my-max
runs the tests

First, we define some tests.

> (in-package :example)
#<PACKAGE EXAMPLE>
> (define-test test-my-max
    (assert-equal 5 (my-max 2 5))
    (assert-equal 5 (my-max 5 2))
    (assert-equal 10 (my-max 10 10))
    (assert-equal 0 (my-max -5 0)))
TEST-MY-MAX

Following good test-first programming practice, we run these tests before writing any code.

> (run-tests '(test-my-max))
Unit Test Summary
 | 0 assertions total
 | 0 passed
 | 0 failed
 | 1 execution errors
 | 0 missing tests

#<TEST-RESULTS-DB Total(0) Passed(0) Failed(0) Errors(1)>

This shows that we need to do some work. Let's see exactly what caused the execution error. If we had many tests defined, we could get a list of the failed tests from the test results object using the accessor failed-tests. In this case, we just want to see the execution error using print-errors.

> (print-errors *)
 | Execution error:
 | Undefined operator MY-MAX in form (MY-MAX 2 5).
 |
TEST-MY-MAX: 0 assertions passed, 0 failed, 1 execution errors.

Let's define a broken version of my-max and set *print-failures* to true so that the failures are printed as the tests are run.

> (defun my-max (x y) x)  ;; deliberately wrong
MY-MAX

> (setq *print-failures* t)
T

Now we run the tests again:

> (run-tests '(test-my-max))
 | Failed Form: (MY-MAX -5 0)
 | Expected 0 but saw -5
 |
 | Failed Form: (MY-MAX 2 5)
 | Expected 5 but saw 2
 |
TEST-MY-MAX: 2 assertions passed, 2 failed.

Unit Test Summary
 | 4 assertions total
 | 2 passed
 | 2 failed
 | 0 execution errors
 | 0 missing tests

#<TEST-RESULTS-DB Total(4) Passed(2) Failed(2) Errors(0)>

No execution errors, but there were 2 failures. Note that the test specific summary printed as well. It prints when any of the lisp-unit print parameters are set to true. In both failures, the equality test returned NIL. In the first case it was because (my-max 2 5) returned 2 when 5 was expected, and in the second case, it was because (my-max -5 0) returned -5 when 0 was expected.

Assertion Forms

The most commonly used assertion form is

(assert-equal value form)

This tallies a failure if form returns a value not equal to value. Both value and test are evaluated in the local lexical environment. This means that you can use local variables in tests. In particular, you can write loops that run many tests at once:

> (define-test my-sqrt
    (dotimes (i 5)
     (assert-equal i (my-sqrt (* i i)))))
MY-SQRT

> (defun my-sqrt (n) (/ n 2))   ;; wrong!!

> (run-tests '(my-sqrt))
 | Failed Form: (MY-SQRT (* I I))
 | Expected 4 but saw 8
 |
 | Failed Form: (MY-SQRT (* I I))
 | Expected 3 but saw 9/2
 |
 | Failed Form: (MY-SQRT (* I I))
 | Expected 1 but saw 1/2
 |
TEST-MY-SQRT: 2 assertions passed, 3 failed.

Unit Test Summary
 | 5 assertions total
 | 2 passed
 | 3 failed
 | 0 execution errors
 | 0 missing tests

#<TEST-RESULTS-DB Total(5) Passed(2) Failed(3) Errors(0)>

The failed forms are printed because *print-failures* is still true from the example in the previous section. However, the above output doesn't tell us for which values of i the code failed. Fortunately, you can fix this by adding expressions at the end of the assert-equal. These expression and their values will be printed on failure.

> (define-test my-sqrt
    (dotimes (i 5)
     (assert-equal i (my-sqrt (* i i)) i)))  ;; added i at the end
MY-SQRT

> (run-tests '(my-sqrt))
 | Failed Form: (MY-SQRT (* I I))
 | Expected 4 but saw 8
 | I => 4
 |
 | Failed Form: (MY-SQRT (* I I))
 | Expected 3 but saw 9/2
 | I => 3
 |
 | Failed Form: (MY-SQRT (* I I))
 | Expected 1 but saw 1/2
 | I => 1
 |
TEST-MY-SQRT: 2 assertions passed, 3 failed.

Unit Test Summary
 | 5 assertions total
 | 2 passed
 | 3 failed
 | 0 execution errors
 | 0 missing tests

#<TEST-RESULTS-DB Total(5) Passed(2) Failed(3) Errors(0)>

The next most useful assertion form is

(assert-true test)

This tallies a failure if test returns false. Again, if you need to print out extra information, just add expressions after test. There are also assertion forms to test what code prints, what errors code returns, or what a macro expands into. A complete list of assertion forms is in the reference and extensions sections.

Do not confuse assert-true with Common Lisp's assert macro. assert is used in code to guarantee that some condition is true. If it isn't, the code halts. assert has options you can use to let a user fix what's wrong and resume execution. A similar collision of names exists in JUnit and Java.

How to Organize Tests with Packages

Tests are grouped internally by the current package, so that a set of tests can be defined for one package of code without interfering with tests for other packages. If your code is being defined in cl-user, which is common when learning Common Lisp, but not for production-level code, then you should define your tests in cl-user as well. If your code is being defined in its own package, you should define your tests either in that same package, or in another package for test code. The latter approach has the advantage of making sure that your tests have access to only the exported symbols of your code package.

For example, if you were defining a date package, your date.lisp file would look like this:

(defpackage :date
  (:use :common-lisp)
  (:export :date->string :string->date))
  
(in-package :date)

(defun date->string (date) ...)
(defun string->date (string) ...)

Your date-tests.lisp file would look like this:

(defpackage :date-tests
  (:use :common-lisp :lisp-unit :date))

(in-package :date-tests)

(define-test date->string
  (assert-true (string= ... (date->string ...)))
  ...)
...

You could then run all your date tests in the test package:

(in-package :date-tests)

(run-tests :all)

Alternately, you could run all your date tests from any package with:

(lisp-unit:run-tests :all :date-tests)

How to Organize Tests with Tags

Tests can also be organized using tags. Tags are a convenient way to create subsets of tests for targeted evaluation. If we were writing separate addition and subtraction functions for integer, floating point, and complex numbers, it would be convenient to organize the tests by operation and by type.

(defun add-integer (integer1 integer2)
  "Add 2 integer numbers"
  (check-type integer1 integer)
  (check-type integer2 integer)
  (+ integer1 integer2))

(defun subtract-integer (integer1 integer2)
  "Subtract 2 integer numbers"
  (check-type integer1 integer)
  (check-type integer2 integer)
  (- integer1 integer2))

(define-test add-integer
  "Test add-integer for values and errors."
  (:tag :add :integer)
  (assert-eql 3 (add-integer 1 2))
  (assert-error 'type-error (add-integer 1.0 2))
  (assert-error 'type-error (add-integer 1 2.0)))

(define-test subtract-integer
  "Test subtract-integer for values and errors."
  (:tag :subtract :integer)
  (assert-eql 1 (subtract-integer 3 2))
  (assert-error 'type-error (subtract-integer 3.0 2))
  (assert-error 'type-error (subtract-integer 2 3.0)))

(defun add-float (float1 float2)
  "Add 2 floating point numbers"
  (check-type float1 float)
  (check-type float2 float)
  (+ float1 float2))

(defun subtract-float (float1 float2)
  "Subtract 2 floating point numbers"
  (check-type float1 float)
  (check-type float2 float)
  (- float1 float2))

(define-test add-float
  "Test add-float for values and errors."
  (:tag :add :float)
  (assert-eql 3.0 (add-float 1.0 2.0))
  (assert-error 'type-error (add-float 1.0 2))
  (assert-error 'type-error (add-float 1 2.0)))

(define-test subtract-float
  "Test subtract-float for values and errors."
  (:tag :subtract :float)
  (assert-eql 1.0 (subtract-float 3.0 2.0))
  (assert-error 'type-error (subtract-float 3.0 2))
  (assert-error 'type-error (subtract-float 2 3.0)))

(defun add-complex (complex1 complex2)
  "Add 2 complex numbers"
  (check-type complex1 complex)
  (check-type complex2 complex)
  (+ complex1 complex2))

(defun subtract-complex (complex1 complex2)
  "Subtract 2 complex numbers"
  (check-type complex1 complex)
  (check-type complex2 complex)
  (- complex1 complex2))

(define-test add-complex
  "Test add-complex for values and errors."
  (:tag :add :complex)
  (assert-eql #C(3 5) (add-complex #C(1 2) #C(2 3)))
  (assert-error 'type-error (add-integer #C(1 2) 3))
  (assert-error 'type-error (add-integer 1 #C(2 3))))

(define-test subtract-complex
  "Test subtract-complex for values and errors."
  (:tag :subtract :complex)
  (assert-eql #C(1 2) (subtract-complex #C(3 5) #C(2 3)))
  (assert-error 'type-error (subtract-integer #C(3 5) 2))
  (assert-error 'type-error (subtract-integer 2 #C(2 3))))

Then we could use run-tags to evaluate only the tests for an operation or type.

> (setq *print-summary* t)
T

> (run-tags '(:integer))
SUBTRACT-INTEGER: 3 assertions passed, 0 failed.

ADD-INTEGER: 3 assertions passed, 0 failed.

Unit Test Summary
 | 6 assertions total
 | 6 passed
 | 0 failed
 | 0 execution errors
 | 0 missing tests

#<TEST-RESULTS-DB Total(6) Passed(6) Failed(0) Errors(0)>

> (run-tags '(:subtract))
SUBTRACT-COMPLEX: 3 assertions passed, 0 failed.

SUBTRACT-FLOAT: 3 assertions passed, 0 failed.

SUBTRACT-INTEGER: 3 assertions passed, 0 failed.

Unit Test Summary
 | 9 assertions total
 | 9 passed
 | 0 failed
 | 0 execution errors
 | 0 missing tests

#<TEST-RESULTS-DB Total(9) Passed(9) Failed(0) Errors(0)>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly