The Allegro CL Test harness

The index for the Allegro CL Documentation is in index.htm. The documentation is described in introduction.htm.

This document contains the following sections:

1.0 The tester module API
   1.1 Test Harness Variables
   1.2 Test Harness Macros
   1.3 Examples

ANSI Common Lisp contains no functionality designed specifically for testing applications. Because testing is an essential part of application development, Franz Inc. is making public the test harness used internally for testing Allegro CL itself. (A test harness is a collection of macros and variables associated with testing, along with templates for test forms.)

The test harness facility was added to Allegro CL in release 6.0. (It was available as a patch for release 5.0.1).

To use the test harness, you must load the tester.fasl module. Do this by evaluating

(require :tester)

1.0 The tester module API

All of the following symbols are exported from the util.test package.

1.1 Test Harness Variables

The test harness API includes following variables, each described fully on its own page and briefly here.

1.2 Test Harness Macros

These macros wrap around a form to be tested and supply the expected value (for the test macro) or the expected behavior, which is encoded in the macro name (e.g. test-error). For example:

(test 1 (+ 0 1))   ;; (testing that the result of (+ 0 1) is
                   ;; the fixnum 1)
(test-error (+ 1 "2"))  ;; (testing that an error is
                                  ;; signaled when a string is 
                                  ;; passed as an argument to +)

Many more examples are given below.

with-tests wraps around a collection of test or test-* forms.

Note that many of the macros have fail-info and known-failure keyword arguments.

Each macro is described briefly here and fully on its documentation page.

1.3 Examples

The following are simple examples using the test harness. The test forms themselves are trivial, and the purpose is to indicate the behavior of the test harness macros.

user(1): (require :tester)
; Fasl loading .../tester.fasl
t
user(2): (use-package :util.test)
t
user(3): (test 1 1)
t
user(4): (test 1 2)
 * * * UNEXPECTED TEST FAILURE * * *
Test failed: 2
  wanted: 1
     got: 2
nil
user(5): (defun foo (x) x)
foo
user(6): (test 1 (foo 1))
t
user(7): (test 1 (foo 2))
 * * * UNEXPECTED TEST FAILURE * * *
Test failed: (foo 2)
  wanted: 1
     got: 2
nil
user(8): (setq *break-on-test-failures* t)
t
user(9): (test 1 (foo 2))
 * * * UNEXPECTED TEST FAILURE * * *
Test failed: (foo 2)
  wanted: 1
     got: 2
Break: *break-on-test-failures* is non-nil.

Restart actions (select using :continue):
 0: return from break.
 1: Return to Top Level (an "abort" restart)
 2: Abort #<process Initial Lisp Listener>
[1c] user(10): :pop
user(11): (setq *break-on-test-failures* nil)
nil
user(12): (test 1 (error "foo"))
Error: foo

Restart actions (select using :continue):
 0: Return to Top Level (an "abort" restart)
 1: Abort #<process Initial Lisp Listener>
[1] user(13): :pop
user(14): (setq *error-protect-tests* t)
t
user(15): (test 1 (error "foo"))
Condition type: simple-error
Message: foo
 * * * UNEXPECTED TEST FAILURE * * *
Test failed: (error "foo")
Reason: an error (of type `simple-error') was detected.
nil
user(16): (setq *error-protect-tests* nil)
nil
user(17): *test-errors*
4
user(18): *test-successes*
2
user(19): (test 1 2 :known-failure t)
Test failed: known failure: 2
  wanted: 1
     got: 2
nil
user(20): (test 1 (foo 1) :known-failure t)
Expected test failure for (foo 1) did not occur.
nil
user(21): (test 1 (foo 1) :known-failure t :fail-info "This is bug666.")
Expected test failure for (foo 1) did not occur.
Additional info: This is bug666.
nil
user(22): (test-error (error "foo"))
t
user(23): (test-no-error (error "foo"))
 * * * UNEXPECTED TEST FAILURE * * *
Test failed: (error "foo")
Reason: detected an unexpected error of type `simple-error'.
nil
user(24): (test-error (car '(10)))
 * * * UNEXPECTED TEST FAILURE * * *
Test failed: (car '(10))
Reason: expected but did not detect an error of type `condition'.
nil
user(25): (test-no-error (car '(10)))
t
user(26): (test-warning (warn "foo"))
t
user(27): (test-no-warning (warn "foo"))
 * * * UNEXPECTED TEST FAILURE * * *
Test failed: (warn "foo")
  wanted: no warning
     got: a warning
nil
user(28): (test-warning (car '(10)))
 * * * UNEXPECTED TEST FAILURE * * *
Test failed: (car '(10))
  wanted: a warning
     got: no warning
nil
user(29): (test-no-warning (car '(10)))
t
user(30): (test-error (error "foo: ~a" 10))
t
user(31): (test-error (error "foo: ~a" 10) :format-control "foo: ~a")
t
user(32): (test-error (error "foo: ~a" 10) :format-control "foo: ~a"
	    :format-arguments '(10))
t
user(33): (test-error (error "foo: ~a" 10) :format-control "foo:  ~a")
 * * * UNEXPECTED TEST FAILURE * * *
Test failed: (error "foo: ~a" 10)
Reason: the format-control was incorrect.
  wanted: "~1@<foo: ~a~:@>"
     got: "~1@<foo:  ~a~:@>"
nil
user(34): (test-error (error "foo: ~a" 10) :format-control "foo: ~a"
	    :format-arguments '(11))
 * * * UNEXPECTED TEST FAILURE * * *
Test failed: (error "foo: ~a" 10)
Reason: the format-arguments were incorrect.
  wanted: (10)
     got: (11)
nil
user(35): (test-error (error "foo: ~a" 10) :condition-type 'condition
	    :include-subtypes t)
t
user(36): (test-error (error "foo: ~a" 10) :condition-type 'simple-break
	    :include-subtypes t)
 * * * UNEXPECTED TEST FAILURE * * *
Test failed: (error "foo: ~a" 10)
Reason: detected an incorrect condition type.
  wanted: simple-break
     got: #<standard-class simple-error>
nil
user(37): (test-error (break "foo: ~a" 10) :condition-type 'simple-break
	    :include-subtypes t)
Break: foo: 10
  [condition type: simple-break]

Restart actions (select using :continue):
 0: return from break.
 1: Return to Top Level (an "abort" restart)
 2: Abort #<process Initial Lisp Listener>
[1c] user(38): :pop
user(39): (test-error (break "foo: ~a" 10) :catch-breaks t
		      :condition-type 'simple-break :include-subtypes t)
t

Copyright (c) 1998-2000, Franz Inc. Berkeley, CA., USA. All rights reserved. Created 2000.10.5.