Common Lisp Package: IT.BESE.FIVEAM

README:

FUNCTION

Public

!

Rerun the most recently run test and explain the results.

!!

Rerun the second most recently run test and explain the results.

!!!

Rerun the third most recently run test and explain the results.

DEBUG! (&OPTIONAL (TEST-SPEC *SUITE*))

Calls (run! test-spec) but enters the debugger if any kind of error happens.

EXPLAIN! (RESULT-LIST)

Explain the results of RESULT-LIST using a detailed-text-explainer with output going to *test-dribble*

GEN-CHARACTER (&KEY (CODE-LIMIT CHAR-CODE-LIMIT) (CODE (GEN-INTEGER MIN 0 MAX (1- CODE-LIMIT))) (ALPHANUMERICP NIL))

Returns a generator of characters. CODE must be a generator of random integers. ALPHANUMERICP, if non-NIL, limits the returned chars to those which pass alphanumericp.

GEN-FLOAT (&KEY BOUND (TYPE 'SHORT-FLOAT))

Returns a generator which producs floats of type TYPE. BOUND, if specified, constrains the ruselts to be in the range (-BOUND, BOUND).

GEN-INTEGER (&KEY (MAX (1+ MOST-POSITIVE-FIXNUM)) (MIN (1- MOST-NEGATIVE-FIXNUM)))

Returns a generator which produces random integers greater than or equal to MIN and less than or equal to MIN.

GEN-LIST (&KEY (LENGTH (GEN-INTEGER MIN 0 MAX 10)) (ELEMENTS (GEN-INTEGER MIN -10 MAX 10)))

Returns a generator which producs random lists. LENGTH must be an integer generator and ELEMENTS must be a generator which producs objects.

GEN-STRING (&KEY (LENGTH (GEN-INTEGER MIN 0 MAX 80)) (ELEMENTS (GEN-CHARACTER)) (ELEMENT-TYPE 'CHARACTER))

Returns a generator which producs random strings. LENGTH must be a generator which producs integers, ELEMENTS must be a generator which produces characters of type ELEMENT-TYPE.

GEN-TREE (&KEY (SIZE 20) (ELEMENTS (GEN-INTEGER MIN -10 MAX 10)))

Returns a generator which producs random trees. SIZE control the approximate size of the tree, but don't try anything above 30, you have been warned. ELEMENTS must be a generator which will produce the elements.

MAKE-SUITE (NAME &KEY DESCRIPTION IN)

Create a new test suite object. Overides any existing suite named NAME.

RESULTS-STATUS (RESULT-LIST)

Given a list of test results (generated while running a test) return true if all of the results are of type TEST-PASSED, faile otherwise.

RUN (TEST-SPEC)

Run the test specified by TEST-SPEC. TEST-SPEC can be either a symbol naming a test or test suite, or a testable-object object. This function changes the operations performed by the !, !! and !!! functions.

RUN! (&OPTIONAL (TEST-SPEC *SUITE*))

Equivalent to (explain (run TEST-SPEC)).

Undocumented

GEN-BUFFER (&KEY (LENGTH (GEN-INTEGER MIN 0 MAX 50)) (ELEMENT-TYPE '(UNSIGNED-BYTE 8)) (ELEMENTS (GEN-INTEGER MIN 0 MAX (1- (EXPT 2 8)))))

GEN-ONE-ELEMENT (&REST ELEMENTS)

GET-FIXTURE (KEY &OPTIONAL DEFAULT)

SETFGET-FIXTURE (VALUE KEY)

GET-TEST (KEY &OPTIONAL DEFAULT)

SETFGET-TEST (VALUE KEY)

REM-FIXTURE (KEY)

REM-TEST (KEY)

TEST-NAMES

Private

ADD-RESULT (RESULT-TYPE &REST MAKE-INSTANCE-ARGS)

Create a TEST-RESULT object of type RESULT-TYPE passing it the initialize args MAKE-INSTANCE-ARGS and adds the resulting object to the list of test results.

MAKE-COLLECTOR (&OPTIONAL INITIAL-VALUE)

Create a collector function. A Collector function will collect, into a list, all the values passed to it in the order in which they were passed. If the callector function is called without arguments it returns the current list of values.

RETURN-RESULT-LIST (TEST-LAMBDA)

Run the test function TEST-LAMBDA and return a list of all test results generated, does not modify the special environment variable RESULT-LIST.

Undocumented

BINDING (X BINDS)

CHECK-REQUIRED (NAME VARS REQUIRED)

IMPORT-TESTING-SYMBOLS (PACKAGE-DESIGNATOR)

LIST-MATCH (X Y &OPTIONAL BINDS)

PARTITION-RESULTS (RESULTS-LIST)

PARTITIONX (LIST &REST LAMBDAS)

PERFORM-RANDOM-TESTING (GENERATORS BODY)

PERFORM-RANDOM-TESTING/RUN-ONCE (GENERATORS BODY)

VARS (MATCH-SPEC)

VARSYMP (X)

MACRO

Public

DEF-FIXTURE (NAME ARGS &BODY BODY)

Defines a fixture named NAME. A fixture is very much like a macro but is used only for simple templating. A fixture created with DEF-FIXTURE is a macro which can use the special macrolet &BODY to specify where the body should go. See Also: WITH-FIXTURE

DEF-SUITE (NAME &KEY DESCRIPTION IN)

Define a new test-suite named NAME. IN (a symbol), if provided, causes this suite te be nested in the suite named by IN. NB: This macro is built on top of make-suite, as such it, like make-suite, will overrwrite any existing suite named NAME.

FAIL (&REST MESSAGE-ARGS)

Simply generate a FAIL.

FINISHES (&BODY BODY)

Generates a pass if BODY executes to normal completion. In other words if body does signal, return-from or throw this test fails.

FOR-ALL (BINDINGS &BODY BODY)

Bind BINDINGS to random variables and test BODY *num-trials* times. BINDINGS is a list of binding forms, each element is a list of (BINDING VALUE &optional GUARD). Value, which is evaluated once when the for-all is evaluated, must return a generator which be called each time BODY is evaluated. BINDING is either a symbol or a list which will be passed to destructuring-bind. GUARD is a form which, if present, stops BODY from executing when IT returns NIL. The GUARDS are evaluated after all the random data has been generated and they can refer to the current value of any binding. NB: Generator forms, unlike guard forms, can not contain references to the boud variables. Examples: (for-all ((a (gen-integer))) (is (integerp a))) (for-all ((a (gen-integer) (plusp a))) (is (integerp a)) (is (plusp a))) (for-all ((less (gen-integer)) (more (gen-integer) (< less more))) (is (<= less more))) (for-all (((a b) (gen-two-integers))) (is (integerp a)) (is (integerp b)))

IN-SUITE (SUITE-NAME)

Set the *suite* special variable so that all tests defined after the execution of this form are, unless specified otherwise, in the test-suite named SUITE-NAME. See also: DEF-SUITE *SUITE*

IN-SUITE* (SUITE-NAME &KEY IN)

Just like in-suite, but silently creates missing suites.

IS (TEST &REST REASON-ARGS)

The DWIM checking operator. If TEST returns a true value a test-passed result is generated, otherwise a test-failure result is generated. The reason, unless REASON-ARGS is provided, is generated based on the form of TEST: (predicate expected actual) - Means that we want to check whether, according to PREDICATE, the ACTUAL value is in fact what we EXPECTED. (predicate value) - Means that we want to ensure that VALUE satisfies PREDICATE. Wrapping the TEST form in a NOT simply preducse a negated reason string.

IS-EVERY (PREDICATE &BODY CLAUSES)

The input is either a list of lists, or a list of pairs. Generates (is (,predicate ,expr ,value)) for each pair of elements or (is (,predicate ,expr ,value) ,@reason) for each list.

IS-FALSE (CONDITION &REST REASON-ARGS)

Generates a pass if CONDITION returns false, generates a failure otherwise. Like IS-TRUE, and unlike IS, IS-FALSE does not inspect CONDITION to determine what reason to give it case of test failure

IS-TRUE (CONDITION &REST REASON-ARGS)

Like IS this check generates a pass if CONDITION returns true and a failure if CONDITION returns false. Unlike IS this check does not inspect CONDITION to determine how to report the failure.

PASS (&REST MESSAGE-ARGS)

Simply generate a PASS.

SIGNALS (CONDITION-SPEC &BODY BODY)

Generates a pass if BODY signals a condition of type CONDITION. BODY is evaluated in a block named NIL, CONDITION is not evaluated.

SKIP (&REST REASON)

Generates a TEST-SKIPPED result.

TEST (NAME &BODY BODY)

Create a test named NAME. If NAME is a list it must be of the form: (name &key depends-on suite fixture compile-at profile) NAME is the symbol which names the test. DEPENDS-ON is a list of the form: (AND . test-names) - This test is run only if all of the tests in TEST-NAMES have passed, otherwise a single test-skipped result is generated. (OR . test-names) - If any of TEST-NAMES has passed this test is run, otherwise a test-skipped result is generated. (NOT test-name) - This is test is run only if TEST-NAME failed. AND, OR and NOT can be combined to produce complex dependencies. If DEPENDS-ON is a symbol it is interpreted as `(AND ,depends-on), this is accomadate the common case of one test depending on another. FIXTURE specifies a fixtrue to wrap the body in. If PROFILE is T profiling information will be collected as well.

WITH-FIXTURE (FIXTURE-NAME ARGS &BODY BODY)

Insert BODY into the fixture named FIXTURE-NAME. See Also: DEF-FIXTURE

Private

AIF (TEST THEN &OPTIONAL ELSE)

Just like IF-BIND but the var is always IT.

DEF-SPECIAL-ENVIRONMENT (NAME (&KEY ACCESSOR BINDER BINDER*) &REST VARS)

Define two macros for dealing with groups or related special variables. ACCESSOR is defined as a macro: (defmacro ACCESSOR (VARS &rest BODY)). Each element of VARS will be bound to the current (dynamic) value of the special variable. BINDER is defined as a macro for introducing (and binding new) special variables. It is basically a readable LET form with the prorpe declarations appended to the body. The first argument to BINDER must be a form suitable as the first argument to LET. ACCESSOR defaults to a new symbol in the same package as NAME which is the concatenation of "WITH-" NAME. BINDER is built as "BIND-" and BINDER* is BINDER "*".

DOLIST* ((ITERATOR LIST &OPTIONAL RETURN-VALUE) &BODY BODY)

Like DOLIST but destructuring-binds the elements of LIST. If ITERATOR is a symbol then dolist* is just like dolist EXCEPT that it creates a fresh binding.

IF-BIND (VAR TEST &BODY THEN/ELSE)

Anaphoric IF control structure. VAR (a symbol) will be bound to the primary value of TEST. If TEST returns a true value then THEN will be executed, otherwise ELSE will be executed.

Undocumented

%IN-SUITE (SUITE-NAME &KEY (FAIL-ON-ERROR T) IN)

ACOND2 (&REST CLAUSES)

BIND-RUN-STATE (REQUESTED-VARS &BODY BODY)

BIND-RUN-STATE* (REQUESTED-VARS &BODY BODY)

DEF-SUITE* (NAME &REST DEF-SUITE-ARGS)

LIST-MATCH-CASE (TARGET &BODY CLAUSES)

PROCESS-FAILURE (&REST ARGS)

WITH-*TEST-DRIBBLE* (STREAM &BODY BODY)

WITH-RUN-STATE (REQUESTED-VARS &BODY BODY)

GENERIC-FUNCTION

Public

EXPLAIN (EXPLAINER RESULTS &OPTIONAL STREAM RECURSIVE-DEPTH)

Given a list of test results report write to stream detailed human readable statistics regarding the results.

Private

%RUN (TEST-SPEC)

Internal method for running a test. Does not update the status of the tests nor the special variables !, !!, !!!

RUN-RESOLVING-DEPENDENCIES (TEST)

Given a dependency spec determine if the spec is satisfied or not, this will generally involve running other tests. If the dependency spec can be satisfied the test is also run.

Undocumented

FOR-ALL-TEST-FAILED-P (OBJECT)

RESOLVE-DEPENDENCIES (DEPENDS-ON)

RUN-TEST-LAMBDA (TEST)

TEST-FAILURE-P (OBJECT)

TEST-PASSED-P (OBJECT)

TEST-SKIPPED-P (OBJECT)

SLOT-ACCESSOR

Private

COLLECT-PROFILING-INFO (OBJECT)

When T profiling information will be collected when the test is run.

SETFCOLLECT-PROFILING-INFO (NEW-VALUE OBJECT)

When T profiling information will be collected when the test is run.

DEPENDS-ON (OBJECT)

The list of AND, OR, NOT forms specifying when to run this test.

SETFDEPENDS-ON (NEW-VALUE OBJECT)

The list of AND, OR, NOT forms specifying when to run this test.

DESCRIPTION (OBJECT)

The textual description of this test object.

SETFDESCRIPTION (NEW-VALUE OBJECT)

The textual description of this test object.

NAME (OBJECT)

A symbol naming this test object.

SETFNAME (NEW-VALUE OBJECT)

A symbol naming this test object.

PROFILING-INFO (OBJECT)

An object representing how much time and memory where used by the test.

SETFPROFILING-INFO (NEW-VALUE OBJECT)

An object representing how much time and memory where used by the test.

RUNTIME-PACKAGE (OBJECT)

By default it stores *package* from the time this test was defined (macroexpanded).

SETFRUNTIME-PACKAGE (NEW-VALUE OBJECT)

By default it stores *package* from the time this test was defined (macroexpanded).

STATUS (OBJECT)

A symbol specifying the current status of this test. Either: T - this test (and all its dependencies, have passed. NIL - this test failed (either it failed or its dependecies weren't met. :circular this test has a circular dependency and was skipped. Or :depends-not-satisfied or :resolving

SETFSTATUS (NEW-VALUE OBJECT)

A symbol specifying the current status of this test. Either: T - this test (and all its dependencies, have passed. NIL - this test failed (either it failed or its dependecies weren't met. :circular this test has a circular dependency and was skipped. Or :depends-not-satisfied or :resolving

TEST-LAMBDA (OBJECT)

The function to run.

SETFTEST-LAMBDA (NEW-VALUE OBJECT)

The function to run.

TESTS (OBJECT)

The hash table mapping names to test objects in this suite. The values in this hash table can be either test-cases or other test-suites.

SETFTESTS (NEW-VALUE OBJECT)

The hash table mapping names to test objects in this suite. The values in this hash table can be either test-cases or other test-suites.

Undocumented

ACTUAL-CONDITION (OBJECT)

SETFACTUAL-CONDITION (NEW-VALUE OBJECT)

REASON (OBJECT)

SETFREASON (NEW-VALUE OBJECT)

TEST-CASE (OBJECT)

SETFTEST-CASE (NEW-VALUE OBJECT)

TEST-EXPR (OBJECT)

SETFTEST-EXPR (NEW-VALUE OBJECT)

VARIABLE

Public

*DEBUG-ON-ERROR*

T if we should drop into a debugger on error, NIL otherwise.

*DEBUG-ON-FAILURE*

T if we should drop into a debugger on a failing check, NIL otherwise.

*RUN-TEST-WHEN-DEFINED*

When non-NIL tests are run as soon as they are defined.

*VERBOSE-FAILURES*

T if we should print the expression failing, NIL otherwise.

Undocumented

*TEST-DRIBBLE*

Private

*FIXTURE*

Lookup table mapping fixture names to fixture objects.

*MAX-TRIALS*

Number of total times we attempt to run the body of the FOR-ALL test including when the body is skipped due to failed guard conditions. Since we have guard conditions we may get into infinite loops where the test code is never run due to the guards never returning true. This second run limit prevents that.

*NUM-TRIALS*

Number of times we attempt to run the body of the FOR-ALL test.

*RUN-QUEUE*

List of test waiting to be run.

*SUITE*

The current test suite object

*TEST*

Lookup table mapping test (and test suite) names to objects.

Undocumented

*!!!*

*!!*

*!*

*INITIAL-!*

CLASS

Private

TEST-CASE (OBJECT)

A test case is a single, named, collection of checks. A test case is the smallest organizational element which can be run individually. Every test case has a name, which is a symbol, a description and a test lambda. The test lambda is a regular funcall'able function which should use the various checking macros to collect results. Every test case is part of a suite, when a suite is not explicitly specified (either via the :SUITE parameter to the TEST macro or the global variable *SUITE*) the test is inserted into the global suite named NIL. Sometimes we want to run a certain test only if another test has passed. FiveAM allows us to specify the ways in which one test is dependent on another. - AND Run this test only if all the named tests passed. - OR Run this test if at least one of the named tests passed. - NOT Run this test only if another test has failed. FiveAM considers a test to have passed if all the checks executed were successful, otherwise we consider the test a failure. When a test is not run due to it's dependencies having failed a test-skipped result is added to the results.

TEST-FAILURE

Class for unsuccessful checks.

TEST-PASSED

Class for successful checks.

TEST-RESULT

All checking macros will generate an object of type TEST-RESULT.

TEST-SKIPPED

A test which was not run. Usually this is due to unsatisfied dependencies, but users can decide to skip test when appropiate.

TEST-SUITE

A test suite is a collection of tests or test suites. Test suites serve to organize tests into groups so that the developer can chose to run some tests and not just one or all. Like tests test suites have a name and a description. Test suites, like tests, can be part of other test suites, this allows the developer to create a hierarchy of tests where sub trees can be singularly run. Running a test suite has the effect of running every test (or suite) in the suite.

UNEXPECTED-TEST-FAILURE

Represents the result of a test which neither passed nor failed, but signaled an error we couldn't deal with. Note: This is very different than a SIGNALS check which instead creates a TEST-PASSED or TEST-FAILURE object.

Undocumented

DETAILED-TEXT-EXPLAINER

EXPLAINER

FOR-ALL-TEST-FAILED

FOR-ALL-TEST-NEVER-RUN

FOR-ALL-TEST-NO-TESTS

FOR-ALL-TEST-PASSED

FOR-ALL-TEST-RESULT

SIMPLE-TEXT-EXPLAINER

TESTABLE-OBJECT

TEXT-EXPLAINER

CONDITION

Private

CHECK-FAILURE

Signaled when a check fails.

CIRCULAR-DEPENDENCY

Condition signaled when a circular dependency between test-cases has been detected.