OpenEuphoria

8.51 Unit Testing Framework

8.51.1 Background

Unit testing is the process of assuring that the smallest programming units are actually delivering functionality that complies with their specification. The units in question are usually individual routines rather than whole programs or applications.

The theory is that if the components of a system are working correctly, then there is a high probability that a system using those components can be made to work correctly.

In Euphoria terms, this framework provides the tools to make testing and reporting on functions and procedures easy and standardized. It gives us a simple way to write a test case and to report on the findings.
Example:

include std/unittest.e

test_equal( "Power function test #1", 4, power(2, 2))
test_equal( "Power function test #2", 4, power(16, 0.5))

test_report()

Name your test file in the special manner, t_NAME.e and then simply run eutest in that directory.

C:\Euphoria> eutest
t_math.e:
failed: Bad math, expected: 100 but got: 8
2 tests run, 1 passed, 1 failed, 50.0% success

===== Test failure summary:
FAIL: t_math.e

2 file(s) run 1 file(s) failed, 50.0% success--

In this example, we use the test_equal function to record the result of a test. The first parameter is the name of the test, which can be anything and is displayed if the test fails. The second parameter is the expected result -- what we expect the function being tested to return. The third parameter is the actual result returned by the function being tested. This is usually written as a call to the function itself.

It is typical to provide as many test cases as would be required to give us confidence that the function is being truly exercised. This includes calling it with typical values and edge-case or exceptional values. It is also useful to test the function's error handling by calling it with bad parameters.

When a test fails, the framework displays a message, showing the test's name, the expected result and the actual result. You can configure the framework to display each test run, regardless of whether it fails or not.

After running a series of tests, you can get a summary displayed by calling the test_report() procedure. To get a better feel for unit testing, have a look at the provided test cases for the standard library in the tests directory.

When included in your program, unittest.e sets a crash handler to log a crash as a failure.

8.51.2 Constants

8.51.2.1 TEST_QUIET

include std/unittest.e
namespace unittest
public enum TEST_QUIET

8.51.2.2 TEST_SHOW_FAILED_ONLY

include std/unittest.e
namespace unittest
public enum TEST_SHOW_FAILED_ONLY

8.51.2.3 TEST_SHOW_ALL

include std/unittest.e
namespace unittest
public enum TEST_SHOW_ALL

8.51.3 Setup Routines

8.51.3.1 set_test_verbosity

include std/unittest.e
namespace unittest
public procedure set_test_verbosity(atom verbosity)

Set the amount of information that is returned about passed and failed tests.

Parameters:

1. verbosity : an atom which takes predefined values for verbosity levels.

Comments:

The following values are allowable for verbosity:

• TEST_QUIET -- 0,
• TEST_SHOW_FAILED_ONLY -- 1
• TEST_SHOW_ALL -- 2

However, anything less than TEST_SHOW_FAILED_ONLY is treated as TEST_QUIET, and everything above TEST_SHOW_ALL is treated as TEST_SHOW_ALL.

• At the lowest verbosity level, only the score is shown, ie the ratio passed tests/total tests.
• At the medium level, in addition, failed tests display their name, the expected outcome and the outcome they got. This is the initial setting.
• At the highest level of verbosity, each test is reported as passed or failed.

If a file crashes when it should not, this event is reported no matter the verbosity level.

The command line switch "-failed" causes verbosity to be set to medium at startup. The command line switch "-all" causes verbosity to be set to high at startup.

See Also:

test_report

8.51.3.2 set_wait_on_summary

include std/unittest.e
namespace unittest
public procedure set_wait_on_summary(integer to_wait)

Request the test report to pause before exiting.

Parameters:

1. to_wait : an integer, zero not to wait, nonzero to wait.

Comments:

Depending on the environment, the test results may be invisible if set_wait_on_summary(1) was not called prior, as this is not the default. The command line switch "-wait" performs this call.

See Also:

test_report

8.51.3.3 set_accumulate_summary

include std/unittest.e
namespace unittest
public procedure set_accumulate_summary(integer accumulate)

Request the test report to save run stats in "unittest.dat" before exiting.

Parameters:

1. accumulate : an integer, zero not to accumulate, nonzero to accumulate.

Comments:

The file "unittest.dat" is appended to with {t,f}

8.51.3.4 set_test_abort

include std/unittest.e
namespace unittest
public function set_test_abort(integer abort_test)

Set behavior on test failure, and return previous value.

Parameters:

1. abort_test : an integer, the new value for this setting.

Returns:

An integer, the previous value for the setting.

Comments:

By default, the tests go on even if a file crashed.

8.51.4 Reporting

8.51.4.1 test_report

include std/unittest.e
namespace unittest
public procedure test_report()

Output test report

Comments:

The report components are described in the comments section for set_test_verbosity. Everything prints on the standard error device.

See Also:

set_test_verbosity

8.51.5 Tests

8.51.5.1 test_equal

include std/unittest.e
namespace unittest
public procedure test_equal(sequence name, object expected, object outcome)

Records whether a test passes by comparing two values.

Parameters:

1. name : a string, the name of the test
2. expected : an object, the expected outcome of some action
3. outcome : an object, some actual value that should equal the reference expected.

Comments:

• For floating point numbers, a fuzz of 1e-9 is used to assess equality.

A test is recorded as passed if equality holds between expected and outcome. The latter is typically a function call, or a variable that was set by some prior action.

While expected and outcome are processed symmetrically, they are not recorded symmetrically, so be careful to pass expected before outcome for better test failure reports.

See Also:

test_not_equal, test_true, test_false, test_pass, test_fail

8.51.5.2 test_not_equal

include std/unittest.e
namespace unittest
public procedure test_not_equal(sequence name, object a, object b)

Records whether a test passes by comparing two values.

Parameters:

1. name : a string, the name of the test
2. expected : an object, the expected outcome of some action
3. outcome : an object, some actual value that should equal the reference expected.

Comments:

• For atoms, a fuzz of 1e-9 is used to assess equality.
• For sequences, no such fuzz is implemented.

A test is recorded as passed if equality does not hold between expected and outcome. The latter is typically a function call, or a variable that was set by some prior action.

See Also:

test_equal, test_true, test_false, test_pass, test_fail

8.51.5.3 test_true

include std/unittest.e
namespace unittest
public procedure test_true(sequence name, object outcome)

Records whether a test passes.

Parameters:

1. name : a string, the name of the test
2. outcome : an object, some actual value that should not be zero.

Comments:

This assumes an expected value different from 0. No fuzz is applied when checking whether an atom is zero or not. Use test_equal() instead in this case.

See Also:

test_equal, test_not_equal, test_false, test_pass, test_fail

8.51.5.4 assert

include std/unittest.e
namespace unittest
public procedure assert(object name, object outcome)

Records whether a test passes. If it fails, the program also fails.

Parameters:

1. name : a string, the name of the test
2. outcome : an object, some actual value that should not be zero.

Comments:

This is identical to test_true() except that if the test fails, the program will also be forced to fail at this point.

See Also:

test_equal, test_not_equal, test_false, test_pass, test_fail

8.51.5.5 test_false

include std/unittest.e
namespace unittest
public procedure test_false(sequence name, object outcome)

Records whether a test passes by comparing two values.

Parameters:

1. name : a string, the name of the test
2. outcome : an object, some actual value that should be zero

Comments:

This assumes an expected value of 0. No fuzz is applied when checking whether an atom is zero or not. Use test_equal() instead in this case.

See Also:

test_equal, test_not_equal, test_true, test_pass, test_fail

8.51.5.6 test_fail

include std/unittest.e
namespace unittest
public procedure test_fail(sequence name)

Records that a test failed.

Parameters:

1. name : a string, the name of the test

See Also:

test_equal, test_not_equal, test_true, test_false, test_pass

8.51.5.7 test_pass

include std/unittest.e
namespace unittest
public procedure test_pass(sequence name)

Records that a test passed.

Parameters:

1. name : a string, the name of the test

See Also:

test_equal, test_not_equal,test_true, test_false, test_fail