OpenEuphoria

7.1 EuTEST - Unit Testing

7.1.1 Introduction

The testing system gives you the ability to check if the library, interpreter and translator works properly by use of unit tests. The unit tests are Euphoria include files that include unittest.e at the top, several test-routines for comparison between expected value and true value and at the end of the program a call to test_report(). There are error control files for when we expect the interpreter to fail but we want it to fail with a particular error message. You may use this section as an outline for testing your own code.

7.1.2 The eutest Program

7.1.2.1 Synopsis for running the tests

eutest [-D NO_INET ] [-D NO_INET_TESTS ]
    [-verbose] [-log] [-i include path] [-cc wat|gcc] [-exe interpreter]
    [-ec translator] [-lib binary library path]
    [optional list of unit test files]

7.1.2.2 Synopsis for creating report from the log

eutest -process-log [-html]

7.1.2.3 General behavior

If you want to test translation of your tests as well as interpreted tests, you can specify it with -ec.

If you don't specify unit tests on the command line eutest will scan the directory for unit test files using the pattern t_*.e. If you specify a pattern it will interpret the pattern, as some shells do not do this for programs.

7.1.2.4 Options detail

• -D REC: Is for creating control files, use only when on tests that work already on an interpreter that correctly works or correctly *fails* with them. This option must come before the eutest.ex program itself in the command line and is the option with that requirement.
• -log: Is for creating a log file for later processing
• -verbose: Is for eutest.ex to give you detail of what it is doing
• -i: is for specify the include path which will be passed to both the interpreter and the translator when interpreting and translating the test.
• -cc: is for specifying the compiler. This can be any one of -wat, djg, or gcc. Each of these represent the kind of compiler we will request the translator to use.
• -process-log: Is for processing a log created by a previous invocation of eutest.ex output is sent to standard output as a report of how the tests went. By default this is in ASCII format. Use -html to make it HTML format.
• -html: Is for making the report creation to be in HTML format
• -D NO_INET: This is for keeping tests from trying to use the Internet. The tests have to be written to support them by using ifdef/end ifdef statements. Since in some Euphoria unit tests "-D NO_INET_TESTS" is used in its place, you must use both options to prevent them from trying to connect through the Internet.
• -D NO_INET_TESTS: See NO_INET

7.1.3 The Unit Test Files

Unit test files must match a pattern t_*.e. If the unit test file matches t_c_*.e the test program will expect the program to fail, if there is an error control file in a directory with its same name and 'd' extension it will also expect it to fail according to the control file's contents found in the said directory. Such a failure is marked as a successful test run. However, if there is no such directory or file the counter test will be marked as a failed test run.

7.1.3.1 A trivial example

The following is a minimal unit test file:

include std/unittest.e

test_report()

Please see the Unit Testing Framework, for information on how to construct these files.

7.1.4 The Error Control Files

There are times when we expect something to fail. We want good EUPHORIA code to do the correct thing and there is a correct thing to do also for *bad* code. The interpreter must return with an error message of why it failed and the error must be correct and it must get written to ex.err. We must thus check the ex.err file to see if it has the correct error message.

If the unit test is t_foo.e then the location for its control file can be in the following locations:

• t_foo.d/interpreter/OSNAME/control.err
• t_foo.d/OSNAME/control.err
• t_foo.d/control.err

The OSNAME is the name of the operating system. Which is either UNIX or Win32.

Now, if t_foo.d/Win32/control.err exists, then the testing program eutest.ex expects t_foo.e to fail when run with the WINDOWS interpreter. However, this is not necessarily true for other platforms. In WINDOWS, eutest runs it, watches it fail, then compares the ex.err file to t_foo.d/Win32/control.err. If they ex.err is different from control.err an error message is written to the log. Now on, say NetBSD, t_file.e is tested with the expectation it will return 0 and the tests will all pass unless t_foo.d/UNIX/control.err or t_foo.d/control.err also exist. Thus you can have different expectations for differing platforms. Some feature that is not possible to implement under WINDOWS can be put into a unit test and the resulting ex.err file can be put into a control file for WINDOWS. This means we do not need to have all of these errors that we expect to get drawing our attention away from errors that need our attention. On the other hand, if an unexpected error message not like t_foo.d/Win32/control.err gets generated in the Windows case then eutest will tell us that.

How do we construct these control files? You don't really need to, you can take an ex.err file that results from running a stable interpreter on a test and rename it and move it to the appropriate place.

7.1.5 Test Coverage

When writing and evaluating the results of unit tests, it is important to understand which parts of your code are and are not being tested. The Euphoria interpreter has a built in capability to instrument your code to analyze how many times each line of your code is executed during your suite of tests. The data is output into an EDS database. Euphoria also comes with a coverage data post-processor that generates html reports to make analysis of your coverage easy.

The coverage capabilities can be used manually, with arguments supplied on the command line, or passed to eutest. Indeed, eutest simply passes these along to the interpreter. The Euphoria suite of unit tests can be run via the makefiles, and there is a special target to run a coverage analysis of the standard library:

Windows:
> wmake coverage

Unix:
$ make coverage

Then, in your build directory, eutest will run the tests to create the coverage database unit-test.edb, and will post-process the results, placing the HTML reports into a unit-test subdirectory from your build directory.

7.1.5.1 Coverage Command Line Switches

• -coverage [file|dir] This specifies what files to gather stats for. If you supply a directory, it recurses on child directories. Only files that are obviously Euphoria are included ( .e, .ew, .eu, .ex, .exw, .exu).

• -coverage-db <file> This one allows you to specify a specific location and name for the database where coverage information is stored. It's an EDS database. By default, the DB is eui-cvg.edb.

• -coverage-erase Tells the interpreter to start over. By default, multiple runs accumulate coverage in the DB to allow coverage analysis based on a suite of unit test files.

• -coverage-exclude <pattern> Specifies a regular expression that is used to exclude files from coverage analysis.

• -coverage-pp <post-processor> Supported by eutest only (i.e., not the interpreter itself). Tells eutest how to post process the coverage data. <post-processor> must be the path to a the post processing application. After running the suite of tests, eutest will execute this program with the path to the coverage db as an argument.

7.1.5.2 Coverage Post-Processing

Once you have run tests to generate a coverage database, the data is not easily viewed. Euphoria comes with a post-processor called eucoverage.ex, which is installed in the bin directory. On a Unix packaged install, you should be able to simply use eucoverage, which is configured to run eucoverage.ex.

The post-processor generates an index page, with coverage stats for each file, and individual html files, linked from the index page, for each file analyzed for test coverage. At the file level, statistics are presented for total and executed routines and lines of code. The files are sorted in descending order of lines that were never executed, in order to highlight the parts of your code that are less tested. The page for each file shows this information, as well as a similar breakdown by routine, displaying the number of lines in each routine that was executed. The routines are also sorted in descending order by the most unexecuted lines.

Additionally, the source of the file is displayed below the statistics. The routines are linked to their place in the code. Each line is colored either green red or white. White lines are those that are not executed. These are typically blank, comments, declarations or "end" clauses of code blocks that do not create any executable code. Red lines are those that were never executed, and lines that were executed are colored green. The line number is displayed in the left margin, and the number of times each line was executed is displayed just to the left of where the source code begins.

Command Line Switches

• -o <dir> Specify the output directory. The default is to create a subdirectory, from the same directory as the coverage database, with the name of the base filename (without extension) of the coverage database.

• -v Verbose output