itl

itl

Integration Test Library

An experiment in writing executable documentation in Clojure.

Project Test Results

This project is self-testing. We run this README file as part of every build.

• Markdown Test Results
  

Prerequisites

I have tested the entire build process using JDK 11

This project’s version of Clojure requires at least Java 8.

You’ll also need to install Leiningen

Building the project

The easiest thing to do is run ./prepare-release to run all the project tests, document generation, and linting. This is the best way to make sure your changes work fine.

Other things you can do:

• lein test runs the itl tests printing full output
• In a REPL you can run one or all itl tests using itl.core/run. There is sample code at the bottom of itl.core

Getting

You can download the source code for itl from its Project Page

Quick Start

Here’s the simplest set of steps you can follow to get going with an itl project (note: all this code can be found in this project and is kept in sync as part of its test suite; see below):

• After satisfying the above Prerequisites, create a new project using lein new quick-start
• Within the quick-start folder, there is now a project.clj file. Edit it and add the dependency for itl. You can get the syntax by clicking on the clojars badge above
• Create a file called simple-example.md
• On the first two lines, enter the following lines:

bind::greeting[Hello World]
should::greeting[Hello Jane]

• Execute: lein run -m itl.cli simple-example.md
• Notice something like the following output:

simple-example.md
| :pass | :fail | :exception | :elapsed-time |        :run-date |
|-------+-------+------------+---------------+------------------|
|     0 |     1 |          0 |            78 | 2018-10-01 09:00 |

• Upon executing ls or dir you’ll notice a new file called test.html in the current directory. Open it in a browser and notice the output showing that :greeting = Hello, World :greeting should be 'Hello, Jane', and was 'Hello, World'

Binding your code

That’s not very satisfying. A bit more is necessary to bind itl to your code. We’ll define a simple addition function and test it:

• Back in simple-example.md add a new statement that will cause itl to load your code, and execute it:

exec::use[ns=quick-start.core]

Given two numbers:
bind::n1[5]

and

bind::n2[6]

after we

op::[add two numbers]

should::result[11]

• In the file src/quick_start/core.clj you can make the file the following:

(ns quick-start.core
  (:require [itl.core :refer :all]))

(defn add-nums [n1 n2] (+ n1 n2))

(defop "add two numbers" [{:keys [n1 n2] :as page-state}]
  (let [n1 (Integer/parseInt n1)
        n2 (Integer/parseInt n2)]
    (assoc page-state :result (str (add-nums n1 n2)))))

• After you execute: lein run -m itl.cli simple-example.md, you’ll notice the following result (-ish):

simple-example.md
| :pass | :fail | :exception | :elapsed-time |        :run-date |
|-------+-------+------------+---------------+------------------|
|     1 |     1 |          0 |            92 | 2018-10-01 09:30 |

• And in test.html:

use: {"ns" "quick-start.core"}

Given two numbers: :n1 = 5 and :n2 = 6, after we add two numbers, :result should be '11', and was '11'

• That’s it! You’ve successfully: ** Bound two variables to the global state, ** Executed an operation on those variables which translates data to/fro Strings and binds the result back to the global state, and ** Asserted that the right value is present therein.

Tables full of assertions

But still, that is not a very pretty setup! Even if you created a table structure that holds all the data you might want to check, why would you want to type all those assertions over and over again?

The answer is table tests. Much like a defop, you can use a deftfn or deftafn to define a table operation, or a table operation that takes arguments. Here’s an example you can paste into your simple-example.md:

| n1 | n2 | result |
|:---|:---|:-------|
| 2  | 3  | 5      |
| 3  | 5  | 8      |
| 5  | 8  | 13     |
| 8  | 13 | 99     |
table::[add numbers]

If you add the following function, you’ll see what the parsed internal structure the table looks like:

(deftfn "add numbers" [page-state table-data]
  (clojure.pprint/pprint table-data))

You should see the following on the console:

$ lein run -m itl.cli simple-example.md
{:labels ("n1" "n2" "result"),
 :rows
 ({"n1" "2", "n2" "3", "result" "5"}
  {"n1" "3", "n2" "5", "result" "8"}
  {"n1" "5", "n2" "8", "result" "13"}
  {"n1" "8", "n2" "13", "result" "99"})}

simple-example.md
| :pass | :fail | :exception | :elapsed-time |        :run-date |
|-------+-------+------------+---------------+------------------|
|     0 |     0 |          0 |           176 | 2018-10-03 09:36 |

Now, let’s get some actual assertions to happen!

(defn- process-row [{:strs [n1 n2 result]}]
  (let [n1 (Integer/parseInt n1)
        n2 (Integer/parseInt n2)
        expected (Integer/parseInt result)
        actual (add-nums n1 n2)]
    (if (= expected actual)
      {"n1" (str n1)
       "n2" (str n2)
        "result" (pass (str actual))}
      {"n1" (str n1)
       "n2" (str n2)
       "result" (fail (str "Expected " expected " but got " actual))})))

(defn- process-add-table [{:keys [rows]}]
  {:rows (map process-row rows)})

(deftfn "add numbers" [page-state table-data]
  [page-state (process-add-table table-data)])

Boy was that a lot of code! Thankfully we have a function we can use to simplify it quite a lot. So instead of what you see above, you can use this!

(defn- calc-value [{:keys [n1 n2] :as page-state}]
  (assoc page-state :result
    (str (add-nums (Integer/parseInt n1)
                   (Integer/parseInt n2)))))

(deftfn "add numbers" [page-state table-info]
  (column-table page-state
                table-info
                {:assign {"n1" :n1, "n2" :n2}
                :exec calc-value
                :asserts {"result" :result}}))

That’s it! What’s happening here is:

• itl invokes the “add numbers” operation for the table
• the operation then calls column-table on the data, telling it that:
  • For each value "n1" assign it to a key :n1 in the current state, and similarly for "n2"
  • Then, execute the function calc-value on the new state, and
  • Finally, look in the new state for a value called :result and compare it against whatever is in the table cell with the label "result".

Just as with before, we need to have string values coming into the function and out of it. itl is very sensitive!

Advanced Usage

Also see our API Documentation. Within that documentation you can find all the code used to run the example below, as well as all the globally accessible fixtures. Those are found in itl.core.

Here’s an example of how to use the library: lein cli README.md, which should execute this document. If you use lein cli -- -h you’ll see some options you can use.

??? abstract “global:ITL Test Suite”

The below table of tests specifies all the functionality of ITL.

File	Result	Pass	Fail	Exception
simple-example.md	output	7	3	0
complete-example.md	output	13	2	4

table::[execute example files, indir=tests, outdir=docs/md, logfile=md-output.log, parallel=true]

Check out the itl.example namespace as well as bin/test-all for how the documentation for this project was executed.

Making it Pretty

You’ll notice this file is pretty nice looking! That’s because we used a nice simple CSS file. The default is in resources/itl-md.css You can specify your own CSS by passing the --css flag to itl.cli.

Tagging

You can tag a section of a document. Then, if you run itl.cli with one or more -t or --tag switches, followed by the name of the tag you want to run, only the sections with the given tags, plus the global tag, will be executed. You can see an example of how this is done in https://git.calmabiding.me/scstarkey/itl/raw/branch/master/tests/complete-example.md and the results of executing the file in https://docs.calmabiding.me/itl/md/complete-example.html

Parallel Column Tables

As of 0.2.8 we have had support for executing each row in a column table in parallel with another. All you need to do is add the arguments :parallel? true to the end of your column-table execution.

The downside: each row that’s executed does not pass its result to the next row. In fact, the entire table will have no effect whatsoever on the overall page-state. That means you don’t get the ability to assign variables (using, for example, uuid>val) and then use those variables in future tables. So, if you are intending to use tables to impact page-state, don’t make your table parallel.

This should be fine -- you can create a single table to assign variables and then another one to use them in parallel for side effects. This is probably preferable to mixing it all anyway.

Deprecated features

• As of 0.4.0, Generative tables (gentable and generative-table) are deprecated.
• As of 0.3.3, AsciiDoc support is deprecated. This means no new features will be added to the AsciiDoc portion of the library, and at some future release it will be purged altogether.

Breaking changes

Upgrading from 0.3.* to 0.4.*

1. The 4-argument run function in itl.core has been removed. Use one of the other options.
2. The CSS_FILE environment variable has been replaced with the --css switch in itl.cli -- you can use itl.markdown/with-css on the REPL instead of using the environment variable.

Upgrading from 0.2.* to 0.3.*

The syntax for '<x' variable interpolation has been removed completely. Use {{x}} instead, and don’t worry about manually resolving it using generated-var. column-table does the resolution for you.

Upgrading From 0.1.* to 0.2.*

1. itl used to be quiet. It didn’t print any status information. Now it does. A lot. If you want it to be quiet again, pass the --quiet or -q switch to itl.cli
2. The execute example files table used to take an outdir, which was where it both looked for the example md files to parse as well as where it writes its file called example-output.log. Now, you would use an indir parameter to specify here md files are found, an outdir parameter to specify where the html files are written, and a logfile parameter for where the output from running the files should go. If no logfile is given, output will go to standard out. You must specify an indir and outdir

Contributing

If you have a contribution to make, you are welcome to! Send an email to stephen@calmabiding.me and we can have a conversation about how you like to contribute!

License

Copyright © 2019 Stephen Starkey

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.