Exam BDD Framework

Overview

Exam is oriented on declarative end-to-end gray-box application testing in a way a manual tester would do it: send request, verify response/database/message queue etc.

Figure 1. Manual testing

Hence, the Exam functionality falls apart in different libraries (plugins) that are tailored for specific kinds of checks: database interactions, message queue interactions, http interactions, file system interactions etc. and may be used separately.

Figure 2. Exam testing

Each library consist of a plugin class that should be configured and attached to ExamExtension and set of commands that can be used for instrumenting .adoc specification files.

Exam could be enabled by implementing io.github.adven27.concordion.extensions.exam.core.AbstractSpecs class:

class Specs : AbstractSpecs() {

    override fun init() = ExamExtension(
        WsPlugin(
            host = "localhost",
            port = 8080
        ),
        DbPlugin(
            driver = "org.postgresql.Driver",
            url = "jdbc:postgresql://localhost:5432/postgres",
            user = "postgres",
            password = "postgres"
        )
    )

    /*  Methods in order of execution: */

    override fun beforeSetUp() {
        // Optional: Run some actions BEFORE Exam set up
    }
    override fun beforeSutStart() {
        // Optional: Run some actions AFTER Exam set up and BEFORE start of a System Under Test (SUT)
    }
    override fun startSut() {
        // Start SUT before specs suite if needed
    }
    override fun stopSut() {
        // Stop SUT after specs suite if needed
    }
    override fun afterSutStop() {
        // Optional: Run some actions AFTER stop of a SUT and BEFORE Exam tear down
    }
    override fun afterTearDown() {
        // Optional: Run some actions AFTER Exam tear down
    }
}

Core

Core commands enable abstract interactions and verifying capabilities.

Commands

e-eq
e-set
e-execute
e-verify-rows
e-run
e-echo
example command
Handlebar support

e-eq

Verifies an equality of instrumented block and instrumentation expression result.

Attributes

`verifier`	Optional. Default: `text`. Content type verifier. Built-in values: `text`, `json`, `xml`.
`awaitAtMostSec`	Optional. Default: 4. How long to wait in seconds before failing the verification
`awaitPollDelayMillis`	Optional. Default: 0. How long to wait in millis before starting verification
`awaitPollIntervalMillis`	Optional. Default: 1000. How long to wait in millis between verification attempts

Waiting is disabled by default. Check will fail immediately in case of mismatch unless at least one of the awaitAtMostSec, awaitPollDelayMillis or awaitPollIntervalMillis attributes are set.

Usage

Given `greeting()` method returns `Hello World!`
expected result is: [e-eq=greeting()]_Hello World!_

Given greeting() method returns Hello World! expected result is: Hello World!

JSON verifying

Given `someJson()` method returns `{"result": 1}`
expected result is: [e-verifier=json e-eq=someJson()]_{"result": "{{number}}" }_. +
Short version: [e-eq-json=someJson()]_{"result": "{{number}}"}_. +
On listing:
[source,json,e-eq-json=someJson()]
--
{"result": "{{number}}"}
--

Given someJson() method returns {"result": 1} expected result is: {"result": "${json-unit.any-number}" }.
Short version: {"result": "${json-unit.any-number}"}.
On listing:

{"result": "${json-unit.any-number}"}

XML verifying

Given `someXml()` method returns `<result>1</result>`
expected result is: [e-verifier=xml e-eq=someXml()]_<result>{{number}}</result>_. +
Short version: [e-eq-xml=someXml()]_<result>{{number}}</result>_. +
On listing:
[source,xml,e-eq-xml=someXml()]
--
<result>{{number}}</result>
--

Given someXml() method returns <result>1</result> expected result is: <result>${xml-unit.any-number}</result>.
Short version: <result>${xml-unit.any-number}</result>.
On listing:

<result>${xml-unit.any-number}</result>

e-set

Sets a specification variable which could be used in other commands expressions.

Usage

The greeting for user [e-set=firstName]_Bob_ will be: [e-eq=greetingFor(#firstName)]_Hello Bob!_

The greeting for user Bob will be: Hello Bob!

Variable can be initialized with text, object and file content.

Initialization with string from template:

[.when]
Set `someText` =  [e-set=someText]`some text`
and set `template` = [e-set=template]`someText: {{someText}}, current timestamp: {{iso (at)}}`
[.then]
`template` equals [e-eq=#template]`someText: {{someText}}, current timestamp: {{iso (at)}}`

Set someText = some text and set template = someText: some text, current timestamp: 2024-02-21T21:22:27.379

template equals someText: some text, current timestamp: 2024-02-21T21:22:27.379

Initialization with object:

[.when]
Set `mapObject` = [e-set=mapObject]`{{map a='1' b='2'}}`
[.then]
`mapObject.a` equals [e-eq=#mapObject.a]`1` +
`mapObject.b` equals [e-eq=#mapObject.b]`2`

Set mapObject = {a=1, b=2}

mapObject.a equals 1
mapObject.b equals 2

Initialization with string from file template:

[.when]
Set `fileTemplate` = [e-set=fileTemplate]`{{file '/data/core/file.txt' v='123'}}`
[.then]
`fileTemplate` equals [e-eq=#fileTemplate]`File template: v = 123, someText = some text`

Given file /data/core/file.txt

File template: v = {{v}}, someText = {{someText}}

Set fileTemplate = File template: v = 123, someText = some text

fileTemplate equals File template: v = 123, someText = some text

Initialization with listing:

.Given `someJson`:
[source,json,e-set=someJson]
----
{ "a" : 1 }
----
Then `someJson` equals [e-eq-json=#someJson]`{ "a" : 1 }`

Given someJson:

{ "a" : 1 }

Then someJson equals { "a" : 1 }

Hidden set

[.when]
Set `hidden` = [e-set=hidden hide]#1#
[.then]
`hidden` equals [e-eq=#hidden]`1`

Set hidden = 1

hidden equals 1

e-execute

Wrapper of concordion execute command

Usage

The full name [e-execute=#result = split(#TEXT)]_Jane Smith_
will be broken into first name [e-eq=#result.first]_Jane_
and last name [e-eq=#result.second]_Smith_

The full name Jane Smith will be broken into first name Jane and last name Smith

In case, when an input parameter occurs after an output we want to check, we can solve this problem by putting an execute command on the paragraph.

Unusual sentence structures

[role=e-execute=#greeting = greetingFor(#firstName)]
The greeting "[e-eq=#greeting]#Hello Bob!#"
should be given to user [e-set=#firstName]#Bob# when he logs in.

The greeting "Hello Bob!" should be given to user Bob when he logs in.

Execute on table

[.given]
Variable `someVar` = [e-set=someVar]`123`

[.then]
.Lowercase test
[%header, cols="1,1,2,2,3,3", e-execute="#r = lowercase(#param)"]
|===
|[e-set=param]#Param#
|[e-eq=#r.text]#Text#
|[e-eq-json=#r.json]#JSON#
|[e-eq-xml=#r.xml]#XML#
|[e-echo=#r.json]#Echo JSON#
|[e-echo=#r.xml]#Echo XML#

|[.json]`ABC {{someVar}}`
|abc {{someVar}}
|[.json]`{"result": "abc {{someVar}}"}`
|[.xml]`<result>abc {{someVar}}</result>`
|[pre language-json details]`{empty}`
|[pre language-xml]`{empty}`

|aBc {{someVar}}
|abc {{someVar}}
|[.json]`{"result": "abc {{someVar}}"}`
|[.xml]`<result>abc {{someVar}}</result>`
|[pre language-json]`{empty}`
|[pre language-xml details]`{empty}`
|===

Variable someVar = 123

Table 1. Lowercase test
Param	Text	JSON	XML	Echo JSON	Echo XML
`ABC 123`	abc 123	`{"result": "abc 123"}`	`<result>abc 123</result>`	`{ "result": "abc 123" }`	`<result>abc 123</result>`
aBc 123	abc 123	`{"result": "abc 123"}`	`<result>abc 123</result>`	`{ "result": "abc 123" }`	`<result>abc 123</result>`

Checking emptiness:

.Empty string test
[e-execute="#result = getEmptyString()"]
,===
[e-eq=#result]#Text#, [e-eq-json=#result]#JSON#, [e-eq-xml=#result]#XML#

,,
,===

Table 2. Empty string test
Text	JSON	XML

e-verify-rows

Wrapper of concordion verify-rows command

Usage

.Set up users
[e-execute=setUpUser(#TEXT)]
1. john.lennon
2. ringo.starr

.Add more users
[e-execute=setUpUser(#TEXT)]
- george.harrison
- paul.mccartney

Searching for [e-set=searchString]*arr* will return:

.Search result
[e-verify-rows="#username : search(#searchString)", e-match-strategy=BestMatch]
,===
[e-eq=#username]#Matching Usernames#

george.harrison
ringo.starr
,===

Set up users

john.lennon
ringo.starr

Add more users

george.harrison
paul.mccartney

Searching for arr will return:

Table 3. Search result
Matching Usernames
george.harrison
ringo.starr

e-run

Usage

link:Nested.adoc[Some nested spec, role=e-run]

Some nested spec

e-echo

Usage

Echo _firstName_: [e-echo=#firstName]_{empty}_

Echo firstName: Bob

example command

Asciidoc named examples interpreted as Concordion examples.

Sidebar with id = before transformed to “before” examples command.

Usage

[#before]
.Before each example
****
Split the full name [e-execute=#result = split(#TEXT)]_Jane Smith_
****

.Dummy
====
Result contains first name [e-eq=#result.first]_Jane_
and last name [e-eq=#result.second]_Smith_
====

[.ExpectedToFail]
.Dummy
====
[.when]
Wrong expectations

[.then]
Result contains first name [e-eq=#result.first]_Wrong_
and last name [e-eq=#result.second]_Wrong_
====

Before each example

Split the full name Jane Smith

Example 1. Dummy

Result contains first name Jane and last name Smith

Log File

!Example 2. Dummy

Example 2. DummyExpectedToFail

Wrong expectations

Result contains first name

Text mismatch Expected: "Wrong" but: was "Jane"
~~Wrong~~Jane

and last name

Text mismatch Expected: "Wrong" but: was "Smith"
~~Wrong~~Smith

Handlebar support

The Handlebars templates may be used for templating values in Exam commands.

Exam provides following built-in helpers groups:

Date helpers: {{at '-2d' '-1m'}} will produce: object of class java.util.Date = "Mon Feb 19 21:21:27 MSK 2024" {{now '-2d' '-1m'}} will produce: object of class java.util.Date = "Mon Feb 19 21:21:27 MSK 2024" {{today '-1y' '-2M' '-3d'}} will produce: object of class java.util.Date = "Sun Dec 18 00:00:00 MSK 2022" {{parse '01.02.2000 10:20' 'dd.MM.yyyy HH:mm'}} will produce: object of class java.util.Date = "Tue Feb 01 10:20:00 MSK 2000" {{shift date '-2d' '1m'}} will produce: object of class java.util.Date = "Sat Jan 01 10:21:00 MSK 2000" (given context has variables: {date=Mon Jan 03 10:20:00 MSK 2000}) {{weeksAgo 2}} will produce: object of class java.util.Date = "Wed Feb 07 00:00:00 MSK 2024" {{daysAgo 2}} will produce: object of class java.util.Date = "Mon Feb 19 00:00:00 MSK 2024" {{format date "yyyy-MM-dd'T'HH:mm"}} will produce: "2000-01-02T10:20" (given context has variables: {date=Sun Jan 02 10:20:00 MSK 2000})
Data matcher helpers: {{notNull}} will produce: "${json-unit.not-null}" (given context has variables: {placeholder_type=json}) {{string}} will produce: "${json-unit.any-string}" (given context has variables: {placeholder_type=json}) {{number}} will produce: "${json-unit.any-number}" (given context has variables: {placeholder_type=json}) {{bool}} will produce: "${json-unit.any-boolean}" (given context has variables: {placeholder_type=json}) {{ignore}} will produce: "${json-unit.ignore}" (given context has variables: {placeholder_type=json}) {{uuid}} will produce: "${json-unit.matches:uuid}" (given context has variables: {placeholder_type=json}) {{regex '\d+'}} will produce: "${json-unit.regex}\d+" (given context has variables: {placeholder_type=json}) {{after (today)}} will produce: "${json-unit.matches:after}2024-02-21T00:00" (given context has variables: {placeholder_type=json}) {{before (today)}} will produce: "${json-unit.matches:before}2024-02-21T00:00" (given context has variables: {placeholder_type=json}) {{formattedAndWithin 'yyyy-MM-dd' '5s' (today)}} will produce: "${json-unit.matches:formattedAndWithin}yyyy-MM-dd|$|5s|$|2024-02-21T00:00" (given context has variables: {placeholder_type=json}) {{within '5s' (today)}} will produce: "${json-unit.matches:within}5s|$|2024-02-21T00:00" (given context has variables: {placeholder_type=json}) {{formattedAs "yyyy-MM-dd'T'hh:mm:ss"}} will produce: "${json-unit.matches:formattedAs}yyyy-MM-dd'T'hh:mm:ss" (given context has variables: {placeholder_type=json}) {{iso date}} will produce: "2000-01-02T10:20:11.123" (given context has variables: {date=Sun Jan 02 10:20:11 MSK 2000}) {{isoDate date}} will produce: "2000-01-02" (given context has variables: {date=Sun Jan 02 10:20:11 MSK 2000}) {{matches 'name' 'params'}} will produce: "${json-unit.matches:name}params" (given context has variables: {placeholder_type=json})
Misc helpers: {{vars v1='a' v2=2}} will produce: "" {{set 1 "someVar"}} will produce: object of class java.lang.Integer = "1" {{fnn var1 var2 "default value"}} will produce: "default value" {{map key='value'}} will produce: object of class java.util.Collections$SingletonMap = "{key=value}" {{ls '1' '2'}} will produce: object of class java.util.Arrays$ArrayList = "[1, 2]" {{json (map f1='1' f2=(ls '1' '2'))}} will produce: "{"f1":"1","f2":["1","2"]}" {{prettyJson '{"a": 1}'}} will produce: "{ "a": 1 }" {{NULL}} will produce: null {{exist obj}} will produce: object of class java.lang.Boolean = "true" (given context has variables: {obj=any}) {{eval '#var'}} will produce: object of class java.lang.Integer = "2" (given context has variables: {var=2}) {{resolve 'today is {{today}}' var='val'}} will produce: "today is Wed Feb 21 00:00:00 MSK 2024" {{file '/hb/some-file.txt' var1='val1' var2='val2'}} will produce: "today is Wed Feb 21 00:00:00 MSK 2024" {{jsonPath '{"root": {"nested": 1}}' '$.root.nested'}} will produce: object of class java.lang.Integer = "1" {{prop 'system.property' 'optional default'}} will produce: "optional default" {{env 'env.property' 'optional default'}} will produce: "optional default"

Data types matchers:

[.given]
.`dataJson`:
[source,json,e-set=dataJson]
{
  "string": "some string",
  "number": 123,
  "bool": true,
  "ignore": "anything 123",
  "regex": "123"
}

[.then]
.`dataJson` matches:
[source,json,e-eq-json=#dataJson]
{
  "string": "{{string}}",
  "number": "{{number}}",
  "bool": "{{bool}}",
  "ignore": "{{ignore}}",
  "regex": "{{regex '\\d+'}}"
}

dataJson:

{
  "string": "some string",
  "number": 123,
  "bool": true,
  "ignore": "anything 123",
  "regex": "123"
}

dataJson matches:

{
  "string": "${json-unit.any-string}",
  "number": "${json-unit.any-number}",
  "bool": "${json-unit.any-boolean}",
  "ignore": "${json-unit.ignore}",
  "regex": "${json-unit.regex}\\d+"
}

Check with matcher and set actual to variable:

.Check and set
[%header, e-execute="#r = lowercase(#param)"]
|===
|[e-set=param]#Param# |[e-eq=#r.text]#Text#

|ABC |{{string}}>>str1
|DEF |{{ignore}}>>str2
|===
Actual value is matched and set to variable: str1 = [e-eq=#str1]`abc`, str2 = [e-eq=#str2]`def`

Table 4. Check and set
Param	Text
ABC	${text-unit.any-string}
DEF	${text-unit.ignore}

Actual value is matched and set to variable: str1 = abc, str2 = def

Date/time matchers:

[.given]
.`dtJson`:
[source,json,e-set=dtJson]
--
{
  "customFormat": "{{format (at) "yyyy/MM/dd'T'HH:mm.ss"}}",
  "isoDate": "{{isoDate (at)}}",
  "iso": "{{iso (at)}}",

  "customFormatAndWithinNow": "{{format (at) "yyyy/MM/dd'T'HH:mm.ss"}}",
  "isoDateAndWithinNow": "{{isoDate (at)}}",
  "isoAndWithinNow": "{{iso (at)}}",

  "customFormatAndWithinSpecifiedDate": "{{format (at) "yyyy/MM/dd'T'HH:mm.ss"}}",
  "isoDateAndWithinSpecifiedDate": "{{isoDate (at)}}",
  "isoAndWithinSpecifiedDate": "{{iso (at)}}",

  "afterSpecifiedDate": "{{iso (at)}}",
  "beforeSpecifiedDate": "{{iso (at)}}"
}
--
[.then]
.`dtJson` matches:
[source,json,e-eq=#dtJson,e-verifier=json]
--
{
"customFormat": "{{formattedAs 'yyyy/MM/dd\'T\'HH:mm.ss'}}",
"isoDate": "{{isoDate}}",
"iso": "{{iso}}",

"customFormatAndWithinNow": "{{formattedAndWithin 'yyyy/MM/dd\'T\'HH:mm.ss' '25s'}}",
"isoDateAndWithinNow": "{{isoDate '1d'}}",
"isoAndWithinNow": "{{iso '25s'}}",

"customFormatAndWithinSpecifiedDate": "{{formattedAndWithin 'yyyy/MM/dd\'T\'HH:mm.ss' '25s' (now)}}",
"isoDateAndWithinSpecifiedDate": "{{isoDate '1d' (now)}}",
"isoAndWithinSpecifiedDate": "{{iso '25s' (now)}}",

"afterSpecifiedDate": "{{after (at '-1h')}}",
"beforeSpecifiedDate": "{{before (at '+1h')}}"
}
--

dtJson:

{
  "customFormat": "2024/02/21T21:22.27",
  "isoDate": "2024-02-21",
  "iso": "2024-02-21T21:22:27.379",

  "customFormatAndWithinNow": "2024/02/21T21:22.27",
  "isoDateAndWithinNow": "2024-02-21",
  "isoAndWithinNow": "2024-02-21T21:22:27.379",

  "customFormatAndWithinSpecifiedDate": "2024/02/21T21:22.27",
  "isoDateAndWithinSpecifiedDate": "2024-02-21",
  "isoAndWithinSpecifiedDate": "2024-02-21T21:22:27.379",

  "afterSpecifiedDate": "2024-02-21T21:22:27.379",
  "beforeSpecifiedDate": "2024-02-21T21:22:27.379"
}

dtJson matches:

{
"customFormat": "${json-unit.matches:formattedAs}yyyy/MM/dd'T'HH:mm.ss",
"isoDate": "${json-unit.matches:formattedAs}yyyy-MM-dd",
"iso": "${json-unit.matches:formattedAs}ISO_LOCAL",

"customFormatAndWithinNow": "${json-unit.matches:formattedAndWithin}yyyy/MM/dd'T'HH:mm.ss|$|25s|$|2024-02-21T21:22:39.979",
"isoDateAndWithinNow": "${json-unit.matches:formattedAndWithin}yyyy-MM-dd|$|1d|$|2024-02-21",
"isoAndWithinNow": "${json-unit.matches:formattedAndWithin}ISO_LOCAL|$|25s|$|2024-02-21T21:22:39.979",

"customFormatAndWithinSpecifiedDate": "${json-unit.matches:formattedAndWithin}yyyy/MM/dd'T'HH:mm.ss|$|25s|$|2024-02-21T21:22:39.979",
"isoDateAndWithinSpecifiedDate": "${json-unit.matches:formattedAndWithin}yyyy-MM-dd|$|1d|$|2024-02-21",
"isoAndWithinSpecifiedDate": "${json-unit.matches:formattedAndWithin}ISO_LOCAL|$|25s|$|2024-02-21T21:22:39.979",

"afterSpecifiedDate": "${json-unit.matches:after}2024-02-21T20:22:27.379",
"beforeSpecifiedDate": "${json-unit.matches:before}2024-02-21T22:22:27.379"
}

Web Service plugin

WsPlugin enables to document REST/SOAP API.

Dependency

testImplementation "io.github.adven27:exam-ws:2024.0.4"

Configuration

class Specs : AbstractSpecs() {
    override fun init() = ExamExtension(
        WsPlugin(
            host = "localhost",
            port = 8080
        )
    )

e-http

Check HTTP interaction

Attributes

`verifier`	Optional. Default: `text`. Content type verifier. Built-in values: `text`, `json`, `xml`.
`awaitAtMostSec`	Optional. Default: 4. How long to wait in seconds before failing the verification
`awaitPollDelayMillis`	Optional. Default: 0. How long to wait in millis before starting verification
`awaitPollIntervalMillis`	Optional. Default: 1000. How long to wait in millis between verification attempts

Usage

.HTTP API check
[e-http=]
--
.Given request:
[source,httprequest]
----
GET /mirror/request
Content-Type: application/json
Authorization: token
Accept-Language: ru
Cookie: cookie1=c1; cookie2=c2
----

.Expected response:
[source,httprequest]
----
200
Content-Type: application/json

{
  "GET": "/mirror/request",
  "headers": {
    "host": "{{ignore}}",
    "content-type": "application/json",
    "authorization": "token",
    "accept-language": "ru",
    "cookie": "cookie1=c1; cookie2=c2"
  },
  "cookies": { "cookie1": "c1", "cookie2": "c2" }
}
----
--

HTTP API check

Given request:

GET /mirror/request HTTP/1.1
Host: localhost:8888
Content-Type: application/json
Authorization: token
Accept-Language: ru
Cookie: cookie1=c1; cookie2=c2

Expected response:

HTTP/1.1 200 
Content-Type: application/json

{
  "GET": "/mirror/request",
  "headers": {
    "host": "${test-unit.ignore}",
    "content-type": "application/json",
    "authorization": "token",
    "accept-language": "ru",
    "cookie": "cookie1=c1; cookie2=c2"
  },
  "cookies": {
    "cookie1": "c1",
    "cookie2": "c2"
  }
}

HTTP/1.1 200 
Content-Type: application/json
Transfer-Encoding: chunked
Date: Wed, 21 Feb 2024 18:22:40 GMT

{
  "GET": "/mirror/request",
  "headers": {
    "host": "localhost:8888",
    "content-type": "application/json",
    "authorization": "token",
    "accept-language": "ru",
    "cookie": "cookie1=c1; cookie2=c2"
  },
  "cookies": {
    "cookie1": "c1",
    "cookie2": "c2"
  }
}

SOAP

.SOAP in collapsible blocks
[e-http=]
--
.SOAP request
[%collapsible]
=====
[source,httprequest]
----
POST /mirror/soap
Content-Type: application/soap+xml

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <ns2:getItemRequest xmlns:ns2="http://ws.io">
            <date>{{iso (at)}}</date>
        </ns2:getItemRequest>
    </soap:Body>
</soap:Envelope>
----
=====

.SOAP response
[%collapsible]
=====
[source,httprequest]
----
200

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <ns2:getItemRequest xmlns:ns2="http://ws.io">
            <date>{{iso (at)}}</date>
        </ns2:getItemRequest>
    </soap:Body>
</soap:Envelope>
----
=====
--

SOAP in collapsible blocks

SOAP request

POST /mirror/soap HTTP/1.1
Host: localhost:8888
Content-Type: application/soap+xml
Content-Length: 259

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <ns2:getItemRequest xmlns:ns2="http://ws.io">
      <date>2024-02-21T21:22:27.379</date>
    </ns2:getItemRequest>
  </soap:Body>
</soap:Envelope>

SOAP response

HTTP/1.1 200 

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <ns2:getItemRequest xmlns:ns2="http://ws.io">
            <date>2024-02-21T21:22:27.379</date>
        </ns2:getItemRequest>
    </soap:Body>
</soap:Envelope>

HTTP/1.1 200 
Content-Type: application/soap+xml;charset=utf-8
Content-Length: 259
Date: Wed, 21 Feb 2024 18:22:40 GMT

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <ns2:getItemRequest xmlns:ns2="http://ws.io">
      <date>2024-02-21T21:22:27.379</date>
    </ns2:getItemRequest>
  </soap:Body>
</soap:Envelope>

Data-driven check

.Data-driven check
[e-http=, awaitAtMostSec=4]
--
.Request
[source,httprequest]
----
{{file req}}
----
.Response
[source,httprequest]
----
{{file resp}}
----

[.where]
,===
,req, resp

GET, /data/ws/get.http, /data/ws/get-resp.http
POST, /data/ws/post.http, /data/ws/post-resp.http
PUT, /data/ws/put.http, /data/ws/put-resp.http
DELETE, /data/ws/delete.http, /data/ws/delete-resp.http
,===
--

Data-driven check

Request

GET /mirror/request HTTP/1.1
Host: localhost:8888
Content-Type: application/json

Response

HTTP/1.1 200 
Content-Type: application/json

{
  "GET": "/mirror/request",
  "headers": {
    "host": "${test-unit.ignore}",
    "content-type": "application/json"
  }
}

HTTP/1.1 200 
Content-Type: application/json
Transfer-Encoding: chunked
Date: Wed, 21 Feb 2024 18:22:40 GMT

{
  "GET": "/mirror/request",
  "headers": {
    "host": "localhost:8888",
    "content-type": "application/json"
  }
}

Request

POST /mirror/request?p=1 HTTP/1.1
Host: localhost:8888
Content-Type: application/json
Content-Length: 21

{
  "a": "ok",
  "b": 1
}

Response

HTTP/1.1 200 
Content-Type: application/json

{
  "POST": "/mirror/request?p=1",
  "headers": {
    "host": "${test-unit.ignore}",
    "content-type": "application/json",
    "content-length": "${test-unit.ignore}"
  },
  "body": {
    "a": "ok",
    "b": 1
  }
}

HTTP/1.1 200 
Content-Type: application/json
Transfer-Encoding: chunked
Date: Wed, 21 Feb 2024 18:22:40 GMT

{
  "POST": "/mirror/request?p=1",
  "body": {
    "a": "ok",
    "b": 1
  },
  "headers": {
    "host": "localhost:8888",
    "content-type": "application/json",
    "content-length": "21"
  }
}

Request

PUT /mirror/request?p=1 HTTP/1.1
Host: localhost:8888
Content-Type: application/json
Content-Length: 21

{
  "a": "ok",
  "b": 1
}

Response

HTTP/1.1 200 
Content-Type: application/json

{
  "PUT": "/mirror/request?p=1",
  "headers": {
    "host": "${test-unit.ignore}",
    "content-type": "application/json",
    "content-length": "${test-unit.ignore}"
  },
  "body": {
    "a": "ok",
    "b": 1
  }
}

HTTP/1.1 200 
Content-Type: application/json
Transfer-Encoding: chunked
Date: Wed, 21 Feb 2024 18:22:40 GMT

{
  "PUT": "/mirror/request?p=1",
  "body": {
    "a": "ok",
    "b": 1
  },
  "headers": {
    "host": "localhost:8888",
    "content-type": "application/json",
    "content-length": "21"
  }
}

Request

DELETE /mirror/request HTTP/1.1
Host: localhost:8888
Content-Type: application/json

Response

HTTP/1.1 200 
Content-Type: application/json

{
  "DELETE": "/mirror/request",
  "headers": {
    "host": "${test-unit.ignore}",
    "content-type": "application/json"
  }
}

HTTP/1.1 200 
Content-Type: application/json
Transfer-Encoding: chunked
Date: Wed, 21 Feb 2024 18:22:40 GMT

{
  "DELETE": "/mirror/request",
  "headers": {
    "host": "localhost:8888",
    "content-type": "application/json"
  }
}

Put interactions to variable

.Put interactions to variable
[source,httprequest,e-http=interactions]
----
GET /mirror/request?p=1
Content-Type: application/json
----

.Check interactions
[%header,cols="4,2,0",e-verify-rows="#i : #interactions"]
|===
|[e-eq=#i.req.raw]#Request#
|[e-verifier=jsonIgnoreExtraFields e-eq=#i.resp.body]#Response#
|[e-eq=#i.resp.statusCode]#Status#

|[pre language-http]`GET /mirror/request?p=1 HTTP/1.1
Host: localhost:8888
Content-Type: application/json
{blank}
{blank}`
|[pre language-json]`{}`
|200
|===

Put interactions to variable

GET /mirror/request?p=1 HTTP/1.1
Host: localhost:8888
Content-Type: application/json

Table 5. Check interactions
Request	Response	Status
`GET /mirror/request?p=1 HTTP/1.1 Host: localhost:8888 Content-Type: application/json`	`{}`	200

Custom content type verifier

import io.github.adven27.concordion.extensions.exam.core.AbstractSpecs
import io.github.adven27.concordion.extensions.exam.core.ExamExtension
import io.github.adven27.concordion.extensions.exam.core.JsonVerifier
import net.javacrumbs.jsonunit.core.Option.IGNORING_EXTRA_FIELDS

class Specs : AbstractSpecs() {

    override fun init() = ExamExtension(
        //...
    ).withVerifiers(
        "jsonIgnoreExtraFields" to JsonVerifier { it.withOptions(IGNORING_EXTRA_FIELDS) }
    )
}

Use custom content verifier

.Custom content type verifier
[e-http=]
--
[source,httprequest]
----
POST /mirror/request
Content-Type: application/json

{}
----

.Check with registered `jsonIgnoreExtraFields` verifier
[source,httprequest,e-verifier=jsonIgnoreExtraFields]
----
200
Content-Type: application/json

{
  "POST": "/mirror/request",
  "body": {},
  "headers": {
    "host": "{{ignore}}",
    "content-type": "application/json",
    "content-length": "{{ignore}}"
  }
}
----
--

Custom content type verifier

POST /mirror/request HTTP/1.1
Host: localhost:8888
Content-Type: application/json
Content-Length: 2

{}

Check with registered jsonIgnoreExtraFields verifier

HTTP/1.1 200 
Content-Type: application/json

{
  "POST": "/mirror/request",
  "body": {},
  "headers": {
    "host": "${test-unit.ignore}",
    "content-type": "application/json",
    "content-length": "${test-unit.ignore}"
  }
}

HTTP/1.1 200 
Content-Type: application/json
Transfer-Encoding: chunked
Date: Wed, 21 Feb 2024 18:22:40 GMT

{
  "POST": "/mirror/request",
  "body": {},
  "headers": {
    "host": "localhost:8888",
    "content-type": "application/json",
    "content-length": "2"
  }
}

Details

How failures look like?

SQL Database plugin

DbPlugin is a wrapper around DbUnit library that enables setting up and verification of database.

Figure 3. Exam integration with database

Dependency

testImplementation "io.github.adven27:exam-db:2024.0.4"

Configuration

class Specs : AbstractSpecs() {
    override fun init() = ExamExtension(
        DbPlugin(
            driver = "org.postgresql.Driver",
            url = "jdbc:postgresql://localhost:5432/postgres",
            user = "postgres",
            password = "postgres"
        )
    )

Commands

e-db-set
e-db-check
e-db-clean
e-db-execute
e-db-verify
e-db-show

e-db-set

Creates DbUnit dataset for specified table and applies it to database.

Attributes

`operation`	Optional. Default: `clean_insert`. DbUnit operation
`ds`	Optional. Default: `default`. Datasource to apply dataset. By default, only 1 datasource with name `default` exists, more can be configured in `DbPlugin`.

Usage

.Set up table `PRODUCT`
[%header,format=csv, e-db-set=product]
|===
id, name, price, rating, disabled, created_at, meta_json
1, Model A, 100.50,       10, false,       {{at '-1d'}},"{
  ""id"": ""a"",
  ""code"": 1
}"
2, Model B,    200, {{NULL}},  true, {{at}},"{
  ""id"": ""b"",
  ""code"": 2
}"
|===

Table 6. Set up table `PRODUCT`
id	name	price	rating	disabled	created_at	meta_json
1	Model A	100.50	10	false	{{at '-1d'}}	{ "id": "a", "code": 1 }
2	Model B	200	{{NULL}}	true	{{at}}	{ "id": "b", "code": 2 }

Table 6. Set up table PRODUCT
id	name	price	rating	disabled	created_at	meta_json
1	Model A	100.50	10	false	2024-02-20T21:22:27.379	{ "id": "a", "code": 1 }
2	Model B	200	(null)	true	2024-02-21T21:22:27.379	{ "id": "b", "code": 2 }

e-db-check

Creates DbUnit dataset for table and verifies it against a database.

Attributes

`where`	Optional. Default: not set. WHERE clause in database-specific SQL format to filter actual dataset
`orderBy`	Optional. Default: columns in order of declaration. List of columns to sort records before verifying
`ds`	Optional. Default: `default`. Datasource to apply dataset. By default, only 1 datasource with name `default` exists, more can be configured in `DbPlugin`.
`awaitAtMostSec`	Optional. Default: 4. How long to wait in seconds before failing the verification
`awaitPollDelayMillis`	Optional. Default: 0. How long to wait in millis before starting verification
`awaitPollIntervalMillis`	Optional. Default: 1000. How long to wait in millis between verification attempts

Usage

.Check table `PRODUCT`
[%header,format=csv, e-db-check=product]
|===
id, name, price, rating, disabled, created_at, meta_json
1, Model A, 100.50,       10, false,       {{at '-1d'}},"{
  ""id"": ""a"",
  ""code"": 1
}"
2, Model B,    200, {{NULL}},  true, {{at}},"{
  ""id"": ""b"",
  ""code"": 2
}"
|===

Table 7. Check table PRODUCT
id	name	price	rating	disabled	created_at	meta_json
1	Model A	100.50	10	false	2024-02-20T21:22:27.379	{ "id": "a", "code": 1 }
2	Model B	200.00	(null)	true	2024-02-21T21:22:27.379	{ "id": "b", "code": 2 }

Check with matchers

.Check with matchers
[e-db-check=product]
|===
|name |price |rating |disabled |created_at |meta_json

|{{string}}
|{{number}}>>priceA
|{{notNull}}
|{{bool}}>>disabledA
|{{within '15s' (at '-1d')}}
|{"id": "{{string}}", "code": "{{number}}"}

|{{regex '.*'}}>>nameB
|{{ignore}}
|{{ignore}}>>ratingB
|true
|{{within '15s'}}>>createdB
|{"id": "{{regex '\\w'}}", "code": "{{ignore}}"}
|===

nameB = [e-eq=#nameB]`Model B`
priceA = [e-eq=#priceA]`100.50`
ratingB = [e-eq=#ratingB]`(null)`
disabledA = [e-eq=#disabledA]`false`
createdB = [e-eq=#createdB]`{{format (at) 'yyyy-MM-dd HH:mm:ss.SSS'}}`

Table 8. Check with matchers
name	price	rating	disabled	created_at	meta_json
Model A	100.50	10	false	2024-02-20T21:22:27.379	{ "id": "a", "code": 1 }
Model B	200.00	(null)	true	2024-02-21T21:22:27.379	{ "id": "b", "code": 2 }

nameB = Model B priceA = 100.50 ratingB = (null) disabledA = false createdB = 2024-02-21 21:22:27.379

Check ordered

[e-db-check=product, e-orderBy='rating, id']
,===
name, rating

{{string}}, {{ignore}}
{{string}}, 10
,===

product
name	rating
Model B	(null)
Model A	10

Check emptiness

.Check table
[e-db-check=product]
|===
|===

Table 9. Set empty table

EMPTY

Table 10. Check table

EMPTY

e-db-clean

Cleans specified tables in a database.

Attributes

`ds`	Optional. Default: `default`. Datasource to apply dataset. By default, only 1 datasource with name `default` exists, more can be configured in `DbPlugin`.

Usage

Clean tables: [e-db-clean]#person, person_fields#

Tables will be cleaned up in the order opposite to declaration, hence the "referenced" tables should go first and the "referencing" - last.

Clean tables: person, person_fields

e-db-execute

Applies specified DbUnit datasets to database.

Attributes

`operation`	Optional. Default: `clean_insert`. DbUnit operation
`ds`	Optional. Default: `default`. Datasource to apply dataset. By default, only 1 datasource with name `default` exists, more can be configured in `DbPlugin`.
`dir`	Optional. Default: empty string. Set directory in `src/test/resources` as working directory so that all datasets from `datasets` attribute will be relative to it: `src/test/resources[dir][item in datasets]`

Usage

Execute datasets: +++<e e:db-execute='adam.xml, bob.json' dir='/data/db/'/>+++

adam.xml

<?xml version='1.0' encoding='UTF-8'?>
<dataset>
  <PERSON ID="1" NAME="Adam" AGE="10" BIRTHDAY="{{today}}"/>
  <PERSON_FIELDS ID="11..19" NAME="a1" VAL="{{a1}}" PERSON_ID="1"/>
  <PERSON_FIELDS ID="11..19" NAME="a2" VAL="{{a2}}" PERSON_ID="1"/>
</dataset>

bob.json

{
  "PERSON": [
    {"id": 10, "name": "Bob", "age": 20, "BIRTHDAY": "2000-10-10 00:00:00"}
  ],
  "PERSON_FIELDS": [
    {"id": "21..30", "name": "b1", "val": "1", "person_id": 10},
    {"id": "21..30", "name": "b2", "val": "2", "person_id": 10}
  ]
}

1 2

Execute datasets:
PERSON
ID NAME AGE BIRTHDAY
1 Adam 10 2024-02-21T00:00:00
10 Bob 20 2000-10-10 00:00:00
PERSON_FIELDS
ID NAME VAL PERSON_ID
11 a1 1 1
12 a2 2 1
21 b1 1 10
22 b2 2 10

PERSON_FIELDS
ID	NAME	VAL	PERSON_ID
11	a1	1	1
12	a2	2	1
21	b1	1	10
22	b2	2	10

e-db-verify

Verifies specified DbUnit datasets against database.

Attributes

`dir`	Optional. Default: empty string. Set directory in `src/test/resources` as working directory so that all datasets from `datasets` attribute will be relative to it: `src/test/resources[dir][item in datasets]`
`orderBy`	Optional. Default: columns in order of declaration. List of columns to sort records before verifying
`ds`	Optional. Default: `default`. Datasource to apply dataset. By default, only 1 datasource with name `default` exists, more can be configured in `DbPlugin`.
`awaitAtMostSec`	Optional. Default: 4. How long to wait in seconds before failing the verification
`awaitPollDelayMillis`	Optional. Default: 0. How long to wait in millis before starting verification
`awaitPollIntervalMillis`	Optional. Default: 1000. How long to wait in millis between verification attempts

Usage

Verify datasets: +++<e e:db-verify='adam.xml, bob.json' dir='/data/db/'/>+++

adam.xml

<?xml version='1.0' encoding='UTF-8'?>
<dataset>
  <PERSON ID="1" NAME="Adam" AGE="10" BIRTHDAY="{{today}}"/>
  <PERSON_FIELDS ID="11..19" NAME="a1" VAL="{{a1}}" PERSON_ID="1"/>
  <PERSON_FIELDS ID="11..19" NAME="a2" VAL="{{a2}}" PERSON_ID="1"/>
</dataset>

bob.json

{
  "PERSON": [
    {"id": 10, "name": "Bob", "age": 20, "BIRTHDAY": "2000-10-10 00:00:00"}
  ],
  "PERSON_FIELDS": [
    {"id": "21..30", "name": "b1", "val": "1", "person_id": 10},
    {"id": "21..30", "name": "b2", "val": "2", "person_id": 10}
  ]
}

Verify datasets:
PERSON
ID NAME AGE BIRTHDAY
1 Adam 10 2024-02-21T00:00:00
10 Bob 20 2000-10-10T00:00:00
PERSON_FIELDS
ID NAME VAL PERSON_ID
11 a1 1 1
12 a2 2 1
21 b1 1 10
22 b2 2 10

PERSON_FIELDS
ID	NAME	VAL	PERSON_ID
11	a1	1	1
12	a2	2	1
21	b1	1	10
22	b2	2	10

e-db-show

Prints content of specified database table and optionally generates DbUnit dataset.

Attributes

`where`	Optional. Default: not set. WHERE clause in database-specific SQL format to filter actual dataset
`ds`	Optional. Default: `default`. Datasource to apply dataset. By default, only 1 datasource with name `default` exists, more can be configured in `DbPlugin`.
`saveToResources`	Optional. Default: not set. Path in 'srs/test/resources/' to store generated dataset

Usage

.Show all and generate dataset
[e-db-show=person, e-saveToResources=/data/db/person.xml]
,===
,===

.Show specific columns
[e-db-show=person]
,===
id, name

,===

.Show specific rows
[e-db-show=person, e-where="name='Bob'"]
,===
id, name

,===

Table 11. Show all and generate dataset
id	name	age	birthday
1	Adam	10	2024-02-21T00:00:00
10	Bob	20	2000-10-10T00:00:00

<?xml version='1.0' encoding='UTF-8'?>
<dataset>
  <person id="1" name="Adam" age="10" birthday="2024-02-21 00:00:00.0"/>
  <person id="10" name="Bob" age="20" birthday="2000-10-10 00:00:00.0"/>
  <person_fields id="11" name="a1" val="1" person_id="1"/>
  <person_fields id="12" name="a2" val="2" person_id="1"/>
  <person_fields id="21" name="b1" val="1" person_id="10"/>
  <person_fields id="22" name="b2" val="2" person_id="10"/>
</dataset>

Table 12. Show specific columns
id	name
1	Adam
10	Bob

Table 13. Show specific rows
id	name
10	Bob

Details

Message Queue plugin

MqPlugin enables to set up and verify the state of some message queue. A message queue is represented as an implementation of the io.github.adven27.concordion.extensions.exam.mq.MqTester interface and that implementation is responsible for interacting with that specific queue.

There are several out-of-the-box implementations that can be used directly or as an example for custom ones:

Dependency

testImplementation "io.github.adven27:exam-mq:2024.0.4"

Configuration

open class Specs : AbstractSpecs() {
    override fun init() = ExamExtension(
        MqPlugin(
            "someRabbitQueue" to RabbitTester(
                port = 5432,
                sendConfig = SendConfig("someRoutingKey"),
                receiveConfig = ReceiveConfig("someQueueName")
            ),
            "dummyQueue" to object : MqTester {
                private val queue = ArrayDeque<Message>()

                /* open connection*/
                override fun start() = Unit

                /* close connection*/
                override fun stop() = Unit

                override fun send(message: Message) {
                    queue += message
                }

                override fun receive(): List<Message> = queue.map { queue.poll() }
                override fun purge() = queue.clear()
                override fun accumulateOnRetries(): Boolean = true
            }
        )
    )

MqPlugin accepts a map of a <queue alias name> to a MqTester implementation. Queue alias is a queue logical name: e-mq-set=out

Commands

e-mq-set
e-mq-check
e-mq-clean

e-mq-set

Sends messages to specified queue.

Usage

:queue2: myAnotherQueue

.Sending message to {queue2}
[source,json,e-mq-set={queue2}]
----
{ "a": 1, "d": "dd" }
----

Sending message to myAnotherQueue

{ "a": 1, "d": "dd" }

Sending message with headers and params which could be used by MqTester:

.Sending message with headers and params to {queue2}
[e-mq-set={queue2}]
--
[.params]
.Params
[cols="h,1", caption=]
,===
key, some-kafka-message-key
partition, 1
,===

[.headers]
.Headers
[cols="h,1", caption=]
,===
h1, 1
h2, 2
,===

[source,json]
----
{"timestamp": "{{iso (at '-1h')}}"}
----
--

Sending message with headers and params to myAnotherQueue

Params
key	some-kafka-message-key
partition	1

Headers
h1	1
h2	2

{"timestamp": "2024-02-21T20:22:27.379"}

Sending several messages:

.Sending several messages to myQueue
[cols="a", grid=rows, frame=ends, caption=, e-mq-set=myQueue]
|===
| Message without headers:
[source,json]
----
{{file '/specs/../data/mq/msg.json' msg=(map v1='a' v2='b')}}
----

| Message with headers and params which could be used by `MqTester`

[.params]
.Params
[cols="h,1", caption=]
,===
key, some-kafka-message-key
partition, 1
,===

[.headers]
.Headers
[cols="h,1", caption=]
,===
h1, {{isoDate (at '-1d')}}
h2, 2
,===

[source,json]
----
{{file '/specs/../data/mq/msg.json' msg=(map v1='c' v2='d')}}
----
|===

Message without headers:

{
  "date": "2024-02-21",
  "var1": "a",
  "var2": "b",
  "var3": "3"
}

Message with headers and params which could be used by MqTester

Params
key	some-kafka-message-key
partition	1

Headers
h1	2024-02-20
h2	2

{
  "date": "2024-02-21",
  "var1": "c",
  "var2": "d",
  "var3": "3"
}

e-mq-check

Check messages in queue.

Attributes

`contains`	Optional. Default: `EXACT`. Messages list verifying mode. Supported values: `EXACT`, `EXACT_IN_ANY_ORDER`.
`verifier`	Optional. Default: `text`. Content type verifier. Built-in values: `text`, `json`, `xml`.
`awaitAtMostSec`	Optional. Default: 4. How long to wait in seconds before failing the verification
`awaitPollDelayMillis`	Optional. Default: 0. How long to wait in millis before starting verification
`awaitPollIntervalMillis`	Optional. Default: 1000. How long to wait in millis between verification attempts

Usage

.Check messages in myQueue
[cols="a", grid=rows, frame=ends, caption=, e-mq-check=myQueue]
|===
| Expected message ignore headers:

[source,json]
----
{{file '/specs/../data/mq/msg.json' msg=(map v1='a' v2='b')}}
----

| Expected message containing headers and params:

[.params]
.Params
[cols="h,1", caption=]
,===
key, some-kafka-message-key
,===

[.headers]
[cols="h,1"]
,===
h1, {{isoDate (at '-1d')}}
,===

[source,json]
----
{{file '/specs/../data/mq/msg.json' msg=(map v1='c' v2='d')}}
----
|===

Check messages in myQueue

{ "date": "2024-02-21", "var1": "a", "var2": "b", "var3": "3" }

Params
key	some-kafka-message-key

Headers
h1	2024-02-20

{ "date": "2024-02-21", "var1": "c", "var2": "d", "var3": "3" }

Check emptiness

.Check myQueue is empty
[caption=, e-mq-check=myQueue]
|===
|===

Check myQueue is empty
EMPTY

Custom content type verifier

import io.github.adven27.concordion.extensions.exam.core.AbstractSpecs
import io.github.adven27.concordion.extensions.exam.core.ExamExtension
import io.github.adven27.concordion.extensions.exam.core.JsonVerifier
import net.javacrumbs.jsonunit.core.Option.IGNORING_EXTRA_FIELDS

class Specs : AbstractSpecs() {

    override fun init() = ExamExtension(
        //...
    ).withVerifiers(
        "jsonIgnoreExtraFields" to JsonVerifier { it.withOptions(IGNORING_EXTRA_FIELDS) }
    )
}

Use custom content verifier

.Check with registered `jsonIgnoreExtraFields` verifier
[cols="a", grid=rows, frame=ends, caption=, e-mq-check=myQueue]
|===
| Expected ISO-formatted `date` and ignore extra fields:

[source,json,e-verifier=jsonIgnoreExtraFields]
----
{"date":  "{{isoDate}}"}
----
|===

myQueue has message

{
  "date": "2024-02-21",
  "var1": "1",
  "var2": "2",
  "var3": "3"
}

Check with registered jsonIgnoreExtraFields verifier
{ "date": "${test-unit.matches:formattedAs}yyyy-MM-dd" }

e-mq-clean

Cleans specified queues.

Usage

Queues are empty: [e-mq-clean]_myQueue, myAnotherQueue_

Queues are empty: myQueue, myAnotherQueue

Details

How failures look like?