Navigation
⇒ Orkney Demo

Welcome to Orkney!

The universal testframework ...

Orkney Logo

Contents

Introduction

Orkney is a test framework created to fulfill the following requirements

The following picture gives on overview about Orkney, its components and its usage.


Orkney Components

Let us introduce and explain the building blocks of Orkney.

Download

Download Orkney from here and extract the contents of the Zip archive to a folder on your computer.

Getting Started

We want to create a short example, so open the destination folder and have a look to the contents. You should find the Orkney Jar file, orkney.jar, a folder lib with several third party jars and two scripts for Unix and Windows, runOrkney.sh and runOrkney.cmd.

In order to get started, create a new folder work with a sub folder tests. Please create a new folder below tests and call it stringtests. Use the text editor of your choice and create a new text file and name it StringTest.yaml. The folders we did create are our test groups. The newly created text file is our first test case.

Test cases are based on test steps. Kind and order of the test steps as well as their parameters are defined by writing a small text file. Orkney uses Yaml, a user friendly alternative to XML. Copy the following code into your first test case.

Note: Do not use tabs! It is very important to use spaces instead of tabs for indentation, failing to do so, will result in errors when Orkney tries to parse the test case!

TestCase.yaml:
# my first Orkney test
description: First Testcase for Stringcomparison. The test case will convert a string to lower case and compare the result against an expected value.
steps:
- step: de.softquadrat.orkney.steps.strings.ConvertToLowercase
  description: Convert a string to lower case.
  in: { sin: Hello world }
  out: { sout: $s }
- step: de.softquadrat.orkney.steps.strings.StringCompare
  description: Compare the result against an exepected value.
  in: { s1: $s, s2: hello world }

First of all Orkney uses Yaml and hence Yaml's comments, which are introduced by a leading #. This helps to document the whole test case and each single step. A test case description is added using the keyword description followed by a colon an the explaining text.

After that the keyword steps is used to introduce the test steps used. Our example test case uses two test steps, ConvertToLowerCase and StringCompare, both from Orkney's own step package. Each step is parametrized with description, input and output parameters.

After the keyword step the class name of the test step is given. Below this an optional description and zero, one or two lines with input and output parameters may follow. Each parameter group, input and output, is introduced by the corresponding keyword in and out respectively.

The parameters in each group are enclosed in curly brackets separated by commas. Note that the names and types of input and output parameters are defined by the test step designer. Consult the documentation of the test steps you want to use in order to find out what must be provided in which format.

The test step ConvertToLowerCase expects one input parameter named sin and returns one output parameter named sout, all of them of type string.

In order to save the output of this step for later use in the test case a variable named s is used, marked with a leading $. This variable is reused in the next step to fill an input variable.

The next test step StringCompare expects two input parameters and does not have any output parameters. The input parameters are named s1 and s2, the first of them is filled with the value of the test case variable s, again marked with a leading $. The second parameter is filled with a string literal.

Note the difference between test step parameters and step case variables. The first step ConvertToLowerCase has an input parameter named sin and an output parameter named sout all coloured green inside the Yaml script. The whole test case uses a variable named s, that we have coloured blue to distinguish it from step parameters.

Each step can reflects failure by throwing an exception. The step ConvertToLowerCase will not throw an exception and succeed by that. The step StringCompare will throw an exception when the comparison fails. The whole test case will succeed if all steps were successful. In our example this means that the test case will be successful exactly when the comparison step ends successfully.

Running Orkney

Lets start Orkney in order to run our first test suite and create the first report. Open a command shell, change to the Orkney folder and use the Orkney scripts for invocation:

runOrkney.sh *(for Unix)*
runOrkney.cmd *(for Windows)*

This simple invocation will use the file structure found in directory "work". We will see later, how to fine tune the invocation.

Orkney will execute all tests found in directory "tests", a sub directory of "work". As a result Orkney should write a small report to the console like:

  Orkney, version 0.0.2
Copyright (c) 2013, 2014 SoftQuadrat Gmbh
Orkney test run completed with result: Success.
1 tests executed.
1 tests run successfully.
0 tests failed.

Orkney Reports

Text Reports

Orkney creates a report directory on each run. You should find this newly created directory named reports under our work directory. Inside reports two subfolders, log and xml, can be found. While the first one contains text logs, one for each test group and test case, the seconds one contains the same in XML format. Explore the directories to understand Orkney reporting, find below a text log file example.

TestCase.log
Name: TestCaseDescription: First Testcase for Stringcomparison
Type: Case
Timestamp: 2014.05.03 21:43:30.282
Result: PASSED
Success: 1
Fail: 0
Properties: {}
Steps:
Name: ConvertToLowercase
Description: Convert Step
Result: PASSED
Name: StringCompare
Description: Compare Step
Result: PASSED
Logs:
2014.05.03 21:43:30.456 | TestCase | Running setup of test case
2014.05.03 21:43:30.456 | TestCase | Running setup of test step ConvertToLowercase
2014.05.03 21:43:30.456 | ConvertToLowercase | Set up
2014.05.03 21:43:30.456 | TestCase | Ended setup of test step ConvertToLowercase
2014.05.03 21:43:30.456 | TestCase | Running setup of test step StringCompare
2014.05.03 21:43:30.456 | StringCompare | Set up
2014.05.03 21:43:30.456 | TestCase | Ended setup of test step StringCompare
2014.05.03 21:43:30.456 | TestCase | Ending setup of test case, initialized=true
2014.05.03 21:43:30.456 | TestCase | Running test case
2014.05.03 21:43:30.456 | TestCase | Running test step ConvertToLowercase
2014.05.03 21:43:30.457 | TestCase | Input Variables: sin=Hello world
2014.05.03 21:43:30.457 | TestCase | Output Variables: sout=$s
2014.05.03 21:43:30.457 | ConvertToLowercase | Running test step
2014.05.03 21:43:30.457 | ConvertToLowercase | Input value: Hello world
2014.05.03 21:43:30.457 | ConvertToLowercase | Result is: hello world
2014.05.03 21:43:30.457 | TestCase | Ended test step ConvertToLowercase
2014.05.03 21:43:30.457 | TestCase | Running test step StringCompare
2014.05.03 21:43:30.457 | TestCase | Input Variables: s1=$s, s2=hello world
2014.05.03 21:43:30.457 | TestCase | Output Variables:
2014.05.03 21:43:30.457 | StringCompare | Running test step
2014.05.03 21:43:30.457 | StringCompare | Input value 1: hello world
2014.05.03 21:43:30.457 | StringCompare | Input value 2: hello world
2014.05.03 21:43:30.457 | StringCompare | Result is: true
2014.05.03 21:43:30.457 | TestCase | Ended test step StringCompare
2014.05.03 21:43:30.457 | TestCase | Ending test case, result=PASSED
2014.05.03 21:43:30.457 | TestCase | Running tear down of test case
2014.05.03 21:43:30.457 | TestCase | Running tear down of test step ConvertToLowercase
2014.05.03 21:43:30.457 | ConvertToLowercase | Tear down
2014.05.03 21:43:30.457 | TestCase | Running tear down of test step ConvertToLowercase
2014.05.03 21:43:30.457 | TestCase | Running tear down of test step StringCompare
2014.05.03 21:43:30.457 | StringCompare | Tear down
2014.05.03 21:43:30.457 | TestCase | Running tear down of test step StringCompare
2014.05.03 21:43:30.457 | TestCase | Ending tear down of test case, result=PASSED

XML Reports

The same log in XML format might look like this:

TestCase.xml
<?xml version='1.0' encoding='UTF-8'?>
<?xml-stylesheet type='text/xsl' href='report.xslt'?>
<orkney xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://orkney.sourceforge.net/orkney.xsd">
<type>case</type>
<name>TestCase</name>
<description>First Testcase for Stringcomparison</description>
<timestamp>2014.05.03 21:43:30.282</timestamp>
<result>PASSED</result>
<count>
<success>1</success>
<fail>0</fail>
</count>
<parents>
<parent>tests</parent>
</parents>
<properties>{}</properties>
<steps>
<step>
<name>ConvertToLowercase</name>
<description>Convert Step</description>
<result>PASSED</result>
</step>
<step>
<name>StringCompare</name>
<description>Compare Step</description>
<result>PASSED</result>
</step>
</steps>
<logs>
<log>
<timestamp>2014.05.03 21:43:30.456</timestamp>
<caller>TestCase</caller>
<message>Running setup of test case</message>
</log>
<log>
<timestamp>2014.05.03 21:43:30.456</timestamp>
<caller>TestCase</caller>
<message>Running setup of test step ConvertToLowercase</message>
</log>
...
</logs>
</orkney>

You can find all information from the text log in the machine readable XML as well.

But all of these logs are not very well suited for human reading. In order to get a quick overview and navigate through the test results we need a better approach. Did you notice the second line of the XML log? It references an XSLT stylesheet that will help to make the XML log better understandable.

Open the root log of the Orkney run from work/reports/xml/tests.xml with your favorite browser. You should get a result like the one below allowing you easily to navigate through the test results by clicking the hyperlinks:

Orkney report
        open with web browser

The page shows our test group named tests, click on the hyperlink for the test case called TestCase.

TestCase XML
        report opened with a web browser

With the hyperlinks for the parent goups in the top of each XML report you can easily navigate back to the parent test groups like navigating through a directory tree. Have a try to get familiar with the visualization of XML reports.

Note:
A full report showing the results of executing the Orkney demo project can be found here. Open it and try to navigate through the test results of the demo in order to understand their structure and the navigation.

Test Run Archiving

A nice feature of Orkney is that a test run archive is created on each invocation. By this you will never loose the results of prior runs when starting Orkney again. In the folder work zip files with timestamp are created that contain the full test run. This will allow you to have a look to earlier Orkney executions and document progress and history of your tests:

work
|___ orkney-20140503-202517.zip
|___ orkney-20140503-214330.zip

Navigation
⇒ Orkney Demo