|
||||
Welcome to Orkney!The universal testframework ... |
Orkney is a test framework created to fulfill the following requirements
Let us introduce and explain the building blocks of Orkney.
We want to create a short example, so open the destination folder
and have a look to the contents. You should find the Orkney Jar
file, orkney.jar, a
folder lib with several
third party jars and two scripts for Unix and Windows, runOrkney.sh and runOrkney.cmd.
In order to get started, create a new folder work with a sub folder tests. Please create a new
folder below tests and
call it stringtests. Use
the text editor of your choice and create a new text file and name
it StringTest.yaml. The
folders we did create are our test groups. The newly created text
file is our first test case.
Test cases are based on test steps. Kind and order of the test
steps as well as their parameters are defined by writing a small
text file. Orkney uses Yaml, a user friendly alternative to XML.
Copy the following code into your first test case.
Note: Do not use tabs! It
is very important to use spaces instead of tabs for indentation,
failing to do so, will result in errors when Orkney tries to parse
the test case!
TestCase.yaml:
# my first Orkney test
description: First
Testcase for Stringcomparison. The test case will convert a string
to lower case and compare the result against an expected value.
steps:
- step:
de.softquadrat.orkney.steps.strings.ConvertToLowercase
description:
Convert a string to lower case.
in: { sin:
Hello world }
out: { sout: $s }
- step:
de.softquadrat.orkney.steps.strings.StringCompare
description:
Compare the result against an exepected value.
in: { s1: $s, s2: hello world }
First of all Orkney uses Yaml and hence Yaml's comments, which
are introduced by a leading #. This helps to
document the whole test case and each single step. A test case
description is added using the keyword description followed by a colon an the explaining
text.
After that the keyword steps is used to introduce the test steps used. Our example test case uses two test steps, ConvertToLowerCase and StringCompare, both from Orkney's own step package. Each step is parametrized with description, input and output parameters.
After the keyword step the class name of the test step is given. Below this an optional description and zero, one or two lines with input and output parameters may follow. Each parameter group, input and output, is introduced by the corresponding keyword in and out respectively.
The parameters in each group are enclosed in curly brackets separated by commas. Note that the names and types of input and output parameters are defined by the test step designer. Consult the documentation of the test steps you want to use in order to find out what must be provided in which format.
The test step ConvertToLowerCase expects one input parameter named sin and returns one output parameter named sout, all of them of type string.
In order to save the output of this step for later use in the test case a variable named s is used, marked with a leading $. This variable is reused in the next step to fill an input variable.
The next test step StringCompare expects two
input parameters and does not have any output parameters. The
input parameters are named s1 and s2,
the first of them is filled with the value of the test case
variable s, again marked with a leading $.
The second parameter is filled with a string literal.
Note the difference between test step parameters and step case
variables. The first step ConvertToLowerCase has
an input parameter named sin
and an output parameter named sout
all coloured green inside the Yaml script. The whole test case
uses a variable named s,
that we have coloured blue to distinguish it from step parameters.
Each step can reflects failure by throwing an exception. The step ConvertToLowerCase will not throw an exception and succeed by that. The step StringCompare will throw an exception when the comparison fails. The whole test case will succeed if all steps were successful. In our example this means that the test case will be successful exactly when the comparison step ends successfully.
Lets start Orkney in order to run our first test suite and create the first report. Open a command shell, change to the Orkney folder and use the Orkney scripts for invocation:
runOrkney.sh *(for Unix)* runOrkney.cmd *(for Windows)*
This simple invocation will use the file structure found in directory "work". We will see later, how to fine tune the invocation.
Orkney will execute all tests found in directory "tests", a sub directory of "work". As a result Orkney should write a small report to the console like:
Orkney, version 0.0.2
Copyright (c) 2013, 2014 SoftQuadrat Gmbh
Orkney test run completed with result: Success.
1 tests executed.
1 tests run successfully.
0 tests failed.
Orkney creates a report directory on each run. You should find
this newly created directory named reports under our work directory. Inside reports two subfolders, log and xml, can be found. While the
first one contains text logs, one for each test group and test
case, the seconds one contains the same in XML format. Explore the
directories to understand Orkney reporting, find below a text log
file example.
Name: TestCaseDescription: First Testcase for Stringcomparison |
The same log in XML format might look like this:
<?xml version='1.0' encoding='UTF-8'?> |
You can find all information from the text log in the machine
readable XML as well.
But all of these logs are not very well suited for human reading.
In order to get a quick overview and navigate through the test
results we need a better approach. Did you notice the second line
of the XML log? It references an XSLT stylesheet that will help to
make the XML log better understandable.
Open the root log of the Orkney run from work/reports/xml/tests.xml with your favorite browser. You should get a result like the one below allowing you easily to navigate through the test results by clicking the hyperlinks:
The page shows our test group named tests, click on the hyperlink for the test case
called TestCase.
With the hyperlinks for the parent goups in the top of each XML
report you can easily navigate back to the parent test groups like
navigating through a directory tree. Have a try to get familiar
with the visualization of XML reports.
Note: A full report showing the results of executing the Orkney demo project can be found here. Open it and try to navigate through the test results of the demo in order to understand their structure and the navigation. |
A nice feature of Orkney is that a test run archive is created on each invocation. By this you will never loose the results of prior runs when starting Orkney again. In the folder work zip files with timestamp are created that contain the full test run. This will allow you to have a look to earlier Orkney executions and document progress and history of your tests:
work
|___ orkney-20140503-202517.zip
|___ orkney-20140503-214330.zip
⇍ |
Navigation |
⇒ Orkney Demo |