The Nerrvana Jenkins plugin allows you to fully automate the testing of your web applications using a Jenkins CI server.
It uses Nerrvana’s API for communication with the Nerrvana cloud.
The plugin implements the following functionality:
- Creates a configuration file in the Nerrvana user interface
- Synchronizes tests extracted from the version control system with tests stored in the corresponding Nerrvana space
- Adds Jenkins build number to the name of the generated test run
- Allows you to dynamically add information to the description of the created test run (for example, a committer name, revision number, or commit message)
- If you use Nerrvana messages, the plug-in can analyse test results and set the status of a build, depending on the level of errors found
Requirements
The plugin uses LFTP to synchronize tests between Jenkins and Nerrvana. LFTP must be installed (yum install lftp) on the same server running Jenkins.
Installation
The Nerrvana plugin is available at http://your_jenkins_instance/pluginManager/available. (If it does not exist there yet, you can download and install it manually from here.)
Configuration file
We recommend that you run your tests from Nerrvana’s UI and verify that Nerrvana can run them before proceeding. The process for running tests manually is described on the ’Get Started’ page.
To create a configuration file for the Nerrvana plugin, go to the page ‘Add new test run’ or ‘Edit test run’.
As shown below, select the desired Space (A), enter the Test Run name (B), and select an executable file (C). (Note: Try not to use a long name for the test run since the plugin will add a Jenkins build number to it,) We recommend that you leave the Description field (D) blank and allow the plugin to add commit information to it dynamically. Select desired platforms (E) and, if your tests can run in parallel, the number of nodes per platform (F). Nerrvana is now able to create a configuration file (G), where you will need to set a few parameters before using it in Jenkins. Please save the file.
Lines 3-20 contain your Nerrvana API access keys. They are added automatically when you generate the configuration file. They are also available on the Settings page, where you can change them, if necessary.
<api-params>
<!-- Address of the Nerrvana gateway. -->
<gateway>https://api.nerrvana.com</gateway>
<!--
User-specific key which identifies user on Nerrvana side.
Available as an 'API public key' on Settings page
(https://cloud.nerrvana.com/user/editAccount) in Nerrvana.
-->
<apikey>2bf7c81-a031f-1ad2-fd3a6-f59b9e0668e</apikey>
<!--
This key is used by the Nerrvana plug-in to create a checksum of API call
parameters to ensure their consistency.
Available as an 'API private key' on Settings page in Nerrvana.
-->
<secretkey>
wctfFwelygx3tXS4TasrsOS4oXV7YadcPppvEnx55WG7qPk6jrAHjJi1TRoLlhrarIlabte4H9zWROXVkLnXto2LlVC47EXx91Uu
</secretkey>
</api-params>
Name of the Test Run is in lines 22-24. The plugin will add the Jenkins build number, thus generating a new name for each new Test Run.
<!-- Parameters related to Nerrvana-driven Selenium tests. -->
<!-- Test Run name template, Jenkins build number will be added to the end automatically. -->
<test-run-name>Answers TRUNK</test-run-name>
The Test Run description is in lines 26-40 – parameter ‘test-run-descr’. You can leave it blank or add, for example, ‘Generated by Jenkins plugin’. Lines 30-38 explain how to use a file to add a description dynamically with each job build.
<!-- Test Run description. All Test Runs created by this Jenkins build step will have this description. -->
<test-run-descr></test-run-descr>
<!--
Content of this file, if not empty, will be added to a description,
defined by 'test-run-descr' parameter above.
File path is relative to the Jenkins job workspace.
During deployment phase you can extract revision number, commiter name and a
commit message from your version control system, put them into this file and
use them as a description.
You can extract and parse SVN information into info.txt file with our little
tool - https://github.com/deepshiftlabs/scm-decorator
You can read how we do it with SVN in our blog http://www.deepshiftlabs.com/sel_blog/?p=2343&lang=en-us.
-->
<test-run-descr-file>info.txt</test-run-descr-file>
Here's how a Test Run created by the plugin will look if:
- Test-run-name – ‘Answers TRUNK MySQL’
- Test-run-descr – ‘Created by Nerrvana-Jenkins plugin’
-
The contents of ‘test-run-descr-file’:
Revision: 142
Committer: igork
Date: 2012-10-17 07:39:20 +0000 (Wed, 17 Oct 2012)
The Jenkins build number (‘build # 51’) was added to the Test Run name. The description is a combination of ‘test-run-descr’ and the content of a file specified by the content ‘test-run-descr-file’ parameter.
Lines 42-43 contain a parameter that defines an executable file we selected – ‘xbuild-mysql.sh’.
<!-- Which executable file Nerrvana should use to start tests. -->
<executable-file>xbuild-mysql.sh</executable-file>
This file should be able to launch your tests. For example:
ant -f build-mysql.xml all > build.log 2 >& 1
Rows 45-62 show the platforms on which the tests will run. All supported platforms are included, with the selected ones uncommented.
<!-- List of platforms to run tests against for this config. -->
<platforms>
<!-- List of available platforms. Uncomment to use. -->
<!--platform><code>centos_58_firefox_36</code><name>Firefox 3.6 (CentOS)</name></platform-->
<!--platform><code>winxp_sp3_firefox_110</code><name>Firefox 11.0 (WinXP)</name></platform-->
<!--platform><code>winxp_sp3_firefox_36</code><name>Firefox 3.6 (WinXP)</name></platform-->
<!--platform><code>winxp_sp3_firefox_150</code><name>Firefox 15.0 (WinXP)</name></platform-->
<!--platform><code>winxp_sp3_ie_8</code><name>IE 8 (WinXP)</name></platform-->
<!--platform><code>winxp_sp3_opera_1162</code><name>Opera 11.62 (WinXP)</name></platform-->
<!--platform><code>winxp_sp3_opera_1202</code><name>Opera 12.02 (WinXP)</name></platform-->
<!--platform><code>winxp_sp3_safari_515</code><name>Safari 5.1.5 (WinXP)</name></platform-->
<!--platform><code>winxp_sp3_safari_517</code><name>Safari 5.1.7 (WinXP)</name></platform-->
<platform><code>winxp_sp3_chrome_2001132</code><name>Chrome 20.0.1132 (WinXP)</name></platform>
<platform><code>winxp_sp3_chrome_2101180</code><name>Chrome 21.0.1180 (WinXP)</name></platform>
</platforms>
In lines 64-65, you can configure the plugin to run tests in parallel. More information about parallel execution is available here. If your tests will not work in parallel, change the number of nodes here to 1.
<!-- How many Selenium nodes should be used for each platform. -->
<nodes-count>2</nodes-count>
The next parameter (lines 67-73) points to the location of the tests in the Jenkins workspace. If the tests are in the ‘tests’ folder, this parameter should be set as follows:
<!-- Parameters related to the transfer of the tests from Jenkins to Nerrvana. -->
<!--
Folder in the workspace of Jenkins job where Selenium tests will be located.
It is assumed that the SCM build step, which always occurs BEFORE other steps,
will put tests there.
-->
<folder-with-tests>.</folder-with-tests>
Space ID, Space name and FTPS folder (lines 75-80) were specified previously during configuration file generation and should be left as they are.
<!-- Nerrvana space previously created by you through the Nerrvana UI. -->
<space>
<id>4028</id>
<name>Answers</name>
<ftp-path>Answers/_files</ftp-path>
</space>
Lines 81-89 define FTPS access credentials. It is necessary to add the password to your Nerrvana account.
<!-- Address and credentials of the Nerrvana FTPS connection.
Note that a system running Jenkins should have LFTP application installed. -->
<ftp>
<server>ftp.nerrvana.com</server>
<!-- Your username -->
<username>demo123</username>
<!-- Replace this value with your password!! -->
<password>[p a s s w o r d]</password>
</ftp>
The parameter ‘skip-tests-sync’ (lines 91-95) allows you to speed up testing by skipping test synchronization when the Nerrvana plugin is called multiple times in a single build. For a practical example, please visit our blog.
<!--
You can skip test sync when you call the Nerrvana plugin many times in a single
job and tests were synced in a previous step.
-->
<skip-tests-sync>false</skip-tests-sync>
The parameters defined in lines 97-124 are relevant only if you are using messages, which Nerrvana can analyze. There is a detailed description of these parameters in the plugin configuration file.
<!-- Execution-specific parameters of the plugin -->
<plugin-settings>
<!--
Job status will be set to FAILED if test execution in Nerrvana is unsuccessful
(tests did not sync, executable file did not start your tests, etc.).
If tests were executed successfully, you have a few options:
1. Mark build as successful in Jenkins and analyse generated reports
for problems your tests discovered.
2. If you want to mark a build as FAILED based on errors your tests found, you can:
(a) Add additional step, load results, parse them to analyse errors your tests
discovered, and mark build as FAILED based on it. Keep this parameter as 'false'
in this case.
(b) Use Nerrvana "Messages" and let the Nerrvana plugin analyse results, and change
build status to FAILED based on it. Keep this parameter as 'true' in this
case and use additional parameter 'message-threshold' to add additional logic.
If you use messages and invoke the Nerrvana plug-in several times in the same build,
keep this option as 'false' for all invocations except the last one to let the build
complete all tests and analyse errors at the very end.
-->
<use-messages-to-set-build-status>false</use-messages-to-set-build-status>
<!--
Defines the level at which a FAILED status is generated. If this value is set to 'WARN',
for example, and your tests generate one or more 'WARN' or higher severity messages
(ERROR or FATAL), Nerrvana execution status, and Jenkins build, will be FAILED.
For the full list of levels, visit http://www.nerrvana.com/docs/using-messages page.
-->
<message-threshold>ERROR</message-threshold>
Valid values for ‘message-threshold’ are listed in the documentation. If your tests generate messages and you want to watch WARN error level and above, then set these parameters as follows:
<use-messages-to-set-build-status>true</use-messages-to-set-build-status>
<message-threshold>WARN</message-threshold>
In this case, the plugin will not only receive and store messages polling Nerrvana while tests are running, but also analyze whether there are messages with the level ERROR or higher (FATAL). If any are found, the build status will be set as FAILED in Jenkins.
If you do not use Nerrvana messages and still want to set the status of the Jenkins build based on test results (rather than whether the run was completed successfully), you can add an extra step in your build after the Nerrvana plugin. Get test results from Nerrvana via FTPS, analyze them, and change the status of the Jenkins build.
The parameter ‘max-execution-time’ will stop tests if, for example, they go into an infinite loop. If you run tests using the Nerrvana UI, you can set the real value with some extra time added.
<!--
Maximum execution time (in seconds). Defines how long the Nerrvana plug-in
will wait for tests to complete. Start by setting to a large value and
adjust accordingly after a few runs.
-->
<max-execution-time>3600</max-execution-time>
The last parameter ‘poll-period’ specifies how often the Nerrvana plugin will poll API about the status of test execution.
<!-- How often the Nerrvana plug-in will update test execution status from Nerrvana (in seconds). -->
<poll-period>20</poll-period>
The plugin flowchart below details plugin logic regarding polling and setting Jenkins build status.
- Starting the test run. In this Nerrvana space, a new test run with the name specified in the plugin configuration file is created. Jenkins build number is appended to the test run name. Tests already in Jenkins workspace are synchronized with tests stored in Nerrvana space using ‘lftp mirror‘ command in 50 parallel threads. Test run starts in Nerrvana.
- Test status polling is carried out periodically and is defined by the poll-period and lasts no longer than the time defined by parameter max-execution-time. If during this time the tests were not completed, the build will be marked as FAILED.
- If you are using Nerrvana messages, they are collected by the plugin during the polling process and stored in ‘results.xml’ file.
- If you are not using Nerrvana messages, when testing finishes, the build is marked as successfully completed and is terminated.
- Contents of the ‘results.xml’ file are analyzed for messages containing the level specified in the parameter ‘message-threshold’ and/or higher levels. If messages are found, the build is marked as FAILED. In this case, the plugin creates a simple report containing errors and links to test results in Nerrvana.
Now we only need to add a step to our build...
...and copy the contents of the configuration file into the plugin configuration window.
You can save changes in a Jenkins job, run the build, and watch your tests running in the Nerrvana cloud.
Below is an example of a Jenkins build console log showing the implementation of the plugin, with comments.
---BEGIN PLUGIN EXECUTION---
2012-10-17 07:39:24 ---INITIALIZATION STARTED---
2012-10-17 07:39:24
---BEGIN PLUGIN SETTINGS---
Nerrvana HTTP address: https://api.nerrvana.com
Nerrvana API key: a6c171d-20eb5-61e3-60991-8a1e523ce
Secret key: z3rCWlLdAP35eCCYAmQZ2kPw9X0LByrcc
Space ID: 4144
Space: Answers
Space path[FTPS folder]: Answers/_files
Selenium nodes per platform: 1
Test run name: Answers TRUNK
Nerrvana platforms:
Opera 12.02 (WinXP)
Firefox 15.0 (WinXP)
Executable file: xbuild-mysql.sh
Nerrvana FTPS address: ftp.nerrvana.com
Nerrvana FTPS user: demo123
Nerrvana FTPS pass: dem0I23
Workspace folder: ./tests
Max execution time: 3600
Poll period: 20
Parse user messages mode[results analyzer]: ON
User message threshold: ERROR
-----END PLUGIN SETTINGS---
2012-10-17 07:39:24 -----INITIALIZATION COMPLETED---
2012-10-17 07:39:24 Generated test run name: Answers TRUNK MySQL build #51
2012-10-17 07:39:24 Generated test run description:
Created by Nerrvana-Jenkins plugin
Revision: 142
Committer: igork
Date: 2012-10-17 07:39:20 +0000 (Wed, 17 Oct 2012)
Launching all tests on two platforms
2012-10-17 07:39:24 ---BEGIN UPLOADING TESTS TO NERRVANA FTPS---
[workspace] $ lftp -f upload-build-51-8414449724631519802.ftp
2012-10-17 07:40:16 -----END UPLOADING TESTS TO NERRVANA FTPS
2012-10-17 07:40:18 Creating and starting test run
via Nerrvana HTTP API call...done.
2012-10-17 07:40:18 New test run ID#884.
2012-10-17 07:40:18 New execution ID#5486.
2012-10-17 07:40:18 ---BEGIN NERRVANA POLLING CYCLE
(waiting for tests to complete)
2012-10-17 07:40:40 Current execution status: run
2012-10-17 07:41:02 Current execution status: run
------ Some оutput removed ---------------------------------
2012-10-17 07:44:17 Current execution status: run
2012-10-17 07:44:38 Current execution status: ok
2012-10-17 07:44:38 -----END NERRVANA POLLING CYCLE---
2012-10-17 07:44:38 Saving execution results into results.xml...
2012-10-17 07:44:38 Done.
2012-10-17 07:44:38 ---BEGIN TEST EXECUTION RESULTS---
2012-10-17 07:44:38 At least 43 message(s) from Nerrvana side reach(es)
or exceed(s) threshold level (ERROR).
Message analyzer marks execution as failure.
2012-10-17 07:44:38 -----END TEST EXECUTION RESULTS---
2012-10-17 07:44:38
-----END PLUGIN EXECUTION---
Build step 'Nerrvana plug-in' marked build as failure
Finished: FAILURE
In the first 26 lines, the plugin outputs the parameters parsed from its configuration file. In lines 28-34, the plugin creates a new test run name and description.
In lines 36-39, you see the synchronization process between tests in the Jenkins workspace and tests stored in the Nerrvana space used for this job. In lines 41-44, the plugin creates a new Test Run in Nerrvana, launches it and gets a Test Run ID and unique Test Run execution ID back from API. This will be used to monitor test status by the plugin later. The same Test Run can be launched several times, but each run will have its own unique internal execution ID. At this stage, the Test Run is visible in the Nerrvana UI. "Answers TRUNK MySQL build # 51" Test Run is created and starts running.
In lines 45-56, the plugin polls test status every 20 seconds (polling parameter), while reading Nerrvana messages, which become available in real time during tests’ execution. Once the tests are completed (lines 56-60), the plugin analyzes the errors (lines 61-65) and on the basis of the analysis sets the build status (lines 70-71). If you’re not using Nerrvana messages, the build would be marked as successful.
Another advantage of using Nerrvana messages is that the plugin can create simple reports (1).
Reports show the status of the build and message threshold level used by the plugin (2) and the description of the Test Run (3). In our example, the description contains information about the commit that triggered the tests. Platform names (4) are linked to the Nerrvana FTPS server, which will contain detailed reports generated by the tests. Finally, clicking on the last column (5) expands the table and shows the messages that your tests sent to Nerrvana, which were received and collected by the Jenkins plugin. Very often, this provides enough information for a developer to understand where the errors are, especially when he or she also maintains tests.
Contact us