The Nerrvana Jenkins plugin allows you to fully automate the testing of your web applications using a Jenkins CI server.
It uses Nerrvana’s API for communication with the Nerrvana cloud.
The plugin implements the following functionality:
The plugin uses LFTP to synchronize tests between Jenkins and Nerrvana. LFTP must be installed (yum install lftp) on the same server running Jenkins.
The Nerrvana plugin is available at http://your_jenkins_instance/pluginManager/available. (If it does not exist there yet, you can download and install it manually from here.)
We recommend that you run your tests from Nerrvana’s UI and verify that Nerrvana can run them before proceeding. The process for running tests manually is described on the ’Get Started’ page.
To create a configuration file for the Nerrvana plugin, go to the page ‘Add new test run’ or ‘Edit test run’.
As shown below, select the desired Space (A), enter the Test Run name (B), and select an executable file (C). (Note: Try not to use a long name for the test run since the plugin will add a Jenkins build number to it,) We recommend that you leave the Description field (D) blank and allow the plugin to add commit information to it dynamically. Select desired platforms (E) and, if your tests can run in parallel, the number of nodes per platform (F). Nerrvana is now able to create a configuration file (G), where you will need to set a few parameters before using it in Jenkins. Please save the file.
Lines 3-20 contain your Nerrvana API access keys. They are added automatically when you generate the configuration file. They are also available on the Settings page, where you can change them, if necessary.
<api-params> <!-- Address of the Nerrvana gateway. --> <gateway>https://api.nerrvana.com</gateway> <!-- User-specific key which identifies user on Nerrvana side. Available as an 'API public key' on Settings page (https://cloud.nerrvana.com/user/editAccount) in Nerrvana. --> <apikey>2bf7c81-a031f-1ad2-fd3a6-f59b9e0668e</apikey> <!-- This key is used by the Nerrvana plug-in to create a checksum of API call parameters to ensure their consistency. Available as an 'API private key' on Settings page in Nerrvana. --> <secretkey> wctfFwelygx3tXS4TasrsOS4oXV7YadcPppvEnx55WG7qPk6jrAHjJi1TRoLlhrarIlabte4H9zWROXVkLnXto2LlVC47EXx91Uu </secretkey> </api-params>
Name of the Test Run is in lines 22-24. The plugin will add the Jenkins build number, thus generating a new name for each new Test Run.
<!-- Parameters related to Nerrvana-driven Selenium tests. --> <!-- Test Run name template, Jenkins build number will be added to the end automatically. --> <test-run-name>Answers TRUNK</test-run-name>
The Test Run description is in lines 26-40 – parameter ‘test-run-descr’. You can leave it blank or add, for example, ‘Generated by Jenkins plugin’. Lines 30-38 explain how to use a file to add a description dynamically with each job build.
<!-- Test Run description. All Test Runs created by this Jenkins build step will have this description. --> <test-run-descr></test-run-descr> <!-- Content of this file, if not empty, will be added to a description, defined by 'test-run-descr' parameter above. File path is relative to the Jenkins job workspace. During deployment phase you can extract revision number, commiter name and a commit message from your version control system, put them into this file and use them as a description. You can extract and parse SVN information into info.txt file with our little tool - https://github.com/deepshiftlabs/scm-decorator You can read how we do it with SVN in our blog http://www.deepshiftlabs.com/sel_blog/?p=2343&lang=en-us. --> <test-run-descr-file>info.txt</test-run-descr-file>
Here's how a Test Run created by the plugin will look if:
Revision: 142
Committer: igork
Date: 2012-10-17 07:39:20 +0000 (Wed, 17 Oct 2012)
The Jenkins build number (‘build # 51’) was added to the Test Run name. The description is a combination of ‘test-run-descr’ and the content of a file specified by the content ‘test-run-descr-file’ parameter.
Lines 42-43 contain a parameter that defines an executable file we selected – ‘xbuild-mysql.sh’.
<!-- Which executable file Nerrvana should use to start tests. --> <executable-file>xbuild-mysql.sh</executable-file>
This file should be able to launch your tests. For example:
ant -f build-mysql.xml all > build.log 2 >& 1
Rows 45-62 show the platforms on which the tests will run. All supported platforms are included, with the selected ones uncommented.
<!-- List of platforms to run tests against for this config. --> <platforms> <!-- List of available platforms. Uncomment to use. --> <!--platform><code>centos_58_firefox_36</code><name>Firefox 3.6 (CentOS)</name></platform--> <!--platform><code>winxp_sp3_firefox_110</code><name>Firefox 11.0 (WinXP)</name></platform--> <!--platform><code>winxp_sp3_firefox_36</code><name>Firefox 3.6 (WinXP)</name></platform--> <!--platform><code>winxp_sp3_firefox_150</code><name>Firefox 15.0 (WinXP)</name></platform--> <!--platform><code>winxp_sp3_ie_8</code><name>IE 8 (WinXP)</name></platform--> <!--platform><code>winxp_sp3_opera_1162</code><name>Opera 11.62 (WinXP)</name></platform--> <!--platform><code>winxp_sp3_opera_1202</code><name>Opera 12.02 (WinXP)</name></platform--> <!--platform><code>winxp_sp3_safari_515</code><name>Safari 5.1.5 (WinXP)</name></platform--> <!--platform><code>winxp_sp3_safari_517</code><name>Safari 5.1.7 (WinXP)</name></platform--> <platform><code>winxp_sp3_chrome_2001132</code><name>Chrome 20.0.1132 (WinXP)</name></platform> <platform><code>winxp_sp3_chrome_2101180</code><name>Chrome 21.0.1180 (WinXP)</name></platform> </platforms>
In lines 64-65, you can configure the plugin to run tests in parallel. More information about parallel execution is available here. If your tests will not work in parallel, change the number of nodes here to 1.
<!-- How many Selenium nodes should be used for each platform. --> <nodes-count>2</nodes-count>
The next parameter (lines 67-73) points to the location of the tests in the Jenkins workspace. If the tests are in the ‘tests’ folder, this parameter should be set as follows:
<!-- Parameters related to the transfer of the tests from Jenkins to Nerrvana. --> <!-- Folder in the workspace of Jenkins job where Selenium tests will be located. It is assumed that the SCM build step, which always occurs BEFORE other steps, will put tests there. --> <folder-with-tests>.</folder-with-tests>
Space ID, Space name and FTPS folder (lines 75-80) were specified previously during configuration file generation and should be left as they are.
<!-- Nerrvana space previously created by you through the Nerrvana UI. --> <space> <id>4028</id> <name>Answers</name> <ftp-path>Answers/_files</ftp-path> </space>
Lines 81-89 define FTPS access credentials. It is necessary to add the password to your Nerrvana account.
<!-- Address and credentials of the Nerrvana FTPS connection. Note that a system running Jenkins should have LFTP application installed. --> <ftp> <server>ftp.nerrvana.com</server> <!-- Your username --> <username>demo123</username> <!-- Replace this value with your password!! --> <password>[p a s s w o r d]</password> </ftp>
The parameter ‘skip-tests-sync’ (lines 91-95) allows you to speed up testing by skipping test synchronization when the Nerrvana plugin is called multiple times in a single build. For a practical example, please visit our blog.
<!-- You can skip test sync when you call the Nerrvana plugin many times in a single job and tests were synced in a previous step. --> <skip-tests-sync>false</skip-tests-sync>
The parameters defined in lines 97-124 are relevant only if you are using messages, which Nerrvana can analyze. There is a detailed description of these parameters in the plugin configuration file.
<!-- Execution-specific parameters of the plugin --> <plugin-settings> <!-- Job status will be set to FAILED if test execution in Nerrvana is unsuccessful (tests did not sync, executable file did not start your tests, etc.). If tests were executed successfully, you have a few options: 1. Mark build as successful in Jenkins and analyse generated reports for problems your tests discovered. 2. If you want to mark a build as FAILED based on errors your tests found, you can: (a) Add additional step, load results, parse them to analyse errors your tests discovered, and mark build as FAILED based on it. Keep this parameter as 'false' in this case. (b) Use Nerrvana "Messages" and let the Nerrvana plugin analyse results, and change build status to FAILED based on it. Keep this parameter as 'true' in this case and use additional parameter 'message-threshold' to add additional logic. If you use messages and invoke the Nerrvana plug-in several times in the same build, keep this option as 'false' for all invocations except the last one to let the build complete all tests and analyse errors at the very end. --> <use-messages-to-set-build-status>false</use-messages-to-set-build-status> <!-- Defines the level at which a FAILED status is generated. If this value is set to 'WARN', for example, and your tests generate one or more 'WARN' or higher severity messages (ERROR or FATAL), Nerrvana execution status, and Jenkins build, will be FAILED. For the full list of levels, visit http://www.nerrvana.com/docs/using-messages page. --> <message-threshold>ERROR</message-threshold>
Valid values for ‘message-threshold’ are listed in the documentation. If your tests generate messages and you want to watch WARN error level and above, then set these parameters as follows:
<use-messages-to-set-build-status>true</use-messages-to-set-build-status>
<message-threshold>WARN</message-threshold>
In this case, the plugin will not only receive and store messages polling Nerrvana while tests are running, but also analyze whether there are messages with the level ERROR or higher (FATAL). If any are found, the build status will be set as FAILED in Jenkins.
If you do not use Nerrvana messages and still want to set the status of the Jenkins build based on test results (rather than whether the run was completed successfully), you can add an extra step in your build after the Nerrvana plugin. Get test results from Nerrvana via FTPS, analyze them, and change the status of the Jenkins build.
The parameter ‘max-execution-time’ will stop tests if, for example, they go into an infinite loop. If you run tests using the Nerrvana UI, you can set the real value with some extra time added.
<!-- Maximum execution time (in seconds). Defines how long the Nerrvana plug-in will wait for tests to complete. Start by setting to a large value and adjust accordingly after a few runs. --> <max-execution-time>3600</max-execution-time>
The last parameter ‘poll-period’ specifies how often the Nerrvana plugin will poll API about the status of test execution.
<!-- How often the Nerrvana plug-in will update test execution status from Nerrvana (in seconds). --> <poll-period>20</poll-period>
The plugin flowchart below details plugin logic regarding polling and setting Jenkins build status.
Now we only need to add a step to our build...
...and copy the contents of the configuration file into the plugin configuration window.
You can save changes in a Jenkins job, run the build, and watch your tests running in the Nerrvana cloud.
Below is an example of a Jenkins build console log showing the implementation of the plugin, with comments.
---BEGIN PLUGIN EXECUTION--- 2012-10-17 07:39:24 ---INITIALIZATION STARTED--- 2012-10-17 07:39:24 ---BEGIN PLUGIN SETTINGS--- Nerrvana HTTP address: https://api.nerrvana.com Nerrvana API key: a6c171d-20eb5-61e3-60991-8a1e523ce Secret key: z3rCWlLdAP35eCCYAmQZ2kPw9X0LByrcc Space ID: 4144 Space: Answers Space path[FTPS folder]: Answers/_files Selenium nodes per platform: 1 Test run name: Answers TRUNK Nerrvana platforms: Opera 12.02 (WinXP) Firefox 15.0 (WinXP) Executable file: xbuild-mysql.sh Nerrvana FTPS address: ftp.nerrvana.com Nerrvana FTPS user: demo123 Nerrvana FTPS pass: dem0I23 Workspace folder: ./tests Max execution time: 3600 Poll period: 20 Parse user messages mode[results analyzer]: ON User message threshold: ERROR -----END PLUGIN SETTINGS--- 2012-10-17 07:39:24 -----INITIALIZATION COMPLETED--- 2012-10-17 07:39:24 Generated test run name: Answers TRUNK MySQL build #51 2012-10-17 07:39:24 Generated test run description: Created by Nerrvana-Jenkins plugin Revision: 142 Committer: igork Date: 2012-10-17 07:39:20 +0000 (Wed, 17 Oct 2012) Launching all tests on two platforms 2012-10-17 07:39:24 ---BEGIN UPLOADING TESTS TO NERRVANA FTPS--- [workspace] $ lftp -f upload-build-51-8414449724631519802.ftp 2012-10-17 07:40:16 -----END UPLOADING TESTS TO NERRVANA FTPS 2012-10-17 07:40:18 Creating and starting test run via Nerrvana HTTP API call...done. 2012-10-17 07:40:18 New test run ID#884. 2012-10-17 07:40:18 New execution ID#5486. 2012-10-17 07:40:18 ---BEGIN NERRVANA POLLING CYCLE (waiting for tests to complete) 2012-10-17 07:40:40 Current execution status: run 2012-10-17 07:41:02 Current execution status: run ------ Some оutput removed --------------------------------- 2012-10-17 07:44:17 Current execution status: run 2012-10-17 07:44:38 Current execution status: ok 2012-10-17 07:44:38 -----END NERRVANA POLLING CYCLE--- 2012-10-17 07:44:38 Saving execution results into results.xml... 2012-10-17 07:44:38 Done. 2012-10-17 07:44:38 ---BEGIN TEST EXECUTION RESULTS--- 2012-10-17 07:44:38 At least 43 message(s) from Nerrvana side reach(es) or exceed(s) threshold level (ERROR). Message analyzer marks execution as failure. 2012-10-17 07:44:38 -----END TEST EXECUTION RESULTS--- 2012-10-17 07:44:38 -----END PLUGIN EXECUTION--- Build step 'Nerrvana plug-in' marked build as failure Finished: FAILURE
In the first 26 lines, the plugin outputs the parameters parsed from its configuration file. In lines 28-34, the plugin creates a new test run name and description.
In lines 36-39, you see the synchronization process between tests in the Jenkins workspace and tests stored in the Nerrvana space used for this job. In lines 41-44, the plugin creates a new Test Run in Nerrvana, launches it and gets a Test Run ID and unique Test Run execution ID back from API. This will be used to monitor test status by the plugin later. The same Test Run can be launched several times, but each run will have its own unique internal execution ID. At this stage, the Test Run is visible in the Nerrvana UI. "Answers TRUNK MySQL build # 51" Test Run is created and starts running.
In lines 45-56, the plugin polls test status every 20 seconds (polling parameter), while reading Nerrvana messages, which become available in real time during tests’ execution. Once the tests are completed (lines 56-60), the plugin analyzes the errors (lines 61-65) and on the basis of the analysis sets the build status (lines 70-71). If you’re not using Nerrvana messages, the build would be marked as successful.
Another advantage of using Nerrvana messages is that the plugin can create simple reports (1).
Reports show the status of the build and message threshold level used by the plugin (2) and the description of the Test Run (3). In our example, the description contains information about the commit that triggered the tests. Platform names (4) are linked to the Nerrvana FTPS server, which will contain detailed reports generated by the tests. Finally, clicking on the last column (5) expands the table and shows the messages that your tests sent to Nerrvana, which were received and collected by the Jenkins plugin. Very often, this provides enough information for a developer to understand where the errors are, especially when he or she also maintains tests.