Testing hybrid applications

Advanced feature

Support for custom testing steps is an advanced feature of Nevercode and will require additional setup.

This is a detailed walkthrough of configuring custom testing for a hybrid mobile application. Advanced users can consult our documentation for custom testing steps instead.

Hybrid mobile applications, such as those created using the Cordova or Ionic platforms, can be tested in Nevercode via custom testing steps. Using custom scripts, you can set up and run any testing framework of your choice as part of your project builds. As long as the test results are reported in the xUnit XML format and directed to the folder specified by our environment variable, Nevercode will parse them, take them into account when setting your build's status, and display them in the Tests tab of the build view.

Configuring test runners

In order to allow Nevercode to discover and process results of your custom tests, your test runners must be configured to output standard xUnit XML files to the destination specified by the $NEVERCODE_XUNIT_RESULTS_DIR environment variable. Generally, this should be as simple as specifying and properly initializing an xUnit reporter for your test runner. Here are two examples of achieving this:

Karma test runner

The Karma test runner needs to use its junitReporter to produce xUnit-style XML output. Begin by adding it as a dependency to your project with

npm install karma-junit-reporter --save-dev

Then, configure Karma to use this reporter with Nevercode-specific settings:

module.exports = function(config) {

  config.set({
		...
    // test results reporter to use
    // possible values: 'dots', 'progress'
    // available reporters: https://npmjs.org/browse/keyword/karma-reporter
    reporters: ['progress', 'junit'],
    junitReporter: {
      outputDir: process.env.NEVERCODE_XUNIT_RESULTS_DIR,
      outputFile: "karma-test-results.xml",
      useBrowserName: true,
    },

    // Continuous Integration mode
    // if true, Karma captures browsers, runs the tests and exits
    singleRun: true,
  })
}

Since you most likely do not wish to run Karma locally with these settings, nor to have the test runner rely on Nevercode-specific environment variables, you have two options. One is to create separate config files and then explicitly pass these as arguments when invoking Karma locally vs. in Nevercode. The second alternative is to take advantage of environment variables $NEVERCODE, $CONTINUOUS_INTEGRATION or $CI, all of which are set to true in Nevercode. You can then set Karma's configuration conditionally within the same config file.

Karma errors

Please note: As of Sept. 2016, the Karma test runner will return an error if the test suite only contains failing tests. This error is either not well formed or is not parsed correctly by the junitReporter. As the result, the produced XML file is perceived as invalid by Nevercode's xUnit parsing library and no test results are reported. To avoid this issue, be sure to have at least one green test in your suite.

Protractor

Protractor requires the use of the JUnitXmlReporter to produce the desired output. Add it to your project with

npm install jasmine-reporters --save-dev

then configure Protractor

var jasmineReporters = require('jasmine-reporters');

var junitReporter = new jasmineReporters.JUnitXmlReporter({
  savePath: process.env.NEVERCODE_XUNIT_RESULTS_DIR,
  consolidateAll: false,
});

exports.config = {
  ...
  // seleniumAddress: 'http://localhost:4444/wd/hub',
  framework: 'jasmine2',
  onPrepare: function() {
    jasmine.getEnv().addReporter(junitReporter);
  },
  ...
};

Removing the setting for seleniumAddress will simplify execution of Protractor tests in Nevercode. With the setting in place, Protractor expects the Selenium server to be running in a separate process. With the setting removed, Protractor will launch an instance of the server on its own and terminate it on testing completion.

Once again, you may wish to set these configurations based on Nevercode's environment variables or in a separate .conf.js file.

Running custom tests in Nevercode

Installing dependencies

In order to run your custom tests as part of a Nevercode build, you first need to ensure that all the dependencies of your testing framework are in place. If you are already running npm install in a NC_POST_CLONE_SCRIPT script and if all your testing dependencies are included in package.json, no additional steps may be necessary. Otherwise, use npm and brew to fetch any missing dependencies. The following components are pre-installed in Nevercode builders:

  • Jasmine
  • Karma-CLI
  • Protractor
  • Chrome
  • Firefox

If you think other common testing dependencies should be included by default as well, please create a request in our forum.

Running tests

Custom tests should be run from the NEVERCODE_POST_BUILD_SCRIPT. Special consideration must be given to the fact that some test runners will exit with a non-zero value if any tests fail. If the post-build script returns non-zero, Nevercode will halt the build and label it as failed. To prevent this, append || : to the end of any test invocation command in the script:

#!/bin/sh

if [ "$NEVERCODE_BUILD_STEP_STATUS" == "success" ]
then
		karma start nevercode_karma.conf.js --single-run || :
		protractor nevercode_protractor.conf.js || :
fi

In the above example, the script takes advantage of the environment variable $NEVERCODE_BUILD_STEP_STATUS, which is exported at the end of the build step. This prevents tests from running in cases of build failures.

Once everything is configured correctly, your next build will include the custom test run, whose results will be reported in the Tests tab and taken into account when determining the build's final status

Custom test results in Nevercode

Custom test results in Nevercode