API Contract Tests

These tests are designed to look for changes to the recipe server API that are not expected.

Before running these tests you will need the following installed:

Before running these tests you need to create a virtual environment using Python 3.6.0 or greater. To create it do the following:

virtualenv -p /path/to/python3.6 venv
source venv/bin/activate
pip install -r requirements/default.txt

This creates the virtual environment, activates it, and installs the Python dependencies needed to run the tests.

To run these tests, use the following command from the root project directory.

py.test -v --server=<server> contract-tests/

where <server> is one of

  • https://normandy.stage.mozaws.net
  • https://self-repair.mozilla.org

If you want to send results of the test run to TestRail, you need the following pre-requisites:

  1. You have VPN access to the staging environment
  2. You have an account on https://testrail.stage.mozaws.net
  3. Copy testrail.cfg.dist to testrail.cfg
  4. Modify that TestRail configuration file with your email, TestRail password and TestRail user ID

Then, for any test case you want to report results for, make sure the script file the test is in imports the proper TestRail module:

from pytest_testrail.plugin import testrail

and then you use a decorator to indicate this test case will be reporting results. Here’s an example:

def test_expected_action_types(conf, requests_session):

The ID for the test case (C5603 in this example) is generated by TestRail when the test case is created.

To run the tests and report the results to TestRail, use the following command:

py.test --server=<server> -v \
--no-ssl-cert-check \
--testrail=contract-tests/testrail.cfg contract-tests/

The values for <server> are the same as before.

–no-ssl-cert-check tells pytest to not check if the SSL certificate for the TestRail install is valid – we are using a self-signed certificate and the underlying code throws an exception unless we tell it not to check.

–testrail= tells pytest that we wish to report the results of TestRail decorated tests and where we can find the configuration file.