How to Integrate Automatic Tests with Xray to shorten release cycles

Written by Rina Nir

Company name

NEC OncoImmunity AS

Industry

Biotechnology

About the company

NEC OncoImmunity AS is an artificial intelligence (AI) driven biotech company offering proprietary machine-learning based software, which addresses the key knowledge gaps in the prediction of bona fide immunogenic antigens for personalized cancer immunotherapy and infectious disease vaccines. The AI technology can be used to identify optimal antigen/epitope targets for infectious disease vaccines, truly personalized cancer vaccines & T cell therapies in a clinically actionable timeframe, and also facilitate effective patient selection for cancer immunotherapy.
NEC OncoImmunity AS is a subsidiary of NEC Corporation, and is based in Oslo, Norway.

NEC OncoImmunity is a team on a mission to improve cancer treatment, so they cannot afford to waste time. Top quality and efficiency are what’s required, driving a strong focus on good processes. It was great hosting them on our recent webinar. Product manager, Susanne Tande, and Frederik Larsen, NEC OncoImmunity’s’ principal software engineer, told us how they integrated their automatic tests with Xray Test Management For Jira to reduce their manual test efforts and speed up their releases. This article is a follow-up article to that webinar.

Xray introduces Native Quality Management, a new, holistic approach that naturally embeds the test management process into the development workflow, right inside Jira. That way teams work side by side to release consistently quality software — together. By integrating the test automation flow with Xray, we take Native Quality and collaboration to the next level.

We provide the technical details and code snippets. Thanks to Fredrik, who provided the details we share here.

Why the verification cycle was sub-optimised and how to fix this?

The team invested heavily in implementing many automatic tests. They implemented the tests in Pytest and ran them with GitHub Actions. When executed against a comprehensive input, these tests provide good coverage for many of the product aspects.

But, how the GitHub Action was set, the tests did not generate a good report. They could not use these executions in the regulatory Technical File. So, the team needed to carry out manual tests. The regulatory verification cycle was entirely manual and provided the evidence required for compliance. This led to a long verification cycle at the end of the development effort. It was also using precious resources without contributing to product quality.

To remove this waste, automatic tests will need to generate regulatory records. The idea was that automatic tests and manual tests are recorded on the same platform, where they are linked to the specifications. Once this data is in Jira, its easy to use them for verification test report and traceability matrix. Its especially easy because they use Jira Snapshots for Confluence to create these reports.

With this setup, many of the manual tests will no longer be required- saving time and freeing resources to focus on more impactful things.

The DevOps pipeline that integrates automatic tests with Xray

With several tools we defined carefully the role of each. Then we designed the handovers from one to another:

What will start the cycle of automatic verification tests? We did not need the verification cycle to run on each build. As the cycle is pretty lengthy we would run it less frequently. We would want to have an easy control over the scope of test. With Xray, we’ll define a Test Execution item in Jira and cherry pick the Tests to execute. Once we link the tests to the Test Execution, triggering of the automatic cycle is only one click away.
How will tests be identified across Jira/Xray and GitHub? In our implementation, we assumed that the test exists both in Pytest and Xray, before the execution. For each automatic tests in our verification suite, we created a Test issue in Jira/Xray. We’ll annotate each automatic test (in the codebase) with the Test issue key that represents it in the Jira/Xray project. (@pytest.mark.xray(“TEST-46”)). This works well with defining the scope of the Test Execution in Jira. In the future, if we need to support other flows, we might expand the implementation. For example, to support the automatic creation of Xray tests for new pytest tests.
Capturing test evidence: From compliance perspective, we needed to include the pytest code and the execution log of the test in Jira/Xray. This required a custom extension, which we added at the final stage when tests results where uploaded into Xray.

So here is the DevOps sequence:

Pipeline to integrate Automatic tests with Xray

We’ll look closer at the steps indicated as 1 to 5, and provide the code snippets and technical juice.

Stepstone 1: Triggering the automatic execution from Jira

To trigger the cycle, we define an automatic action in Jira Automation. A Jira user triggers the automation manually from the Test Execution. The automation action is of type “send web request”. Jira Automation sends the web request into GitHub and triggers a GitHub action. This is the template for the command:

To authenticate with GitHub, we used GitHub Personal Access Token (The value field for “Authorization” needs to be “Bearer <PAT>” where <PAT> is replaced by your github personal access token). The only parameters that are sent to GitHub are the Test Execution Issue Key and Issue ID.

Stepstone 2: Starting the test execution in GitHub

The GitHub action starts by authentication against Xray. This authentication sets the baseline for the next stepstone, where the GitHub action retrieves data from Xray. On setup, we generated Client ID and Client Secret in Xray. These are then preconfigured as secrets in GitHub and are available in the GitHub action workflow, where they are set as environment variables Note that the XRAY_CLIENT_ID and XRAY_CLIENT_SECRET environment variables are used by the pytest-jira-xray plugin (discussed later ), in addition to the more visible usage by the get-jira-keys command.

Here is the GitHub actions workflow:

name: Verification Test


on:
  workflow_dispatch:
		inputs:
			xray_execution_issue_key:
			description: 'Xray execution key'
			required: false
			default: 'no_execution_key'
			xray_execution_issue_id:
			description: 'Xray execution id'
			required: false
			default: 'no_execution_id'


jobs:
	build:
		runs-on: self-hosted
		permissions:
			contents: read
			id-token: write
	env:
			PYTHON_VERSION: '3.10'
			XRAY_CLIENT_ID: ${{ secrets.JIRA_CLIENT_ID }}
			XRAY_CLIENT_SECRET: ${{ secrets.JIRA_CLIENT_SECRET }}
			TEST_EXEC_KEY: ${{ github.event.inputs.xray_execution_issue_key }}
			XRAY_API_BASE_URL: 'https://xray.cloud.getxray.app'
	steps:
		- uses: actions/checkout@v3
		- name: Set up Python ${{ env.PYTHON_VERSION }}
			uses: actions/setup-python@v4
			with:
				python-version: ${{ env.PYTHON_VERSION }}
	- name: Install dependencies
		run: |
			make dev
	- name: Run verification tests
		run: |
			JIRA_KEY_FILTERS=$(venv/bin/python xray/xray_tools.py get-jira-keys \
			--test_exec_key "${{ github.event.inputs.xray_execution_issue_id }}" \
			--client_id "$XRAY_CLIENT_ID" \
			--client_secret "$XRAY_CLIENT_SECRET")


			make verification-test TEST_EXEC_KEY="$TEST_EXEC_KEY" JIRA_KEY_FILTERS="$JIRA_KEY_FILTERS"

Stepstone 3: In GitHub, fetch lists of tests from Xray

In order to fetch the list of tests (their corresponding Jira issue keys), an access token for the Xray API is needed. Note that the authentication endpoint is a REST API endpoint, while the remaining endpoints discussed are GraphQL:

BASE_URL_REST = "https://xray.cloud.getxray.app/api/v1"
BASE_URL_GRAPHQL = "https://xray.cloud.getxray.app/api/v2/graphql"

Once the access token has been obtained, we fetch the list of tests to run. This is done in two steps, against the Xray API:

First, we get the list of issue ids for the tests that are part of the test execution we triggered.
Second, we obtain the issue keys that map to each of the issue ids we just fetched.

Note:
The second step to obtain the Jira issue keys is needed, since that is how we are marking our tests.

@click.command(name="get-jira-keys")
@click.option("--test_exec_key", required=True, help="Xray Test Execution key.")
@click.option(
"--client_id", required=True, help="Client ID for xray API authentication."
)
@click.option(
"--client_secret", required=True, help="Client secret for xray API authentication."
)
def get_jira_keys(test_exec_key, client_id, client_secret):
    access_token = get_token(client_id, client_secret)
    jira_keys = get_test_keys_from_execution_id(test_exec_key, access_token)
    jira_key_filters = ",".join(jira_keys)
    click.echo(jira_key_filters)

def get_token(client_id, client_secret):
    auth_url = f"{BASE_URL_REST}/authenticate"
    auth_data = {"client_id": client_id, "client_secret": client_secret}
    data = make_post_request(auth_url, {}, auth_data)
    access_token = data.strip('"')
    return access_token

def get_test_keys_from_execution_id(test_exec_key, access_token)
    test_issue_ids = get_test_issues(test_exec_key, access_token)
    jira_keys = get_jira_keys_from_issue_ids(test_issue_ids, access_token)
    return jira_keys

def get_test_issues(test_exec_key, access_token):
    headers = {"Authorization": f"Bearer {access_token}"}

    # fetch the tests issue ids associated with the test execution
    test_exec_query = """
    query TestExecutionQuery($testExecKey: String!) {
        getTestExecution(issueId: $testExecKey) {
            tests(limit: 100) {
                results {
                    issueId
                }
            }
        }
    }
    """
    variables = {"testExecKey": test_exec_key}
    data = make_post_request(
	      BASE_URL_GRAPHQL,
        headers,
        {"query": test_exec_query, "variables": variables},
    )
    test_issues = data["data"]["getTestExecution"]["tests"]["results"]
    test_issue_ids = [issue["issueId"] for issue in test_issues]

    return test_issue_ids

def get_jira_keys_from_issue_ids(test_issue_ids, access_token):
    headers = {"Authorization": f"Bearer {access_token}"}


    # fetch test details (jira keys) using GetTestsByIds query
    test_details_query = """
    query GetTestsByIds($issueIds: [String]) {
        getTests(issueIds: $issueIds, limit: 100) {
            total
            start
            limit
            results {
                issueId
                testType {
                    name
                }
                status {
                    name
                }
                jira(fields: ["key"])
            }
        }
    }
    """
    variables = {"issueIds": test_issue_ids}
    data = make_post_request(
        BASE_URL_GRAPHQL,
        headers,
        {"query": test_details_query, "variables": variables},
    )
    test_details = data["data"]["getTests"]["results"]
    jira_keys = [test["jira"]["key"] for test in test_details]


    return jira_keys

Stepstone 4: In GitHub, run tests on Pytest

Moving on to execute only the requested tests.

Here is how our pytest command looks like:

pytest --log-cli-level INFO --jira-xray --cloud --execution "$${TEST_EXEC_KEY}" --client-secret-auth -m "verification" --test-cases "$${JIRA_KEY_FILTERS}" tests

In order for pytest to parse our –test-cases argument, we implemented the pytest_addoption method in the test root conftest.py file to store the supplied test case names (issue keys), and then filter the list of tests in the added the pytest_collection_modifyitems hook.
By keeping the pytest command inside a make target, we are able to keep the interface quite simple still:

.PHONY: verification-test
verification-test:

$(venv)/python -m pytest --log-cli-level INFO --jira-xray --cloud --execution "$${TEST_EXEC_KEY}" --client-secret-auth -m "verification" --test-cases "$${JIRA_KEY_FILTERS}" tests

Enables:

name: Run verification tests
run: |
	JIRA_KEY_FILTERS:$(venv/bin/python xray/xray_tools.py get-jira-keys \
--test_exec_key "${{ github.event.input.xray_execution_issue_id}}" \
--client_id "$XRAY_CLIENT_ID" \
--client_secret "$XRAY_CLIENT_SECRET")

make verification-test TEST_EXEC_KEY="$TEST_EXEC_KEY" JIRA_KEY_FILTERS="$JIRA_KEY_FILTERS"

Here are the two confest.py snippets in play. Together the are responsible to fetching only the tests that are in scope:

def pytest_addoption(parser):
    parser.addoption(
        "--test-cases",
        action="store",
        default=None,
        help="Run only specified tests (comma-separated).",
    )

def pytest_collection_modifyitems(config, items):
    def filter_test_cases(item):
        xray_marker = item.get_closest_marker("xray")
        # Check if there is an Xray marker
        if not xray_marker:
            return False

        return True if xray_marker.args[0] in selected_test_cases_option else False

        # Get the custom marker from the command line
        if config.getoption("--test-cases"):
            selected_test_cases_option = config.getoption("--test-cases").split(",")
            selected_test_cases = list(
				        filter(lambda test_case: filter_test_cases(test_case), items)
			      )
			      items[:] = selected_test_cases

Stepstone 5: In GitHub, import test results into Jira/Xray

Xray provide official support for Java.
As NOI are working with Pytest, they opted to use this third party plugin, that largely provides similar features.
Once the plugin is installed, each test can be annotated like this:

@pytest.mark.xray(“TEST-46”) where “TEST-46” is the related test in Jira/Xray.

While most of the import is done by the pytest-jira-xray plugin, we needed to extend the implementation so that the test command, test code and log output will be added to Xray as evidence.
First, we set a Pytest fixture that runs the cli command to produce output that is asserted. It attaches the log output and test command to the pytest item so that it is available later for inclusion in the test report.

tests/verification/conftest.py

@pytest.fixture(scope="session")
def cli_output(request) -> Generator:
    """
    pytest fixture for running a cli command
    and obtaining the output.
    """


    params = request.param


    output = run_cli(**params)
    log_string = ""
    if output["stdout"]:
    log_string += output["stdout"]
    if output["stderr"]:
    log_string += output["stderr"]


    # Store the stdout and cli command in the pytest session
    request.node.session.stdout_capture = log_string
    request.node.session.test_command = " \\ \n".join(output["cmd"])
    yield output

In the pytest_runtest_makereport hook, the test command, test code and log output are fetched from the pytest object and attached to the pytest report generated for each test. It’s now ready for being submitted via the Xray API. This is handled by the pytest-jira-xray plugin.

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()
    evidences = getattr(report, "evidences", [])
    if report.when == "call":
    if not hasattr(report, "wasxfail"):
        test_code = inspect.getsource(item.function)
        command = item.session.test_command
        test_code_with_command = test_code + "\n\nTest Command:\n\n" + command
        test_name = item.name
        evidences.append(
            evidence.text(
                data=test_code_with_command, filename=f"test_code_{test_name}.txt"
            )
        )


        log_string = item.session.stdout_capture
        evidences.append(
            Evidence.text(data=log_string, filename=f"log_{test_name}.txt")
        )
			  report.evidences = evidences

From test results in Jira/Xray to release documentation and technical file

Once the sequence is completed, all the execution information is stored safely in Jira. There is no longer need to repeat the same test coverage with manual tests. The manual test cycle now includes only those tests that are not appropriate for automation, or any scope that is not yet covered by automatic tests.
These tests are designed and reported directly in Jira and Xray.
Once the cycle is complete, for both manual and automatic tests, we generate the test documents in Confluence. And that’s the breezy part of the process.
This last step is done in Confluence, and achieved with Jira Snapshots for Confluence (and Jira Snapshots Traceability Extension). With these Apps, it’s easy to create the test reports and the traceability reports that are required for the regulatory Technical File. Automatic and manual test are “equal citizens” in this report, complementing each other in creating the full picture of all Verification Tests.

Don’t miss these