From automated tests to test reports with Xray for Jira and Jira Snapshots

Having a wide collection of automated tests can boost your release cadence. But what if running the automated tests is not enough? If you are in a regulated industry, you need to show test coverage. A traceability report that links specifications with tests and test runs. Most DevOps pipelines for automated tests produce test reports that are not linked to your stories. Consequently, you have to cobble reports from different files and platforms, scrambling to close the formatting and context gaps.

Let me show you a better way.

Here are the ingredients for a DevOps pipeline that takes you all the way from specification to test coverage report:

1. Jira: to manage our stories and specifications;
2. Xray test management for Jira: To manage tests and their links to the stories and specifications, so we have complete test coverage data. Xray provides a unified platform to manage manual tests and automated tests. Like this, test coverage can leverage both manual and automated tests.
3. GitHub or another code repository platform with DevOps capabilities. It hosts the code for automated tests and also runs them. The code for automated tests is also linked with their respective Xray test issues.
4. Jira Snapshots for Confluence: Aggregates data from Jira and Xray into beautiful coverage reports in Confluence. Note, that on Cloud, you’ll also need Traceability Extensions for Jira Snapshots. The Traceability extension provides the integration of Snapshots with Xray.

This post celebrates the recent addition of Jira Snapshots to the DevOps toolchain. It is now possible to trigger a snapshot through a secure API call. This single addition to Snapshots opens a world of possibilities. From automatic weekly status reports to automatic release documentation.

Let’s look at how to set up a pipeline for seamless traceability reports. We’ll demystify the setup one building block at a time, in the order of the pipeline.

In Jira: Select the automated tests you need and trigger the pipeline

Jira is where we manage our specifications. You may use Story issues, or a dedicated issue type like “Functional specification”. For simplicity, We’ll refer to “Story”.

Test coverage starts by linking each Story with an Xray Test issue, via the link “is tested by”:
the Story → is tested by → the Test

Xray differentiates between manual test scripts and automated tests by the field “Test Type” . Automated tests have Test Type=”Generic”. Manual tests are executed entirely within Jira, while the “Generic” tests are proxies, representing code executed elsewhere. In our case, our automated tests use the Pytest framework.

We establish the link between the test code in Pytest and the Jira issue representing it via a Pytest annotation preceding each test. Check out more details about the GitHub pipeline implementation in this post: How to Integrate Automatic Tests with Xray to shorten release cycles.

To execute the automated tests, we create a Test Execution issue in Xray, and add to it the Tests in scope. All those tests will be of type “Generic”. We label the test execution with the label: “run-automation”. In our setup, this label flags to Jira Automation that this Test Execution issue triggers the GitHub Action.

To trigger the test execution, the user transitions the Test Execution issue from status “To Do” into “In Progress”. A Jira Automation rule listens to the transition events and fires the GitHub Action.

The Jira Automation Rule

The structure of the rule:

1. When: Issue transitioned
2. IF: Two conditions that filter only for:
   1. Test Executions issues
   2. Labeled with ‘run-automation’
3. Action: Send the web request to trigger the GitHub action. See more details about the web trigger
4. Action: Transition the Test Execution issue into DONE

 

The Web Request to trigger a GitHub Action

The payload that is sent to GitHub includes:
The test execution issue: we need this to fetch the list of tests that are in scope.
The Snapshots API endpoint and API key: GitHub uses these to update the traceability report.

The authorisation of the web trigger is achieved via a GitHub personal access token.

A GitHub Action runs the test, creates the new traceability report in ConfluenceThe GitHub Action handles the heavy lifting of the pipeline. Beyond running the automated tests, there are three crucial integration steps:
It fetches the tests that are in scope of the test execution.
Once the test runs complete, the Action imports the result into Confluence. This includes any test evidence, such as code snippets, results text and screenshots.
The last step is a POST command that triggers an update of the traceability table (see the code snippet in our documentation). The traceability table is built by the Jira Snapshots macro in Confluence. Read more details in the next section.

We provided the complete details of the first two steps in this post: How to Integrate Automatic Tests with Xray to shorten release cycles.

Using Jira Snapshots to define a traceability report exactly as you need it

Now, we’ll bring this full cycle and have all the data packaged into a Traceability report. In Confluence, we create a Confluence page with a Jira Snapshots macro. We configure the macro to reflect our hierarchy. Three levels like this:

1. Stories are tested by
   1. Test issues that are executed, resulting with
      1. Test runs. We include in our coverage the most recent test run.

With the Confluence page created, we only need to enable the api endpoint so that we can call it from our GitHub action.

Conclusion

And what’s the cherry on top? When Jira Snapshots updates the page, it creates a new page version. Confluence notifies stakeholders who are watching this page. All the different bits working together in harmony.
First, no one had to author this new traceability report. Second, users can now check the coverage status in Confluence. They did not need to understand yet another tool or browse through log files. It’s all there in their familiar tool, contextualized into the product specifications.
With so much goodness, it’s no wonder we’re seeing more and more users using the Jira Snapshots API endpoint. I hope you’ll check this out too!