Testing is not a solitary effort focused on a single person’s goal. Whether you use Excel, Xray for Jira, or another tool, each executed test is a pillar that attests to the quality of the product and helps decide if it’s ready to ship.
On 19 July 2024, CrowdStrike released a content configuration update for its Windows sensor. The impact was felt around the world. The system crash brought travel to a screeching halt and prevented hospitals from administering care. This experience was very humbling and a stark reminder that moving fast cannot be decoupled from safety and quality. Later, CrowdStrike said the bug was caused by a simple mismatch between the number of input fields the sensor expected and the number provided by the update.
Its conclusion was simple: we need more testing and testing visibility.
In this article, we’ll take a closer look at how testers and managers can enjoy superior test visibility, right within Jira. It’s based on a recent webinar, where RadBee CEO Rina Nir was joined by Swarmit ‘s Anja Fuchs and Xray’s Tonislava Docheva to learn how you can leverage the Atlassian Ecosytem to transform your testing experience.
What does good test management look like (and what it doesn’t)
Good test management requires collaboration, visibility, and coordination across many disciplines, including software quality assurance, product managers, developers, project managers, and senior management. This requires access to a shared source of truth. But the experience on the ground is often far from this ideal.
“The reality looks like Excel spreadsheets,” said Anja. “These are sent out through Outlook and sent back, and it’s some poor person’s job to sum everything up.” Organizations using PowerPoint and Excel will spend hours and hours creating reports and copying graphics. But then, when they want to look back months later, they’ll be wading through multiple sub-folders to find what they need.
Archaic methods aren’t the only issue. Sometimes teams do have special tool sets for testing, but these tools aren’t connected or integrated, making traceability just as difficult. And in pharmaceutical organizations in particular, there’s a prevalence of expensive ALM systems, like HP ALM or DOORS. These limit who has access to the system, creating another layer of siloed working between development – which is typically taking place using Jira – and test management.
“Sometimes organizations have invested so much in these systems that it hurts them to accept that they really need to go and get some new tools.” But if CrowdStrike has taught us anything, it’s that testing visibility should take priority, and finding a streamlined, traceable, integrated solution is essential.
Test management on Jira – same place where work happens
The Atlassian Ecosystem lets you bring your testing into Jira – the same place your developers are organizing their projects – helping eliminate siloed working. Jira is a fantastic agile tool, and it’s even more powerful when used with Xray.
Using Xray for Jira ensures testing quality isn’t only the responsibility of your QA team. It gives your developers one cohesive tool stack to manage the full software development life cycle, enhancing collaboration, and improving visibility. Our solution also includes Snapshots of Jira Data into Confluence, which adds even more reporting capabilities with one-click release documentation. And even better, test automation frameworks can be seamlessly integrated so they feed into the same reporting tools (and here). Visibility across all your test modalities.
Let’s take a closer look at how Xray, Jira, and Snapshots work together to transform your testing experience.
Xray: the smarter way to manage testing
“Xray uses Jira native items for managing testing entities,” explained Tonislava. “And this gives you full traceability between stories, requirements, tests, and test executions.” It also offers instant reporting and compliance support. And, importantly, it’s easy to customise to fit your needs.
“You’re able to have live data for real-time visibility and customized dashboards. And you’re able to see the different levels and statuses of your testing progress and your projects in general. This level of traceability is very important for companies in highly regulated industries or wherever its vital to ship stable products.”
Story-centric test management
Unlike working in Excel files, Xray helps you separate your test management from test execution. You have a work item for the test and you can manage its life cycle – From the moments a scope for the test is defined, through its development and then- when it can be used for testing the product. A test can remain valid over many product release (for regression tests), and then eventually be deprecated when the aspects it tests are changed. A test execution is a seperate work item, documenting an even where a bunch of Test issues have been executed against a version of the product. The test execution has test runs where you see if the test actually passed, or if there are any defects.
With Xray you can define a requirement within the context of your Jira issue types – so that can be a story, an epic, a feature, or any other work item type. “You can also include “parental” connections, for example, between epics and stories,” explained Tonislava, “where an epic will be fully tested, based on the requirements within the stories.”
This is really useful for developers and product managers, who want to get a story-centric view of what’s happening with testing. “They can have full traceability – seeing how a story is tested, the test coverage, and its status – without having to leave their view,” explained Tonislava. “A story can be tested by more than one test, and there might also be scenarios where a test will apply to more than one story.”
Xray reports we love
For good est management visibility you need good reporting. With all your information centralized in Jira, reporting is what enables you to make decisions and answer key questions, including, are you ready to make a release? What is the status of your release? And are there any areas in which we need to improve? Xray makes finding these answers much easier.
There are plenty of reporting options to explore out of the box. Here are three to focus on:
- Test coverage: “This report just gives you a very powerful overview of your coverage,” said Tonislava. “So you can see if you still have high-priority tests that are failing.” It gives you visibility into your coverage, which you can group how you like, for example, by priority. You’ll get a good idea if there are bugs that still need to be resolved and if you’re ready to make a full release.
- Traceability: This report is really great for sharing compliance documentation. You can set it up to suit your needs, such as by final status, choose the project you want to focus on, and generate the report in an instant. It will give you an overview of everything that’s happening in your project, with filtering options to get clearer analysis. If a bug’s been found, for example, you can easily see which version it was in.
- Test runs list: This report lets you explore all your different test plans and their statuses. “You can see just the amount of test executions that you have, the test runs, the tests and test type, and of course the defects,” said Tonislava. This report is really great for companies that are working on multiple test plans or multiple projects that have separated it into multiple test plans.
Follow the North Star with Xray dashboards
As soon as you install Xray, you can use the dashboard gadgets related to your test management data. Dashboards create a unified, shared, and consistent format for viewing and understanding progress. Over time, teams get used to seeing a certain dashboard and that becomes its North Star.
Anja shared a few of her favourite dashboard gadgets:
Different stakeholders, different dashboards
It’s useful to prepare dashboards so different stakeholders can find the information they need easily. Just like with test coverage reports, you can configure gadgets in a way so you can view testing coverage in the dashboard across projects based on priority, status of requirements, or something else.
Testing overview in your dashboard
A test manager may want to see all test plans, when tests were started, when they were due, when testing was finished, and the overal status. And then the execution within those tests. With Xray, Jira dashboards can provide that detailed level of visibility. Test managers can easily scan test execution lists and the status of test runs within each execution and environment.
“Using filters you can view the data that’s most important to you,” explained Anja. “You can then jump into a specific execution or test right from the dashboard.”
Helping testers stay on track
Anja also pointed out the dashboard advantages for testers. As a testing user, you can see all the tests you need to work on, the environment in which you need to run them, and the date when the tests are due by. You can also see all the tests you’ve already completed, so you don’t have to manually keep track of everything.
Bring your Jira data into Confluence
With so much information inside Jira, you might be tempted to keep it all there, but there are two really important reasons to consider using Snapshots of Jira data into Confluence. First, your stakeholders might struggle to find the correct dashboard or run a specific query in Jira, but they still need visibility of your testing process. Snapshots provides a clear view that is friendly to non-Jira users. Second, it’s a super-speedy way to create release documentation.
Snapshots is installed on Confluence and retrieves data from Jira into Confluence so anybody with access to the page sees the data.
The visibility they need
A Snapshots table inside Confluence shows a hierarchical view with all your epics, which are the parents of your requirements, and the stories. You can see the relevant tests against each story and its status. Each time you create a new snapshot, it’s stored in Confluence and you can compare it with the previous snapshot with just one click, making it really easy for stakeholders to see what’s changed.
Ramping up release documentation
Because it’s a static take of your Jira data, no matter what updates are made in the future, you can use the snapshot to demonstrate full traceability from stories all the way down to test runs. You can easily define which levels you want to see and which fields you want to capture in each of those levels.
“We have reports from customers who say doing their traceability matrix this way is six times faster,” explained Rina. “If you think about it, it’s six times faster at the very end of the release. That’s when you don’t want to get delayed and you need to see that you’ve covered all your requirements. You can document it easily and send the release out.”
Conclusion
If you’re using Excel, Outlook, and PowerPoint to keep track of tests and handle reports, there is a better way. Incidents like CrowdStrike have shown the critical nature of effective test management and visibility. By using Jira as your centralized test management system, supported by Xray and Snapshots, you have a single integrated platform that brings requirements engineering, development, test management and reporting together in one place.
And with Xray and Snapshots’ built-in features, traceability is a given. Once you’ve got the right requirements and created tests, Xray takes over and links everything together, giving you the overview you need to make decisions. With Snapshots, you also get one-click reporting, saving you so much time, and a user-friendly way for all stakeholders to access your data and stay up to date.
