Test coverage and code quality are two of the most important indicators for analyzing, tracking, and evaluating the success of an IT project or campaign.
In a manner that few other measures are, test coverage and code quality are inextricably intertwined. One of the ways we estimate code quality, for example, is by looking at test coverage. However, there are concerns about the effectiveness of using test coverage measures to assess code quality.
So, in this post, we’ll examine this practice critically. We’ll do so by going over the widely held belief that test coverage metrics are a good way to measure code quality, as well as how to implement a solution that works for you.
Elephant in the room – Test Coverage
Let’s start with the elephant in the room, which is Test Coverage.
“What Is an Appropriate Level of Test Coverage?”
True, having enough test coverage increases your chances of catching all critical/high-priority bugs. This isn’t just a grandmother’s tale; it’s also a pretty good measure for determining the code quality of your IT system.
However, the main point of contention in the industry regarding test coverage appears to be the question of “how much is adequate?”
Theoretical percentages range from 60% to 80% to even 100%. Each proponent has a compelling case for why they believe a given level of test coverage is sufficient or insufficient.
The one thing no one seems to be talking about is how you measure test coverage.
Don’t get me wrong: everyone has a standard method for calculating test coverage: the number of lines of code in the software under test.
Is that, however, the appropriate measure? That is precisely the question we will try to answer today.
“What Do You Measure Test Coverage Against?” is a question that no one seems to be asking.
Test coverage formula – How it is measured against lines of code
So, what is the formula for test coverage and how does it work? All you have to do is:
(A) the total amount of lines of code in the piece of software you’re testing, (B) the total number of lines of code all test cases currently execute, and multiply (B divided by A) by 100 to get your test coverage percent.
As an example,
If a system component has 1000 lines of code and 650 lines of code are actually executed through all current test cases, then your test coverage is:
(650 / 1000) * 100 = 65%
When measured by the number of lines of code executed, what is considered “adequate” test coverage? The general agreement is roughly 80%, with important systems having a higher percentage (definition of critical may vary by industry, geography, user base, etc.).
The crucial question is whether or not this statistic is effective.
That’s a difficult one. I looked for an answer on the internet forums and discovered that most people believe 80 percent is sufficient. Then there’s the issue of whether you’re using decent tests rather than ones that aren’t particularly beneficial for coverage.
What constitutes a good test?
That is not a tough question to answer. A good test will look to see if a requirement has been met. Flows that are both cheerful and unhappy can be good testing.
So, how do you make sure you have mostly good tests in order to improve test coverage?
Before we get to it, let’s take a look at the more significant question.
Test coverage metrics in the world of Test-Driven Development
Sprint – Day 1
The planning session is used by my scrum team to identify the top x stories in the backlog for delivery. Each tale has been fleshed out and story points have been established in preparation for the planning meetings (through ongoing backlog grooming sessions). A Scrum Tester (or testers) takes the newly created sprint backlog and maps out all of the test scenarios that their test cases will cover. These situations are then sent to the developers, who are already hard at work writing code.
Sprint – Day 2
By this point, scrum developers have made some progress with dev planning and kick-off, and have had a chance to talk with the BA and testers about the sprint’s requirements and test scenarios. To cover all test situations, the testers have finished scripting new test cases or copying old test cases from a case repository, if appropriate. They’ve also discovered and prioritized both Happy and Unhappy flows. Finally, they established traceability by mapping the test cases back to the original need. The testers will know if all of the requirements for the current sprint have been covered by test cases at this time.
Sprint – Final Day
Scrum developers employed test scenarios and thorough test cases to guide their coding efforts, and they also used these test cases to unit test the code. Most of the easy-to-find, unsightly defects have been detected by the time they are deployed to an Integrated Test Environment for end-to-end functional testing. Scrum Testers have also started to use these test cases to run end-to-end tests, assisting in the identification of high/critical issues that prohibit a story from being delivered in that sprint. Most high and critical bugs will have been corrected by the time we get to the Demo and Retrospective, with other unresolved bugs being prioritized for future resolution as needed.
As you can see, towards the end of a Sprint, or a Release, my team has a system in place to ensure that all requirements are adequately covered by tests and that we achieve Exit Criteria from Testing to Release (e.g., no high/critical problems, 100% test case execution, 95% tests passed).
This is entirely possible with Test-Driven Development. And if you can present a working, almost-bug-free product every Sprint, followed by mop-up Spikes to repair the remaining unseemly problems, my experience suggests you’re ready to release.
So, What Should Code Quality be really measured against?
My opinion is that both coverage and the number of lines of code executed are required.
Why? To be honest, I believe that achieving 100 percent coverage of standards is sufficient. After all, with Agile, you’re intended to spend all of your time working on the highest-priority needs for a release, decreasing waste – in terms of effort, time, and money.
Humans, after all, aren’t perfect. It’s possible that bad requirements will find their way into your scope. It’s possible that important requirements will be overlooked. It’s possible that you won’t realize this until the end of your project. With requirements, we can’t always guarantee 100 percent accuracy.
On the other hand, I’ve found that looking at test coverage in terms of lines of code executed is a good way to measure code quality on a surface level. For example, even if your tests cover 100 percent of your requirements and satisfy all exit criteria for release, only 65 percent of your code is covered in your tests.
I agree that this is an extreme example. However, this is very likely. It could also be a combination of shoddy requirements and squandered development time.
It’s not uncommon to come across projects where engineers have created far more code than is required. There could be various reasons for this, some of which are valid, but I consistently find that ‘excess code’ is an issue for many of my clients.
Developers waste valuable resources writing unneeded lines of code when their coding effort deviates from the requirements.
Is it true, as a consequence, that if a developer closely adheres to the agreed-upon scope, all of the code will be covered by tests? Certainly not. Again, this is due to a lack of training for developers on how to design efficient code – that is, just write code that is required to fulfill a requirement. Even so, there may be circumstances in which 100 percent coverage is neither possible nor necessary.
What did we learn?
Everyone is concerned about the quality of their code. In an era when resource costs are rising and budgets are tightening, it’s critical to analyze and measure code quality to better understand your team’s efficiency and where they can improve.
To reach your desired code quality metrics, you should use every tool at your disposal. One way is to compare the number of executed lines of code to the total lines of code. However, this alone will not assist you in determining proper test coverage or real code quality.
You should use additional, and sometimes more sophisticated, methods to assess code quality and test coverage. You can easily do this by including requirements coverage in your tests, which will result in greater code quality almost immediately!
For more info: https://www.qaaas.co.uk/testing-services/
Also Read: https://www.guru99.com/software-testing.html