One of these software testing situations may be recognizable to you.
After a year of watching the tests slow down week after week, someone finally understands that testing is limiting delivery and insists that something be done.
Congratulations. You are a unique individual.
Or perhaps you’ve just begun a test tooling journey, and the tests are going smoothly, and you’d like to keep it that way.
In any case, we thought you may use some assistance.
Remove sleeps and triage tests
Teradata’s staff software developer, Dave McNulla:
I despise having to wait for test results! If tests are taking too long, I try to execute them in parallel (or expand parallel execution), which necessitates the tests being totally self-contained. It costs money to add more systems, whether cloud or on-premises hardware, but it also costs money to make developers wait for results. This includes labor expenditures, lost opportunities, and the cost of a company delay. It’s a triple whammy.
I frequently hunt for “sleeps” in test-ware to speed up tests. They are a waste of time and a sign of a shaky test. Because the test author has no idea what to expect, he or she ends up waiting for a long time. If the test is critical, there are methods for polling more frequently so that the test can restart sooner.
Following that, I look for low-value tests that I can run less frequently. The likelihood of failure is combined with the priority of a defect discovered by a failure to determine the value. We can run a test less frequently, say overnight or weekly, if it never fails or only finds low-priority defects (or is redundant with another test).
Push tests down the pyramid
Tim Ottinger, Industrial Logic’s senior technical consultant:
One way to speed up slow tests is to sacrifice integration. If a database query is required by the system under test, replace it with a mock that returns a prefab dataset. This saves network and database time, and it should enough as long as your dataset fulfils the requirements your database query requires. Slow tests should be moved down Cohn’s test pyramid. Could it be an integration test if you’re performing a system test? Could a few good microtests take the place of the integration test? The faster an exam is, the lower the level it is.
Also, don’t just use Gherkin to control the GUI. We’ve found that Gherkin-based tests are a wonderful way to test an application’s user interface. It will call the same functions as a genuine GUI, but at a considerably faster speed.
Also, keep in mind that if your tests include launching a whole application with all of its accompanying services, you’ll be dealing with lengthy startup and shutdown times. If at all possible, avoid creating a full environment, or do it only once for the entire test run, rather than for each test case.
The faster your tests run, the more frequently they’ll be run, which is an important performance parameter for engineers. They don’t conduct the tests because they don’t catch any abnormalities, and the less often they run them, the more difficult it is to figure out what’s wrong. Fast tests are run frequently, which allows developers to work in smaller, more secure increments.
Finally, keep small datasets and in-memory databases in mind. This one simple trick has paid off handsomely.
Shift together
Micro Focus’s chief technologist for application delivery management, Don Jackson:
Some argue that shifting testing to the left, toward application development, is the best way to speed up testing. Others believe it’s moving to the right. What must happen, though, is a collective adjustment. Organizations must automate testing and include it into their continuous integration and delivery (CI/CD) pipelines. All three personas with a stake in test automation must contribute to that automation: the traditional QA automation engineer, the domain expert/business analyst (shift right), and the developer (shift left).
To limit business risk and guarantee that checkpoints are not executed more than once for the same item in the same state, companies should focus on business-critical process flows/paths while creating tests. Furthermore, the pipeline must make use of distributed testing ideas, which allow the suite of tests to be run simultaneously on several nodes. Finally, exploratory testing should take place during the team’s execution of the pipeline, providing the human/unpredictable element to the testing and catching those pesky bugs before they’re released.
Create faster feedback loops
CloudBees’ vice president of product marketing, Marisa Shumway:
Feature flags can help you improve your testing in a number of ways. QA, for example, can utilize the flags to disable behavior so that it can be compared to previous releases. Getting a developer to alter the code and make a fresh build without this could entail a wait of hours or days, and the test may never run. The flag override view allows QA to reduce the feedback cycle by allowing them to alter flags in the environment with a single click. QA may now work alone, without interrupting developers’ focus or wasting valuable business cycles. If there is an issue once the flag is set, a rollback is a straightforward configuration update. This eliminates the need for retest/rebuild, allowing for a rollback in seconds rather than hours.
Time tests for quick performance indicators
Synopsys Software Integrity Group senior software engineer Philip Ross:
Once you’ve discovered sluggish tests, go over each one and, if there isn’t a clear bottleneck, run the test using a profiler to identify where the time is spent. With that information, you should be able to identify any hot spots and work on resolving them.
Consider examining your configuration to see if the test environment has any developer-like performance options. Some of these developer-friendly options may have sacrificed runtime performance in favor of compile-time performance or anything else, which you may not want on a static CI test run.
Also, see if there are any hard-coded constants that need to be set per environment. In a test environment when you’re not hitting a real endpoint, for example, a sleep for retrying HTTP requests should be set to 0; you may remedy this by transferring the sleep amount into your settings and setting it to 0 in the test environment.
Test should check one thing well
The Testing Show: Michael Larsen, senior automation engineer and Socialtext show producer
Examine what your tests are actually accomplishing. A reasonable rule of thumb is that unless otherwise specified, any test should only accomplish one thing. This atomic aspect is critical for writing unit tests, and it’s also critical when writing additional integration or end-to-end tests. Slow tests are frequently overloaded, attempting to accomplish too many tasks. By reducing the number of steps required for each test, the overall time for each test is reduced, and there is typically a synergistic impact. Performing numerous single-purpose tests may be faster than running a single overloaded test.
Run tests in parallel
PrecisionLender’s lead software engineer in test, Andrew Knight:
Parallel testing is one of the most effective strategies to speed up tests. Attempting to remove one second from this test and three seconds from that test hasn’t had the same impact as running two tests concurrently rather than serially. Parallel capabilities are already embedded into most test frameworks. Often, enabling parallel execution is as simple as changing a configuration or using a command-line option. However, avoiding collisions is the main challenge with parallel testing. If tests share resources like users, services, or databases, there’s a potential they’ll all access them at the same time, resulting in “colliding.” In a retail app, for example, if one test attempts to retrieve an order after another test has deleted it, the first test will fail.
The best strategy to avoid collisions is to make sure that no shared data is modified during tests. If a test needs to update data for testing purposes, it should create new data specifically for that purpose. In the case of the store app, tests should always make new orders rather than reusing old ones.
Make sure CI is accelerating things, not creating drag
Lalitkumar Bhamare, XING’s senior software test engineer; Tea-time with Testers magazine’s CEO, co-founder, and editor
Many organizations, in my experience, use CI to create more problems than they solve. Poorly executed CI jobs and not giving enough thought to how much automation should be done at what layer are two CI-related items that, in my experience, drag organizations behind. Furthermore, shoddy infrastructure can cause flakiness, and constantly failed automated checks that prevent deployment create an ongoing puzzle to solve. This can result in engineering time and productivity being lost “fixing” the CI pipeline, as well as firefighting caused by environmental/infrastructure issues that aren’t related to genuine production failures.
If this describes you, one simple workaround I can suggest is configuring CI jobs to only conduct selective automated checks—for example, only for the areas affected by a specific code change. This will expedite the test run and eliminate false failures, and it will be inexpensive. Another alternative is to disable build triggers for non-production code changes, such as modifications to “helper” scripts or automated checks.
See the whole board
Satisfice: James Bach, consulting software tester
The first thing I do is take a deep breath and focus myself. I remind myself that I might be a bottleneck after all. Then I remember that being a bottleneck isn’t always a terrible thing. (At airports, security is a bottleneck.) Is anyone suggesting, “Let’s get rid of security because I only have 15 minutes to spare?”). However, I consider how I may not be good enough for my job and how the notion that my work is too slow may be accurate.
After I’ve regained my composure, I’d like to investigate the perception. What exactly appears to be slow? To whom does it appear to be slow? What makes you think you’re slow?
This is a procedure for entering the system that I must complete.
As I learn more about the issue, I may come to the same conclusion. However, a common issue is that those measuring the speed of testing haven’t considered the process’s worth. Testing is similar to insurance in that it is not purchased with the intention of profit. You buy insurance to protect yourself from financial ruin. In exchange for that effort and time, testing has a cost: we acquire a chance of identifying critical and evasive flaws. We make every effort to match the investment to the risk involved.
Lighten your load to go faster
When it comes to the test process, there are things that must run all of the time (such as the CI pipeline), the standard testing procedure, and the occasional processes, such as a security audit, that can slow things down if they are removed.
The advice of our specialists boils down to a few things: remove things from the pipeline, speed up the pipeline, better the overall process, or lessen the impact of special/odd things like security and performance testing. It may be conceivable to incorporate them into the continuous integration pipeline or to allow developers to do so locally before the pipeline runs.
In any case, you can lower your load or strengthen your muscles to travel faster. Congratulations if you employ sleep statements or high-level tests on a regular basis. You might be able to come up with some quick fixes.
Everyone else might have to do some hard lifting and flexing.
Special thanks to Lee Hawkins, whose suggestions to execute in parallel, use mocking, test at the appropriate level, eliminate waits, and do one thing well bolstered the contributors’ ideas.
For more info: https://www.qaaas.co.uk/testing-services/
Also Read: https://www.guru99.com/software-testing.html