It’s bee traditional thinking to…
- One to one mapping between manual tests and automated tests.
- Trying to have as much requirements and test case coverage as possible.
- Automated tests should be able be reproducible manually.
But over the years, I’ve found that having 1000’s of test cases I tend to cause even more problems.
1. Test code is harder to maintain which becomes production code.
Besides the technical aspects like having to bend over backwards to accommodate changes in the software under test, it is also the lack of support from business and development for test code.
For example, very few shops will hold up a release or delay development if there are broken tests or flaky tests. The default is generally to comment those tests out..
2. The need for quicker feedback of Failures.
3. When writing many tests, you may not have spend as much effort writing them for debug-ability
what should be done for correct automation ?
- Write fewer tests which are more static in nature
- Focus on performing complete workflows over single actions. Testing single actions in isolation will lead to number of tests creeping up. Working workflows will keep it at a smoke test level.
- Write code with the intention of debugging every single line. Pay close attention to your logging and exception handling. Make sure a failure at any point will return you a reason.
- Use good programming patterns and test architecture to make it easier to refactor.