Key Elements of Automated Tests

This content is syndicated from On Agile Leadership by Manfred Lange. To view the original post in full, click here.

This time I'm writing about an item that is admittedly very specific to software development. More than once when I spoke to a members of a development team I was told "yes, we have an automated test suite". And yet, further along the conversation it turned out that despite a significant test suite the resulting quality wasn't where all the efforts put into creating those tests indicated it should be. And in all these cases when we then took a closer look at the tests themselves it turned out that at least one key element was missing.

That begs the question: What makes up a good test? What key characteristics should a good test have?

Setup, Execute, Validate

The first key ingredient is that a test consists of three parts: The first parts sets up the data that is needed for the test. This could be restoring a particular database content, it can be setting up a few objects in your programming language, it can be launching a particular user interface (e.g. browser) and many more. The second part is the actual execution of the test. This means you invoke functionality that modifies data. In the final third party you validate whether you have the expected outcome, e.g. the actual data is equal to the expected one.

Occasionally I've found, though, that people forget about the third step. I don't have data but suspect that this happens when people come from a background where executing a piece of code without crashing is almost a success. Think early/mids 90s of last century. C and C++ were still very dominant in the PC industry. Exceptions in the midst of running a program were nothing completely out of the ordinary. (Maybe you are the only one who never experienced them?) However, we can do better. Just because it doesn't crash with a nasty null pointer exception doesn't mean it performs as expected. Therefore at the end of a test always validate the outcome! The typical tool for that are the various assertions that come as part of test tools.


Not strictly a requirement but there are quite a few scenarios where running the same test more than once reveals and thereafter prevents certain bootstrapping type issue from happening. Assume your service implementation does some sort of housekeeping upon startup. The first time you invoke an operation on the service everything is still fine. But then perhaps as you repeat the same test (or set of tests) using operations on that service things go off track. Maybe connections are not properly closed. Maybe the service cannot handle more than 10 open connections at a time (rightly or wrongly). By repeating the same test over and over again chances increase are that discover a hidden issue and resolve it before your product is shipped.

Random Order

Tests should not depend on each other. A test should not require a different test to run first. If they do changes to one test may trigger further changes to other tests in the suite thus making changes more expensive and time consuming. You don't want to lose time. You want to be fast.

For example, lets assume you are working on a system that has Project as a concept and the name of the project becomes a unique identifier for each project. If all tests use the same project name for their tests, then each test would have to check during setup whether the project already exists. If it doesn't it would create it. The alternative would be to use a generated name in each test such as a string with the value "ProjectName" + RandomNumberAsString(). That way you make the tests independent from each other.

A corollary to this is that you can run each test in isolation, meaning you can run just that test focusing on the specific task at hand. You don't have to run other tests first and you don't have to remember the sequence for those other tests. You can - and probably want - to run the entire suite anyways once you have finished the coding part of your task.

Fast Equals Cheap

Why do tests to be fast? To an engineer time is money. The longer a test or test suite needs to execute the less likely it is that people will run the suite or portions of it. It a suite runs takes 1 minute to complete you probably run it before each commit. If a suite takes 5 hours you won't run it before each commit. So keep them fast, for example work with the smallest possible dataset, avoid or mock costly operations like talking to remote systems, databases, filesystems, anything that requires mechanical parts to move. Use in-memory databases (e.g. SQLite) instead of client-server systems such as Microsoft SQL Server or Oracle.

You may also want to continuously refactor your tests as well. Keep them lean and mean. Split the test suites into the ones you definitely want to run each time before you commit and those that are more expensive in terms of duration. Platform tests or scalability tests fall into the latter category.

Automated And Integrated

Tests have a value only when they are executed. As long as they are just sitting in your version control system they are useless. Make them work. Make them work hard. Use all those machines that sit idle while the developers using them during daytime are at home enjoying live. Integrate the execution of your automated test suites into your development processes. When tests and their execution are automated and integrated they are executed more frequently and each time a change in the code base has been committed.

Are Automated Tests Orphans In Your Team?

Automated tests are first class citizens and equally valuable as the product that you ship. Don't even think for a second that they are just an unavoidable side affect. It's not a tax report that you do because law says so. Instead fully integrated automated testing is the mechanism that allows your team to operate at full speed. Depending on your team size maintaining the testing infrastructure can well turn into a full-time job for a motivated engineer.

Treat your testing infrastructure at least as well as the other parts of your development infrastructure. Make all of it part of an end-to-end integrated development process that starts with a customer suggestion and ends with a deployed system operational for the very same customer. Automated testing is a key ingredient and the rules are simple to follow. No excuse any more for not improving software quality. Happy testing!

Leave a Reply

Your email address will not be published. Required fields are marked *

16 − 16 =

There are 101 ways to do anything.
To find the best way, sometimes you need expert help

What People Say

“Kelly was engaged as a Program Director on a complex business and technology transformation program for Suncorp Commercial Insurance. Kelly drew on his key capabilities and depth of experience to bring together disparate parties in a harmonised way, ensuring the initiate and concept phases of the program were understood and well formulated. Excellent outcome in a very short time frame. ”


“Kelly and I worked together on a very large project trying to secure a new Insurer client. Kelly had fantastic commercial awareness as well as his technical expertise. Without him I would never had secured this client so I owe a lot to him. He is also a really great guy!”


“I worked with Kelly whilst at Thoughtworks and found him to be a most inspiring individual, his common-sense approach coupled with a deep understanding of Agile and business makes him an invaluable asset to any organisation. I can't recommend Kelly enough.”


“Kelly is an Agile heavy-weight. He came in to assess my multi-million $ Agile development program which wasn’t delivering the right throughput. He interviewed most of the team and made some key recommendations that, when implemented, showed immediate results. I couldn’t ask for more than that except he’s a really nice guy as well.”


“Kelly revolutionised the way our digital department operated. A true advocate of agile principles, he quickly improved internal communication within our teams and our internal clients by aligning our business and creating a much enhanced sense of transparency in the decisions the business was making. Kelly also introduced a higher sense of empowerment to the development teams...”


“Kelly’s a leading program director with the ability to take charge from day one and keep strong momentum at both a program and project level driving prioritisation, resourcing and budgeting agendas. Kelly operates with an easy-going style and possesses a strong facilitation skill set. From my 5 months experience working with Kelly, I would recommend Kelly to program manage large scale, complex, cross company change programs both from a business and IT perspective.”


“I worked with Kelly on many projects at IPC and I was always impressed with his approach to all of them, always ensuring the most commercially viable route was taken. He is great at managing relationships and it was always a pleasure working with him.”


“Kelly was a great colleague to work with - highly competent, trustworthy and generally a nice bloke.”


“Kelly was a brilliant CTO and a great support to me in the time we worked together. I owe Kelly a great deal in terms of direction and how to get things done under sometimes difficult circumstances. Thanks Kelly.”


“Kelly came to the department and has really made a huge impact on how the department communicates, collaborates and generally gets things done. We were already developing in an agile way, but Kelly has brought us even more into alignment with agile and scrum best practices, being eager to share information and willing to work with us to change our processes rather than dictate how things must be done. He is highly knowledgable about agile development (as his active blog proves) but his blog won't show what a friendly and knowledgeable guy he is. I highly recommend Kelly to anyone looking for a CTO or a seminar on agile/scrum practices - you won't be disappointed!”


“Kelly is an extremely talented and visionary leader. As such he manages to inspire all around him to achieve their best. He is passionate about agile and has a wealth of experience to bring to bear in this area. If you're 'lucky' he might even tell you all about his agile blog. Above all this, Kelly is great fun to work with. He is always relaxed and never gets stressed - and trust me, he had plenty of opportunity here! If you get the chance to work with Kelly, don't pass it up.”



To explore how we can help you, please get in touch