The Journey of Test Automation

Software testing is a necessary process and everyone seems to agree on that. Where people differ is the timing and depth of the initiative. The industry has long moved from a stage when testing was considered a post-development, maintenance activity. We gradually interleaved the development and testing process. Both the development and the testing process tend to merge, bringing both new benefits and new challenges. The obvious benefit is fast feedback. We need to check an error as quick as possible. The not-so-obvious challenge is maintenance and overhead. This will come to haunt us later.

Long since the early days, the industry adopted Continuous Integration. Whenever a developer is ready to push some code, he first has to pass it through the Unit Tests. If all is green then only can he actually commit the code. Even per-commit jobs are gaining popularity. People are delegating the responsibility of code validation to the build server itself. The idea is to integrate the Code - Build - Test - Deploy process in a single pipelined workflow. Every stage in the pipeline provides immediate feedback to the preceding stage.

On the developers' front automated Unit Tests have established themselves as an invaluable checkpoint. What about the managers and the business users? Everyone wants things to be well tested. But Unit Tests and their reports are not much meaningful for them. The unit tests are also not reliable in integration testing. Components having passed their unit tests may still fail to integrate. Here comes the need of functional testing. Organizations have dedicated departments of manual-testers. They validate the product at whole build level. Detailed reports are generated, modules' health is monitored and bug tracking is integrated to this process.

The Present

Bulk of functional testing has always been manual. A big reason was lack of tools which can automate applications, detect and report the failures in doing so. But the last decade saw an exponential growth the automation sector. The authority of pioneering products was challenged by emerging tools. A majority of them were open source, promoting accessibility to even small companies. So when we can automate the functional testing process, why not integrate it to the CI server? Why not unit test and functional test every code batch? Here comes the hurdle which we knew but overlooked at the initial stages - maintainability.

Automation scripts are traditionally codes. They are codes which test and validate other code; and they being code, need to be designed, written, maintained and reviewed. But they have a big advantage - they can execute just as any other code. That's why JUnit and similar frameworks are so popular. Problem is they don't scale well with functional testing. For one thing, functional tests are very broad, covering many workflows, scenarios, test cases and unique steps. Such big a code is difficult to maintain and soon outgrows the product itself. Another thing is being code; they don't truly lie in tester's domain. They need seasoned developers. You can have programmers write the test scripts but then a load mismatch remains.

Welcome the era of Keyword Driven Automated Functional Testing. Encapsulating the programming process in test scripts gives many benefits. First and foremost, we no longer need hard programmers for testing task. Any programming-literate can do the task. The Test scripts are more maintainable and language agnostic. Now we can actually have both unit tests and functional tests which can be triggered on each code check in. While unit tests validate code conformity, functional tests validate product's behavioral consistency. And both work unattended. Developers can get feedback meaningful to them from their unit tests. Business users can get module health status meaningful to them from their functional tests, all in near real time.

The Future

There is no limit to innovation. Imagine the development of a web application. Let's say the application needs to be supported on all major browsers, IE, FireFox, Chrome, Safari, and Opera. Across all Platforms: Windows/Linux/Mac Desktops/Laptops. Let us push the limits, ask for support on all major mobile devices: Android, iPhone, Blackberry, Windows; talk about orientations, resolutions. Now lets us ask the test engineers to automate the whole thing. Run all regression flows on all Platforms/browsers on all check-ins. After all we want a strict quality control.

Difficult to accept but this simply cannot be done! Not without an automation framework which can do the heavy lifting. None of tools or frameworks in the market can cope with the requirements that we will be facing in coming years. What the industry needs is a test automation framework which:

Can integrate with the existing CI solutions.
Perform functional tests as seamlessly as Unit Tests. Run a test every night, every check in or on every Run-Test request.
Scripting process is better to be keyword driven than language based. We want one simple ClickOnButton; not a bunch of find, hooks, performs, again recheck hook and so on.
Provides high maintainability of Scripts. Should not need programmers to modify that validation.
Can gracefully orchestrate multiple tools to perform activities which otherwise don't come under ambit of any one single tool. If Selenium cannot click on an image, get it done with Sikuli.
Can launch multiple configurations of a same test in parallel. We want to take the same test on all five major browsers.
Gives a uniform interface over heterogeneous tools. Tools may vary in language support even on platform support but we need it. We need to test our application on both Linux and Windows.
Generates business oriented reports. We want reports in terms of Business flows not individual test cases. We want health check of modules. We want basic metrics. We want to visualize defects over last five builds.
Drill down to step level failures, so that both testers and developers can get workable information regarding the failure.
Integrate with major Defect management tools like Bugzilla, Jira etc. Should be capable of automatically logging confirmed bugs.
Integrate with major Application Lifecycle Management software like HP ALM.
Now that is a whooping wish list and no product in the market claims to do it all. But this wish is soon going to be our requirement. Success and failure of products will depend on timely validation on all the target environments. Even if we somehow manage the sheer volume of test scripts and all the plumbing, we are stuck at the test infrastructure. We need to test on five browsers, each on latest version and three backward versions. We need to test those browsers on Windows, Linux and Mac. Now that makes about a hundred parallel executions on a hundred infrastructure. How are you going to manage that infrastructure? What happens when you need a bunch of Mobile devices? Are you going to purchase it all? Rent it all? Of course not.