Making Exploratory Testing Interesting and Effective For Your Testing Community
It is a popular myth in the testing industry that there is no end to testing a product or an application. Testing as a discipline has evolved in the last 2+ decades giving itself a formal structure. Various facets of software testing such as quality consulting, test strategy, test planning, traceability and requirements analysis, largely contribute towards breaking the myth. Newer and more innovative testing techniques are evolving by the day, to help the test team confidently sign off on a product release amidst the ever shrinking product release timelines.
Exploratory testing is one such technique that has withstood the test of time yielding very good results and also helping the tester think out of box. In my past assignments, I have seen exploratory testing being relied upon completely over formal testing techniques, in situations where the product requirements were not very clear or where the release timelines were very short and did not have much flexibility. When such is the importance and value of exploratory testing, how can you encourage your teams to use this technique more frequently and how can you better interpret & communicate the results of exploratory testing in making the very important decision of "test sign off"?
Here are some simple yet effective techniques to motivate your team into exploratory testing and get the most out of it:
Often times, there is no room for exploratory testing since the team is very busy running scripted/planned tests. Try and set aside about half hour each day for the team to take on exploratory testing. This is the time where they are not going to be mandated by any formal tests. They are encouraged to be creative and putting themselves in the shoes of the end user in trying out varied scenarios. Depending on your product cycle, this may not be required on a daily basis. Dial it up or down at periodic intervals.
If the product under test happens to be heavily role based that may also need simultaneous interactions - e.g. a learning application where you have a student role, teacher role, admin role etc., assign roles to your testers during the exploratory phase and also periodically swap roles so a tester does not play the same role each time
Whether you are a product or a services company, take the initiative to conduct periodic bug bashes. These go a long way in adding value to your overall project engagement with your customer. Give prizes to testers for various categories of bugs such as most number of bugs, most severe/ship stopper, bug that has the most customer impact etc. The prize could even be appreciating the tester in front of the entire team or giving him/her a certificate - these are gestures that go a long way.
Cross assign testers across products - In situations where the project that is in progress does not have restrictions imposed on external team members (from your own company) testing your product, encourage such cross sharing of testers to promote more creativity. This not only motivates the tester by breaking his monotony of testing the same product but also brings in fresh pairs of eyes and perspectives to test the product.
Also, depending on the product under test, involve non testers to come play around with the product. In my past engagements, where we tested a mobile application, we had test managers and directors come participate in the bug bash which helped us emulate some very realistic user scenarios
Encourage exploratory testing for all kinds of testing assignments - black, gray and white box. It is often misconstrued that exploratory testing can be done only black box. If you understand the definition of exploratory testing as something that promotes the tester to be extempore rather than being bound by a formally written test case, you will appreciate this technique in all testing assignments.
Having talked about promoting the use of exploratory testing in your teams, I next want to touch upon how to interpret the results of exploratory testing as unless this is done in an informed manner, to the untrained eye, it may reflect badly on the overall test coverage:
Timing of the exploratory testing is very important in interpreting its results. I've seen some teams take on this testing technique even before a formal test pass can start, as this gives them an opportunity to play around with the product and flush out bugs as early as possible before they are given formal tests to run. I've also seen teams take on this testing after formal testing has been completed, thus using this as a supplemental method to find bugs that may not have been caught formally. Most teams that adopt an agile methodology follow the former approach. Regardless of the approach, be cognizant that in the former, most bugs would be found through exploratory testing. This is in no way reflective of poor test coverage.
Often times, tests are designed using the product specifications or requirements, so to some extent scripted testing will follow the path of the stated specifications. However exploratory testing knows no bounds and often finds bugs in the unbeaten path. This should be looked at in positive light and that the combination of scripted and exploratory testing has helped drive the product towards a quality bar that will be well received by the end user. To make these tests repeatable and constantly work on improving test coverage, the tester should add test cases for valid exploratory bugs filed. This will help ensure they are not missed during the regression cycle.
https://canvas.rice.edu/eportfolios/769/Hayley/Ultimate_ProfessionalCloudNetworkEngineer_Exam__PDF_Questions_for_Passing_in_the_First_Attempt https://canvas.rice.edu/eportfolios/769/Hayley/Unique_ProfessionalCloudSecurityEngineer_Exam__PDF_Questions_for_Passing_in_the_First_Attempt https://canvas.rice.edu/eportfolios/771/Peugh/Unique_ProfessionalCollaborationEngineer_Exam__PDF_Questions_for_Passing_in_the_First_Attempt https://canvas.rice.edu/eportfolios/769/Hayley/Tested_ProfessionalDataEngineer_Exam__PDF_Questions_for_Prep https://canvas.rice.edu/eportfolios/769/Hayley/Unique_ProfessionalMachineLearningEngineer_Exam__PDF_Questions_for_Good_Result
Bugs found from bug bashes especially towards product release, greatly determine the product's ship date. Whether towards the end date or not, when analyzing bugs from bug bashes create and involve a triage team, representing the program & project mgmt teams, development and test teams. This is very important to truly analyze the bug's end user impact.
Having talked of all the value that exploratory testing brings to the table, we also need to discuss one important limitation for which we need to implement a work around, failing which, exploratory testing may be seen as an overhead by the rest of the product team. In case of scripted testing, very formal design and review methods are often implemented, where test cases are vetted for validity before they are executed. Due to such stringent reviews the bugs found through such test cases have a very high validity rate. However, since exploratory testing does not follow such set bounds, there is a higher chance for false positives in the bugs filed. This may bring down the value of the test team and the exploratory test technique amongst the rest of the team, especially when everyone is working on tight deadlines. To ensure such false positives are kept low, peer reviews or reviews with the test lead / manager is often recommended for exploratory bugs. One may also attempt to introduce some formality yet not losing the spirit of exploratory testing through test techniques such as "session based testing viz. sessions and charters". This will also help the test lead/manager tangibly analyze the results of exploratory testing and focus on specific areas that need more testing attention.
The right combination of scripted and exploratory testing is the secret recipe to balance the time available on hand to test and strive to improve product quality and end user acceptance. Exploratory testing cannot be taught / explained formally. It is an art and a science that one needs to pick up through experience. Herein I've shared with you all some approaches I've used on my past projects. Try these techniques on yours too and let me know if you have any suggestions, comments, questions.
Rajini Padmanaban, Director Engagement - Global Testing Services. She has years of experience in Quality Assurance and providing software testing services. She has been involved in providing exploratory testing to many technically challenging projects.