Test automation is like many political issues—there are several camps with varying degrees of bias. PMs and testers with little programming experience are sometimes intimidated by automation. Engineers, especially engineers only a few years out of school, think automation will solve the world’s problems.
If you peek at someone’s resume, you can often predict their feelings about test automation without talking to them.
“The smartest developer on your team should be the one writing your tests.” — Simon Stewart, a core member of the Selenium team at Google, now at Facebook
Sometimes this bias comes from previous experiences, such as successful software companies that saw great success with little automation, or efficient test-driven development teams that needed no manual or user testing. As with most politics, the correct answer usually “depends”–the truth lies somewhere in the middle and is nuanced based on the context.
Here are a few questions can quickly identify if, and how much, you should invest in test automation for mobile app testing:
1. Are you a bank? Does your app deal with money? Do people’s lives depend on your app?
Automate. You should be building a full suite of regression automation for your app.
2. Does your app simply show data pulled from your website? Is it a simple news reader?
Little need to automate unless most of your business, engagement, or money flows through the app.
3. Is your app experimental? Are you still looking for product-market fit?
Don’t automate yet—wait until you know what you are building. Automating now will distract from iterating on the product, and much of your code would likely be thrown away as you iterate on your design before you got a return on your automation investment.
4. Is your app valuable independent of a website? Are you a hot new mobile messaging startup flirting with a $30M or $2B valuation?
Automate a bit. It will keep you from experiencing a dramatic FAIL, and make you look semi-professional to investors and potential acquirers during due diligence.
5. Do you work at an old school software company where the only road to promotion in the testing organisation is to show that you can write some code?
Automate until you get promoted, then stop.
6. Is your app for an internal, small (<1,000) person audience? Is the app critical to your sales force to demonstrate your companies’ value to prospects while on the road? Could your app’s failure mean an indirect loss of revenue or sales?
Automate a bit. When your app does eventually fail, the witch-hunt will ask this question. Automation may not have prevented a failure, but it’s better to conduct due diligence in advance than during a post-mortem.
7. Is your app “done”? Do you expect your app to enter “maintenance mode” soon?
Automate a little bit so you don’t have to worry about it.
8. Do you have daily or hourly app builds? Does your team practice “Continuous Integration” (CI)?
Automate as much as you can bear. To reap the full benefit of continuous integration, you need to catch bugs as soon as they occur, and only automation will be fast enough to catch most regressions before the next build. If you catch a bug quickly in a CI environment, it is far easier to isolate the offending code check-in because there are relatively few diffs in each build.
Other Test Automation Considerations
When in doubt, most app teams should eventually automate a basic set of tests, but only a few. Only build extensive tests after you know the app, in its current form, is going to be important. The calculus for test automation in the world of mobile apps is significantly different than for traditional web or desktop apps.
There are many factors to both motivate and suppress investments in the mobile app world:
Speed Causes Breaks: The mobile world is typically faster paced than the web world. With more design and functional iterations, there is less time to reap the rewards for investing in automation — your tests may fail because the product changed, not because there is a bug.
Opportunity Cost: Anyone capable of writing test automation could likely be writing features for your app. There is a shortage of mobile app developers today, so even when you do invest in some testers that code, they are often quickly lured away to work on important new features or bug fixes within your company. And more often than not, lured straight to other companies.
Speed Means No Time To Wait: The speed of agile and CI environments means that time is at a premium. If you have a new build every hour, the only way you will know if the build is good or bad is by having automation running in the background — you don’t have time between builds to wait for manual testing.
Immature Infrastructure: Test automation infrastructure is still maturing. Until recently, many frameworks were either platform-specific (didn’t support both Android and iOS), ran exclusively on simulators or on real devices, or were closed-source proprietary systems that you shouldn’t rely on in the long term. In a future post, we will dive more deeply into the world of test automaton frameworks.
This Post was orginally publised by Appdiff.
