Paul Hammant's Blog: I think it is time for QTP to die*
Agile practitioners such as myself
are finding it really really
hard to get skilled staff
for our QA needs. Clients past and present are finding the same
thing. There are plenty of candidates but their experience does
not overlap with the needs of the Agile industry much.
In this blog posting, I am going to make a case for the testing industry to give up their love of QTP (and other legacy packages). Implicitly, a call for the professionals formerly adept with those tools (and manual testing) to rejuvenate their careers by mastering some new testing tools and programming languages.
* it's a shrink-wrap product, not a person, don't go nuts on me as if I'd said something about an individual!
Let me go into some detail..
In common with colleagues across the Agile industry I have run into Quick Test Pro a few times. At least, I have seen the evidence for its existence the whole time: the Agile dev team reluctantly throws functionality over the fence to testing, but we're never allowed to join in. There will always be a dedicated team of testing professions that use QTP as it is deemed "too expensive" for developers to have installed on their dev workstations, or manual testing that's a deal of effort for the client. In a nutshell we finish a piece of development work, it gets thrown over the fence, and defects or approvals come back. For most projects we will have done our own automated functional tests before that handover. Because of that we will have some confidence about as to whether the testing department will approve changes. Sometime those departments are on the ball (can accept changes for testing the day they're finished). In the worst case the it could be a year behind schedule, which is not the immediate feedback an Agile team wants. Either way, we're not part of the evolution, maintenance or running of those functional test suites.
Former ThoughtWorker Bret Pettichord, who's somewhat of an authority on this, poses a controversial thought in one slide from a conference presentation in 2004:
Cut a ThoughtWorks colleague and you will observe that a small "testing framework" falls out. Cut some longer served ones and dozens will spill onto the floor, written in a variety of enterprise, legacy or esoteric languages. For a smaller number of ThoughtWorkers the framework may actually be finished.
One could subdivide the ThoughtWorker made frameworks into xUnit style, mocking technologies, and story/scenario and well as functional-UI-clicking types. We have started some notable UI testing technologies. Selenium, WebDriver (thanks Google for 'announcing' it a couple of years later) and Sahi. Brett Pettichord was a ThoughtWorker in the early days of leadership of the Watir project. Aslak Hellesoy (Cucumber) and Dave Astels (RSpec) were Thoughtworkers previously. If we go past web-apps, and look at desktop apps, Vivek Singh started White for .NET. We started Marathon for Java/Swing, then the newer Frankenstein (Vivek Prahlad) for the same combination, and SWTBot by Ketan Padegaonkar for Java's antidote to Swing - SWT. All mentioned so far are open source, but we even have a commercial product in Twist that's worth mentioning because iti s a small fraction of the cost of QTP, and pushes the bar in itself.
Automated testing of UIs is in our blood, and we make them developer tools with testers leveraged for their expertise.
But there is still this gap in that QA professionals from the 'old world' tools like QTP don't make the jump to the newer more agile tools. So I am thinking it is time for the remainder of the professional testing community to be forcibly respectfully dragged to the modern era of smaller test tools. Tools that are checked in to source control, run on the command line when needed, or from continuous integration boxes tens of thousands of times after initial creation. Tools that use test scripts written in modern scripting languages. I'm also saying it is time for the industry to stop commissioning large manual testing phases at the end of development, or buying expensive testing technologies that are alien to developers.
Selenium, for one, has been maturing since 2004. It was used internally in ThoughtWorks for a while before we open sourced it. It is now a existing under the auspices of the Software Freedom Conservancy, having pulled back at the last minute from being its own 501(c)3 non-profit foundation. There's plenty of profit there. Selenium Co-founder Jason Huggins is in a startup Sauce Labs. Java Open Source legend Pat Lightbody has his own too - BrowserMob. Both leverage Selenium for their cloud testing capabilities. There are another couple with funding that exist on the periphery of the core Selenium gang, so the ecosystem is quite healthy commercially.
In terms of jobs, let these six year charts courtesy of Indeed.com speak for themselves (Selenium orange, QTP blue):
(click to zoom)
Sure both QTP and Selenium are growing, but Selenium is eating QTPs market share over time.
How you compose your test scripts for driving of Selenium (or other) is key too. Developers would traditionally use a programing language. As a consequence those tests would remain opaque to testers (and managers, business analysts, etc). So instead of trying to promote Selenium in that style, we are going to suggest that you can compose understandable scripts nearly plain English.
Let us assume you are interested...
So to that end, I'd like to see the Agile dev/test community make some simple examples of their preferred technologies. With colleagues and buddies, I have made a modern functional test suite for a website called Etsy.com. It is a shopping site for awesome craft products. The site itself is made in PHP, and I've never seen the source for it and I've no idea who developed it. I hope Etsy gets some sales from this, as the underdog should always be lifted IMO.
Anyway, the tool-chain I am most familiar with is JBehave 3.3 with Selenium 2.0. (Note: I am a committer for both projects)
The style of plain-English test scripts is BDD which is a whole ideal in itself that we will skim over for now.
Note: I'd like to see other tool combinations like Cucumber, Sahi, Watir aded to this list. I'd also like to see some non-BDD examples too. In fact I'll update this entry to link to examples as they come in.
There are other tools that you would use to learn more about the page being tested. A short list is Firebug, XPath Checker and Selenium IDE (all Firefox plugins) that helps you record short scripts, giving you insights about the page in question again.
The technologies above can be mastered by QA professionals, as long as there are good examples to copy. Your developers will do the environment setup for you, and make the first test script. They will also be on hand to help you overcome issues.
I am trying to put together a one-day course in Dallas, Texas on at least JBehave3 + Selenium2, and Cucumber + Selenium2. With ThoughtWorks colleagues, we run a presentation interleaved with small team-centric experimentation in the tools listed. Delegates would come with a PC (Windows, Linux or Mac). They would have admin rights so that they can install the software we provide on thumb-drive or CD. I am thinking some time in April and priced at $99 per attendee or thereabouts.
The event will amount to a kick-start for attendees into a world of more marketable skills. Self-learning will have to continue of course.
If you are interested in attending, email me phammant@thoughtworks.com so that we can gauge numbers.
Recruiters are increasingly being asked to find resumes with skills outlined above, and coming up with very few candidates. Be one of the first to rejuvenate your testing technology skills, and get past the increasingly technical interviews that Agile teams are going to subject you to.
In this blog posting, I am going to make a case for the testing industry to give up their love of QTP (and other legacy packages). Implicitly, a call for the professionals formerly adept with those tools (and manual testing) to rejuvenate their careers by mastering some new testing tools and programming languages.
* it's a shrink-wrap product, not a person, don't go nuts on me as if I'd said something about an individual!
Let me go into some detail..
Legacy testing practices, to kill off
In common with colleagues across the Agile industry I have run into Quick Test Pro a few times. At least, I have seen the evidence for its existence the whole time: the Agile dev team reluctantly throws functionality over the fence to testing, but we're never allowed to join in. There will always be a dedicated team of testing professions that use QTP as it is deemed "too expensive" for developers to have installed on their dev workstations, or manual testing that's a deal of effort for the client. In a nutshell we finish a piece of development work, it gets thrown over the fence, and defects or approvals come back. For most projects we will have done our own automated functional tests before that handover. Because of that we will have some confidence about as to whether the testing department will approve changes. Sometime those departments are on the ball (can accept changes for testing the day they're finished). In the worst case the it could be a year behind schedule, which is not the immediate feedback an Agile team wants. Either way, we're not part of the evolution, maintenance or running of those functional test suites.
Former ThoughtWorker Bret Pettichord, who's somewhat of an authority on this, poses a controversial thought in one slide from a conference presentation in 2004:
It is not just Quick Test Pro that needs to go the way of the dodo,
it is Test Director, Quality Center and other tools that.
New Lightweight Testing Tools (Free and Low-cost)
Cut a ThoughtWorks colleague and you will observe that a small "testing framework" falls out. Cut some longer served ones and dozens will spill onto the floor, written in a variety of enterprise, legacy or esoteric languages. For a smaller number of ThoughtWorkers the framework may actually be finished.
One could subdivide the ThoughtWorker made frameworks into xUnit style, mocking technologies, and story/scenario and well as functional-UI-clicking types. We have started some notable UI testing technologies. Selenium, WebDriver (thanks Google for 'announcing' it a couple of years later) and Sahi. Brett Pettichord was a ThoughtWorker in the early days of leadership of the Watir project. Aslak Hellesoy (Cucumber) and Dave Astels (RSpec) were Thoughtworkers previously. If we go past web-apps, and look at desktop apps, Vivek Singh started White for .NET. We started Marathon for Java/Swing, then the newer Frankenstein (Vivek Prahlad) for the same combination, and SWTBot by Ketan Padegaonkar for Java's antidote to Swing - SWT. All mentioned so far are open source, but we even have a commercial product in Twist that's worth mentioning because iti s a small fraction of the cost of QTP, and pushes the bar in itself.
Automated testing of UIs is in our blood, and we make them developer tools with testers leveraged for their expertise.
But there is still this gap in that QA professionals from the 'old world' tools like QTP don't make the jump to the newer more agile tools. So I am thinking it is time for the remainder of the professional testing community to be forcibly respectfully dragged to the modern era of smaller test tools. Tools that are checked in to source control, run on the command line when needed, or from continuous integration boxes tens of thousands of times after initial creation. Tools that use test scripts written in modern scripting languages. I'm also saying it is time for the industry to stop commissioning large manual testing phases at the end of development, or buying expensive testing technologies that are alien to developers.
Is This Stuff Even Viable?
Selenium, for one, has been maturing since 2004. It was used internally in ThoughtWorks for a while before we open sourced it. It is now a existing under the auspices of the Software Freedom Conservancy, having pulled back at the last minute from being its own 501(c)3 non-profit foundation. There's plenty of profit there. Selenium Co-founder Jason Huggins is in a startup Sauce Labs. Java Open Source legend Pat Lightbody has his own too - BrowserMob. Both leverage Selenium for their cloud testing capabilities. There are another couple with funding that exist on the periphery of the core Selenium gang, so the ecosystem is quite healthy commercially.
In terms of jobs, let these six year charts courtesy of Indeed.com speak for themselves (Selenium orange, QTP blue):
 |
 |
 |
Sure both QTP and Selenium are growing, but Selenium is eating QTPs market share over time.
A choices you have to make for the style of test
How you compose your test scripts for driving of Selenium (or other) is key too. Developers would traditionally use a programing language. As a consequence those tests would remain opaque to testers (and managers, business analysts, etc). So instead of trying to promote Selenium in that style, we are going to suggest that you can compose understandable scripts nearly plain English.
Let us assume you are interested...
New Technology Tutorials for Newbies
So to that end, I'd like to see the Agile dev/test community make some simple examples of their preferred technologies. With colleagues and buddies, I have made a modern functional test suite for a website called Etsy.com. It is a shopping site for awesome craft products. The site itself is made in PHP, and I've never seen the source for it and I've no idea who developed it. I hope Etsy gets some sales from this, as the underdog should always be lifted IMO.
Anyway, the tool-chain I am most familiar with is JBehave 3.3 with Selenium 2.0. (Note: I am a committer for both projects)
The style of plain-English test scripts is BDD which is a whole ideal in itself that we will skim over for now.
| Technology
Mix |
Script Language |
Metaphor
|
Videos |
Tool-chain
for running tutorial |
Tutorial homepage, for those that know how to 'checkout' | Ready to go Zip to download if you don't know how to do SCM | Who
did this demo and/or blog entry |
| JBehave3
& WebDriver
|
Groovy 1.8 |
Plain text BDD
+ 'page objects'. |
checkout and run
(HD) |
Java5 or above, Maven
2.2.1 or above |
GitHub page for project | TODO |
Me, Mauro and the JBehave team |
| Cucumber & Selenium2 | Ruby 1.8 |
Plain text BDD + 'page objects; | here |
Ruby & Rake |
GitHub page for project |
TODO |
KK
Sure |
| Cucumber
& Sahi |
Ruby 1.8 |
Plain text BDD + 'page objects' | TODO |
Ruby and Rake |
TODO |
TODO |
Narayan Raman or Tyco.com |
| Cucumber, Capybara
& WebDriver |
Ruby 1.8 |
Plain text BDD + 'page objects' | TODO |
Ruby and Rake |
GitHub page for project |
TODO |
KK
Sure |
| Cucumber
& Watir |
Ruby 1.8 |
Plain text BDD + 'page objects' | TODO |
Ruby and Rake | TODO |
TODO |
Looking for a volunteer. |
| Twist |
Groovy 1.8 |
Plain text BDD + 'page objects'. | TODO |
Java5 or above, Eclipse, Ant |
TODO |
TODO |
ThoughtWorks Studios team |
| SpecDriver and WebDriver |
C# |
Plain text + |
TODO |
.Net 3.5 or above |
GitHub
page for project |
TODO |
Alister
Scott |
Note: I'd like to see other tool combinations like Cucumber, Sahi, Watir aded to this list. I'd also like to see some non-BDD examples too. In fact I'll update this entry to link to examples as they come in.
There are other tools that you would use to learn more about the page being tested. A short list is Firebug, XPath Checker and Selenium IDE (all Firefox plugins) that helps you record short scripts, giving you insights about the page in question again.
The technologies above can be mastered by QA professionals, as long as there are good examples to copy. Your developers will do the environment setup for you, and make the first test script. They will also be on hand to help you overcome issues.
Formal Retraining of QTP Engineers with New Skills
I am trying to put together a one-day course in Dallas, Texas on at least JBehave3 + Selenium2, and Cucumber + Selenium2. With ThoughtWorks colleagues, we run a presentation interleaved with small team-centric experimentation in the tools listed. Delegates would come with a PC (Windows, Linux or Mac). They would have admin rights so that they can install the software we provide on thumb-drive or CD. I am thinking some time in April and priced at $99 per attendee or thereabouts.
The event will amount to a kick-start for attendees into a world of more marketable skills. Self-learning will have to continue of course.
If you are interested in attending, email me phammant@thoughtworks.com so that we can gauge numbers.
Recruiters are increasingly being asked to find resumes with skills outlined above, and coming up with very few candidates. Be one of the first to rejuvenate your testing technology skills, and get past the increasingly technical interviews that Agile teams are going to subject you to.
Published
March 14th, 2011