Discussing Tester's Value, Beyond the Echo Chamber

I've finally come to a conclusion. I spend a lot of time thinking about why I feel like a lot of the advice to testers feels common sense, and why I don't feel like my thoughts are novel or unique. Here's the problem: the testing community is in some senses an echo chamber. The same thoughts get shared within the community, and people hear, agree and repeat. What I've personally neglected to do is reach outside the tight testing community to listen, learn and share. What do people in other engineering roles think the value of testers is? Or maybe they think there's no value, and I should seek to understand why. Sharing my thoughts on the value of the tester role and engaging in discussions when opposition is raised is only going to serve to improve bring clarity to my points.

Here's a couple things I've noticed in discussions with people in non-testing roles who think testing isn't valuable:

"Testers don't understand code. They rarely have technical understanding of how software works."

Admittedly, this one pisses me off more than most. Understanding how software works and understanding code don't go hand in hand. And being able to understand what a snippet of code is doing doesn't necessarily involve knowing how to code in that specific language either. Having said that, I've seen many organizations add fuel to this fire by hiring large numbers into testing roles who truly do not understand software. Like at all. We're talking applications-as-a-black-box, tell-me-what-to-click-to-test mentality. Check the box, move on. I've got plenty of tips and tricks I'll share with you (in person, over social media, or in a later blog post) if you are a software tester who is afraid of code, or doesn't know how to grow beyond the role of a pure blackbox tester. Having a blackbox hat on your rack is a great skill...but don't let it be your only skill.

"Testing is slow. We should automate it all so it's fast and we won't need human testers."

Well we're actually partially agreeing here. We should automate regression coverage so that devs can be sure their feature continues to work as expected going forward, without relying on testers having to revisit the feature and all valuable scenarios after every change. That would be slow, tedious, and not a good use of anyone's time. But if we can automate all that away, it's a good use of skilled testers time to explore the feature - exercise workflows as a user would, look for edge conditions and risky areas, think about the feel of the feature, consider performance, accessibility, security....
Good testers don't want to do the boring, repetitive shit. So let's not make them.

"Testers are only needed to execute testing steps. Therefore they're a cheap safety net for organizations."

Have I ever mentioned how much I hate seeing testers referred to as "cheap"? I read a response to a LinkedIn post the other day, and someone said: they don't want their developers responsible for doing any testing because "why pay a skilled developer $120/hr to test their feature when a tester could do it for less than half that"...or something to that effect. Wow. 
This is akin to the whole "anyone can test" or "testing isn't a real skill" types of arguments. You also hear a lot of people repeating things like "this is why companies like Google or Facebook don't have testers". (Newsflash by the way...they DO have testers, despite publishing all kinds of things saying they don't. Is it just cool to say you're hip and modern and don't have testers?)
I think this argument is perpetuated by people that have only come into contact with bad testers, or been at organizations that foster poor testing practices. Testing requires skills in functional decomposition, risk assessment, specific types of communication and technical writing. But we as testers need to prove that by fighting back against bad testing.

The value testers bring...

It started with a simple question that I posed in the Ministry of Test #general slack channel:

Hi all. Does anyone have any sort of resources to point to for “proving the value of testers”? I’ll try to explain: I’m looking for resources that would help when having a conversation trying to explain that testers do more than just sit in a seat and execute test steps. Along with that, I’d like to try and justify reasonable pay for testers. I know how I feel about it all, but having something to point to beyond my own opinion would likely go a long way.

Well, fortunately for me, Mr. Michael Bolton (someone who's lessons I've followed closely throughout my career as a tester) was watching. He pushed back on me to provide my thoughts on what value testers add. I typed what came to my head first, with no censoring or editing:

Personally I believe testers’ value comes at many stages of development, which is why I’ve always been a proponent of them being involved in every stage. Customer advocates, connectors (of people and information), informants (informing the org of their observations on quality, risk, …), analysts (analyzing change, risk areas, issues from the field…and distilling that data to inform changes in process/development in hopes of improving efficiency of development with quality)…and likely definitely more

At this point, Michael offered to role play with me some of the arguments I could expect to encounter when typically having this discussion. I'll paste the majority of the conversation here. It's a long read but those who have read it have expressed interest in the content. I'll summarize with a TL;DR for now, and then likely provide updates on my follow up thoughts soon.

michaelabolton

(In character:)  Don't you think the product owner is a customer advocate?  Isn't the business analyst a customer advocate?  Heck, aren't the programmers customer advocates?

I don't see programmers say "screw the customer", except as a joke sometimes.  Why do we need testers?

(Be prepared; these are perfectly legitimate questions.  AND there are good answers for them.)

Graeme 

Sure. And I believe programmers often do advocate for the customer and weigh in on discussions to that end. But programmer's primary skill sets lie in finding solutions to specifically defined problems, and implementing those solutions. We could take some of their time away to investigate how user's use the software and features, and analyze the types of bugs coming in from the field…but that's a lot less time they'd be spending in their primary skillset.

Does that seem valid?

michaelabolton 

That's pretty good, but not as strong as it could be.

Graeme 

I won't argue that. This is my first attempt at writing my thoughts on the matter beyond having them in my mind or rambling them verbally to an ally

michaelabolton 

(In character) The programmers have a lot of knowledge about the product already.  AND they know how to code.  And the testers don't, by and large.

(C) So the programmers will find most of the important problems anyway.

(C) The programmer's primary skill set is building a quality product.  Who would know more about how to do that than they would?

Graeme 

The programmers have a lot of knowledge about their specific domain of the product. In our case, programmers are focused on the windows app, the iOS app, the Android app, the Web app…the APIs, the worker services…all specifically. Testers are actually one of the only roles in the engineering org that understand the customer's usage and interaction with the apps from end to end. 

A programmer mindset is typically one that is able to prove there aren't problems with the solution the way they decided to implement and use it.

I don't know how to say this not anecdotally, but if I received demos of features from programmers where we didn't encounter bugs during the demos, I'd be more inclined to believe they're good at finding the important problems.

michaelabolton 

(NOT in character) Demos are funny like that.  They're intended to be demonstrations, but they turn out to be experiments.  :slightly_smiling_face:

But here's the thing:  it's important to acknowledge that developers are really good at finding most of the problems.  Almost all of them.  What kinds don't they find, and why not?

Graeme 

Speaking generally, it seems as though developers find important problems in the primary, happy path workflow. But typically not edge cases, or at integration points (especially when integrating with a feature outside of their primary focus). I don't think most developers are mindful in a way that allows them to understand how the user might use their feature beyond the way they expect a user to use it.

michaelabolton 

(C) So... we'll get product managers and BAs to test that stuff. 

(C) And after we've done that, we'll ship the product and if there are problems, we'll fix them right away?

Graeme 

Product managers don't have the same curious mindset that testers bring, and thus don't know how to experiment and investigate in risk areas of the product. In my experience, while testers may not always know how to write code, they do often know how to analyze it and determine technical risk areas. So we could have PMs and BAs try to test the primary workflows…but again that's time that they're not spending thinking about future roadmap features and products, or working to understand customer needs better.

michaelabolton 

(C) Why not just tell them to do that?

(C) Who says that testers are the only curious people?

Graeme 

As for the ship and fix problems…we do roll out most of our features in a way that allow us to catch issues before the entire customer base encounters them. However, for the issues to be found to be fixed, someone must encounter them. Hard to ask for money for a product where you expect the users to be your line of QA.

Who are the people teaching PMs how to experiment and assess risk areas/find issues at the integration points? Surely that's the job of a tester, right? Informing about those areas and providing that information back to people who can consume that information?

michaelabolton 

(C) Hey, it's not the job of a tester to tell the PMs how to do their jobs!

(C) As for the release-to-production strategy:  not much can go wrong with our product. We're really good at this stuff.  We've been doing it for years.

Graeme 

I mean…we have data to prove we're not really good at this stuff.

We can ask any one role to go do all of the things. We wouldn't need PMs if we just told developers to go figure out what the problems our customers are having that need to be solved. But I can't imagine a lot of actual development would get done. In the same vein, while PMs can go do that, asking them to also do the work of a tester is time spent that the PM isn't building roadmaps and deciding product direction to keep making the company money.

michaelabolton 

(NC) Very good.

(C) We don't have roles here.  Everybody does everything.  Why not just get rid of roles?

Graeme 

(NC) In this case, we are so much the other direction of having specific developers from different stacks, focus areas, etc…and product having specific focus areas…I'm not concerned about this argument. But will address anyways since its good to know how to.

michaelabolton 

:slightly_smiling_face:

Graeme 

(C) Again it comes back to mindsets and skillsets. We can either have specialists do what they're best at to build the strongest product possible, or we can have teams of people doing a mediocre job at everything, make a mediocre product, have a mediocre customer base…and make a mediocre amount of money…

michaelabolton 

(C) All right.  So what is it that testers do that's special?  Distinct? 

(C) And why can't people just switch from one mindset to the other?  We've got really smart people here.

Graeme 

testers will experiment and test (both in manual and automated fashions) to gather and analyze data, distill that data and provide direct recommendations on issues to fix, not just in the product, but in the processes of development. This will allow us to ship quality products to customers, and do so with even more speed and efficiency in the future.

michaelabolton 

(C) I still don't see why the programmers and the PMs can't do that.

Graeme 

Hmm…I think this is where I could use some Michael Bolton wisdom and insight.

michaelabolton 

(NC) OK.  Not in character unless explicitly tagged that way. (edited)

The answer is that they can do that.

Anyone can switch roles and mindsets to some degree.  Anyone can learn to program.  Anyone can learn how to use tools.

The issue, it seems to me, is this:  it's hard.  And it's probably slow, too.

Shallow work is easy to do.  Deep work is harder.

Going through the motions and rituals is easy.  Skilled, expert, focused work is harder.

Graeme 

Awesome, that's what I was trying to reach for mentally…but couldn't articulate the way you are

michaelabolton 

Recognizing many of the problems in something you wrote just now is easy.  Recognizing other problems takes time, or distance, or both.

Graeme 

It's much more about providing T shaped value to the teams than about just doing a bit of everything with little focus and expertise

michaelabolton 

That whole T-shaped business sounded tired to me on the first day.

I'm not sure why.

Graeme 

I guess since the org is talking about building "T-shaped" developers anyway…its common language I can use

Even if its not ideal language in general

michaelabolton 

Yes.  It feels kind of naff to me, but it is what it is. Nonetheless:  it's reasonable to believe that everyone should have some degree of general skills outside their speciality.

The testers' speciality, it seems, is this:  living and working at close social distance, but farther critical distance than other people on the team.

Graeme 

We often have programmers "test" each other's work by pulling branches in code review and doing shallow testing. Yet bugs still get out, so obviously there's evidence there that having a non-expert do a shallow test doesn't solve the quality problem.

michaelabolton 

Exactly.

It helps to have someone at some distance from the work to evaluate it IF you need serious evaluation.

Graeme 

Right

michaelabolton 

Rather than having someone adopt that mindset, it's a powerful heuristic to have someone inhabiting that mindset.

Graeme 

So…to the people who say "engineers" deserve more money because their coding skills are hard, measurable skills…how is the monetary value proven in the tester's skills where you can't look at code output and "grade" it directly?

michaelabolton 

Coding skills are not hard and measurable... without testing.

(They're not even measurable with testing, but testing can tell us something about the quality of the product that non-empirical approaches can't.) 

Graeme 

interesting

michaelabolton 

You can't measure quality.  Measurement might inform some aspects or some attributes of quality.

Graeme 

Yes, that I am aware of and believe.

michaelabolton 

Quality is not about measurement.  It's about assessment and evaluation.

Literally, about how we value something; that's the value in "evaluation".

And that takes us right to the tester's role.

The tester's role is to focus on threats to the value of the product.

The tester's role is to focus on investigating the product to discover threats to its value.

The tester's role is to bring expertise and focus to that task.

For me, the TL;DR of this conversation boils down to two (or potentially combined to one) tweet-sized quotes from Michael.

The testers' speciality, it seems, is this:  living and working at close social distance, but farther critical distance than other people on the team. 

The tester's role is to focus on threats to the value of the product.

It's been a minute...and a lot has changed

Well, it's been a minute since I blogged here. To be more exact, it's been almost 3 years.

In those 3 years, a lot has changed. I've grown in my career, from Test Specialist to Test Strategist to SDET to Engineering Manager. I'll touch briefly on what I spent my time doing in each of these roles.

Test Specialist

My time as a Test Specialist was as one on a team of 4 as part of an engineering team of less than 20 people. These were times of a rapidly growing team (and company!) who was trying to find their way into more modern agile development practices with shorter release cycles and more automated processes. The team was just ramping up on things like CI, unit testing, and more automated tooling and scripting.

After learning the product, I helped the team move away from documented test scripts and test cases to a more context-driven model. We worked to shorten time needed for testing by performing tests in more isolated builds and environments. I also had a large hand in standing up the first automated UI tests for the product. As the team continued to grow and we broke development down into smaller feature focused agile teams, I began to recognize a need for the testing across teams to share good patterns, lessons learned and work towards and larger goal of a quality product offering, not just individual quality products. 

This is when I proposed the role of Test Strategist...

Test Strategist

I spent the next 2.5 years at the company in this position. While I continue to be an individual contributor, I was also tasked with attempting to bring some order and consistency to the company's testing practices across the teams. During this time, I wore many hats where I performed manual functional testing for multiple teams, wrote automation scripts, designed and built automation frameworks and tools, acted as a release manager, and "unofficially" product managed the STS team's work (Software Tooling and Support - essentially our version of an automation and release operations team). I also mentored junior testers, provided coaching and direction on good unit testing practices, advised on quality metrics provided to Execs, and much, much more. In this time, the company grew from 2 feature focused agile teams to 8+ agile development teams - most of which having embedded functional testers, and all of which employing CI processes and incorporating some level of automated testing. 

6 months ago I left that role to move across the continent from Waterloo, Ontario to San Francisco, California...

SDET (Software Development Engineer in Test)

My official title here was "Software Engineer", but I was asked to focus first on building/improving the Windows UI Automation framework (since I had experience in such things - my last company's main offerings being primarily Windows Desktop based products). I quickly went to work not only implementing this, but also working to reorganize the way QA was performed and structured at the company (essentially putting my Strategist hat back on, in a way). My biggest regret in this time is creating the SDET role officially at the company. I now believe the correct approach was to just continue calling the role Software Engineer, but giving them automation specific tasks to work on (more on this on another blog post - OMG I have content worth sharing again...after my 3 year funk! *squeals with glee*).

A little context - our company is a cloud based offering, which allows customers to access their projects via web browser, iOS app, Android app or Windows Desktop clients. The means we have teams building features in 4 platforms, plus APIs, workers, services, databases and all the other fun stuff that comes along with large cloud-based offerings. 

My proposal was to move QA back under one "umbrella", with a QA Manager at the top. "Why?" you might ask. With most companies moving to embedding testers into agile, or "no specific role", feature based teams - the "QA Team" model seems obsolete. Well, a year ago I would have agreed with you. However, as it turns out, there are some issues with that. In our case, we had a consistency problem. Features implemented across multiple platforms were developed and tested in isolation from each other, essentially causing the platforms to feel like different products. On top of this, testers often lack direction and career growth when not exposed to a manager that actually knows how to manage them and help the grow in QA. The outcome of this are things like stagnant testing practices, and often a very high attrition rate - either losing testers to other companies, or to other roles within the company (this is not inherently a bad thing, as people should be able to move to various roles but ideally this isn't happening because people feel a lack of growth in their current role after a relatively short period of time).

Oh, by the way, testers are on the QA team as far as reporting and feeling like they belog to a team, but they are each given a feature set to own, and are embedded in the respective feature teams to be the testing lead on that team. There is no old school, traditional dev - test wall that work gets thrown over. We still follow the model of testers being involved as early as design and planning of features, through development, deployment and monitoring in production together with the rest of the development team that owns that feature.

Part of this move to a QA team was the establishment of SDETs, who build and maintain frameworks that the development teams can leverage for various levels of automated testing. They also help implement quality of life tools and other offering to help teams deliver products of higher quality at greater speed. That's always the goal, enable the teams to deliver faster without sacrificing quality. Arguably the SDETs could go either way with respect to reporting to the agile teams' engineering managers or the QA manager. We chose the latter. 

It was decided there should be a manager with strong automation knowledge for the SDET team...and thus my seemingly quick move to...

Engineering Manager (Platform Automation Team)

I now currently manage 2 SDETs plus a part-time contractor (long story, but it's working out wonderfully for us - hmmm, I smell even more blog content), and have positions open for at least two more SDETs. Besides the team management responsibilities, I still contribute to the building and maintenance of automation frameworks and CI tooling (though sometimes I look at the open instance of VS Code on my monitor, realize that it has been days since I typed anything into it and cry a little inside), as well as advise the QA team in a strategist-like capacity on process and mentorship, and unofficially "product manage" the growing automation backlog. 

Because I get asked this a lot, I'll touch on this here for anyone interested in following up - here are the tech stacks we're currently using to automate our UI/E2E testing:

• XCUITest for iOS
• Espresso for Android
• An in-house wrapper around FlaUi Automation Library for Windows (WPF)
• Cypress.io for Web
• ...with more to come as the team grows.

Wow it's been a crazy 3 years! And the adventure still feels like it is just beginning. Feel free to reach out to me with any questions, thoughts or challenges you have for me in regards to any of this. I'm always happy to share my thought process, experiences and to learn more from others' experiences as well.