The value testers bring...

It started with a simple question that I posed in the Ministry of Test #general slack channel:

Hi all. Does anyone have any sort of resources to point to for “proving the value of testers”? I’ll try to explain: I’m looking for resources that would help when having a conversation trying to explain that testers do more than just sit in a seat and execute test steps. Along with that, I’d like to try and justify reasonable pay for testers. I know how I feel about it all, but having something to point to beyond my own opinion would likely go a long way.

Well, fortunately for me, Mr. Michael Bolton (someone who's lessons I've followed closely throughout my career as a tester) was watching. He pushed back on me to provide my thoughts on what value testers add. I typed what came to my head first, with no censoring or editing:

Personally I believe testers’ value comes at many stages of development, which is why I’ve always been a proponent of them being involved in every stage. Customer advocates, connectors (of people and information), informants (informing the org of their observations on quality, risk, …), analysts (analyzing change, risk areas, issues from the field…and distilling that data to inform changes in process/development in hopes of improving efficiency of development with quality)…and likely definitely more

At this point, Michael offered to role play with me some of the arguments I could expect to encounter when typically having this discussion. I'll paste the majority of the conversation here. It's a long read but those who have read it have expressed interest in the content. I'll summarize with a TL;DR for now, and then likely provide updates on my follow up thoughts soon.

michaelabolton

(In character:)  Don't you think the product owner is a customer advocate?  Isn't the business analyst a customer advocate?  Heck, aren't the programmers customer advocates?

I don't see programmers say "screw the customer", except as a joke sometimes.  Why do we need testers?

(Be prepared; these are perfectly legitimate questions.  AND there are good answers for them.)

Graeme 

Sure. And I believe programmers often do advocate for the customer and weigh in on discussions to that end. But programmer's primary skill sets lie in finding solutions to specifically defined problems, and implementing those solutions. We could take some of their time away to investigate how user's use the software and features, and analyze the types of bugs coming in from the field…but that's a lot less time they'd be spending in their primary skillset.

Does that seem valid?

michaelabolton 

That's pretty good, but not as strong as it could be.

Graeme 

I won't argue that. This is my first attempt at writing my thoughts on the matter beyond having them in my mind or rambling them verbally to an ally

michaelabolton 

(In character) The programmers have a lot of knowledge about the product already.  AND they know how to code.  And the testers don't, by and large.

(C) So the programmers will find most of the important problems anyway.

(C) The programmer's primary skill set is building a quality product.  Who would know more about how to do that than they would?

Graeme 

The programmers have a lot of knowledge about their specific domain of the product. In our case, programmers are focused on the windows app, the iOS app, the Android app, the Web app…the APIs, the worker services…all specifically. Testers are actually one of the only roles in the engineering org that understand the customer's usage and interaction with the apps from end to end. 

A programmer mindset is typically one that is able to prove there aren't problems with the solution the way they decided to implement and use it.

I don't know how to say this not anecdotally, but if I received demos of features from programmers where we didn't encounter bugs during the demos, I'd be more inclined to believe they're good at finding the important problems.

michaelabolton 

(NOT in character) Demos are funny like that.  They're intended to be demonstrations, but they turn out to be experiments.  :slightly_smiling_face:

But here's the thing:  it's important to acknowledge that developers are really good at finding most of the problems.  Almost all of them.  What kinds don't they find, and why not?

Graeme 

Speaking generally, it seems as though developers find important problems in the primary, happy path workflow. But typically not edge cases, or at integration points (especially when integrating with a feature outside of their primary focus). I don't think most developers are mindful in a way that allows them to understand how the user might use their feature beyond the way they expect a user to use it.

michaelabolton 

(C) So... we'll get product managers and BAs to test that stuff. 

(C) And after we've done that, we'll ship the product and if there are problems, we'll fix them right away?

Graeme 

Product managers don't have the same curious mindset that testers bring, and thus don't know how to experiment and investigate in risk areas of the product. In my experience, while testers may not always know how to write code, they do often know how to analyze it and determine technical risk areas. So we could have PMs and BAs try to test the primary workflows…but again that's time that they're not spending thinking about future roadmap features and products, or working to understand customer needs better.

michaelabolton 

(C) Why not just tell them to do that?

(C) Who says that testers are the only curious people?

Graeme 

As for the ship and fix problems…we do roll out most of our features in a way that allow us to catch issues before the entire customer base encounters them. However, for the issues to be found to be fixed, someone must encounter them. Hard to ask for money for a product where you expect the users to be your line of QA.

Who are the people teaching PMs how to experiment and assess risk areas/find issues at the integration points? Surely that's the job of a tester, right? Informing about those areas and providing that information back to people who can consume that information?

michaelabolton 

(C) Hey, it's not the job of a tester to tell the PMs how to do their jobs!

(C) As for the release-to-production strategy:  not much can go wrong with our product. We're really good at this stuff.  We've been doing it for years.

Graeme 

I mean…we have data to prove we're not really good at this stuff.

We can ask any one role to go do all of the things. We wouldn't need PMs if we just told developers to go figure out what the problems our customers are having that need to be solved. But I can't imagine a lot of actual development would get done. In the same vein, while PMs can go do that, asking them to also do the work of a tester is time spent that the PM isn't building roadmaps and deciding product direction to keep making the company money.

michaelabolton 

(NC) Very good.

(C) We don't have roles here.  Everybody does everything.  Why not just get rid of roles?

Graeme 

(NC) In this case, we are so much the other direction of having specific developers from different stacks, focus areas, etc…and product having specific focus areas…I'm not concerned about this argument. But will address anyways since its good to know how to.

michaelabolton 

:slightly_smiling_face:

Graeme 

(C) Again it comes back to mindsets and skillsets. We can either have specialists do what they're best at to build the strongest product possible, or we can have teams of people doing a mediocre job at everything, make a mediocre product, have a mediocre customer base…and make a mediocre amount of money…

michaelabolton 

(C) All right.  So what is it that testers do that's special?  Distinct? 

(C) And why can't people just switch from one mindset to the other?  We've got really smart people here.

Graeme 

testers will experiment and test (both in manual and automated fashions) to gather and analyze data, distill that data and provide direct recommendations on issues to fix, not just in the product, but in the processes of development. This will allow us to ship quality products to customers, and do so with even more speed and efficiency in the future.

michaelabolton 

(C) I still don't see why the programmers and the PMs can't do that.

Graeme 

Hmm…I think this is where I could use some Michael Bolton wisdom and insight.

michaelabolton 

(NC) OK.  Not in character unless explicitly tagged that way. (edited)

The answer is that they can do that.

Anyone can switch roles and mindsets to some degree.  Anyone can learn to program.  Anyone can learn how to use tools.

The issue, it seems to me, is this:  it's hard.  And it's probably slow, too.

Shallow work is easy to do.  Deep work is harder.

Going through the motions and rituals is easy.  Skilled, expert, focused work is harder.

Graeme 

Awesome, that's what I was trying to reach for mentally…but couldn't articulate the way you are

michaelabolton 

Recognizing many of the problems in something you wrote just now is easy.  Recognizing other problems takes time, or distance, or both.

Graeme 

It's much more about providing T shaped value to the teams than about just doing a bit of everything with little focus and expertise

michaelabolton 

That whole T-shaped business sounded tired to me on the first day.

I'm not sure why.

Graeme 

I guess since the org is talking about building "T-shaped" developers anyway…its common language I can use

Even if its not ideal language in general

michaelabolton 

Yes.  It feels kind of naff to me, but it is what it is. Nonetheless:  it's reasonable to believe that everyone should have some degree of general skills outside their speciality.

The testers' speciality, it seems, is this:  living and working at close social distance, but farther critical distance than other people on the team.

Graeme 

We often have programmers "test" each other's work by pulling branches in code review and doing shallow testing. Yet bugs still get out, so obviously there's evidence there that having a non-expert do a shallow test doesn't solve the quality problem.

michaelabolton 

Exactly.

It helps to have someone at some distance from the work to evaluate it IF you need serious evaluation.

Graeme 

Right

michaelabolton 

Rather than having someone adopt that mindset, it's a powerful heuristic to have someone inhabiting that mindset.

Graeme 

So…to the people who say "engineers" deserve more money because their coding skills are hard, measurable skills…how is the monetary value proven in the tester's skills where you can't look at code output and "grade" it directly?

michaelabolton 

Coding skills are not hard and measurable... without testing.

(They're not even measurable with testing, but testing can tell us something about the quality of the product that non-empirical approaches can't.) 

Graeme 

interesting

michaelabolton 

You can't measure quality.  Measurement might inform some aspects or some attributes of quality.

Graeme 

Yes, that I am aware of and believe.

michaelabolton 

Quality is not about measurement.  It's about assessment and evaluation.

Literally, about how we value something; that's the value in "evaluation".

And that takes us right to the tester's role.

The tester's role is to focus on threats to the value of the product.

The tester's role is to focus on investigating the product to discover threats to its value.

The tester's role is to bring expertise and focus to that task.

For me, the TL;DR of this conversation boils down to two (or potentially combined to one) tweet-sized quotes from Michael.

The testers' speciality, it seems, is this:  living and working at close social distance, but farther critical distance than other people on the team. 

The tester's role is to focus on threats to the value of the product.