One of the more interesting characters I’ve encountered in my wanderings through the internet is a man by the name of Jocko Willink. He’s the author of Extreme Ownership and a business consultant. Oh, and an ex-Navy SEAL and a black-belt in jiu-jitsu. So, the man knows a thing or two about getting shit done under arduous circumstances. And his personal mantra is “Discipline equals freedom.” And the more I study operation science and the more I learn about software development, the more I see his point. So, in this post, I’m going to walk you through a multi-step process for testing code and how a little QA discipline can avail a lot of freedom.

Previously on “Game Planning With Science!”: Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 Part 7 Part 8 Part 9 Part 10 Part 11 Part 12

---

By Reading This Post, You Will Learn:

• Why discipline provides freedom
• Why its cheaper to fix issues as soon as possible
• A 6-step script for running a lean QA process
• How to reduce QA cycle times

---

What Does He Mean By “Discipline Equals Freedom”?

Simple: if you’re disciplined in how you handle your responsibilities, you will have more freedom to roll with what life throws at you. If you are disciplined about eating a healthy diet and getting regular exercise, you will have the physical freedom inherent in good health (injuries and disease not withstanding). If you are disciplined about studying consistently throughout the school term, you will have the freedom to sleep the night before your final exam, because you won’t need to cram. And, if you have the discipline to follow processes that eliminate waste, you will have the freedom to spend more time creating value for gamers rather than fighting fires.

That is the mindset you need to have when thinking about defects in your game code.

The Muda of Defects

In the intro to lean, i described the seven forms of muda and how they map to game development. From an end-user perspective (and, in most cases, a developer perspective), defects are the most painful. And defects breakdown into two broad categories: bugs and missed acceptance criteria. But whatever form they take, defects are best fixed when it is cheapest: AS SOON AS POSSIBLE. I cover this topic in more detail in my post on the time value of fixes, but, in short, a fix today is more valuable than that same fix in the future.

First, because you don’t know if it will actually be the same fix in the future. The longer it lingers, the more code gets built around it, and the greater the chance the scope of the fix will increase. And that risk bears a cost. Second, if the resulting fix does increase in scope as a result of lingering in the repository, you also bear an opportunity cost. The extra time required to resolve the issue if you defer it is time you could have spent creating more content.

So, a truly lean approach to QA would be focused on finding defects as soon as possible using a defined and disciplined process

A Lean Approach To QA

A lean approach to QA breaks down into five stages, with each stage justifying the next (eg, it’s not worth performing Stage 3, until the submission has passed Stages 1 & 2).

Stage 1: Buddy Tests

The first step of a lean QA process is a buddy test. Quite simply, when one developer is ready to submit his changes to the repository, he grabs another team member – another dev, a QA member, a producer, anyone – and asks her to check his work locally on his machine. This is essentially a smoke test: is there anything glaringly wrong with the potential submission? Have all of the acceptance criteria been met?

Stage 1 Goal: Ensure that the submission meets a basic level of quality and catch any glaring issues before the submission goes any further

• If submission passes the buddy test, move to Stage 2
• If it fails, address outstanding issues, then repeat Stage 1

Stage 2: Automated Testing

Before the developer requests that the submission gets merged into the build, he runs the relevant suite of unit, integration, regression, and functional testing (depending on the nature of the submission). The goal here is as described in the post on jidoka: leverage the speed and precision of automated testing to perform brute-force validation testing on the submission.

Stage 2 Goal: Let the machines catch any issues they can before you spend further valuable dev hours on QA’ing the submission.

• If submission passes automated testing, developer requests a build merge and moves to Stage 3
• If it fails, resolve issues, then repeat Stage 2

Stage 3: Peer Review

Before the developer’s merge request is approved, another team member reviews the changes in the repository. This is similar to a buddy test, except that a) it should be performed intra-disciplinarily (designers should review design submissions, engineers should review code submissions) and b) the review is at a deeper level of inspection. You’re actually reviewing code, scripts, Unity scenes, etc.

Stage 3 Goal: Let domain experts double check submissions to catch any technical issues or risks before they can contaminate the build

• If submission passes peer review testing, submission is approved and merged into the next build; proceed to Stage 4
• If it fails, reviewing team member identifies issues to fix, developer resolves issues and repeats Stages 2 & 3

Stage 4: Continuous Integration/Build Verification

The continuous integration process merges the code with its next build push. If the build fails, the CI machine notifies the team, so they can identify and fix any and all offending submissions.

Stage 4 Goal: Ensure the most recent submissions result in a stable build before moving forward

• If submission passes continuous integration, proceed to Stage 5
• If it fails, identify and correct the offending submission then repeat stages 2 and 4 (you can skip Stage 3 unless the necessary correction is large)

Stage 5: Manual QA Testing

QA manually tests the submission according to its own testing plan and scenarios. The objective here is to find the kinds of obscure defects that automation scripts would have trouble finding. QA should also test against the designated acceptance criteria.

Stage 5 Goal: Maintain a high confidence level that any code that goes to a lead for review is defect-free. Remember the that your leads’ time is generally in the highest demand, and thus is the most valuable. Therefor, it behooves you to now consume any of that time until you have the highest confidence that the submission is ready for review.

• If QA signs off on the submission, it goes to the lead or other person who needs to sign-off on the work
• If QA finds defects, the relevant developer fixes them and repeats stages 2 through 4 (again, you can skip Stage 3 unless the fix is notably large or risky)

Stage 6: Final Review

The person who requested the feature/user story reviews the submission from an end user perspective. Does it satisfy all of the acceptance criteria and technical requirements? Are there any lingering defects that the previous stages missed?

Stage 6 Goal: Verify that the submission satisfies the design, content, and/or technical needs for which it was specified.

• If the reviewer is satisfied, the feature/user story/submission is considered complete
• If the reviewer finds missing acceptance criteria, developer addresses and repeats all 6 stages
• If the reviewer finds lingering defects, developer addresses and repeats stages 2,3, and 5 (and 3 if the fix is large)
• If the reviewer finds that the submission is defect free (both in terms of bugs and missing acceptance criteria) but the feature, as spec’ed, doesn’t answer the original need:
  • Decide whether to overhaul the current submission, or
  • Scrap the submission, re-write the original request and start-over

---

Resources That Informed And Influenced This Post

  

If you have ad blockers turned on, you may not see the above links. Breaking The Wheel is part of Amazon’s Affiliate Program. If you purchase products using the above links, the blog will get a portion of the sale (at no cost to you).

---

A Lean Approach To QA?!

Here’s the trick will calling this stuff “lean”. Some of you might read the term “lean” and balk at a 6-stage process. Because, many of us, when we hear the term lean, think of Mick Jagger or David Bowie. We think of someone slender – without a lot of meat on his/her bones. But I like the body-builder definition of lean, meaning “no fat”. So, when I think “lean”, I think Jean-Claude Van Damme in blood sport. Beefy as all get-out, but with a minimum of fat.

Lean is not about reducing work. It’s about reducing waste. And this beefy, disciplined approach to QA can do just that.

Think about how the stages are arranged:

• Ensure you have something that is not obviously broken and has addressed all of the acceptance criteria before you waste any time testing it
• Find and resolve all of the issues automation can find before you consume any dev hours with testing
• Perform due diligence to ensure a submission is technically sound before you merge it into a build
• Verify that the submission does not hose a build before you have QA take a look
• Ensure that the submission is defect free before you ask a lead to look at it
• Verify that the submission suits both the intention of the original feature request and the needs of the game before you let it linger in the repository

But A Process Like This Takes Lots Of Time!

It can. But, going back to Scylla and Charybdis yet again, don’t just consider the costs, consider the savings. You expend known effort now to avoid variant disaster later.

Further, you have a measurable cost of time, which means you have an improvable cost (thanks, Peter Drucker!).

In other words, rather than stressing about how long a single QA-pass per submission takes, put your effort into reducing the time per QA cycle. Basically, a rigorous QA process is your insurance against build-melting disaster later. And the faster you can move submissions through the cycle, the lower the cost of your insurance.

Practical Ways To Reduce QA Cycle Times

Write Better User Stories/Feature Specs

The clearer you can be about acceptance criteria and technical requirements for each story, the easier it will be to buddy test, peer review, and manually test the resulting submission.

Write Smaller User Stories/Feature Specs

The smaller the story, the less time it takes to develop and test it. This particularly impacts the time to buddy test and the time to run a manual QA test. This means shorter cycle times and more rapid iteration on the build overall.

Give Your QA Team Members Bandwidth To Develop Testing Plans For Each Pending Story

If they can create testing plans ahead of time (eg, while the developer executes the story), then they can hit the ground running when the story moves to Stage 5.

Use Proper Task Tracking Software

Purpose-built services like JIRA and Hansoft make it easy to track user stories, from the basic stuff (owner and status) to the technical (build numbers, environment deployments, etc). The easier it is to maintain situational awareness, the easier it is to keep stories moving. Further, proper task tracking software makes it simple to track the average time user stories spend in each status.

Optimize Your Build Machine

Faster builds mean less time waiting for builds.

Increase The Rate Of Continuous Integration

If you are pushing builds more often, you will make stories available to QA with less leadtime (ie, less time waiting for the next build), and troubleshooting efforts on broken builds will be more efficient (fewer new submissions per build means fewer stones to overturn).

Prioritize the QA Process

Give team members the bandwidth they need to perform proper buddy tests and peer reviews. Better yet, establish a culture where such supporting efforts are as important as feature development. And don’t bog down QA team members with other work. Think of the effort as an investment rather than a cost.

---

Further Reading If You Enjoyed This Post

The Time Value of Fixes, Or: A Fix In The Hand Is Worth Two In The Bush

Discovery Versus Process – A Preface To Game Planning With Science!

Kanban: The Counter-Intuitive Value Of Pull-Based Production

---

Where Do We Go From Here?

Some grizzled vets might be asking “What’s wrong with doing QA at the end of the project? What’s wrong with saving the last few months of production for ironing our the defects?”

A lot.

First, recall the time value of fixes. The longer you leave this issues in the database, the more expensive they become. The second reason is a little more esoteric – namely, if you leave all of your defects to the end, you are “batching” the work in the extreme. And batching is bad. To understand why, we need to delve into our next topic: heijunka.

---

Key Takeaways

• Maintaining disciplined processes can free up time over the length of a project by stopping major production issues before they occur
• A lean approach to QA involves multiple steps: buddy testing, automated testing, peer review, continuous integration, QA review, and lead review
• The order of these stages ensures that no QA operation initiates until it has been justified by prior stages
• Once the process is in place, your focus should be on minimizing the cycle time per pass

---

If You Enjoyed This Post, Please Share It!

“The Muda of Defects, Or: Finding Freedom Through Discipline – Game Planning With Science! Part 13” by Justin Fischer is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

SaveSaveSaveSave