It struck me one day that “Game Planing With Science” has a glaring omission: the value of scrapping a plan. The goal of “Game Planning With Science” is to forecast, not predict. It’s to estimate and understand, but not to codify. You can’t codify the creative process, or the future for that matter. Just as important is the fact that life doesn’t care about your plans. Reality is going to be what it’s going to be. You can’t change reality to fit your plan, so modifying your plan to fit reality is the only path forward. As Dwight Eisenhower, one of the most immensely quotable people ever, once said, “Plans are useless, but planning is everything.”

Image source: Graphic Stock

By Reading This Post, You’ll Learn:

• Why it’s important to deal with reality and accept bad news
• Why you can’t trust yourself to be objective about whether your idea is actually good
• The value of early and regular testing
• Why successful entrepreneurs leverage the scientific method
• The formal definition of a sunk cost, how to identify one, and why they should not be factored into strategic decision making

A Lesson From Agile

The last of the four tenants of the Agile Manifesto states that responding to change is more important than following a plan. This doesn’t mean that following a plan is a negative or worthless activity. The manifesto specifically states that there is value in following a plan. But responding to change is more valuable.

There are two sayings that go hand in hand in my mind, and both are critical to management. The first comes from the Nobel Prize winning physicist, Marie Curie: “Nothing in life is to be feared. It is only to be understood.” The second comes from the Marine Corps: “Adapt and overcome.”*

We shouldn’t be afraid of bad news. Fear causes us to avoid bad or disconfirming information. But we need to know what the bad news is so that we can understand what it means. Once we understand it, once we know what reality is, we can stop wasting our time being afraid. We can start focusing on adapting.

Don’t fear the results of a bad playtest session. Understand what the bad news means. You now have something valuable: data. What was once unknown is now known – your game isn’t fun. What’s more you might have sense as to why it isn’t fun, which gives you something more valuable still – action items.

Likewise, don’t fear the emergence of a similar game releasing alongside yours or bad reviews or low retention. Understand what this emergent information means, and adapt to it.

The Ugly Baby Phenomenon

Objectivity is a hard thing to maintain when it comes to your own creative output. It’s easy to fall in love with a design, or mechanic, or gimmick, or moment. It’s easy to tell yourself that everyone will love it as much as you. Start-ups face this problem all the time: “Everyone will think this idea is as awesome as we do. We’re going to make all of the money!”

A serial entrepreneur I know phrases it this way for first-time and aspiring entrepreneurs: “Your baby is ugly.”

So, let me be upfront: your baby is ugly. Like, FUGLY.

Don’t assume that people will think your game is as fun as you do. Find out. Put a build in a player’s grubby mitts, and watch what happens. Watch. Don’t curate. Let him experience the game as he would if he had actually purchased it.

• If he can’t get figure out how to get through the game without you guiding him, he’s either not a very bright or your shit is hosed from a UX perspective. Assume it’s the latter until additional data say otherwise.
• If he gets frustrated while playing the build, he’s either not your target skill level or your shit is too hard. Assume it’s the latter until additional data say otherwise.
• If he gets bored while playing the game, he either has ADHD or your shit is not as fun as you think it is. Assume it’s the latter until additional data say otherwise.

Test Early, Test Often

While my honeymoon with scrum is long over, I do like the notion that every development cycle (or sprint, as it’s known in the parlance of scrum) should produce a working product that could be sold. That’s easier said than done, obviously, but it’s a useful benchmark to strive for. Every cycle – typically two to four weeks in scrum – you have a self-contained thing.

1. It provides a regular break point that you can use to stop, assess, and course-correct
2. It provides a regular output of a testable “product” that you can put in a prospective customers hands

In other words, this iterative cycle has a built in cadence of Understand => Adapt => Overcome. Even if you don’t use a scrum/agile framework, “test early, test often” should be your mentality for gathering user data.

Taking a Page from Entrepreneurs

Smart entrepreneurs are scientists at heart. They find the smallest possible tests to validate business ideas and then apply the scientific method: Question => Research => Hypothesis => Test => Observe => Anaylize => Repeat. The tests can be anything: focus interviews, surveys, Google ad campaigns, works-like prototypes, etc.

They start with the fundamental questions (Do people really have this problem? If not, what problem do they actually have? Does our product reliably solve that problem?) and iterate to the more nuanced ones (What’s the most effective channel for advertisements? What’s the optimal pricing model?).

Professional, serial entrepreneurs don’t fear change (or pivoting, as it’s known in the start-up world). In fact, they welcome it. I’d go so far as to say many entrepreneurs would be suspicious if a start-up didn’t pivot at some point. As the old military saying goes, “If your advance is going too well, you are walking into an ambush.”

I’ve long held that all (or at least most) game development is an entrepreneurial endeavor: lots of risk, no clear path forward, solving problems as you go, and irascible investors of one kind of another. So, why not use the techniques that successful entrepreneurs use?

Understand, adapt, overcome, repeat.

Pivoting: Not Just for Start-Ups

Pivoting can work wonders. Halo was once an RTS game. Twitch.tv mutated from one of the founders live-streaming his entire life. Nintendo started as a card company.

You initial design doc is just the starting point. It’s nothing more than an informed set of assumptions and intuition. Treat it as such. You should view your GDD as an elaborate hypothesis: “I think this will be fun”.

Your priority should not be to simply build whats in the doc, but to figure out how to test your hypothesis. What are the most fundamental aspects of the hypothesis, and thus the highest priority for testing? What is the smallest, fastest test you can run to verify those aspects of the hypothesis?

As you test and adapt designs, move from the most fundamental aspects to the most nuanced.

This does not mean let testers design for you, or designing for the lowest common denominator. Quite the opposite.

Test To Your Target Audience

When gathering data, it’s important to interpret that data relative to your target video game market segment. Let’s say you want to design a game targeted at the hardest of the hardcore. You have 10 players to test it, 9 of whom are casual to mid-core, and one of which is a hardcore gamer.

If all of the casual and mid-core gamers hate your game, but the hardcore gamer loves it, that doesn’t mean that you should re-balance for the casual set. It means you are on-target for your desired audience. Your hypothesis has been verified. The fact 9 out of 10 gamers hated it doesn’t invalidate your hypothesis – it supports it. Your observations indicate that your design is tailored for hardcore gamers.

Alternatively, if 9 casual and mid-core players love it, but the hardcore gamer is bored out of her mind, you should not view that outcome as success. You are off-target. Even though 90% of your testers loved the game, relative to your target audience, your hypothesis doesn’t hold water. You either need to pivot on your target audience (match the target to the game) or pivot on your design (match the game to the target). That’s not a bad thing. It just means you need a new plan.

Resources That Informed And Influenced This Post

If you have ad blockers turned on, you may not see the above links. Breaking The Wheel is part of Amazon’s Affiliate Program. If you purchase products using the above links, the blog will get a portion of the sale (at no cost to you).

 A Pox On Your Sunk Costs

A “sunk cost” is money spent that you can’t get back. Sunk costs tend to trigger a nasty little phenomenon known as the “sunk cost fallacy”, typically defined as “throwing good money after bad”.

In short, the mentality of someone falling victim to a sunk cost fallacy goes something along the lines of “Well, we’ve already spent so much money on this. It would  be a shame if all of that went to waste, so let’s spend some more and get it done.”

Sunk cost fallacies come in many forms. Many consider The Vietnam War as a prime example – “We’ve lost so many lives. We can’t let those deaths be in vain!”. But it also occurs in less atrocious settings. If you’ve ever waited 20 minutes for a bus, and said “Well, I’ve already waited this long…”, that’s a sunk-cost fallacy. Maybe you read half a book, realized you hated it, but finished it purely based on the time you spent getting to the half-way point. Or you purchased food, thought it was awful, but still ate it for no other reason than the fact that you spent money on it.

Or, for instance, if you continue following a plan simply because you spent so much time planning it.

How to Think about Sunk-Costs

One of the first and most oft-repeated lessons you receive in business school is to never fall for the the sunk-cost fallacy. The clearest way to think about sunk-costs comes from my finance text book. “If our decision cannot affect the cash flow, our cash flow should not affect the decision.”*

By definition, a sunk-cost is gone forever. Nothing you do can get it back, so it should not have any bearing on anything you do.

You should always make decisions on the optimal course for your game, studio, and career based on where you are RIGHT NOW, and the best information you have about the future. The past should inform your decisions in the form of experience, but you should base your decisions squarely on the future.

Sunk Costs aren’t Just in the Past

One important aspect of sunk-costs is that they can occur in the future too. For instance, if you have a contractual obligation to make some form of payment (for rent, or software licenses, or some employment benefits related cost), and you have to pay that price no matter what you do. That would be a future sunk-cost. And while an impending sunk cost might limit your options financially, it shouldn’t influence your decision criteria.

Don’t stick with a software package just because you are going to have to pay for it. The money is going out the door either way, so focus on what’s best for you and/or your studio.

Plans Are Useless, But Planning Is Everything.

I’m repeating that quote because it’s so crucial to managing complex processes. Planning is critical to an organization. A plan communicates an understanding of where you’re going and what you need to do. Plans align the efforts of team members. But, as Helmuth von Moltke said, ” No plan of operations extends with any certainty beyond the first contact with the main hostile force.” (often paraphrased more succinctly as “No plan survives contact with the enemy.”)

Reality will not cooperate with you, so just as you need to be willing to plan, and you need to be willing to scrap that plan and create a new one. Then do it again. And again. And again. This applies to production and design alike.

Nothing in life is to be feared, it is only to be understood. So understand, adapt, and overcome.

Key Takeaways

• Planning is valuable, but the existence of a plan does not make that plan more valuable than responding to reality
• Don’t fear disconfirming information – embrace it
• Understand that your baby is ugly – it has flaws and issues that you can’t see
• Don’t assume everyone likes your ideas as much as you, find out
• Apply the scientific method in all aspects of discover
• Be ready to pivot when observation discomfirms your hypothesis
• Interpret results based on your target audience, not popular opinion
• “Sunk costs” are expenditures that are unrecoverable
• Because you cannot recover a sunk cost, you should exclude it from any decision-making

*OK, it’s actually “Improvise, adapt, and overcome.”

Looking for more info about process management? Check out the Management & Operations Resources Page!

Return to the “Game Planning With Science” Table of Contents

“Sunk-Costs and Ugly Babies: On The Value of The Scientific Method – Game Planning With Science! Part 8” by Justin Fischer is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Related posts:

Character Art Pipeline Capacity Charts – Game Planning With Science! Part 2 Scheduling Video Games Scientifically! – Game Planning With Science! Part 7 Discovery Versus Process – A Preface to Game Planning With Science! Lean Development for Games – Game Planning With Science! Part 9 GDC17 Feedback and Responses Destruction and Creation: John Boyd’s Analysis and Synthesis Loop