I give a lot of presentations on growth, but my favorite topic of all covers “Smoke Tests” — a simple way to validate an idea before actually building product.

Why do I like Smoke Testing so much?

Because Smoke Tests have saved me $1000s of dollars, and saved my first business from going under.

Because I think it’s a method that’s widely relevant to startups and teams of all sizes, and yet it’s often forgotten about when it comes to GROWTH.

So, as Borat would say “I like a the smoke test!”

Smoke Tests are more than just 404 tests. They’re an actual process that, if done well, becomes ideologically engrained into your growth and your product team’s daily habits.

The fundamental thesis of Smoke Testing is:

Don’t let your product roadmap hi-jack growth, rather let growth drive your product roadmap.

Let me first explain the process I now use at 500 Startups as part of the 500 DISTRO team to help companies learn faster, grow bigger, and save money. Then, I’ll share some real life examples of smoke tests I’ve used.

SMOKE TESTING 101

What Is A Smoke Test

A simple way to validate an idea before actually dedicating technical resources to building product.

Why Do I Use Smoke Tests

You should use smoke tests when you believe that a new product/feature/function will increase your growth rate, but in order to accomplish this “new idea” you have to dedicate additional resources.

Use a smoke test to validate this idea and prove it to yourself rather than building product — because that way you don’t stunt growth.

How Do I Figure Out What To Test To Run

So you know that Smoke Tests are a good idea, but what tests should you run? Not knowing your business, it’s impossible for me to tell you exactly what to test.

Here are the questions you should ask when coming up with ideas for what to smoke test:

• What are the bottlenecks in my business and what do I think could solve them? Is it customer acquisition, is it activation, retention, etc? Figure out the highest impact change you could make then Smoke Test it.

• Where are resources most strapped? Is your product backlog just a blackhole of “ideas?”  If so, try to validate more of them before passing them to product team. But, don’t waste time trying to validate anything that isn’t directly correlated to growth.

• What is a product feature I’ve recently released that didn’t have the impact on Growth I expected? Make a list of iterations and Smoke Test those.

How Do I Run A Smoke Test

Always make sure that you prove your ideas to yourself before involving others. As Parker Thompson says :

“ If you can’t clearly explain why you’re doing something, then you don’t know what the fuck you are doing.”

I couldn’t agree more.

Next is to figure out how to basically implement these ideas and try them by yourself or with no engineers (unless you have an unlimited amount of cheap engineering talent that can pump ideas out in hours, not days — probably not the case!).

Make sure you run your test properly too (i.e., that the people you’re running the test on actually represent the correct audience). Don’t cheat yourself and just pick people it will work with and don’t cheat your company by being sloppy and not investing the time to figure out the right traffic/customer base.

It’s obvious, but I have to say it because so many people forget it.

KEEP SCORE.

If you don’t keep score, nothing matters. So, make sure you have the right tracking setup. This can even be something really simple like Google Analytics.

Once you have your tracking in place, then you’ll be able to use them to optimize. MOST SMOKE TESTS AREN’T SUCCESSFUL OR EVEN ACTIONABLE ON THE FIRST ITERATION.

As a result, making a decision is really tough. So a lot of this has to do with your confidence you tested the idea the best way possible using this process.

Now… the stories that share some of the reasons I LOVE SMOKE TESTS 🙂

The first, and hopefully last, time I almost went bankrupt 🙂

My first business, GarmentValet, was hemorrhaging money — around $30K/mo actually.

My founder and I, along with the entire team, were subject to groupthink and collectively decided that we should wait on “marketing” or in my co-founder’s terms “opening the floodgates of customers waiting for our service” until we had quality completely 100% squared away.

How would we do that? Automate the shit out of every process from top to bottom.

I’m talking silly stuff. We built our own CRM, we built out apps that drivers would use to scan in bags, we built out apps that ran conveyor systems based on barcodes we’d heatseal onto garments — the whole 9 yards.

When we were finally “done” we “turned on marketing” — haha.

Turns out that there were NOT floods of customers waiting to use our service.

Turns out that growing a laundry and dry cleaning delivery customer base is actually quite painful because no one cares enough to change until their cleaner fucks up their order and no one trusts you that your service is “better.”

The good news is that we had some business already, around $100k/mo. But, we were spending $130k to service them and pay our way through the credit card debt my cofounder and I racked up to build all these “cool” systems.

Sounds like you were failing DistroDom, what does this have to do with Smoke Tests?

So this one night I was working with Ryan Bloomer and we figured out that all of our profitable revenue came from customers in 4 zones, all where the highrises were. So we decided we needed to cut everything else and just get more of those customers.

Pretty much overnight we did these three smoke tests:

Smoke Test 1: Reduce/Cancel Service To Non-Condo Customers

Rather than engineering some complicated service schedule, we just deactivated all the buildings we didn’t want to service that didn’t have active orders and sent all the customers in those buildings a nice email explaining we no longer serviced them with a link to other cleaners (hyperlink to google search Dry Cleaners Near Me).

RESULT: A few angry customers, but no more driving out to random neighborhoods for a $20 order.

Smoke Test 2: Get Condo Customers to Refer More Customers

We printed out all these Willy Wonka style tickets with manual coupon codes we set to only work at those addresses and sent them out to each of the customers in our buildings.

The call to action was simple: “Invite your neighbors by giving them this card and you’ll both get $50 in cleaning.”

Since we had drivers going to these buildings we literally did this super manual and relied on concierges to give them to the customers.

RESULT: We increased the # of customers we had in desirable buildings by 50% in one month, so each trip was significantly more profitable.

Smoke Test 3: Increases the Order Minimum for non-condo customers in condo zones

We told customers in areas with Condos we were upping our delivery minimum from $20 to $35.

RESULT: Some complaints, but we didn’t actually lose more than a handful of customers because revenue actually grew and customer attrition stayed steady.

SMOKE TESTING – DO IT.

So, in summary, I really believe that all great founders should quickly validate their ideas to prevent their product backlog from stunting growth.

I think the best way to do this is run Smoke Tests, for GROWTH.

Have you used Smoke Tests for growth marketing? Let me know @distrodom or in the comments below.