Episode 145 | Does Lean Startup Work for Bootstrappers?

• Facebook
• LinkedIn
• Twitter
• Email

Show Notes

• WordPress Digital Signature Plugin
• Nice slideshow overview of Lean Startup Tenets
• Lean Startup Wikipedia page
• Lean Startup book on Amazon
• Lean Startup website

Tenets discussed:

• #1: Startup Defined
• #2: Validated learning
• #3: Build-measure-learn feedback loop
• #4: Minimum Viable Product (MVP)
• #5: Innovation accounting
• #6: Pivot

Lightning Round

• #7: Customer development
• #8: Problem-solution fit & Product-market fit
• #9: Continuous Deployment
• #10: Split testing
• #11: Actionable metrics (vs. vanity metrics)

Transcript

[00:00] Mike: On today’s episode of Startups for the Rest of Us, Rob and I are going to be talking about lean startup. This is Startups for the Rest of Us: Episode 145.

[00:07] Music

[00:14] Welcome to Startups for the Rest of Us, the podcast that helps developers, designers and entrepreneurs be awesome at launching software products, whether you’ve built your first product or you’re just thinking about it. I’m Mike.

[00:22] Rob: And I’m Rob.

[00:23] Mike: And we’re here to share our experiences to help you avoid the same mistakes we’ve made. What’s the word this week, Rob?

[00:27] Rob: I noticed that you have a really cool email opt in widget on auditshark.com as well as a nice email mini course that looks like you launched this week.

[00:36] Mike: I did. Did you like that?

[00:36] Rob: I did. Mike is now using Drip. I logged into the Drip dashboard and I noticed that his account was all setup went to Audit Shark and it looks like your conversion rate is going really well. Your subscription rate is going well and your course is frankly one of the better ones that I’ve seen. Yours has a lot of data and a lot of research and I’m looking forward to the rest of it. I‘ve only seen the first email at this point.

[00:56] Mike: Yeah. I’ve sent it out to a few different people and they forward it on to some other contacts in the industry. I think that might be influencing the conversion rate a little bit but you and I did track through and it looks like the conversion rates are pretty good anyway but it’s nice to see that – and just in talking to a couple of people, the information they’re seeing in there is really good so far.

[01:15] So I kinda want to wait ‘til people get through the entire course to give me feedback on the entire thing but I’ve re-read everything a couple of times, gone back and forth through some of the emails and the sequence, just to make sure that I’m not repeating myself too much and that it naturally leads from one email into the next.

[01:32] Rob: Yeah. Very good. What’s nice is every subscriber you get to that would’ve just come and left the site even if you get 20 or 30. I mean there are some people getting a few hundred a week, yeah, just nice to be able to follow-up with them later. It’s kind of building your early access or your beta list depending on the stage you’re at.

[01:47] Mike: And of course the obvious benefit since I’m seeing their email addresses is to be able to track them back to see the types of companies that are taking a look at it. Obviously you can’t really do that with Gmail account. But when you start getting corporate domains in there for the email addresses, it’s really nice to go in there and say they’re a PCI compliance company. Maybe they’ll look and use Audit Shark for their customers or they’re a health insurance company and it looks like maybe they want to lock down their servers and do HIPAA compliance on those machines.

[02:13] Rob: Yeah I’ve been doing the same thing. So Drip is continuing to move forward there at the point where I frankly don’t even have account on the number of people who are in there using it. It’s in the low to mid 20’s at this point. Another paying customer this week, the trial ended. Like I said, I’m doing trial period based on I log in and I say hmm, I think they’ve received enough value from this and I’m going to email and say do you think you’ve received enough value? Are you interested in paying? We’ve got one more this week and think I’m going to email another guy today. It’s cool.

[02:42] So revenue’s in the mid three figures, that’s the SAAS ramp. That was in July and I’d expect to double or triple that this month and then move on from there. The plan is in the next – we have about 2-3 weeks left of features that we need to get in because to this point I’ve been doing so much stuff one off in terms of on boarding people and the trial emails and just all that touching base. So we need to get all of that automated because I want to email a couple hundred people on the launch list. And before I do that, I’m going to need to get stuff in place to automate it because I just can’t do that part manually. So that’s about it.

[03:19] So hopefully it’ll probably be a couple more weeks before things really start to move with Drip. Busy behind the scenes building a lot of features but nothing new to report for a couple weeks until that email goes out. And then I’ll be running frantically.

[03:30] Mike: Yeah. I’m probably a bit behind you. I’ve been going through – and there’s about a little bit more than a dozen machines that are currently being audited through Audit Shark. It’s good to see that those things are going in there. And I’m talking to two different people that have been added in this week. It’s kind of funny that there are sections of Audit Shark that I spent just a phenomenal amount of time and effort and making sure that they would work and that they were simple because they are far from simple by any stress of the imagination.

[03:57] And both of the early access people who went in this past week, both of them just kind gave it a hand wave or one liner and said “oh, well I did this and then now what do I do?” And they just completely glossed over all the complexity of that. And on one hand it’s like “oh, I can’t believe you didn’t notice how hard that is” but at the same time it’s also a really good feeling to say the time and effort that I spend on that was justified because it’s so incredibly simple to use that they didn’t even notice how hard it was.

[04:24] Rob: Right. You don’t want them to notice how complex it was.

[04:26] Mike: Right. Going the big things I’ve gotten back is that they’re interested in immediately skipping to the remediation pieces which is not something that I had really planned on tackling right away because remediating people’s servers is kind of a risky thing anyway. But it seems to me like they’re really looking for that guidance and input and to what they should be doing, how they should be prioritizing things and what their next steps are once they get the information back.

[04:52] So it looks like I’m going to have to go through and start implementing those remediation steps that I was thinking that people just would not be interested in because they’re production level servers and it turns out they are interested in that. So I’ve got my work cut out for me there.

[05:04] Rob: Right. And by remediation you mean your auditing software finds that these things are wrong and now how do we fix them right?

[05:10] Mike: Yeah.

[05:11] Rob: I received an email from a listener this week. He had just kind of a success story to tell. He’s building a SAAS app and then he decided after reading my book and kind of fall in love with you to build a WordPress plug in to get an early win. He said “here’s a short back story about how you guys have altered the course of the project.”

[05:28] “Initially I started building a SAAS app for approval management and that’s at approveme.me” he said “but after $1,500 and a development cost and not knowing at the end of the day if anyone actually wanted what I was building I came across your writings, the chapters about…” he calls out specific essays and then he says “I immediately put the brakes on the development of the SAAS and I started brainstorming about a better way to build a list in the same market space before trying to tackle the goliath of SAAS.” He said this is how WP digital E signature was born. So it’s a WordPress plug in to help people do digital signatures.

[06:00] He says “per your advice, I outsourced the programming, built the marketing landing page alongside my team, ones who got the very first premium release with bugs and all. I setup the store just to see if anyone was interested. One guy emailed. I gave him a 50% discount and he’s been super helpful. Just the other day we received our very first real order from start to finish on the product without any direct human interaction. It’s the best feeling in the world to know that someone somewhere was willing to pay money for a product that I’ve created. Taking your advice has helped save thousands of hours and brought us much closer to the starting line.”

[06:31] That was from Kevin Gray thanks a lot for writing in Kevin, definitely good to hear. Like I said, his SAAS app is at approveme.me and from there he has a linked to WP digital E signature.

[06:41] Mike: That’s a really cool idea for a WordPress plug-in. I hadn’t even thought of that.

[64:45] Rob: Yeah I know. I’ve been seeing a lot of web apps that are then being encompassed or built into a WordPress plug-in. So there’s like kind of an event registration system that’s in a WP plug-in. You can take almost any SAAS idea and put it into a WP plug-in. It’s a pretty interesting concept for finding new businesses I think.

[07:02] Mike: Definitely. The only there news I have is MicroConf Prague is now sold out.

[07:07] Rob: Yeah that’s very cool. It was by 8 or 9 days. So sold out house at this point. We’ll see you in October if you’re going to show up otherwise maybe you can make it Vegas next April.

[07:16] Music

[07:19] This week we’re going to be giving our thoughts on lean startup. This topic has been requested in the past but I think since we just talked about Dan Norris’s stuff about his article of is startup validation bullshit, we received another email. We received a voicemail basically saying you guys commented on lean startup. We’d love to hear your thoughts on. I guess the pieces that we feel that apply to our businesses, the pieces that maybe we feel are less applicable to bootstrappers, less applicable to people launching small apps.

[07:48] The challenge with this and the reason we’ve never done this before even though people have requested it is that lean startup is huge and it’s getting bigger all the time and its changing constantly. So lean startups started with a few key concepts and its now – frankly its more than we could define in a podcast episode much less, actually give our thoughts. And so this makes it hard not only because it’s a big subject but it just seems to be a moving target.

[08:10] So basically, what we’ve done, I went through all the lean startup principles and tenants and try to boil them down to the ones that we know the most about. Frankly since it is so big, neither you and I are experts in it. You’ve read most of the book. I read the book. We’ve both seen Eric Riese speak and give his overview of it and certainly you just hear people talking about it and all that stuff.

[08:33] So we know the concepts of validated learning and minimum viable product and the buzz words in that. But I think we want to get into giving our thoughts on specifically on some topics and some will have to define and then others will just kind of run through.

[08:47] I guess to start off with, if you’re only kind of familiar with lean startup, it basically started in 2008, Eric Riese standing on the shoulders of people like Steve Blank who came up with customer development, Marc Andreesen who coined the phrase product market fit, agile software development practices, lean manufacturing which is used by Toyota, lean management which was morphed off lean manufacturing.

[09:10] Eric Ries took all of that and complied it into kind of a single methodology and so he started blogging about it and then it started picking up steam. He knew Steve Blank. I think he was a student of Steve Blank. So Steve Blank encouraged him to push this forward. So he was blogging about it then he did a lot of speaking and then finally he wrote the book.

[09:30] Mike and are going to be talking about 11 topics lean startup topics and giving our thoughts on them. The first six we’re going to define because we think some people may not what they are. The last five were just going to run through them in a lightning round and kind of give our thoughts. Now the challenge here is that if you work in – let’s say you’re in a huge enterprise and you’re trying to launch a new product, well lean startup says it can apply to you.

[09:54] If you’re launching Brick and Mortar lean startup says it can apply to you. If you’re launching a bootstrap startup or a venture funded startup, lean startup tries to apply to all of those. So we are going to give one point view on each of these topics but I would never go out and say well validated learning never works or it’s a dumb idea because maybe it just doesn’t work for this specific group of people or in my specific experience.

[10:14] The first thing I want to talk about is how lean startup defines a startup. The quote is “a startup is a human institution designed to deliver a new product or service under conditions of extreme uncertainty.”

[10:28] Mike: If you take a look at that exact definition then sure it can apply but I almost feel like when you start talking about startups, you’re really talking about – I guess my world is more of the software startups but it just seems so incredibly overly broad. I think that’s my big issue with it. And pretty much anything is when you’re building a new product or service, doesn’t a new product or service kind of define itself as being under extreme uncertainty. I mean how can you have a new product or service that doesn’t have a lot of uncertainty that goes along with it?

[11:02] Rob: I think that’s what the definition is saying though. I think this definition in particular is clever. I think he encompassed a lot of possibilities with this single sentence and in a way that no one had done before. I like the definition. I agree with you that it is brought by saying human institution because he’s covering his basis and saying well, it could be a nonprofit. It could be someone doing it within Proctor and Gamble or another Fortune 500 company or it could be you starting a new bar down the street.

[11:28] Lean startup principles have been applied to all of these business types and their examples given in different books on customer development and they do cover all those businesses. I think that would be my one quip. Maybe not what this definition itself, but with lean startup is that by being so broad and by trying to encompass every potential thing of anyone launching a new product, I think it actually provides less value.

[11:51] Mike: So topic number two is validated learning. The idea here is everything that a startup does should be an experiment and whether that’s marketing campaign, your ability to feature, the entire product itself. Any effort that’s not absolutely necessary for learning is wasteful and needs to be eliminated. And really what you’re trying to do here is you’re trying to ask questions about what are we trying to accomplish? How are we going to measure whether or not that was successful?

[12:18] What you’re really trying to do is make sure that when you’re doing stuff, so move the product forward that you’re making the right measurements that need to be made so that you could repeat those and scale them up and things will not fall apart on you later on because you are measuring the wrong things. You’re not making assumptions but you’re using hard data to make decisions that are going to move you forward.

[12:38] Rob: Yeah. I think the key sentence here is the effort that is not absolutely necessary for learning is waste and should be eliminated. This is an insight that I hadn’t heard. I wasn’t aware of before at lean startup and I’m pretty sure this came from lean manufacturing. I think that Eric did a really good job with this, that validated learning was it was always present in startups and that’s what a lot of the lean startup stuff is. It’s just putting a label. Its putting a name to something that already existed but that we didn’t quite know how to talk about.

[13:07] I think that maybe the number one thing that lean startup has done well is its accomplished something, it’s given us a common vocabulary for a lot of things that didn’t have names before. So take something like validated learning, problem solution fit, product market fit are two that I use all the time on the podcast. And MVP is another. All these concepts existed before lean startup but now I can say two words and most people in our world know what you’re saying. Lean startup accomplished that and I also think that validated learning as a concept and the fact that it now has a name I think for me has been helpful.

[13:43] Mike: Yeah I think one of the things you have to keep in mind when you’re thinking about validated learning is that let’s say you’re building a new marketing campaign and its going to be email marketing. Well if you don’t necessarily know what you’re doing then you’re going to have to try a bunch of things.

[13:58] The key to this validated learning piece is to figure out what your theories are, what your hypotheses are. figure out how to go about testing them. Make sure that the results that you’re getting back can be quantified and that you’re learning something from it. And anything that you’re doing that you can’t quantify or you’re not going to be able to learn from or that you are not learning from, just get rid of it.

[14:19] And the fact is that this doesn’t apply to scaling the business. this just really applies to finding out what works and what doesn’t. Once you figure that out, then you can basically blast it out to the masses, scale it up and do it very, very quickly. But this is kind of the fundamentals that you’re working on here with validated learning. You need to identify what works, what doesn’t, and then you start throwing resources at it.

[14:42] Rob: So topic number three is the build, measure, learn feedback loop. This concept basically says you should build something. You should build the minimum that you need to build in order to learn. Then you should so something and measure it. So if it’s a landing page then you launch it and you send traffic to it and you split test it or if it’s a new feature then you see how many people use and see their reactions and see whether they like it.

[15:05] Then you learn from that and you look back and build again. I like this topic. I have kind of mixed emotions about this. I literally have this exact diagram that Eric Ries uses in my notebook that I had written in like 2007. So it was before lean startup was around at all. I didn’t use build measure learn but I was thinking to myself what were the systems that I’m doing over and over? What is this cycle? And I realized I was basically building something – I think I put build, test, adjust or something like that but it’s the same concept.

[15:35] It’s like you build it and then you get someone to use it and then you adjust and tweak and come back in a circle. I’ve done it, a lot of people have done it and this is another place where I think Eric did a job of putting a label, a common label on something that already existed and that people were doing a lot I startups already. So not a new insight, maybe just a new label for it.

[15:56] Mike: Yeah I almost feel it’s a new application of that terminology because if you go into any sort of electrical engineering there’s all these feedback loops. I mean the one most people are going to be familiar with is like cruise control on a car. If you go too fast and there’s a feedback loop that says hey the car needs to slow down a little bit and if you’re going too slow there’s a feedback loop that says hey you need to speed up a little bit in order to get to that steady state speed.

[16:20] And this is really just the exact same thing. You’re saying build something, put it out there, measure, figure out what needs to be done to fix it or improve it and then go through that process again. I think the thing that Eric has done that’s really important is that he’s takeng those concepts and ideas and apply them to startups where I just don’t think that’s really been quantified or codified in terminology before.

[16:42] So our next topic is minimum viable product. The minimum viable product or MVP is the fastest way to get validated learning so you can make it through that feedback loop. You need to be able to have a product that some software or whatever your service is in place in order to get that process started because you can make all the theories in the world. You can make all the assumptions that you want and talk to all these different people.

[17:07] But until you actually have a product or a process or service in place and start implementing it, you’re not going to know what the deficiencies are. You’re not going to know how things actually workout. So just for example, let’s say that you’re trying to sell something for – you think its going to cost you $60. Well until you build it and you start going through and the process of building it and figuring out oh, you sure it only cost $60 to build one but 1 out of every 10 is a bad widget and we have to throw it away. So it actually increases your cost of production by 10%. So it’s no longer $60. Its $66.

[17:42] But until you go through that process you wouldn’t know what your defect rate is. So those are the types of things that are really important to find out but you don’t know those until you’ve built your minimum viable product.

[17:50] Rob: So minimum viable product is interesting because some people think that the MVP is as simple as putting up a landing page and figuring out if people want to buy your product. And that is like the way over simplified version of it. Really what lean startup is saying is the MVP is the minimum step you can take in the next week or two weeks to get more validated learning.

[18:12] So depending on what phase you’re at, the minimum viable product can be just a single feature or it can be just a single adjustment or it can be something that you do manually. Actually there was an example I heard the other day on Steve Blank himself who basically came up with customer development was saying that your MVP isn’t a prototype. It doesn’t have to be a prototype of what you’re trying to do. It can be a manual version of the same thing.

[18:38] And we’ve talked about this on the podcast in the past of like don’t build a whole SAAS app to do X. Go ahead and do X manually or hire a VA to do it. Send out emails manually. I mean that’s basically what you and I are doing with early access right now. We haven’t built all the trial emails and the on boarding emails unless we’re sending them manually. And that essentially is a minimum viable product and for Drip to have 20 something people using it right now but really to not have a billing engine built but I’m still billing people. And then not have trial emails going out but they’re still receiving them means that I took that minimum step to get there.

[19:41] So again, the interesting thing is I was doing this before it was named right? Before MVP, a lot of us were doing that. And so I don’t think this was some – it wasn’t a brand new concept, however him putting a title to it that we can all talk about I think is something that’s very valuable and I also think that people who weren’t doing it have started to do MVP type stuff and I think that’s helpful.

[19:37] On the flip side, this is what maybe gets people in the most trouble is your judgment of what an MVP is, putting out a crappy product and sending your entire email list to it and seeing who converts is not the right way to do it either. Right? That’s a big mistake and it can cost you a lot of money and a lot of time if you do it that way. And there’s nothing in lean startup that says don’t do that. There’s no explicit kind of warnings so people I think are making a lot of mistakes with this concept in particular.

[20:05] So I think it’s a pretty dangerous concept if you don’t have a high standard of what you need to deliver and if you don’t take it pretty seriously and really understand what an MVP is rather than read the ones in this description and then go off and think that kind of building a crappy product is enough because you can call it an MVP.

[20:21] Mike: I guess the confusion and the mistake that people make is the word product is in the name so they think oh well I have to create a version of the product that needs to do X,Y and Z and that’s how I’m going to put this out there and that’s how I’m going to be successful. And if you read the definition which from Wikipedia’s page, it says a minimum viable product is the “version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort.”

[20:49] Really what that can mean though is you’re just building a single feature and that’s the next version of your product because its different than the previous one. And adding this new feature is really the key to that new product and that’s what you’re testing. It’s not you’re testing the whole thing. You’re just testing this one little piece of it which is why terms like minimum viable feature have come out of lean startup because minimum viable product itself is just kind of confusing people, what constitutes a product. And as you said, that’s where people just get into trouble especially if they don’t have this minimum quality bar that they’re adhering to.

[21:23] Rob: Right. I think that’s one of the tricks with lean startup is it is quite academic. This might be one of the negatives of it. It comes out of academia which is different than actually being in the industry and building startups. So a lot of the stuff is complex, its nuanced and it’s hard to take complexity and nuance and turn that into an actual piece of code or an actual decision of what then do I build? So the nature of it being high level almost allows it to avoid taking responsibility for the actual steps you take.

[21:53] Lean startup, I mean I guess I’m going into a con here. Its not super actionable and I think that’s probably a side effect of it being created more by academics rather than people who are actively day to day launching startups. Because most of the writing, most of the books that are being written on lean startup and lean entrepreneurs and that kind of stuff are being written by people who aren’t launching startups now. For the most part they’re not as experienced as some of the people who read day to day who are just blogging about startups.

[22:20] Our next topic is innovation accounting. Innovation accounting refers to the rigorous process of defining impeccably measuring and communicating the true progress of innovation such as customer retention and usage patterns whether for startup companies, for new products or business units within established companies.

[22:38] Basically, typical accounting metrics often don’t work for startups. You have to adapt different metrics and make sure they’re not vanity metrics. So that’s the concept of innovation accounting.

[22:48] Mike: I was one of the reviewers of the Lean Analytics book by Ben Yoskovitz who was also a MicroConf speaker this year. This topic is very well covered in that book. There are tons and tons of different ways of measuring things and based on what if you’re trying to gather, what the underlying purpose of gathering that information is, I mean you can look at the same data and slice it 10 different ways and have 10 different – even 15 different conclusions that come out of that set of data just because of the types of things that you’re looking at or you’re that you’re trying to do.

[23:22] So innovation accounting I would recommend just going out and getting the Lean Analytics book to figure out how it is that you go about measuring things but I like the idea of innovation accounting because you’re not looking at these vanity metrics. You’re not looking at things that they make it feel like you’re growing your product but you’re really not.

[23:39] One of the things for example is how many people are coming to my website? People say well my product is growing I’m getting a lot more people to my website and the fact is you may not be because as you add customers to your product, if they’re coming to your website to log in then they are indirectly counting towards your statistics towards your new visitors. And that’s just not accurate at all. It’s not accurate to include those people who’ve already been to your website and are actively using your product in your website to visit your account.

[24:07] Rob: I have mixed feelings about innovation accounting. I like the concept that if you’re in a Fortune 500 company that maybe you shouldn’t be held accountable to the standard accounting principles for the first 6 months or a year because you’re looking at different metrics and that you should avoid vanity metrics. That I all agree with.

[24:24] The problem is this doesn’t help anyone who is starting, trying to bootstrap a SAAS app today. If you’re going to bootstrap a SAAS app, don’t try to understand what innovation accounting is. It’s basically an astronaut’s view of what are the key metrics that you need to look at? Like Mike said I’d recommend buying Lean Analytics and reading it but I’d also say let’s say you’re going to launch SAAS app, here’s what I would look at. Not unique visitors to website but I’d say what’s a percentage that that are converting to your email list? What’s the percentage that are converting to trial? What’s the percentage that you’re converting to paid? What’s your lifetime vale per customer and what is your return rate?

[24:59] With those five things, if you could show that to me, just those five numbers, I would have no idea about your business, I can tell you whether or not your business is doing well, whether its going to do well, whether you should drive more traffic or less traffic. So that in my opinion is the short version and that’s where a more specific laser focused resource that’s actually talking about these individual things I think could be more helpful because for bootstrappers, bootstrapping, a SAAS app, that’s what they need to know.

[25:25] What about someone with venture fund and go after a B to C app? There’s a different list. Well what is that list then? You can define innovation accounting but that doesn’t help the guy who’s sitting there trying to figure out what things do I need to look at. And if you’re in that Fortune 500 company trying to launch a new product out of Proctor and Gamble, what is the list that I need to look at because I bet you could get pretty close.

[25:45] I bet you could define a short list of things that everybody needs to know and that’s where I wish innovation accounting had actually taken that next step and gotten more specific for people who actually want real actionable answers rather than a high level definition of something.

[25:59] Mike: I think the other problem with innovation accounting is the fact that because of the way that its phrased, it makes it seem like if you’re following lean startup principles then you should be doing innovation accounting. And there’s this massive chapter in the book about innovation accounting and how to use it, how to leverage it. And the fact is depending on where you are in your business, it has completely different meaning. It has a completely different application and a scope of application in your business.

[26:28] In the early stages you’re just trying to figure out what works, what people are interested in so that your minimum viable product will actually get some traction. But when you’re much, much further on, it’s a completely different story. You’re going to be measured all these different things. As you said, when you’re first launching a SAAS app there’s like five things that you’re looking at. When you get to the point where you have hundreds of thousands or millions of dollars from revenue, you’re looking at probably 30 or 50 different things and they all mean something different.

[26:55] There’s a scale that you have to take into account from when you first get started to when you’re much further down the road and you’re trying to scale the business. I don’t think the lean startup book really covers that or explains that very well and says hey look, if you’re in your early stages, you don’t need to worry about most of this stuff. Know that it exists but just kind of ignore it for now until you get to this stage.

[27:15] Rob: Yeah I think that’s an issue. Lean startup is so big and has so many concepts but some of them are completely not relevant for bootstrappers on the web or bootstrappers doing mobile stuff and others are and there’s no real place that designates that that I know of. So you kind of get information overload with all these concepts. Maybe I should call it concept overload where you’re just trying to get your head wrapped around them and it can be more of a distraction for someone.

[27:40] If you’re really boots on the ground and you’re trying to launch something tomorrow, I would go to the Wikipedia page and I would read through these concepts and I’d listen to these podcasts and maybe read like a summary of the concepts but just getting deeper into them than that doesn’t help you take that next action of what you actually need to get launched.

[27:56] Mike: So topic number 6 is to pivot. This really is based on validated learning. It really requires somebody to take a look at the information and you need to be able to decide whether or not you’re going to pivot or whether you’re going to continue down the path you’re on. It’s a very hard thing to quantify into a book and say well if you’re in this situation, you should pivot. If you’re on this other situation, you should persevere. It’s really kind of a gut feel and I think that’s one of the things I don’t like about this is because there’s no real hard rules around it.

[28:28] I think part of the reason I went into computers because there’s 1’s and 0’s and its some very easy to define when you should do what. But when you get into other subjects like this, it can be very, very difficult to make those decisions especially because you don’t have a lot of information. You’re making a decision based on incomplete information and sometimes that’s very difficult and sometimes you’re going to make the wrong decision but most of the time it’s really just the gut feel of where you want to go and what the data is telling you.

[28:54] Rob: Yeah. I’ve heard the phrase pivot or persevere and that’s your choice at any given moment. As soon as you do that build, you measure and you learn. At that point you choose to pivot or to persevere. And this reminds me of the concept of The Dip that Seth Godin had a book called The Dip. And the number one question he’s asked about that book is when is the dip? How do I know that I’m going to make it through the dip or that it’s not a long term thing? And that’s the question I see asked about pivots.

[29:18] How do I know if I should pivot or not? And so I think that’s a challenge of a concept like this. I think the introduction of this concept I was intrigued by it. I liked that it was basically defined that you could be actually going along with a certain business model or you’re building a certain app that you expect people to use a certain way. And then when they don’t, what do you?

[29:28] I’m not sure that I had ever actually talked about wow I could take this app that’s going to be a time tracker and suddenly turn it into an invoicing app or an expense tracker instead because everybody wants to use it like that. And so the concept of a pivot, my mind was opened when I first read about it. So I do like it and I think it expands your thinking but I also think like anything, this gets over used and people are talking about starting a game.

[30:02] I think the story is that Flickr started as some game and then became a photo sharing app and they’re completely different. I totally don’t see that as a pivot. It’s more like it can be used as an excuse for yeah, we just failed at one business and we started another one under the same shell company name with the same funding and we’re going to now call it a pivot.

[30:21] So I’ve also seen I think this has been used by venture capitalists or people who are starting VC backed companies as kind of a reason or as an excuse for you being able to just abandon the idea all together and go to another direction. The pivot is supposed to be based on validated learning and that you’re only supposed to pivot if you have data pointing you in that direction. There is a gut feeling component to it but I don’t think everybody’s doing that from my observation and what I’m hearing called pivots.

[30:46] Mike: I know exactly what you’re saying. I mean you hear a lot of these VC backed companies and they pivot into some completely different direction or its kind of tangentially related but there’s no clear evidence of why they would have gone in that direction. A lot of times just the plug isn’t pulled on them because the VC’s have invested in the team not necessarily the technology or the product and they’re investing the team to say hey, we trust you guys to make the right decisions to go in the right directions that you feel are appropriate and that’s what we’re investing in. We’re not necessarily investing in technology or the software that you’re building. it’s what you guys think is best and in you guys.

[31:24] Music

[31:28] Rob: Alright, so right now we’re going to dive into the lightning around where we’re going to cover five topics and we’re not going to define them because we’re going to assume that you can either go look them up or that they’re common enough that you know what they mean. We’re basically just going to give our thoughts on what we think about them. Do we think they’re valid for bootstrappers and so on.

[31:44] So topic number seven is customer development originally developed by Steve Blank and included the umbrella of lean startup. I am a big fan of customer development. I think it’s very hard to get right but I do think there are resources out there that you can buy that have more specifics, that have those tactical things like what interview questions should I ask?

[32:02] You can look at Running Lean by Ash Maurya and you can look at coldcallingbook.net by Robert Graham who we’ve had on the show. I think both of those are exceptional resources for learning how to do in person customer development. There’s also some pretty good examples on the web of how to do email customer development. That’s something I’ve done a lot with Drip and maybe I’ll wrap it up at a later point. This may be my favorite concept in all of lean startup. This is the idea of customer development.

[32:26] Mike: I really love the idea of customer development but I also see a lot of people who are doing it wrong or just not doing it or thinking that they’re doing it when they’re really not. I think there’s a lot of misconception around how to do it right and I think one of the big issues that most people run into, when you get into customer development, it’s very easy to look back into the past and say oh I should have asked that.

[32:50] And unfortunately when you’re in the position of trying to ask the questions, you don’t always know what questions to ask. So you do the best that you can and then in retrospect, hindsight is 20-20 so it’s obvious that you should have asked a specific question because you went through this process and it failed miserably and you say oh, I didn’t realize that this was important. I should have asked that. And it’s really hard to figure out what questions you should be asking upfront.

[33:16] And the types of people who are successful are the ones who are asking those questions who are kind of insightful and I don’t know if there’s a good way to teach that. I really wish there was or I wish there was some material out there that I could find that says these are the types of questions you should ask based on these types of businesses but that’s just a giant matrix and obviously product types are changing all the time so it’s not feasible to come up with that kind of a list.

[33:37] But I do think there are probably different types of questions that people could ask or you can put together in a spreadsheet and say this is why you would ask this question and it will help you avoid this particularly situation. And then based on the type of business you’re building, you can look at those and say okay I’ll ask this and this but not that one because the end result of that is not applicable to me.

[33:57] Rob: Our next topic is problem solution fit and product market fit. After customer development, these might be my next favorite concepts. I’m a big fan. You hear me talk about them a lot on the podcast. The steps of starting with a problem that a group of people have and striving for problem solution fit meaning you’re trying to solve a problem and at a certain point you know that you’ve done that, that’s a milestone. Then the next phase you’re trying to do is you now have a product and you’re trying to find the market for that.

[34:27] So I love thinking of that as steps and I think the 2 and 3 phase idea of solving a problem and then moving onto marketing and learning during that, really reshapes the way I thought about how software products are developed when this concept was introduced.

[34:43] Mike: I really like this as well but the problem I have with it is its very difficult to do both at the same time and I would be so nice to be able to shortcut a lot of the time that you waste if you can do them both at the same time.

[34:55] Rob: Yeah. I think you’re supposed to do them in sequence. That’s my understanding. But it takes a long time. That’s why there are Audit Shark early access and the Drip early access are months long because you’re still trying to figure out how we’re really solving a problem.

[35:07] Our next topic is continuous deployment. And you remember this one? This one was a big part of lean startup early on and now it’s almost nowhere to be found. I see it on the Wikipedia page but it’s really downplayed in a lot of the places that I looked.

[35:20] Mike: I remember hearing Eric talk specifically about continuous deployment and how they would go push things into production and how its jarring as a developer to you write some code and then you check it into the repository and it just gets pushed out into production. I can see that as a developer being a little bit schizophrenic about that especially if you’re a product manager saying okay who pushed that feature out and why did they go out today? We wanted to really push that out as part of this marketing effort.

[35:49] And I think that’s one of the things that makes continuous deployment a little difficult because there’s certain types of things that you want to push out and you don’t want to immediately deploy it because you want to build a marketing buzz around it. You want to be able to solicit feedback from people and have things go out on a specific schedule.

[36:05] You launch a new feature or a new version or something like that solves this new problem and you want to tell everybody about it. But if you’re doing continuous deployment it gets pushed out there and then hopefully nobody makes a big deal out of it and you don’t lose control of the story. I think that’s why this has been downplayed a little bit.

[36:23] The other reason I think that it’s been downplayed is that it’s actually very difficult to do continuous deployment and it’s a significant engineering task and it’s not something I would probably do in most cases just because of the fact that you have to do a lot of engineering effort to be able to make it so that things can get pushed out and do all the tests such that if it fails, then you are able to automatically roll back because that’s where continuous deployment to me kind of falls on its face is because that engineering effort is so incredibly complex and difficult, I mean you’ve got enough problems just building the product and making sure that it’s what people want.

[36:58] Do you really want to add in all the complexity of creating all these unit testes and everything else? I mean it slows down the process. I think that’s where it falls short is it’s so hard. It’s so complicated. It slows down your iteration process. If you push things out and they fall in their face, you have to rip them out of production, fine go ahead and do it. Do it manually. But the idea would be if you’re pushing things out on a regular basis, hopefully you’re making things better over time and you don’t have to resort to completely ripping them out of production.

[37:27] Rob: Yeah. I think some would argue that continuous deployment is intended to speed up your cycle rather than slow it down.

[37:33] Mike: I agree that’s the intent but I don’t think that it does because as part of continuous deployment and Eric Ries specifically talked about this was writing unit tests such that it would detect whether or not let’s say you push out some new code that affects the sign up process. Well as part of that you’re going to have code in place that measures the rate of signups that you’re getting. Well if the rate of signups drops from 50 per hour to 0, then the code that you pushed out probably broke your sign up process and it needs to be rolled back.

[38:02] So you have to take those things into account when you’re pushing that out and you have to build all of those cheeks that will verify whether or not a number of signups in the past hour drops to 0 or drops by some significant margin. And that’s the other piece of continuous deployment is being able to make measurements to say is it making the product better? Is it increasing performance? And if it’s not, then roll it back. And because of all these things, it just makes things more difficult.

[38:27] Rob: I think continuous deployment is a nice theoretical idea but like you I’m not a fan of it in practice. When I first heard it I was very skeptical as a software developer. I’ve never done it myself but I have used a couple apps from two startups that were doing continuous deployment and maybe it’s just selection bias but both of the apps were really crappy like they were very buggy. These were funded startups with teams and they were doing lean startup stuff.

[38:53] There were just bugs all over the place and I was like this is really crazy. I think a big part of that is they were pushing, they were moving so fast and they were trying to really espouse the MVP and the containers deployment. So that point I kind of wrote this off. I agree with you. I think the amount of instrumentation and reporting and testing that you need to make this work is probably impractical.

[39:15] So our next topic, second to last topic is split testing. I’m not sure this is even a tenant of lean startup as much as it was included in a lot of the talks and the books and such. I’ve been split testing for 10 or 11 years and even doing the manual, the poor man split testing of modifying something from one week to the next before we had good tools to do it. But lean start did include this in their concepts of validated learning and the build measure loop.

[39:40] Mike: I think split testing is a good idea. If you have any questions about split testing, I would go to the guru of split testing which is Patrick McKenzie. Check out his blog. He has a lot to say about it. He’s got a ruby gem out there that will help an AB test. And there’s all these different frameworks out there for doing AB testing and there’s very clear research and results that show the AB testing provides measurable improvements in a business. So I don’t think there’s any question that split testing is kind of a fundamental thing that everyone should be doing whether you’re doing lean startup or not.

[40:12] Rob: Indeed and to know we just launched split testing inside Drip which I’m very excited about. I already have a test running right now on some subject lines. Our last topic for today is actionable metrics versus vanity metrics. I think it’s a valuable concept and I think it helped a lot of people when it came out that realizing that certain metrics are what he calls vanity metrics that they’re just things that are useless like page views how often times its like time onsite, really aren’t that actionable.

[40:41] Whereas actionably seeing your churn rate or your trial to paid conversion, that kind of stuff, something that’s much more actionable. I would give this concept, this idea a thumbs up.

[40:51] Mike: Yeah, this concept just talked about a lot in the Lean Analytics book that I talked about earlier. They do devote some time to figuring out what is an actionable metric versus what is not. One of the things that really comes out of that is the rate of change of something. So just knowing that your number of website visitors this month or this week is 1,000 and next week say that its 2,000 that’s not necessarily important in it of itself that you went from 1,000 to 2,000. It’s just really the rate of change over the course of that week is what she should be looking at.

[41:23] So rate of changes become very, very important when you’re talking bout action ability because let’s say that you went from 501,000 to 502,000. Well grant it you’re going up and you went up by 1,000 visitors on a weekly basis but the rate of change over the course that week with regards to how much traffic you had is almost insignificant. So those are the types of things that really become important. I think this is a very important concept.

[41:48] Rob: So we’ve covered 11 topics under the lean startup umbrella. I feel we’ve kind of scratched the surface and gotten into it a bit. Hopefully it’s been helpful if you’re listening that it gives you an idea of how we think about it from a bootstrapper’s perspective. I think we raise a lot of things that we like about lean startup. I think overall, lean startup has moved that higher understanding of a startup and thinking of it as you can’t do a blue print for it but you can start putting these words and concepts in place that make it easier to understand the moving parts of it.

[41:28] I also think we talked about some things that we don’t necessarily like about lean startup like that it’s not focused enough and that it tries to cover bootstrappers, venture backs, dry cleaners, new product launch inside a Fortune 500 company. I think that’s a real drawback to it.

[42:34] I was actually put off by the first 3 or 4 examples in the lean startup book were like Intuit and other massive companies and I almost stopped listening because I tend – in general, if it hadn’t been this book, I would’ve turned it off because at that point, I typically bailed and said this book is not going to have anything insightful for me.

[42:50] We also talked about the language feeling maybe high level and academic and that it can’t quite be laser focused and provide action for people. I also think anything that gets this big, certain people want to rebel against it and when you hear the word like MVP and pivot and these concepts that are thrown out so much, it does get a little irritating. It’s like a hit song that you hear too many times. And you kind of want it to stop.

[43:12] I think the other common criticism in  lean startup is that it pulled together a lot of existing stuff, maybe some things that were already obvious and just pulled them together under one roof, I don’t think that’s necessarily a bad thing but it is lean startup itself. the concept is unique but most of the concepts underneath it were not original. But most of them existed maybe under different names before this.

[43:35] So certainly I’d love to hear your feedback as well. Feel free to send us a voicemail, email or post a comment on this episode. And you can do that by calling our voicemail number at 1-888-801-9690 or you can email it to us at questions@startupsfortherestofus.com. Our theme music is an excerpt from “We’re Outta Control” by MoOt used under Creative Commons. Subscribe to us in iTunes by searching for startups or via RSS at startupsfortherestofus.com where you’ll also find a full transcript of each episode. Thanks for listening. We’ll see you next time.