Nov 13

How To Make Meaningful Estimates For Software Products

A few thoughts on estimating. I had a conversation with someone yesterday who asked me how I worked with the engineers on estimates. My answer shocked him, I think. I wanted to expand here on what was a throwaway conversation:

  • My favorite story about estimates is about the Sydney Opera House, as told by Nicholas Nassim Taleb in The Black Swan. First, you should know that construction is incredibly well-understood and for some types of projects builders can repeatedly complete them within 5% of the estimated time.

    The Sydney Opera House, started in 1959, was scheduled to be completed in 1963 for $7M (Aus). Actual construction took nearly four times the original estimate – it actually finished in 1973 (10 years late!) – and it cost more than 12 times the original budget at $104M (Aus). And of course, the Opera House was only 1/3 of the original project. If builders can be that far off, simply because it’s never been done before, why should we think that we should be able to estimate software, which always by definition has never been done before?

  • There is a fundamental disconnect between estimates and interesting things. Interesting things are unpredictable. User stories are estimatable, therefore not interesting. 
  • Estimates are not a standard distribution. They are really screwed up distribution where the likely value is way the heck out there beyond the value you think it should be. (And very occasionally, extremely rarely, things go a lot faster than you expect.)
  • I prefer timeboxes, and for interesting things, we get done what we get done in the timebox. The art of product management is figuring out what to do in the timebox. Note: this works much better in software than in construction. Buildings have to obey the laws of physics, but software doesn’t. There is no such thing as a Minimum Viable Product in construction – you can’t build a fancy roof until you build the structure to support it. But you can do that in software. There’s a lot of software out there that is essentially fancy roofs floating in the air.
  • Think about failure, which is so important in innovation. Failure is of course immune to estimates, by definition.

    For example, let’s assume I can get a decent estimate for doing something interesting (which we know I can’t, but hang on). Then we do it. It only takes twice as long as we estimated! (That’s a great result.) Unfortunately, given reality, it’s wrong, and has to be done again. It was a failure, but it was a productive failure. We learned a lot. We didn’t get the feature to market when we expected to, but if we’d put that version into the market, it would have been bad in oh so many ways.

    So we start doing it again, and mostly we have to start from scratch, but we did learn some things in version 1. We also realize we can get a little bit of version 2 out to early adopters. It’s definitely not a full feature – they have to do manual work to get the value, but they are willing because it’s so useful to them. And we learn some stuff, and we end up building version 3, instead of version 2, because we got some great feedback that makes it even better. Versions 1 and 2 are sunk costs, and they are PAINFUL, but because we did them, we have version 3, and it’s beautiful. And it only took us four times as long to get the feature out as originally estimated, which is actually a pretty good result. 

The title of this post might have been a little misleading. I suspect I may have created a firestorm. I can’t wait to hear what you think!

May 13

A Radically Different Manifesto For Agile and Lean

I do not like the agile manifesto. It does not resonate with me as a product manager – someone responsible for guiding successful products to market. It’s very focused on the needs and desires of developers, and much less so on my needs and desires, or more importantly, on those of my customers and prospects. But, I love agile, and I believe agile – and its follow-ons, like lean and kanban – are critical to enable people like me to create and deliver new products that create value in the world.

My Six (Not 99) Theses

What is it about agile that turns me on, while the agile manifesto itself leaves me cold? I’ve made a stab below (and made other points elsewhere) at a high-level explanation for why I like agile and why I think it works. It’s based on some tenets or axioms that lead to the conclusion that an approach like agile is generally superior for delivering value than more traditional approaches like waterfall. I guess these are my version of a manifesto, one that leads to agile and lean when they are applied to product development:

  1. It’s fundamentally better to not try to predict the future. Do your best to predict only as little of the future as you can get away with.
  2. It’s fundamentally better to be responsive to changes, especially as change happens so fast.
  3. It’s much easier to predict how long it will take me to do one thing than it is to predict how long it will take to predict multiple things.
  4. Don’t try to predict where you’ll be a long time in the future. Instead, keep your prediction interval as short as possible – you’re much more likely to be correct. (E.g., if I travel at 50 mph on this road for 15 minutes I can predict with great certainty where I’ll be, but if I have to predict where I’ll be in two weeks’ time, I could literally be anywhere!)
  5. Predicting what the market will want far in the future is extremely difficult, so instead do your best to only have to predict what the market will want in the near future.
  6. “Work in progress” is fundamentally a type of waste – this is a tenet from lean manufacturing and the Toyota Product System, but it also applies to software. Software that’s been developed but is not in use by customers – the definition of “work in progress” for software – is wasteful. Even if it eventually provides value for customers, it’s not providing value right now, leading to an overall loss of total value, which is undesirable. Note: This also applies to requirements – a fully fleshed out requirement that no one is working on is waste, so my recommendation is “don’t do that.”

If you combine those tenets “agile” naturally pops out: Don’t focus on optimizing a large set of work, instead focus on completing the most important piece of that work, as quickly as you can, and deliver it. Since you’re only delivering a small chunk, you don’t have to predict very far into the future. And you’ve chosen the most valuable thing to work on, so you know your market will want it. Because you are working in small chunks, if the market changes its mind, you can quickly change your plans in response. And by always having only a small amount of work in progress, you reduce waste and create more value.

Agile, Upon Analysis, Turns Into Lean and Kanban

In this analysis, though, it’s not really agile that pops up, it’s lean, or more accurately, “Kanban” (at least as practiced in software development). No matter how long your sprints are, some of your features will take longer than the sprint length to deliver to the 80% level. This is a big problem for agile methodologies, because a feature that’s only at the 60% level (which perhaps you can get done in one sprint) doesn’t provide significant business value, and usually neither you nor your customer gains by delivering it at that point. (There’s clearly another blog post in this topic of 60% value versus 80% value.) And therefore Kanban’s approach, which I will simplify as “take your three most important features and work on each one feature until it’s deliverable, then deliver it, and then start on another feature,” does a better job of delivering value to the market and reducing work in progress.

I’ve used the term “lean” above, and I’m really talking about the concepts that come from lean manufacturing. But these ideas are also intimately tied to the “lean startup” mindset as well. One of the key insights of the lean startup is that instead of guessing or predicting the future (all the things I’ve said you don’t want to do), you create hypotheses and test them, getting real data about how customers respond to your product and its features. Agile and lean and kanban again perfectly align with that approach:

  • Get “finished” work out to the customer as soon as possible to find out if they like it – after all, there was a hypothesis that drove the feature or whatever it is, and you want to test whether the hypothesis was correct or not. If so, yay!, and your customers are going to be more successful faster. If NOT, then yay! You have learned something that you can immediately address without letting the issue wither on the vine.
  • Instrument the product so that you can understand how it’s used over its life. This way you can find out what users do with the product, how they’re using it, where they’re running into problems (using the same techniques as webmasters use to assess the usage and obstacles of their websites).
  • Getting features out earlier means that you can get paid for those features faster, whether this is due to existing customers expanding their installations, or renewing their maintenance, or due to new customers coming on board because they heard from their friends (or via an evaluation) that the product does more of what they need better. In other words, more features earlier means a better top line. And the top line is the one that grows the economy – both yours and ours.

Do We Need A Product Managers’ Agile Manifesto?

I don’t know why I feel the need to reinvent things, like the agile manifesto, but there you go. I’d love to start a conversation about this, and if other product managers resonate with my ideas here, perhaps we can refine them and put out our own manifesto on how to deliver more value to market, faster.