Story Points, T-shirt Sizing and Time Buckets: How Tech Companies Do Engineering Estimates
Out of my curiosity, I emailed the Product Habits subscribers and asked everyone a question about how we all do engineering estimates.
I had a feeling I’d get a few responses, but not much more than that since estimation is a somewhat dry topic. But boy, was I wrong!
I’ve got nearly 100 responses.
It turns out that most of the folks who responded are really passionate about whatever their approach is to estimates. I even got a few responses that were over 1,000 words. And not just that, everyone is curious about what other people are doing so that they can improve their own process.
Here’s how we dug into the 11,000+ words from the responses:
- Read every response and highlighted notable information.
- Each response was re-read and grouped by a methodology type.
- Identified patterns in responses to understand what everyone is doing, regardless of the exact estimate methodology used.
How are estimates done today?
There wasn’t a single distinct method that most people used. Instead there were lots of different methods. We barely saw anyone use the exact same formula.
To our surprise, not a single person was using the waterfall model. Waterfall is a sequential non-iterative software development methodology that used to be very popular. It typically involves very little customer contact during the development process. Two people did mention the method, but neither used it – one was researching it, and the other pointed out how his process differed from waterfall.
As a whole, the approaches fell into two distinct buckets.
Agile
The Agile Manifesto was created in 2001 to put a name to what had been happening since 1957 when software development started to become more iterative.
Over a quarter of the people who responded said they use some form of what would be considered an Agile methodology to do estimates.
People who mentioned Agile typically fell into three groups:
- Story points – Points are used to assign each engineering task a set amount of time. People commonly mentioned using Fibonacci series numbers to help offset the uncertainty that comes with larger stories. Fibonacci uses 1, 2, 3, 5, 8 instead of using 1, 2, 3, 4, 5 for assigning points.
- T-shirt sizing – Assigning small, medium or large labels for each engineering initiative to indicate its complexity.
- Time buckets – Simply using days and hours to estimate, instead of t-shirt sized buckets or story points.
Respondents who use Agile varied – from die hard evangelists of the methodology who follow every aspect religiously, to those who take a few pieces from Agile and apply them to their process.
Here’s an example of someone using the exact prescription of Agile:
“We use Jira to track all of our stories. Everything boils down to a point count. Using Fibonacci series numbers, we estimate points based on number of hours it would take a lead engineer to do something.”
There were a few folks like this, but others have also taken elements of Agile and then used them in their own way…
“I’ve implemented the t-shirt sizing estimation methodology which is based on estimating the complexity of the task, not how much time it would take. And by doing that for a few sprint cycles, we’ve been able to establish a certain velocity to our engineering process.”
Some folks who use pieces of Agile were against certain tactics like points, calling them “unnecessarily removed from reality.”
And which of these methods were the most common for measuring Agile?
Agile-inspired t-shirt size bucketing into categories like ‘Small’ ‘Medium’ ‘Large’ or full Fibonacci estimates were the most popular.
Here’s how one person described their use of buckets:
“I’ve tried many things over the years but what seems to work the best is asking the team to individually come up with their estimates into three buckets.”
Just about everyone in the Agile segment is implementing t-shirt sizing or a Fibonacci sequence. But… everyone executes it slightly differently. In fact, we couldn’t find a pair of responses that did it exactly the same way.
So while there’s a few general trends, each of us is indeed a unique snowflake.
Take t-shirt sizing for example. Some folks started with hours and ended with 1 or more weeks, like this:
Small = < 4 hours
Medium = 1-2 days
Large = 1 week or more
While others used larger day and week increments:
Small: < 1 day
Medium: < 1-2 weeks
Large: < 2 weeks
And the points-based folks broke things down into smaller chunks compared to those who used t-shirt sizing buckets by using hours and days as their time metric with no mention of weeks.
1 point: < 30 minutes
2 points: < ~3 hours
3 points: A full day
Here’s another example:
1 point: 2 hours
2 points: 5 hours
3 points: < day
5 points: > day
8 points: 2 days
13 points: 3.5 days
No Estimates
Is everyone using Agile? Not quite.
Haters of estimates, newcomers to estimates and those who had been burned too frequently in the past have chosen to skip estimates entirely. The second-largest group of respondents seem to have developed an allergy to estimates. Agile isn’t in their vocabulary, unless they’re hating on it.
One person even introduced the hashtag #noestimates, which we thoroughly enjoyed 🙂
There are two distinct categories that people who don’t do estimates fall into.
Trust > estimates
Getting work done was enough for some respondents. They described estimates as unnecessary. As long as everyone is doing what they say and code is being shipped, why burden people with extra stress and work?
“I am monitoring daily each developer activity, so as long as they work, I do not stress about estimation and I do not stress them also.”
Another respondent felt that estimates were a violation of trust between Product and Engineering. Instead of focusing on estimates, the focus should be on making sure the most important, impactful and meaningful work is being completed.
Estimates get outdated fast
After being burned by estimates in the past, this crew decided to ditch them. They had used estimates before, but found that they never quite matched reality.
Things are constantly evolving, causing estimates to become out of date, they argued:
“We generally try and avoid estimates as they are often out of date the moment you start developing :)”
“Engineering estimates are as good as any plan before they hit reality. Unless it is a faithful copy of a previous project, plans and estimates are very inaccurate.”
Instead of continuing to provide flawed estimates, they decided to stop doing them altogether.
One habit that came up over and over again
No matter the engineering estimation method used, one habit kept cropping up repeatedly.
Adding buffers on top of estimates is the #1 habit. Over half the responses mentioned it.
The formula we kept seeing over and over again goes like this:
Estimate multiplied by X = Final shareable estimate.
The three most common reasons people added padding included:
- Unforeseen scope changes that happen during a project.
- Accounting for things that were missed that might come up during build out.
- Avoiding possibly disappointing management by not hitting estimates.
Buffers ranged in size – from adding 15% to five times the original estimate! Here are just a few of the buffers respondents shared:
- We add 15% as a margin for unknown tasks and coordination activities.
- Add about 10% for buffer and about 10% for management etc.
- Add 20% to account for scope changes and other things that come up.
- Add a buffer of let’s say 20%.
- Add 25% to whatever date comes out the other end.
- I add 50% on top for things we didn’t think of.
- Multiply it by two. 🙂
- Make wild guesses at how long I think things will take me to code and then multiply that guess by 2.
- Just try to estimate how much time I think it will take and then multiply that number by 4.
- If it’s a whole new system with no similarities to past systems I will take a guess and multiply by 5.
Everyone recommends adding a buffer. So we’re all lying to each other all the time.
I know that I’ve even added another buffer whenever folks have told me estimates in the past, I’m willing to bet you do too.
So that means someone thinks of an estimate. Then they add a buffer. Then they tell that estimate to someone else. And then that other someone adds another buffer.
Lunacy.
Think of all the costs that come from all this miscommunication. It kinda sounds like we’ve all accepted that just guessing about our engineering estimates is the best that we can do.