Software Estimation Review #1

One of my recent projects had this burndown:

At the start of the project, the “desired” hours were coming down at the speed of the “high” estimate.  If I had used the first 5 data points to predict the end of the project, it would have been near T=7, at a cost of 6N or so, and the estimate was only for 5N.   Not good.

Luckily, things started to sort themselves out as the project progressed.   However, I wanted to understand what was going on.

At work, I track every slice of time at a project and “top level task” level.   I also note down what exactly I’m doing (and usually its outcome – success, interrupt, break, etc – especially useful if there is fragmentation, you can see how interrupted you were – a metric to show pain).

I exported this detail into Excel, and created an item by item grid of time taken for each line item  in the original estimate.

In the process I found several items that I worked on that didn’t have a home in the estimate, and I got “actual hours worked” for each of the line items.   It was shocking.  Here is an abridged version:

• I saw tasks much larger than the original estimate.
• I saw some tasks much smaller than the original estimate.
• I missed several features.
• I left out entire categories such as refactoring (as the project grows), documentation,  styling and usability.

Possible Interpretations

I did well.

The project was done within the time allotted.  This was a success.  Perhaps the individual line items were not, but the overall project was – I was lucky that I overestimated several items.  If I had tried to squeeze every line item, I would have been in trouble.

Could more Up Front Design yield a better estimate?

The Waterfall thought would be I didn’t do enough up-front design.  However, I spent longer putting this estimate together than I should have – because in this job, putting together an estimate is not billable time!  If I had spent more time on it, I would have had more line items – yielding a higher estimate – yielding a greater risk that the client would not approve the budget – and I would have still gotten it done on time – perhaps even less time.

I have done this before = takes less time

I had previously done a similar project.  As a result,  I estimated several items fairly low. Examples:

• 5 to 8 (actual: 23)
• 5 to 7 (actual: 12)
• 5 to 7 (actual: 14).

This is equivalent of an handyman stating: I’ve built a house before, so hanging drywall won’t take as long.   Incorrect!   It takes time to build software, even if you know exactly what you are doing.

 “Sure that Technology will be easy to integrate”

My coworker (on loan to me for two weeks) spent a lot of time trying to get a grid to work correctly.   Actually, the grid worked fine, the helper classes to interface it with MVC did not.     Trying to get them to work took the majority of the hours on one of the pages.

In the end, he scrapped the helper classes he had downloaded – they worked only in specific circumstances – and rolled his own – and coded the rest of the page in record time.

I’m not sure which way to go on this.  Perhaps having a bucket of “research” hours?    Timebox the items that we research to save research budget in case its needed later?   It seems every project I’ve been on, there’s been one or two things that took some time to get the approach figured out.

I left out parts of the project from the estimate

There were several items that I had not estimated up front that are a part of any project.  See below for a running checklist.

I skipped parts of the estimate

Some functionality turned out to be trivial, or unnecessary. For example, we went straight from a grid of Endor to editing Endor without a stop at a detail page for Endor.   Or, one of the setup pages – thought to be three seperate pages – collapsed into a single grid.

These “mistakes” are what saved the project.

Overall Analysis

I think my mistake was trying to specify everything in the estimate at a line by line level.  As DPR207 at TechEd 2010 pointed out on slide 16, there are different levels of uncertainty as a project progresses.  I was estimating at too granular a level too early – guaranteeing I would miss items, as well as specify some items that were unnecessary.

Doing it Differently

In this particular project, how else could I have estimated it?

Agile, Story Points, etc.

Using Story Points and approaching a project in an agile way (Scrum-like) are my favorite ways of handling project estimation and management.  However: In my role as consultant, with clients pre-approving blocks of hours (based on a high end estimate), I don’t get to do that.   So I’ll pass on that solution.   I HAVE to have a number of hours up front, or I need to find a very understanding client.

Estimate at a higher level

Rather than estimating at a very detailed level, I could estimate in bigger chunks like this:

Level 1: Systems	“A Website”“A Batch App”	50,100,150,200,300,500 hours
Level 2: Functions / Features / Pages	Grid of FooEditing Foo Selecting Bars for a Foo. Reading in an xxx file. Validating xxx information	10-20, 20-30, 30-50 hours

Fibonacci Numbers

I will definitely continue using Fibonacci numbers for estimation.  It saves me brain power – Its either a 5 or an 8.  I don’t have to argue with myself whether its a 7 or an 8 or a 9.

I use a slightly modified sequence:   0.5, 1,2,3,5,8,15 … somewhere in there, I jump into adding zeros:  5,10,20,30,50, etc.

Going Forward

Jim Benson (@ourfounder) gave a presentation at CodePaLousa: “We are Not Engineers“.   (I tried to find a link to a powerpoint, and then I remembered, it was truly a powerpoint free presentation. I did find similar materials here)  The gist of the presentation was:  “Humans are horribly designed for doing estimation”.   If I estimate based on gut feel, I’m going to be wrong.   Thus, for my job, I either need to stop estimating, or start to build a toolkit.

This is the toolkit so far:

Keep track of all time and match it back up against the estimate at a line by line level.

It takes about 15-30 minutes to export from Harvest and match things up in Excel.   Its a form of instant feedback.

Estimate in larger chunks.

Rather than counting the number of fields on a page (which is what I did previously), I need to find the average time to write a page, and then double it if it’s a big page.    Try not to estimate in anything smaller than, say, 10 hours, for a chunk – a page, a file processing feature, etc.

Keep a checklist

My checklist at the moment:

• Initial project setup
  • Machine setup
  • Project setup – layers (DAL, etc) – into SVN
  • Stub version of project that can be deployed
• Test data strategy (in support of integration testing)
  • Test data creation and deletion
  • Setting up automated integration tests to run automatically
• Individual Items (Project specific)
  • Larger chunks
  • Include testing time for each item – writing unit or integration tests, and testing by hand.
• Growing pains
  • Refactoring is inevitable (and not specific to any particular item)
  • Configuration and Deployment (as the project enters different environments)
  • Merging between branches if branches are called for (any time there is a phased release, or a separation between UAT, QA and DEV)
  • Necessary Documentation (after complicated things get done)
• Project Management
  • Dependent on each client. 10%  of a 35 hour week = 3.5 hours of email, status updates, demo prep, and demoing – which is about right for a good client.  Other clients, the number has been closer to 20%.
• Research bucket
  • For the unfortunate item(s) that will pop up that just don’t work
  • Also can be used to take extra care to simplifying something so it stays more maintainable.
• Usability, Look and Feel
  • Cross browser testing
  • “Making it Pretty” – fonts, colors, grids
  • “Making it more functional” – renaming things, page organization, number of clicks to do something
• I18N and L10N
• • Dependent on client and project.

The Request

How do you estimate your projects?  How close do you get?   I truly want to know.  Shoot me a comment or write it up and add a link.