Tag: software-engineering

On Transitioning Assets between Environments

I’ve previously talked about unit tests VS integration tests and various database-inclusive testing strategies.  I have recently been dealing with some pain on making sure everything needed for some code is moved between environments – a process fraught with human intervention and forgotten steps and mistakes and ambiguity. In an IM conversation with one really smart guy about this (Hi Josh!), an Idea was born.

Preliminary to the Idea

We’ve gotten pretty good at getting code between environments.  There’s all kinds of things in source control for branching, and merging.  There’s different flavors of branching, and different ways to package together code to send it places, my personal favorite is using a build server to do the actual deployment (thus documenting it). 

However, every customer I’ve worked with, struggles with how to get [the right] database changes into an environment. None of them think they struggle with it, but they all do.  (well, 3 of the last 4).  A lot more than developers struggle with getting code moved around.   The DBA’s are usually saying “all you have to do is X”, and the developers are saying “you want me to do WHAT?”, and they meet somewhere in the middle.

The standard idealist’s view of this world is: check your database into source control.  The tricky bit becomes: using which solution?  Visual Studio SQL Server Database Projects?  Redgate SQL Source Control? E/R Studio Repository? RoundhousE?   I have not yet met a client who does this; usually there is some really weird way that the description of what needs to change gets handed to the DBA, who then takes the changes the rest of the way.

To broaden the scope, the kinds of assets that I’ve seen companies struggle with:

  • Database schema changes
  • Database data changes
  • Configuration entries per environment
  • The right list of components being deployed per environment
  • All of the above when adding a new environment (essential for feature branching)

Target Audience

Folks working at small companies, where they usually have full access to the production server, might scoff at this; sorry guys you are not my current target audience. 

The target audience is folks working at large enterprise companies, that host their own software — usually those that have Sarbanes-Oxley compliance that states: NO DEVELOPER CAN TOUCH THE PROD BOXES, EVER.    [sidebar: maybe it doesn’t actually say that, but it’s the excuse given every time].  It’s a black box. 

The target audience is also NOT people writing sellable software, where hopefully 500 customers will run an install 500 times – in those environments, there’s usually a chunk of resource per release to get the installers right.   And a lot of VM’s to run installs on.   Over and over. 

Personal peeve: I ended up trying to communicate with other folks on what needs to happen, using a very limited vocabulary, often with edge cases that don’t fit their language.  Sometimes, the chasm between our experiences was even larger, for example, in production they had a web farm behind a certain kind of load balancer with a firewall, and in lower environments where I had access, they didn’t (too expensive, not in the the department’s budget).   But I certainly was not allowed to log in to those things to get a “feel” for them, and how to work with them; yet I was expected to to tell them exactly what they needed to get it running.    Personal solution:  Cookies.  Shared.

The Birth of the Idea:

At the previously salaried job, I was in charge of adding some stuff to a website so that it could communicate to a web service. This web service was in charge of taking uploaded documents and processing them further, then depositing the results in a shared folder. I added a Test<ReleaseName>.aspx page to that website, where it talked to the TestWebService.asmx page that sat next to the normal web service, and they talked about stuff like: 

  • What identity was the service running under?
  • Did the service have read/write/delete access to the drop folder?
  • could the service transfer a LAAAARGE file without dying over the network?

Within the website and the web service, I made sure to limit the request so that it would only “do work” if the sender was from the developer’s subnet, specifically excluding the subnet that the internet came in on (after they got through the firewall).   Everything not recognized was immediately 404’ed.

The result was I could walk over, hang out with the build/deploy guy (without touching his keyboard and breaking compliance), and have him browse to the test website – and it told him everything that was going on, especially what was broken. We spent a LONG time trying to get the identity to run as the right person with the right access to the right folders, and if I hadn’t built the test pages, it could have gotten really frustrating.   (Rather than BUILDGUY=>”no”=>DEVELOPER=>”try”=>PERMISSIONSGUYS=>”done”=>DEVELOPER=>”again?”=>BUILDGUY, it was [DEVELOPER+BUILDGUY]=>”try”=>PERMISSIONSGUYS=>”done”=>[DEVELOPER+BUILDGUY]=>”yay!”)

The Idea: Create Tests

Here’s a sample of the kinds of tests that I’m talking about:

  • Can I communicate to W1 web service correctly?
  • Does table T1 exist?
  • Does table T2 have an entry for E1,E2?
  • Does table T3 have a column C1?
  • Is there any data that matches this known pattern SQL1 which should no longer exist?
  • Does Table T4 have an index I1? 
  • Does stored procedure SP1 exist?
  • Does stored procedure SP1 have xyz in its definition?   (this could get fancier in your environment if you have stored procedure revision numbers; mostly trying to ensure that a stored procedure got updated)
  • Does any stored procedure have any references to XXX in their definition? (when dealing with the case where many stored procedures need to be modified because a table is going away)
  • Do at least N stored procedures have any references to XXX in their definition? (when dealing with multiple stored procedures being updated because something required got added to a table)
  • Can I read the configuration entry for E1?
  • Do I have {x/y/z} permission to network path N1?

The Proposal

  • To add a test/configuration channel to every component of what assets are required for that component to function
  • The structure could be like this (C# psuedo-code, written in notepad:)
    • public class Requirement { 
	public string Name { get; set; }            // to help us all talk about the same requirement
	public string ReleaseName { get; set; }     // in which release was this introduced
	public DateTime CreationDate { get; set; }  // when was the requirement created? 
	public string Description { get; set; }     // the "why" of the test, documentation of the asset needed or what role it plays
}

public class RequirementResult { 
    public Requirement Requirement { get; set; }  // which requirement are we talking about? 
	public bool? Success {get; set; }  			  // was it successful?  null = don't know
	public string Result { get; set; } 			  // the result of the test, what's broken, or what worked
	public IEnumerable ChildResults { get; set; }   // sub-component results
}
  • If a component depends on a child component, have it be able to query the child component’s test channel (example: web service calls to a web service in our control).
  • To ensure that this test/configuration channel is locked down by a simple, shared piece of code that determines if the test/configuration channel can run (on which the enterprise can sign off as a whole), and provides the structure of the test results.

Summary

I think it would be pretty useful.

It would get even more useful over time – with the system getting more documented with each additional release.  This is the kind of detailed documentation many folks would drool over – much more useful than the documentation provided by writing unit tests, which in my experience, only 10% of developers think are useful.

I think it would help all the folks involved – developers, DBA’s, Network guys, Permissions guys, Deployment guys – talk the same language and have the same set of tools to determine if the work was “done”.  Rather than being at odds with each other: “you need to communicate with me in MY language!” 

The next project I get to start from scratch, I’m going to try this.

Randomness: WordPress Fail, Feature Branching

  • I tried editing a post that I had started from Windows Live Writer, in the WordPress iPad app. Something about how the HTML got formed killed the editor there, as well as the rich text box editor on the WordPress site. Not having a laptop with me, I tried Remote Desktop from the iPad to my work computer, but that experience was too hard to do any real content creation. So I gave up and am writing this post instead.
  • I have had a rough patch at work the last few weeks. End of Release, lots of defects. We did a post-mortem, traced down each defect and where it came from, and found out:
    • Only about 10%were actual code errors. Yay, professional pride intact.
    • The rest were of the nature that no matter how much we looked at it, we would not have detected it was not what the customer wanted. The customer had to look at it, give us feedback, unfortunately in the form of a defect, because that is the stage of the SDLC we were in.
    • We are trying to find ways to get the customer to give us their eyeballs earlier in the process. The good news: The customer “gets it”, and now one of their style stars is hanging out on campfire where we can trade screenshots. Hopefully they will also be able to use the CI build and deploy to “play” with the latest code.
    • Part of the learning experience here is that the customer is going from a single pipeline of development to “Feature Branching”, and everybody is learning this new paradigmn. I could probably do a blog post on things to watch for with Feature Branching.
    • To be fair, we also missed a lot of things that if we had read every word of the requirements very carefully, we would have gotten. Yet, with feature requirements changing often, it was very easy to go with what we thought was common consensus rather than being word nazi on the requirements that may not have been up to date. Possible solution: as our perspective of things drifted, we could add notes to the requirements, as addendums.
  • I have not had a chance to get back to crunching car stats. the next thing I want to do there is visualize tracks in 3D using Processing. I have something working, but its not pretty enough yet.
  • A friend asked me for advice on how to do a timelapse of some work their employer is getting done. I hope I get to help!

Oops look at the time I am needed elsewhere. Publish without Pretty!

Developing as a Team

Update: This entry was re-posted on my company blog, here:   http://ignew.com/2013/02/16/developingasateam/  — with some updates.   

In October, I started on my current project as a solo affair. In November, we went up to 2, and then shortly afterwards 3 developers, and we had to figure out how to work together effectively.

I’ve had some good experiences in the past working as a team, and some bad ones.   I was eager to try to craft a good working experience, and with the help of my colleagues, I think we mostly succeeded.  We just did an internal presentation to my company on the subject, so now it feels right that I can blog about it.

We started collaborating on Campfire

At one person on a project, well, there’s just myself and myself.  Not much to do there; I can email the customer questions, I can save the emails I get back for stuff that’s important.

At two people, I can communicate with my teammate over IM (if they’re in), or yell across the hall.    Maybe try to maintain a wiki, and forget to do it.   Very often, one person takes primary in client communication, and the other person takes secondary.

At three people, things get a bit different.  Multi-person IM can be a pain.   One primary and two secondaries leads to a lot more communication overhead for the primary.

Enter Campfire.   http://campfirenow.com/

With Campfire, communication can be asynchronous – as long as everybody agrees to check the history of campfire to see what they missed.   We can work on different schedules.  (Its still more fun when we’re on at the same time).

  • Campfire became the place to start asking “I wonder if the customer meant this or that”
  • Campfire became the history of the project
  • Campfire became the place to state what we were working on, and to trade pieces of work (“while I’m in there, I could take care of xxx”)
  • Campfire became the place to let everybody know if one was working from home, or the office, or other
  • Campfire became the place to paste code for a quick code review
  • Campfire became the place to share cute jokes, and side technical things, as we got to know each other more

Note:There are many alternatives to campfire; that’s just what we used.  IRC is how we could have done it in the old days, although it would take some setup to get the history to save.

We centralized our communication with the client via Basecamp

If we had a question to ask the client, we would use a basecamp discussion.

  • We would create a discussion in basecamp, with a subject of “Question: xxxxx”, and the body being the question.
  • We would “loop the client in” (specifying their email addresses) to the discussion.  The client folks would get an email with the discussion thread.  We would also add ourselves to the discussion.
  • When the client responded, it got automatically added to basecamp, and we got notified by email.
  • Once the question was answered, we would rename the subject to “Answered: xxxx”.

This gave us:

  • A central place where we could see “what decisions had been made”
  • An easy way to figure out “what has not yet been answered”

I would recommend this even for a single person working on a project, because of the second bullet point.

Once again, I’m sure there are alternatives that could be used here; although I’m not sure which ones would provide a way to log customer responses with such ease, while presenting the questions in a format that they would easily understand: email.

We collaborated on Google Documents for Digesting Requirements

Wen we received the equivalent of 15-20 pages of requirements to digest and estimate, with links to external resources (mockups) and other “stuff”, we floundered for about 2 hours.  This was our (eventual) solution:

  • One of us pulled ALL the information into a SINGLE document, in Google Docs.
  • We individually went through the document, adding comments at places that we had questions, highlighting things that we found interesting, etc.
    • In reality, we did this together in almost- real time.  Maybe not on the same page, but I could see my teammates putting in comments, and I could even respond to them, as we all looked at the document together.
  • We then sat down face to face (although we could have done it over a phone call), and walked the document from top to bottom – everybody talking about the stuff they found interesting, how they thought it would affect the architecture, etc.
  • We then created a spreadsheet, and started filling out the “details” of the work.  We all collaborated on it as we were typing it up:
    • By the time we left that document, I *knew* that we all agreed on the same way to do the work, and the architecture.
    • The breakdown was more rigorous than I could have done myself (everybody’s rigor got UNION’ed together)
  • During the collaboration process, we also figured out the how to divvy the work up.  In this case, we decided to have one spearhead doing UI work, and another one coming up behind and connecting the UI to the underlying layers.

Google Docs is just what we had available to use at my company; One could also edit office documents in SkyDrive in a similar way.  As long as you can see what everybody is doing in real time, this works.  

We Used Planning Poker and Google Spreadsheets To Estimate

Example planning poker cardsOnce we had a work breakdown, we would use the planning poker process to figure out Low-end and High-end estimates on the individual items.

  • We did this face to face with a real planning poker deck, once.   (Thank you Microsoft)
  • We did this online, in a google spreadsheet, once.
    • We hid the numbers with white-on-white, and ran through the spreadsheet, and called out “done” when we were ready.
    • We revealed all the numbers
    • And then visited line by line to find any differences.
    • Sadly, our estimates were way off – high – but I don’t think it was because of the process.

Beware, with multiple people, you will get a higher number. This is because those who are more sure and bid lower, give in to the folks who bid higher.

We started collaborating on Google Documents for Demo Notes

Our process involves a weekly demo to show the client where we are at.     These demos were on Friday’s at 11am.   Thus, usually on Thursday night (Depending on who got to a stopping point first), or on Friday morning, we would create a “Demo Notes” document

  • We had a “team level” update which was usually put together by the “most chatty” of us. (me)
  • Each of us had a section, which we would fill out – of the stuff we wanted to show the customer, that we had gotten done.
    • We each had a different level of detail.  That’s fine, and in time, comes to be something we celebrate.
  • In each section, we had questions set aside like this, of stuff to ask the customer.   It was clear that there was an answer missing, there was a spot to type in the answer.
    • Question:  Why did the chicken cross the road?
    • Answer:
  • We could have added a section on “pending questions”.  Our client was excellent in responding, though, so we didn’t need to.
  • At the end of the document, we added a “Decisions” section.

During the demo:

  • We would openly share the demo notes document, using it as an agenda.  Whoever’s screen was being shared with the client, the client would see the demo notes somewhere in there.
  • As one person presented, if the client had feedback, or answers, another of us would take notes on their behalf.
    • It gave all of us a job.  With three of us on the call, we had two people listening in and ensuring that anything that was important got captured.   We helped each other.  We came to trust each other in a very “you’ve got my back” kind of way.
  • When the client stated something that was even a bit complex, we would type it into the demo notes, highlight it, and ask: “Did I capture this correctly?”
  • If decisions were made, we added them into the Decisions section.

Soon after the demo:

  • We would scrub the document (a little bit), to clean up the mess that sometimes happens during the demo.
  • We would export it as a PDF and email it to the client.

After the demo:

  • If something was noted that we needed to take care of during the demo, we would add it as a “TODO: xxx” in the demo.
  • When it was done, one could go back to those demo notes, and change it to “DONE: xxx”.  (we didn’t all do this; maybe it was just me – but that’s the beauty of a live document, it can represent “now” rather than “at that time”).

We started sharing administrative tasks

In a past team environment, I made the mistake of “volunteering” to be the only person who did administrative tasks, like merges, status updates, etc.  I was “team lead”, after all, wasn’t I supposed to do this?  In doing so, under the guise of “protecting” my teammates, I signed up for all kinds of pain.

In this incarnation, we’re going with the philosophy “Everybody is capable of, and willing to do, everything”.

The first part, “capable”, meaning:

  • If there’s something that one of us doesn’t know how to do, we’re willing to learn
  • We don’t have to be awesome at it.   As long as we can get it done well enough to move the project forwards.

The second part, “willing”, meaning:

  • Nobody has to be saddled with exclusive pain.
  • I know my teammates have my back.  They are capable and willing to take on my pain.

So, we’ve ended up at this:

  • We have a round robin order, for the weekly administrative tasks:
    • Preparing and sending the Weekly Status Report
    • Sending the Demo notes
    • Updating the Project Burndown

Additionally, we started doing automatic merges from the parent branch into our branch.   At first, we tried a “weekly” approach to it – but that ended up being WAY too much pain in a week.   So, if we hit that again, we might be doing a “its your turn today” round robin approach instead.

We started using Google Docs for Status Report Documents

The person saddled with the status report, would create the document, and look through our time tracking system / the commit log to see what people did, and take a first stab at the contents of the report.

Then, they would invite the rest of us in to the report.  We would update our individual sections, and add a “sign off” at the bottom of the document.

Once the document had everybody’s sign off, it got sent.

There’s More To Learn

Even with all of the above awesomeness, we have room to grow.    My teammates will probably either groan or cackle with glee when they read this, but here’s what I’m thinking:

  • There are other admin tasks we were lax on.   These could be added in to the admin-task-monkey’s list of stuff to handle.
  • We might start writing test cases together.
  • We might start running each other’s test cases [more often]
  • We might use a better way of breaking down the available work – in such a way that more than one person can get their feet wet in a feature.
  • We’ve now invited the client into some of our collaboration – might learn some things there.

The underlying thing is that we were willing to do what was necessary to have a good team working environment, and we did it.   And for that, I am grateful.

Software Estimation Review #1

One of my recent projects had this burndown:

At the start of the project, the “desired” hours were coming down at the speed of the “high” estimate.  If I had used the first 5 data points to predict the end of the project, it would have been near T=7, at a cost of 6N or so, and the estimate was only for 5N.   Not good.

Luckily, things started to sort themselves out as the project progressed.   However, I wanted to understand what was going on.

At work, I track every slice of time at a project and “top level task” level.   I also note down what exactly I’m doing (and usually its outcome – success, interrupt, break, etc – especially useful if there is fragmentation, you can see how interrupted you were – a metric to show pain).

I exported this detail into Excel, and created an item by item grid of time taken for each line item  in the original estimate.

In the process I found several items that I worked on that didn’t have a home in the estimate, and I got “actual hours worked” for each of the line items.   It was shocking.  Here is an abridged version:

  • I saw tasks much larger than the original estimate.
  • I saw some tasks much smaller than the original estimate.
  • I missed several features.
  • I left out entire categories such as refactoring (as the project grows), documentation,  styling and usability.

Possible Interpretations

I did well.

The project was done within the time allotted.  This was a success.  Perhaps the individual line items were not, but the overall project was – I was lucky that I overestimated several items.  If I had tried to squeeze every line item, I would have been in trouble.

Could more Up Front Design yield a better estimate?

The Waterfall thought would be I didn’t do enough up-front design.  However, I spent longer putting this estimate together than I should have – because in this job, putting together an estimate is not billable time!  If I had spent more time on it, I would have had more line items – yielding a higher estimate – yielding a greater risk that the client would not approve the budget – and I would have still gotten it done on time – perhaps even less time.

I have done this before = takes less time

I had previously done a similar project.  As a result,  I estimated several items fairly low. Examples:

  • 5 to 8 (actual: 23)
  • 5 to 7 (actual: 12)
  • 5 to 7 (actual: 14).

This is equivalent of an handyman stating: I’ve built a house before, so hanging drywall won’t take as long.   Incorrect!   It takes time to build software, even if you know exactly what you are doing.

 “Sure that Technology will be easy to integrate”

My coworker (on loan to me for two weeks) spent a lot of time trying to get a grid to work correctly.   Actually, the grid worked fine, the helper classes to interface it with MVC did not.     Trying to get them to work took the majority of the hours on one of the pages.

In the end, he scrapped the helper classes he had downloaded – they worked only in specific circumstances – and rolled his own – and coded the rest of the page in record time.

I’m not sure which way to go on this.  Perhaps having a bucket of “research” hours?    Timebox the items that we research to save research budget in case its needed later?   It seems every project I’ve been on, there’s been one or two things that took some time to get the approach figured out.

I left out parts of the project from the estimate

There were several items that I had not estimated up front that are a part of any project.  See below for a running checklist.

I skipped parts of the estimate

Some functionality turned out to be trivial, or unnecessary. For example, we went straight from a grid of Endor to editing Endor without a stop at a detail page for Endor.   Or, one of the setup pages – thought to be three seperate pages – collapsed into a single grid.

These “mistakes” are what saved the project.

Overall Analysis

I think my mistake was trying to specify everything in the estimate at a line by line level.  As DPR207 at TechEd 2010 pointed out on slide 16, there are different levels of uncertainty as a project progresses.  I was estimating at too granular a level too early – guaranteeing I would miss items, as well as specify some items that were unnecessary.

Doing it Differently

In this particular project, how else could I have estimated it?

Agile, Story Points, etc.

Using Story Points and approaching a project in an agile way (Scrum-like) are my favorite ways of handling project estimation and management.  However: In my role as consultant, with clients pre-approving blocks of hours (based on a high end estimate), I don’t get to do that.   So I’ll pass on that solution.   I HAVE to have a number of hours up front, or I need to find a very understanding client.

Estimate at a higher level

Rather than estimating at a very detailed level, I could estimate in bigger chunks like this:

Level 1: Systems “A Website”“A Batch App” 50,100,150,200,300,500 hours
Level 2: Functions / Features / Pages Grid of FooEditing Foo

Selecting Bars for a Foo.

Reading in an xxx file.

Validating xxx information

 10-20, 20-30, 30-50 hours

Fibonacci Numbers

I will definitely continue using Fibonacci numbers for estimation.  It saves me brain power – Its either a 5 or an 8.  I don’t have to argue with myself whether its a 7 or an 8 or a 9.

I use a slightly modified sequence:   0.5, 1,2,3,5,8,15 … somewhere in there, I jump into adding zeros:  5,10,20,30,50, etc.

Going Forward

Jim Benson (@ourfounder) gave a presentation at CodePaLousa: “We are Not Engineers“.   (I tried to find a link to a powerpoint, and then I remembered, it was truly a powerpoint free presentation. I did find similar materials here)  The gist of the presentation was:  “Humans are horribly designed for doing estimation”.   If I estimate based on gut feel, I’m going to be wrong.   Thus, for my job, I either need to stop estimating, or start to build a toolkit.

This is the toolkit so far:

Keep track of all time and match it back up against the estimate at a line by line level.

It takes about 15-30 minutes to export from Harvest and match things up in Excel.   Its a form of instant feedback.

Estimate in larger chunks.

Rather than counting the number of fields on a page (which is what I did previously), I need to find the average time to write a page, and then double it if it’s a big page.    Try not to estimate in anything smaller than, say, 10 hours, for a chunk – a page, a file processing feature, etc.

Keep a checklist

My checklist at the moment:

  • Initial project setup
    • Machine setup
    • Project setup – layers (DAL, etc) – into SVN
    • Stub version of project that can be deployed
  • Test data strategy (in support of integration testing)
    • Test data creation and deletion
    • Setting up automated integration tests to run automatically
  • Individual Items (Project specific)
    • Larger chunks
    • Include testing time for each item – writing unit or integration tests, and testing by hand.
  • Growing pains
    • Refactoring is inevitable (and not specific to any particular item)
    • Configuration and Deployment (as the project enters different environments)
    • Merging between branches if branches are called for (any time there is a phased release, or a separation between UAT, QA and DEV)
    • Necessary Documentation (after complicated things get done)
  • Project Management
    • Dependent on each client. 10%  of a 35 hour week = 3.5 hours of email, status updates, demo prep, and demoing – which is about right for a good client.  Other clients, the number has been closer to 20%.
  • Research bucket
    • For the unfortunate item(s) that will pop up that just don’t work
    • Also can be used to take extra care to simplifying something so it stays more maintainable.
  • Usability, Look and Feel
    • Cross browser testing
    • “Making it Pretty” – fonts, colors, grids
    • “Making it more functional” – renaming things, page organization, number of clicks to do something
  • I18N and L10N
    • Dependent on client and project.

The Request

How do you estimate your projects?  How close do you get?   I truly want to know.  Shoot me a comment or write it up and add a link.