Carbon Five wrote an article on "Rewrite or Rescue", speaking to legacy Java apps to which they lent their expertise to save.

However, their comments on the habits formed by a fire-fighting development culture and the steps required to transition an organization out of that toward a focus on quality are spot-on for any development platform.

They summarize their thoughts as such: "All too often, the source of poor software quality is poor process and practices. Unless you fix those problems it is not worth embarking on a new software development effort."

This all jives completely with my experiences with system/component rewrites and organizational culture change for development teams. If you have low quality code coming out of your teams, no amount of rewrite will produce a better quality product. You have to address the problem at the core, which is the developers themselves.

Sometimes you have good coders who are failing because they are working in the wrong environment. This is usually very easy to fix. In my personal experience, and from what I've heard often in conversation with industry peers over the years, the usual suspects are (a) insufficient time constraints for architectural planning and (b) throwing out basic quality controls (such as code reviews, and/or the time necessary for writing automated unit tests). These items are usually sacrificed in the name of speed, but (as has been written about countless times before by people smarter than me) the hour or two spent in architectural brainstorming and the extra 25% dev time to write automated unit tests is recouped as soon as the second dev cycle hits, which is usually right after the code goes to production and you get your first batch of new feature requests.

The rest of the time, you just have "bad" coders. I say "bad" because sometimes you have a good coder who has become jaded or lazy and just needs a kick in the ass; in remaining cases you have people who really are not very detail/quality oriented as part of their personality. I'd say for every five "bad" coders I've come across in my career, only one is salvageable; the rest are clock-punchers that sap the productivity of your good coders who have to spend their valuable time cleaning up the other guy's code (amounting to negative effort). The only solution with the truly bad coders is to fire them and seek better talent. Unfortunately, with the state of the tech industry being what it is, it's always a challenge to find better talent. Pick your poison.

Carbon Five speaks directly to the fire-fighting situation, too: "To be successful in a rescue mission or even a rewrite, you have to turn the firefighting culture into one where developers value quality and work for it daily. They should be excited to make things better and be engaging each other with ideas and practices to get there."

As to how to get there: "Again, this can be a very difficult effort. Sometimes it requires dramatic measures. In our experience this includes:

- Changing a workspace to remove barriers to casual conversation
- Relocating developers to the same physical location
- Hiring new blood and firing those resistant to change
- Pair programming
- Book clubs and study groups"

I generally agree with these statements, but have a few thoughts on specifics. I already spoke to hiring new blood and firing the bad, so here are my thoughts on the other items.

Workspace/Relocating Developers
Workspace is definitely a huge factor in developer productivity. I was recently given the option to relocate my development team to another part of the building where we'd be split two to a room. I declined because our current space is perfect for the size of my team: we can all see each other with a twist of the head, be at each other's desks within one chair-roll, and when a meeting is required I just ask everybody to reach a reasonable stopping point after which we have the meeting in five minutes or less from our desks.

Obviously, this won't scale as the team grows. As we get bigger, I plan to slice the team in to squads (of around four coders or less) that can maintain the same gelling qualities as a small team while accommodating the split-up layout of their environment. Experience has taught me that big dev teams are never productive without masses of process to keep everybody in line, which is wasteful and time-consuming. In contrast, squads can be given goals and responsibility for hitting them, after which you can rely on them working effectively as a unit through professionalism and in-team communication. You also get some benefits of having multiple groups relying on each other for components, since the pressure is on to not be the group that misses the deadline (thus holding up the project) or the group that writes the bit that breaks/is buggy (pointing to skill or attention-to-detail issues on a particular squad).

Pair Programming
The XP-based pair programming has never seemed that great to me personally. I just can't see the business justification for two developers working on the same piece of code the whole time.

However, when you have a gelled team, they pair program automatically at the appropriate junctures, for no longer than the time required to get the benefit. When somebody on my team doesn't know how to do something, or can't decide between several techniques, they vocalize the problem to the room and usually get an answer within a few minutes of discussion. In some cases the topic is sticky enough to require a whiteboarding session, which in itself is essentially an impromptu mini-architectural discussion. We already know from experience that architectural discussions yield better end products.

The key take-away I want to present here is that if the environment is right and the team is gelled, you get the benefits of pair programming automatically without needing to formalize or force the issue.

Book Clubs and Study Groups
I'm a big fan of inter-resource training and company-sponsored developer libraries. However, whether or not a company needs to set aside time in a developer's week for clubs and study groups depends largely on the skill set at hand. This sort of thing is a must when you have a lot of junior level coders, but is of much less value with a seasoned team. Seasoned developers will usually initiate their cross-training fix once or twice a month on their own, or at local user tech groups.

One thing they do at Highwinds that seems kind of cool is a bi-annual in-house conference, where all the remote developers are flown in to the mothership and we share the nitty-gritty behind-the-scenes details of the entire system. I'll get to see my first one of these in two weeks, so I'll let you know how it goes.

Conclusions
In closing, we've been approaching some rewrites with StrikeTracker and our web service tier since I got to Highwinds in April. We have not had time to address some of the larger-impact work yet, but we've had good success by identifying a high-level plan of where we want to go and writing new system components/organizing incremental cleanup in that vein.

We identified a few basic tenants: (a) components are developed as agnostic libraries as much as possible, (b) automated unit tests are mandatory, (c) inter-component contracts must be developed as part of the design process and strictly adhered to during implementation, and (d) a new hub will eventually combine the components in a loosely-coupled way. So far, we've hit three out of the four goals without much effort. I've worked on several projects over my career where we've been able to develop new code with this style of plan and plug it in to the old app framework with minimal effort, so I know it works.

Once again, I can only speak from my experiences, but a full system rewrite rarely proves to be a good idea. It's always better to tackle things incrementally so you can course-correct as you go and start extracting value from the new code being produced. Of course, if you're not going to hold quality in high regard as you go, you might as well just stick with the old code and keep on fightin' those fires!