I'm with you - a background in maintenance (or production support) is a great way to learn programming. Maintenance programming is constrictive in a lot of good ways, so it teaches ingenuity and pragmatic problem solving. I think it can be a great apprentice / junior level job provided there's a senior dev overseeing the work and helping out when things get hairy. It's a great way to find out that professional programming is nothing like gee-whiz magazine article code or Compiler Theory 401. It's where the rubber meets the road.

I often ask developers about their background in maintenance or production support during interviews, and I'm a little wary of dev's who've never maintained a system that was in real life use.

While I agree that maintenance coding is a great way to hone programming skills, as well as familiarising oneself with a given software system, it can be heavyweight and spirit-crushing at times.

Of course, whether it is unpleasant very much depends on the quality of the code you are maintaining, and the scope of your work.

As such, I very much agree with your points about the balacing act of maintenance work; the need to assign good coders to such work, and the importance of being able to decide that, if necessary, a ground-up re-write is appropriate.

Two personal examples spirng to mind; my first 'heavy' maintenance work, and my most recent.

The first time, I had to make substantial changes to a script that was years old, had been updated dozens of times, by many different developers, all with their own unique styles. The code quality was horrendous; dozens of functions that mediated 80% of their activity using global variables, and which imported many modules, all of which exported all their symbols by default. Reams of redundant code that had never been removed, no formal documentation, and entirely out-of-date code comments. I wasn't allowed to re-write the script from the ground up.

It took a couple of days of doing nothing but reading and re-reading the code (while drawing insane diagrams and flow-charts) before I felt confident enough to be able to change anything without breaking it in hard-to-debug ways.

It seemed, at the time, to be a soul-destroying activity. After many days I felt I'd done as much as I could. The code was in much better shape, but it all felt somewhat unrewarding at the time.

More recently, things were in my favour. Once more, major changes to some fairly old scripts that had gone through many revisions. The scripts were infamous for being hard to work with (a 'well' deserved reputation - its amazing what a few hundred 'I'll just bang it in quickly' changes can do to code clarity); but this time, I was authorised to re-write if necessary. While you are spending most of your time replacing extant functionality, it still seems to carry some of the 'buzz' you get when you are working on actual new stuff.

Ultimately, if I were 'breaking in' a new member of the team I'd give them a supervised mix of maintenance and new-build. That is, some maintenance work to really excercise their skills and get them familiar with the systems, and other new-build work to keep them enthusiastic and energised.

The crux of the problem, it seems to me, can be seen by how little shows up when you search for "software archeology".

Given that these languages are "simple" enough for computers to "understand", one might expect more tools to extract information, structure, internal relationships from the code itself, but this is clearly not so. Some languages support refactoring tools better than others. It seems to be easier to automate reasoning about statically typed languages, but easier for programmers to be creative with dynamically typed languages. Combine this with people's (I include myself here) ability to write badly in any language, and it is rather tricky to solve.

Given the costs of maintenance programming, why isn't there more work on moving from "computer literacy" to "appreciation of computer literary criticism"?

I think that the biggest problem with a lot of existing systems is that they don't have extensive tests. When you have tests that tell you when a change is bad, you can step out of "if it's not broke, don't fix it" mode and rejuvenate sections of your code. That's when maintenance can start to become fun. You can refactor and make the code/design better for the next guy (or yourself).

Re finding material on this, yes, it's hard. I wrote the book 'Working Effectively with Legacy Code' and I considered using the words 'archeology' or 'maintenance' in the title, but I figured I that they wouldn't be any less of a turnoff than the word 'legacy' :-)

Halfway through writing, I discovered 'Object Oriented Reengineering Patterns' by Serge Demeyer, St?phane Ducasse, and Oscar Nierstrasz. It's a great book and its targeted toward issues of understanding code. I just never thought to search for 'reengineering.'

I think that the biggest problem right now is language designers think about features that make it easy to write code but they don't seem to be thnking about features that make it easy to maintain. If we could start to raise that awareness, we might be able to make life much easier for the people who follow us.

> I wrote the book 'Working Effectively with Legacy Code' and I considered using the words 'archeology' or 'maintenance' in the title, but I figured I that they wouldn't be any less of a turnoff than the word 'legacy'

Hi Michael, I'm glad you posted a comment -- I meant to reference your (excellent) book in this post:

Working Effectively With Legacy Code
https://www.amazon.com/gp/product/0131177052/

And here's the book you cited. It's a little pricey, though:

Object Oriented Reengineering Patterns
https://www.amazon.com/gp/product/1558606394/

> language designers think about features that make it easy to write code but they don't seem to be thnking about features that make it easy to maintain

It sounds like what we need most is first-class support for testing integrated in the language and the IDE..

I'm with Andy and Dave. After 16 years programming I know I'll never write production ready code the first time. Once I've written a block of code it's straight into maintenance mode and I'm testing, reworking, refactoring, tweaking until it operates as expected and reads well. But renovation _is_ part of that process, I wouldn't distinguish a rewrite from maintenance.

Joel seems to take a rather subjective stance that one developers flower bed is another developers crap heap, but if that were the case as a community we could never agree on acceptable standard approaches to common problems... maybe he's got a point!

I worked for many years as a consultant wherein we certainly gave maintenance headaches to the junior folks, who often responded in the best way they could - which was hacking the code. Successive waves of hacking "petrified" the architectre further and further until it inevitably became a legacy system; Depetrifying legacy systems turns out to be horrendously difficult (indeed it was the subject of my PhD thesis - and consumed four years of my life). What I know look for is developers who understand the need for evolving architectures to reflect evolving assumptions about points of commonality and variability in requirements. With this frame of mind, "maintenance" becomes ongoing architecural evolution - which seems more "respectable" work to the top flight developers I have working for me.

To manage this balance, one thing I've done is to say that if you wrote the code, you maintain it. It's far to easy to write sloppy if you know someone else is going to have to clean up your mess - same deal with coders throwing buggy code over the wall to Q/A. I like having people incented to write clean and maintainable (for themselves later) from the get-go.

Also, that way everyone writes new code and everyone maintains - so no bickering!

I recently "inherited" a few applications from a coworker who was leaving the company. When I found out that I was going to have to refamiliarize myself with VB6 after a year of steady C#, I immediately recalled a description used by a character in Neal Stephenson's Cryptonomicon: making license plates. Which is exactly what I'm doing now. The work is neither exciting nor innovative. It is, however, necessary. Bummer... ;-)

Code will always need to be maintained, thats a given. What I don't like is hiring a group of people to develop the application and then when it is completed, a different group of people to maintain it. Many systems are made in this fashion, often with costly mistakes. Contractors and programmers should be on the project for at least one or two maintenance cycles before moving on. People who build the system, should also maintain it for a certain length of time, hopefully they would learn from thier mistakes before implementing the next system (Reap what you sow). Maybe it wasn't such a good idea to put the entire application's objects in session....

The reverse hold true for maintenance programmers. I heard this before from a number of managers, don't you get satisfaction for fixing bugs? Yeah, it was fun for the first 100, but I've been maintaining now 6 months. Give me something new to do!

I think you need tours of duty in both (new and maintenance) to become a complete developer. There should be no bias between the two.

I have personally adopted the Boy Scout motto of camping with regards to maintenace programming: "Always leave a site better than you found it."

I agree with you in that best developers should work on Software maintainance. But, since most developers don't like it, what should Project managers do ? They rotate people - from maintainance to dev and vice versa. This ofcourse has its own advantages and disadvantages.

> What I don't like is hiring a group of people to develop the application and then when it is completed, a different group of people to maintain it.

Excellent point.

I enjoy maintaining code I've built. It's my personal responsibility to my users. On the other hand, maintaining code I didn't build-- and living with the consequences of decisions I had no hand in making-- is very painful for me.

Make people live with the consequences of their decisions by supporting and maintaining the app *they* built, at least through version .1. It's irresponsible not to!

This is also why I'm a big fan of developers handling tech support calls for their software, at least a part of the time. It really motivates them to address the pain points users are experiencing, because they become shared pain points. ;)

"Of course, whether it is unpleasant very much depends on the quality of the code you are maintaining, and the scope of your work."

All too often it is an experiment in sacrificing an innocent persons reputation by having them take responsibility for code that does not work which they did not create that cannot be declared broken for political reasons (because someone in management has already lied about it working, either to themselves or to others).

I have lost track of the money, health, time and private life I have sacrificed trying to rescue people from incompetantly built systems.

I do not mean "different to how I would have built them", I mean "random scribble, not working, undocumented, code files scattered to the wind across the network and partially lost, impossible to fix regardless of effort and expense".

Sadly, this is more common than not from where I'm sitting (10 years in the industry). Perhaps this is why so many IT professionals go overseas.

I totally agree with you on this one Jeff, but there are some issues you need to account for.

If some developer(s) keep doing the same project from development to maintenance, then you have the problem that you depend on the developers because nobody else knows how the project is working. It is especially dangerous in small projects.
I think it is very much a management problem. Developers should not switch when some estimated goal is reached, but instead the new developers should be introduced in the development/maintenance phase. And if a non experienced developer is to take over, then have some other experienced developer or manager closely follow the progress. In any case the developer should be held responsible for his/hers monster creation.

In my experience, the attributes associated with maintenance programming primarily come from organizational culture and development culture in the organization. In my experience, maintenance programming is significantly less of a core if management and team leads see maintenance programming as a necessary task that helps improve the code base, fosters growth of developers (learning new techniques and learning which techniques to avoid), and improving knowledge of existing code bases.

It is rather unfortunate that management and even too many team leads fail to understand the importance of maintenance programming as a learning and knowledge sharing tool or as a tool to improve the quality of existing code bases. Too many times I have seen developers sacrificed to maintain code bases that are horribly written with severe time restrictions to add functionality.

In my experience, part of the failure comes organizational cultures that fail to understand the need for quality controls through design and code reviews, progress monitoring and review, and peer development. Another factor is excessively strict timelines or poorly estimated timelines that encourage poor programming practices to get the product out the door.

I actually enjoy maintenance programming, sometimes more so than new development, even when the code base is a prime candidate for The Daily WTF. The more challenging the code base, the more satisfaction I derive from pulling out a clean design that is easier to maintain by someone else. I have yet to determine if this is because I want to make the world a better place for my peers or because I'm just weird.

All developers should be forced to do some maintenance. If only to see how well or badly others code!

I was hired as one of the senior developers in my dept. about 10 months ago. Since then I have primarily only maintaining (others) code. I have complained but it appears to have got me no where. Because of all this I'm now looking for a new (more creative) job else where.

If you force people to maintain others systems for long periods of time I'm sure most will simply leave.

I know of many progams written by a very junior programmers , that just happen to come at the right time, that takes even the most season developer weeks to figure out the basic of what it is doing.

Some of my jobs been exactly that, I will relate this one event in my life. This 19 year old highschool dropout, works at a call center, he writes a basic program in clipper to manage the call records. It gets overly complex for him, and it starts to show its cracks. (This is back when MFC 4.x was just released) What does he do , he goes tells the owner, he should manage and hire "maintance programmers". As it turned out, they eventually hired the consultanting firm i was working for. I looked at the code, and sat the CEO down and put it to him simple. What do you expect from a guy who first program was this? verus someone who done this 5+ years? This is a rewrite, there is no way this is even maintainable. I got let go :) called me back after 4 weeks, and well i was there for a year cleaning up the mess made by some 19 year old :)

Maintance programming is harder than new code, mainly due to you are cleaning up someone else crap, and its generally someone that is VERY junior.

When I do software maintenance I call myself a software mechanic.

You always end up fixing other people's crap, (maybe the janitor picture is appropriate, but he should be holding a toilet plunger).

My coworker, Angie, and I started this board where we posted all the examples of shtty code we found. Some of them were real classics.

We used to say wooo hooo, some day we can get promoted to principle or fellow engineer, or even be a software manager -- just like these folks did.

Somebody has to keep things running after others leave. And it's quite unfair to broadly say it's their crap that is left behind. For example I've had the position of maintaining Jeff's work for the past year. And I wouldn't consider it crap at all even when I've really wanted to write him up and ask "Now why did you do that?" sometimes.

A head hunter mentioned to me not long ago that COBOL programmers may again be in demand for high dollar soon since so many will be reaching retirement. The systems still rely on it heavily, especially ERP packages and mainframe systems. I ask how many of you would be willing to stoop to that when the pay could be so good? Or are we at the point that we just code for fun and not for a living? I may have to consider it one day; I certainly wouldn't be above failing back on it.

Hi,
a very good article. IMHO, the main fear of maintenance is loss of design specifications. As long as you have papers detailling a system's architecture, you can at least somehow understand what the guy was thinking.

By the way, I wrote about that a bit of time ago myself, as I had to go over an old project of mine where the documentation had gone astray:
https://tamspalm.tamoggemon.com/2006/06/16/on-taking-over-foregin-code/

Best regards
Tam Hanna

I think every developer should spent some time on legacy systems. I've worked with baselines that are ~10 years old on a few occasions. These opportunities are great learning experiences. Unfortunately, the lessons are usually in what not to do. It has made me think of the poor developer after me when I'm writing new code, and at times, will drive my implementation.

A clever, or complicated, design is not always code that is easy to maintain. A balance needs to be struck.

Whenever I write code I find it beneficial to go through a maintenance phase before going to production.

Simply putting on the side for a while then doing some 'maintenance' - tidying up, refactoring, whatever - with a fresh pair of eyes - can reveal things that you didn't see when you first wrote it.

It allows you to look at the code from a purely maintenance point of view - something that is not usually included in a development cycle. If you have to do it yourself first, it improves the chances of someone else being able to.

I guess it has parallels with Pair Programming in the agile world.

My very first internship was with a state-owned steel company. It was meant to be for one year. The first two weeks we were given a complete orientation in to the entire company's policies, work habits, etc. Even though we were EDP staff, we had to go into the shop floor, see how the furnaces worked, take samples to the lab (in a crucible, with tongs and wearing asbestos-lined gloves!) etc.

After that orientation, I was glad to be in the air-conditioned comfort of the EDP department.

"Welcome from the orientation. As a new intern, we want you to become familiar with our payroll system. Here's the code; study it, understand it, document it as best you can, and get back to me when you finish."

And he dumped a three-foot binder full of greenbar on the desk, and walk out. The thought of those sweaty, beefy, built-like-Popeye-the-Sailor-Man steelworkers beating down my door because I made a change to the code that controlled the payroll system, thus depriving them of that month's salary, was motivation enough to prevent me from breaking any of the code. Which was written in COBOL, of course.

Moral of the story: yes, code maintenance CAN be soul-destroying work. But, with proper motivation, even junior workers can be tasked with doing so, without fear of them breaking the code.

And, in hindsight, it was a good experience. It taught what NOT do when it came to coding, documenting, and suchlike. Most programmers hate maintenance simply because no one takes a class in code maintenance in university. It is like a combination of spelunking and architecture. And it can be fun, as long as you don't have to do too much of it at a time.

Whether writing new code or maintaining old code, you're still writing new code. Either way is just as creative. I think it's a much larger challenge to be able to shoehorn your 'new' maintenenace fixes into old code and you can learn a lot from others code. The difference is like building a new automobile from the ground up or repairing one as it speeds down the highwah at 60MPH. Which is more exhilerating? *g*

every one has been speaking about how terrible or great it is to do code maintenance. Could please anyonegive me some tips about it as far as how to do it efficiently. I have always done new development and all of a sudden I have to work with different legacy codes.What should be my strategy?
sincerely sa

Hi Sunny, there's a good book on this topic: Working Effectively With Legacy Code, by Michael Feathers.

https://www.amazon.com/exec/obidos/ASIN/0131177052/codinghorror-20

>> I actually enjoy maintenance programming, sometimes more so than new development, even when the code base is a prime candidate for The Daily WTF. The more challenging the code base, the more satisfaction I derive from pulling out a clean design that is easier to maintain by someone else. I have yet to determine if this is because I want to make the world a better place for my peers or because I'm just weird.

>>Lothan on June 14, 2006 02:40 AM

I was reading the comments above and wondering what could be the WTF mean. This is a general rule to explain an abbreviation before you use it in your writing. This showed that you've presumed that everybody understand what WTF means. And this is the reason why maintaining a code is so frustrating.

I have done mostly maintenance programming in my brief career and I definitely enjoy some things about in much more than new development. With new development, there are so many different possible ways to accomplish something, and many seem perfectly reasonable until, for instance, the requirements change slightly.

With maintenance programming, your possibilities are constrained somewhat, but it is usually possible to improve things by some reasonably objective measure of how well it works and/or how readable the code is. It can be satisfying knowing that you have gotten things right, or at least more right than when you started.

Since it is often very rare that anyone can get things completely right on the first try, you might not get this kind of satisfaction from new development.

P.S. Masud: WTF means What the F***?

Personally, I like maintenance. The problem space is generally well defined; i.e. it has to do what it does now, but faster, or more scalable, or with some new feature, etc. (I suppose then, as other have pointed out, that all development is maintenance in some respect.) I know that I'd rather write and improve code than try to write documents and collect requirements.

It can be tough, however, and mostly due to personality, I think. The first thing is that it can be miserable working on someone else's code. The easiest way around this, I've found, is to make the code mine. If it's someone else's, then why am I maintaining it? I'll ask questions, I'll get suggestions, but then I'll do what I think is best. I've got to. The flip side of this, of course, is that if someone else begins to maintain code I've written, I've got to let them do it. I can answer questions, I can make suggestions, but at the end of the day, they're doing the maintenance, and I've go to let them do it the way they see fit.

In actuality, we should all be working towards the same goal, so all the code is mine, and none of it is. I think that it's harder for most people to let go of code they've written. People also don't want to step on each other's toes, so it can be difficult to refactor something that needs it if the person who wrote (with probably valid reasons for doing that way) if they are right there with you, and maybe haven't bought into the plan.

I do know that if you can combine communication, testing, and source control, together with the will to refactor constantly and mercilessly, you can write some really good software. The sense of camaraderie and satisfaction of a job well done has always followed along for me as well.

@masoud and Jim: "The Daily WTF" is a very enjoyable web site where people relate short stories about the crazy code they've worked with, usually with examples. It's at https://thedailywtf.com/. The official expansion of "WTF" in this context is "worse than failure".

Masud,
If you come across a word or acronym you don't recognise such as WTF, just google "define wtf" (without the quotation marks). Works almost every time.

Michael Feathers,
I learned a reat deal reading your book "Working Effectively With Legacy Code" - time and money well spent. The book placed much of what I have been doing the last 10 years in context and has helped me doing it better. I am the "expert" on an old in-house application and not get to do much new stuff, but now I do not feel bad about it. I stopped long ago just patching code - I always try to leave the system in a better state, even if it is only by a little bit. No-one else sees this but it gives me personally much satisfaction.

Jeff,
Welcome post on a subject close to my heart - thanks!

I use the term software archeology on my CV but it certainly hasn't helped me get a new job. I also think there is a certain amount of psychology in maintaining legacy code. I work on the assumption that surely it must have worked in the past and so what was in the developer's mind when they wrote that (crap)? What has changed in system requirements to make it look so wrong now? Often these insights help to uncover part of the original architecture and come up with a reasonable solution rather than have to scrap huge chunks of it.

@mosaic - I'm with you - always try to leave it in a better state than you found it.

Large majority of books, tools are all geared for write-from-scratch. Open any book and the first chapter is about installing. You have to look deeply before finding out how to query, if it's even there.

I have done software maintenance in the past and it is honestly the biggest pile of shit I have ever undertaken.

1. The most commom problems are giant routines that completely loose their sense of identity, do multiple things and become a sphagetti mess.

2. operations nested to too many layers. 3 is plenty, but 19 just takes the p***.

3. No documentation or design diagrams - not that design diagram would be any use to something that has no structure anyway.

4. Sometimes we don't even know where the code is or there is mutiple versions and no way of knowing which version is which.

5. BUT the WORST thing by far about software maintenance is the complete lack of interest or cooperation from managers. I tried to push managers to introduce coding standards where I work as I was sick of working with shitty, uncommented code. Their attitude was I needed to improve my ability at working with shit code. I thought F*** you and decided to leave the software scene altogether.