(By the way, when I say "design" I mean the entire application. Not just the Java or C# code, but also the build environment/tools, database scripts, shell scripts, etc.)
So why do so many projects end up with a horrible design and a mountain of technical debt?
No team sets out to create a poor design. But many (perhaps most) projects end up being difficult to understand and modify. Some eventually end up with so much technical debt that the team or management decide it's better to start from scratch. (Which is often another sort of mistake.)
So where does bad design and tech debt come from? Here are some causes I'm familiar with:
- Time constraints. In the midst of getting an application to build and work right, we often take shortcuts. Another name for 'shortcut' is 'technical debt'. Technical debt is usually rationalized as being necessary to meet a deadline. Sadly, just about every excuse or justification we use to "save time" winds up costing us or our beloved colleagues much more time tomorrow as we saved today. Another way of saying "we don't have time to clean up the code" is "it's going to take twice as long to add a new feature in six months as it does today".
- Personnel changes. New team members can be great contributors to bad design, by writing code that is incompatible with or breaks the existing design. Maybe because they don't understand it, maybe because they think they have a better idea (ego-driven programming), maybe because they're thrown into the fire without any help. "Okay Steve, your first assignment is to add history tracking to changes to the model so the user can revert commands. I'd work with you on it but don't really have time. You can stop by if you have a question, though."
- New requirements. Sometimes a new requirement comes along which is incompatible with the existing design. The right way to handle this is to refactor the existing code towards a new design that fits both the existing features and the new one. The more common way to handle this is to "tack on" the new feature by hot-gluing it to the side of the existing code and slapping some paint on it.
- Inexperience and ignorance. Whether it's a new technology we haven't used, a domain we're not familiar with, or a general lack of experience in development, not knowing the "right" way to get it done will almost certainly contribute to a bad design and/or technical debt.
- Familiarity. Familiarity tends to create blind spots and acceptance of bad design and technical debt. You get used to dealing with the randomly failing unit tests, or the bizarre class hierarchy, or the 15 minute deployment cycle. And you forget that it's really pretty messed up and needs to be fixed.
Nobody prioritizes bad design or technical debt; it happens as a side effect of decisions we make every day. Sometimes it's an intentional decision, possibly made with an understanding of the consequences and with the best of intentions to go back and do the rework necessary to clean things up later. Sometimes the decision is made unconsciously or in ignorance of the consequences.
The only way I know of to prevent and eliminate technical debt is to be intentional about removing it when you see it. It's like housekeeping and home maintenance: you dust and vacuum and sweep, pick up and put away tools and movies and toys, and replace the old weatherproofing around the door. The most successful teams I've been part of are good at housekeeping: they have been disciplined about using best practices to keep technical debt at bay and maintain a good design. (And by the way, when I say "team" I mean everyone who is part of the project -- programmers, documentation, QA, UX, BA's, and management.)
So do your chores and keep the house clean. It's worth the effort.