One thing I realized that sometimes engineers go extreme in designing things/code for future cases which are not yet known. Many times these features don't even see any future use case and just keep making the system complex.
At what point we should stop designing for future use cases ? How far should we go in making things generic ? Are there good resources for this ?
They're afraid.
Fear: If I don't plan for all these use cases, they will be impossible! I will look foolish for not anticipating them. So let's give into that fear and over-architect just to be safe. A bit of the 'condom' argument applies: better to have it and not need it than to need it and not have it.
But the reality is that if your design doesn't match the future needs really well, you're going to have to refactor anyway. Hint: there will always be a future need you didn't anticipate! Software is a living organism that we shape and evolve over time. Shopify was a snowboard store, Youtube was a dating website, and Slack was a video game.
So my answer: relentlessly cut design features you don't need. Then relentlessly refactor your code when you discover you do need them. And don't be afraid of doing either of those things because it turns out they're both fun challenges. The best you can do is to try to ensure your design doesn't make it really hard to do anything you know or suspect you'll need in the future. Just don't start building what no one has asked for yet.
This is knowledge most engineers I've met completely ignore in favour of the superstitions, personal opinions, and catchy slogans that came out of the '90s and '00s. It's common to dismiss the early software engineering approaches with "waterfall does not work" – as if the people in the '60s didn't already know that?! Rest assured, the published software engineers of the '60s were as strong proponents of agile as anyone is today.
Read this early stuff.
Read the reports on the NATO software engineering conferences.
Read the papers by David Parnas on software modularity and designing for extension and contraction.
Read more written by Ward Cunningham, Alan Perlis, Edsger Dijkstra, Douglas McIlroy, Brian Randell, Peter Naur.
To some extent, we already know how to write software well. There's just nobody teaching this knowledge – you have to seek it yourself.
The mid-career developer knows this from experience and sticks to the minimum necessary to meet the requirements. Fast and efficient in the short term.
The senior developer mostly agrees with that but also knows that there is nothing new under the sun and all software ideas repeat. So from experience they can selectively pick the additional implementation work to be done ahead of time because it'll save a lot later even if it's not needed yet.
As an aside, this is one of the many reasons why I interview based on discussing past projects and don't care for algorithm puzzles. Unless I specifically need an entry-level developer, I'd prefer to have the person who has written a few silly (in hindsight) complex frameworks and has painted themselves into a corner a few times with overly simplistic initial implementations. That's the person I know I can leave alone and they'll make sane decisions without any supervision. The algorithm puzzle jockey, not so much.
If you're writing code for a device that's going to hang out in the forest for 10 years with no updates, write just enough code to solve the problem and make it easy to test. Then test the fsck out of it.
If you're writing code for a CRUD web service that you know will get rewritten in 10 months, write just enough code to solve the problem and make it easy to test. Then, test the fsck out of it.
If you're writing an Enterprise app that will be expanded over the next half decade, write just enough code to solve the problem and make it easy to test. Then, test the fsck out of it. You simply cannot know how your code will have to change so you cannot design an architecture to make any arbitrary change easy. Accept that. The best you can do is write clean, testable code. Anyone... anyone who thinks they can design a system that can handle any change is flat wrong. And yet... time and again, Architecture Astronauts try it.
A junior engineer recently presented me with some completed work. There was a data schema change on one interface and an API version change on another. He had created a version of the microservice he was working on which could be configured to work on either the old or new version of each of these, with feature flags, switching functionality and maintaining the old and new model hierarchies for both. He viewed this as an achievement. While technically it was, the result was code bloat and unnecessary complexity. The API upgrade and schema change were happening at the same time, and would never be rolled back, his customisability was a net negative.
Code is a liability. Be sparse. Build with some flexibility but overall just build what is required and when the future comes, change with it. Don't build to requirements you can imagine, because the ones you don't will kick your ass.
Period.
Instead, only build what you need to solve the problems you have in-hand, but do so in such a way that your software and your systems are as easy to change as possible in the future.
Because you very, very rarely know what your future use-cases really are. Markets change over time, and your understanding of the market will also shift, both because some of your assumptions were wrong, and because you can never really have a total understanding of such a complex phenomenon.
Your business needs will change as well; your top priority from 2019 is likely very different than it is today.
That is why you build to embrace change, rather than barricade against it.
As to how, I'd start with Sandi Metz's 99 Bottles of OOP: https://www.sandimetz.com/99bottles
Learning to write readable code is also pretty important; Clean Code is a good starting point (https://amzn.to/3168z3A), but I'd be keen to know of any shorter books that cover the same materials.
Growing Object-Oriented Software Guided By Tests (GOOSGT) is a good read as well: https://amzn.to/3du1sEL
Over-engineering can be avoided by carefully sticking to a budget.
For a lot of developers, there's a trade-off between rationality (in the sense of ROI) and feeling good. It doesn't feel good to make every engineering decision against a budget. My dopamine levels [0] skyrocket when I visualize making some component generic, or future proof. I come crashing back to earth when I realize the budget is for a driveway. Budgeting earns the bosses / customer / capital a better return. Engineering for future use cases or making things generic (or the mere anticipation thereof) is an easy way to get a massive hit of neurotransmitter that makes you feel good.
[0] Not a neuroscientist. But I think it's useful to label that spike of feel good and motivation. It may have nothing to do with dopamine.
Better is to design for change. Keep everything modular. Keep your concerns separated. When something needs to change, you can just change that thing. When the whole basis of the system needs to change, you can still keep all the components that don't need to change. The only future use case you should design for is change, because that's the only thing you can be certain of.
So don't make things more generic than you need today, but make sure it can be made more generic in the future. And in that future, don't just add new features all over the place, but reassess your design, and look if there's a more generic way to do the thing you're adding.
I do this sort of stuff all the time on my current project, and it works quite well. There's no part of the system we didn't rewrite at some point, but all of them were easy to rewrite.
“An architect's first work is apt to be spare and clean. He knows he doesn't know what he's doing, so he does it carefully and with great restraint.
As he designs the first work, frill after frill and embellishment after embellishment occur to him. These get stored away to be used "next time." Sooner or later the first system is finished, and the architect, with firm, confidence and a demonstrated mastery of that class of systems, is ready to build a second system.
This second is the most dangerous system a man ever designs. When he does his third and later ones, his prior experiences will confirm each other as to the general characteristics of such systems, and their differences will identify those parts of his experience that are particular and not generalizable.”
The overall advice is to practice self-discipline:
“How does the architect avoid the second-system effect? Well, obviously he can't skip his second system. But he can be conscious of the peculiar hazards of that system, and exert extra self-discipline to avoid functional ornamentation and to avoid extrapolation of functions that are obviated by changes in assumptions and purposes.
A discipline that will open an architect's eyes is to assign each little function a value: capability x is worth not more than m bytes of memory and n microseconds per invocation. These values will guide initial decisions and serve during implementation as a guide and warning to all.
How does the project manager avoid the second-system effect? By insisting on a senior architect who has at least two systems under his belt. Too, by staying aware of the special temptations, he can ask the right questions to ensure that the philosophical concepts and objectives are fully reflected in the detailed design.”
By refactoring what I mean is continually revisiting the architecture of your code, identifying common functionality, better organisation of code, better abstractions. The right decisions for your codebase change as it evolves and so you need to keep reexamining these (implicit) decisions.
Continuously incrementally refactoring your code enables so many things to work better. Your example here is a great one - don't implement something you don't have a direct need for. Don't design for it. If you have a culture of refactoring then you can be confident that in the future if that need crops up, you can implement it with appropriate refactoring that the result will look at least as good as if you designed it in up front. If you don't refactor as a team, then you have to do it now, or putting it in later will result in a mess.
Even then, I don't dig too deeply into things. I look for two things:
* What a migration path to realistically expected features _might_ look like. If I'm debating between two column types or table setups, go with the more flexible option.
* What scenarios will be extremely hard to get out of. Avoid those when reasonable.
----
In other words, instead of proactively planning for some feature to evolve into the unknown in some way. I'm looking to make sure I'm keeping flexibility and upgradability.
Building something “generic” to anticipate “future use cases” is not design; it’s the postponement of design.
If you anticipate some unpredictable future feature, then either you do not understand the design space yet, or you are following the directive of a business which does not understand it.
Startups (prior to product/market fit) have a legit reason for not understanding their design space; they’re still exploring it. That makes design pretty hard. Instead of creating The One Generic System To Rule Them All, I’d recommend small, less-generic, low-risk prototypes, that can be easily replaced or refactored. Keep them uncoupled. Meanwhile, use those prototypes to build up your understanding, so that you will be able to make informed design choices at a more mature stage.
I’m speaking in broad strokes; reality may apply.
My rule of thumb for this is to repeat yourself at least 3 times before trying to build an abstraction to DRY them up.
3 use cases is obviously not always sufficient inform the design of a good abstraction, but IME:
- abstracting at 2 use cases is very often premature and results in leaky/brittle abstractions where the harm from added coupling between fundamentally incompatible usages far outweighs whatever little harm can be introduced by keeping the 2 usages as repeated code, and
- with any more than 3 use cases, the maintenance cost of keeping the usages in sync starts to scale non-linearly, so at the very least thinking about a design for a potential abstraction at that point becomes a worthwhile exercise, even if we end up deciding it's not yet appropriate (hence the "_at least_ 3 times").
In 2017 I started https://wordsandbuttons.online/ as an experiment in unchitecture. There are no generic things there whatsoever. There are no dependencies, no libraries, no shared code. Every page is self-sustaining, it contains its own JS and CSS, and when I want to reuse some interactive widget or some piece of visualization somewhere else, I copy-paste it or rewrite it from scratch. In rare cases when I want multiple changes, I write a script that does repetitive work for me.
This feels wrong, and I'm still waiting when it'll blow up. But so far it doesn't.
Sure, it's a relatively small project, it has about 100 K lines along with the drafts and experiments. It is although inherently modular since it consists of independent pages. But still, this kind of design (or un-design) brings more benefits I could ever hope for.
Since I only code that I want on a page, every one of my pages is smaller than 64KB. Even with animations and interactive widgets.
Since all my pages are small, I have no trouble renovating my old code. Since there is no consistent design, I forget my own code all the time, but 64 KB is small enough to reread anew.
And since there are no dependencies, even within the project, I feel very confident experimenting with visuals. Worst case scenario - I'm breaking a page I'm currently working on. Happens all the time, takes minutes to fix.
I still believe in the golden middle, of course, I'll never choose the same approach in my contract work; but my faith is slowly drifting away from "design for the future" in general and "making things generic" specifically. So far it seems that keeping things simple and adoptable for the future is slightly more effective than designing them future-ready from the start.
IMO, good future-proof design is about putting in place good components and system boundaries.
Those components and boundaries can be highly specialised and have as few options as possible - it's much easier to make a system boundary more complex than to make it simpler. So start as simple as possible!
Now, with those boundaries, you can easily write tests, and iterate on the different parts of the system. Bad code in one component doesn't "infect" bad code in another part of the system.
Most "balls of mess" systems that I have seen came down to not having clear boundaries between components of the system, rather than being too generic or not generic enough.
2. Understand it's very often easier, faster and better to write something trivially but twice than writing it well once;
3. Realize no matter how hard you try, unless this is something you've already done once (which brings you back to #2 above), then your information of the problem is incomplete and therefore any all-encompassing solution you may have will essentially be partial and based on extrapolation of your knowledge and guesswork.
The above also deals with a significant cause of procrastination - an internal subconscious feeling we can't do what we're about to do well, therefore we'd rather not even start it. Just sit down and tell yourself, "today I'm going to write a bad module X", and with enough time you'll be able to sit down again and write a good X.
The one thing you should spend a bit more time on is interfaces and decoupling, but as for internal implementations of things, or even entire system blocks that you can swap later on - don't bother too much. It'll be OK at the end, and you'll get there faster. Even if you need to rewrite the whole god damn thing.
1. Keep It Super Simple (KISS).
Implement the solution which works the easiest first. Copy-pasting code is ok at this point. So if an "if" will do it, use an "if". (Don't start with a BaseClass with an empty default method and a specialized class which overrides it).
Once you have something working (hopefully with some automated tests?) you are allowed to refactor and abstract. But see next point.
2. The Power of Three!
You are only allowed to abstract code once you have copy-pasted it in at least three places. If you only have two, you must resist the urge and move on. Maybe add a comment saying "this is very similar to this other part, consider abstracting if a new case appears".
After abstracting stuff, run tests again (even if they are manual) to make sure you have not broken anything.
Be warned that this method is not guaranteed to produce the "best possible code for future you". If you keep doing this long enough, you might get stuck at "local maxima" in design. Future you might need to do big refactors. Or not. That is the nature of programming IMHO.
I would look to senior people who have boots-on-the-ground experience delivering and maintaining projects/products, ask them if they think you're overdoing it.
edit: I want to add, get input from non-engineers too. Ask them "in your experience, how far have projects diverged from the initial requirements/purpose? I'm trying to plan for the future but not over-complicate things"
If I had to boil the first part down, I'd say you need to focus your engineering on the interface points - network protocols, schemas, API sets, and persistence formats immediately come to mind. It's a truism of the profession that those are the most expensive aspects of a system to change, and therefore they're the least likely to be mutable down the road.
That's really the easy part... the harder part is maintaining the team and self discipline to keep things simple. For better or for worse, engineering job placement (and oftentimes personal satisfaction) is highly resume-keyword driven. Organizations and people all tend to chase the next resume keyword on the assumption that it'll help them deliver more efficiently or get the next job placement or write the next blog post or whatever else. The net of this is that there's a very strong built in tendency for projects to veer in the direction of adding components, even before considering whether or not they're appropriate for use. So keeping your eye on actual, real project requirements over all else is both important, and can require convincing and political work throughout a team and organization.
That said, I think there are two steps in designing abstractions:
The first is to split up and isolate both code and data into small, simple pieces. These can easily be non-DRY (structurally) and often are. You'll have cases where you often (always?) pass things or do things in sequence in the same way.
The second is to merge the pieces with parametrization (or DI in OO), defining common interfaces, polymorphism etc.
The first part is very often beneficial and makes code more malleable, which makes future features/changes less cumbersome.
The second part however is the dangerous but also powerful one. It can lead to code that is much more declarative, high-level and adding new features becomes faster. But the danger is that a project isn't at the stage where these abstractions are understood well enough, so you end up trying to break them up too often.
I try to default to writing dumb, simple small pieces and deferring abstractions until they become provably apparent. Fighting the urge to refactor into DRY code.
Now there is also another issue with writing "future proof" code, which doesn't involve abstractions but rather defensive programming, which is an entirely different issue.
The followup is, if we have to change/replace said system, how difficult would that be? If scaling the simplest solution would be difficult/painful, then one can start look to higher complexity solutions. The idea isn't necessarily to always choose the simplest solution, but having it in mind can be helpful. It crystalizes one of the endpoints of the simple/complex spectrum and helps weigh the pros/cons of various approaches.
As a side note, it's sort of amusing that a lot of design focused interview questions are around things like "design a twitter/instagram like system that'll scale to billions of requests per day." I've never had to do anything like that IRL, but no interviewer has ever asked me to design, say, an invoicing system that gets called rarely. So perhaps one of the reasons the arc of software engineering bends towards complexity is that we're continuously rewarding a "build massive scalable systems" mindset?
I don't know that there is any readable wisdom that will teach you how not to overengineer or underengineer, such that with that knowledge, you automatically know how to achieve the right balance. It is likely a necessary part of the process to build out software projects that are in fact overengineered or underengineered and to intimately learn from that process as well as the aftermath how to tune one's own process to strike an artful balance between the two extremes.
put another way, the MS and Google teams you worked with were screwing up, but screwing up in a way that is necessary for people to learn, if they do in fact learn (they might not learn).
all of that said, starting out by intentionally underengineering, if you know that you are one to start overbuilding, might be a good strategy. but you might have to do some really huge refactorings subsequent to that when you learn your architecture is insufficient.
When everything is DRY and functions follow SRP, it's hard for your code not to be "future-proof."
Tests only when they make sense, not when you are mocking half the app.
Guessing the future and engineering to cope with it is a risky, error-prone business. Good engineering practice should always seek to minimise reliance on unreliable (prediction) data to create future-proof designs.
So I'd go with stopping as soon as it meets current use cases AND then shift gears to make it easy for someone else to pick-up in the future, e.g. write whatever tests are necessary and document thinking.
If it's easy to extend now, it will still be so in the future. Plus, there will be whatever learning has occurred since to make an even better job of it.
> How far should we go in making things generic?
Only if: 0) It's not only about guessing future needs. 1) The maintainers are over 90% certain it will make it easier for them to maintain and test now.
> Are there good resources for this?
There's a mountain of media on "Agile" software development, and it's different flavours. It's not particularly Software Engineering focused, but I enjoyed the "The Lean Startup" by Eric Ries.
Good luck and have fun.
It is more difficult to persuade others not to over-engineer. I have tried and failed many times to do so. In fact, if you try too hard you may just make them hate their job. Folks get into software development for a whole host of reasons and only one of them is shipping. They may not be satisfied with a job where they funnel requests from a PM directly into code with little creative input. I'm not sure I can give good advice on this front, other than to look out for certain red flags during hiring.
You aren’t gonna need it.
As with anything tho, ruthlessly cutting stuff or rounding corners is in tension with supporting future capabilities, sometimes you do need it it turns out.
I believe this is well documented today, but maybe not that much in academia.
What they need to avoid this trap is to talk with people who got burned earlier.
As a rule of thumb, don't design for future use case, ever.
Future-proofing is a fool errand.
I've also found that designing for unknowns can sometimes be resolved by asking questions about the unknowns until they're known. Sometimes, business stakeholders have an answer, they just weren't asked.
For example, in the 80's the linked list was my go-to data structure. It worked very well. But with the advent of cached memory, arrays were better. But my linked list abstractions were "leaky", meaning the fact that they were linked lists leaked into every use of them.
To upgrade to arrays meant changing a great deal of code. Now I encapsulate/abstract the interface to my data structures, so they can be redesigned without affecting the caller.
If you are doing a rewrite of an existing system or have many years of experience with a similar product then thinking ahead can save time. Otherwise you are probably better of not trying to be too clever.
The difficult part for most people is actually being honest in the process :)
Also Kent Becks timeless advice is good to keep in mind
1) make it work
2) make it right
3) make it fast
In that order. You might not need all three :)
And this is not an obligation. If you are really sure about the design of some component, you can also do it right the first time. But in most places, you usually don't know the design well enough.
Also, the imperfect solution can help as a validator. Have a problem that has a unique solution? Make a slow algorithm for it which you are sure is correct, then build a faster version and use the slow algo to compare for correctness. You can also use the slow algorithm to tell you about non-output values that both algorithms compute implicitly or explicitly.
I usually try to support the opposite idea by bringing agile to the table. Agile doesn't say "please don't overengineer" but "this can change a lot in a couple sprints, requirements may be different, there are more (or none anymore) use cases now" and hence why to spend so many resources doing something that not only doesn't cover all potential unknown use cases but also adds overhead when reading the code for no benefit. For some teams this has clicked very well. Appropriate agile training can help a lot.
However it doesn't always work and if the person(s) doing this have higher ups backing it up (even if unintentionally) you're fighting the wrong battle. If you can't change it, move teams or to a company where money is a bit tighter and/or where these behaviours aren't tolerated - in other words, where things needs to get done and there's no time for experimenting with things that most likely won't yield returns.
Then understand a new problem.
Repeat.
If you then start doing things like externalizing inputs, writing tests then when your past solutions become future problems.. You will have created the guide rails on how to solve both the past problem and the future problem together. This is the futureproofing that you should be aiming for. Make sure you write at least a few unit tests that mock out the important logic parts, and write up a short 1 page document about the intent of the program, maybe with a picture and drop that in the README.
* Focus on the current assignment. Implement it using clean code principles, don't overthink the problem.
* Rather than spending time on design decisions, allocate time on handling edge-cases. These will save you from PageDuty alerts.
* Plan excessive for future use-cases only around data models that insert/read into the database. Data migration is super-painful. A more generic design around your database is almost always preferable.
* Have a feature toggling[1] service in your codebase. It will provide you with a better understanding of how you can implement new features alongside existing codebase in the same branch. Releasing features from long-running separate branches is almost always a wrong decision.
* Always keep in mind time-constraints and how urgent is the requested functionality. Don't let perfection be the enemy of productivity.
* Have a process in place that allows for the technical debt tasks to be tackled independently. It helps fix some bad design decisions, which become apparent in light of new requirements.
Use TDD and only write only just enough code for the currently known required functionality. When you get to know more required functionality the tests protect you from breaking existing functionality and you can extend your code to support the new use cases as well. At this point you should make your code just generic enough to support all known use cases without code duplication or too much boilerplate code. If you can support functionality in more than one way you can decide what way to choose based on what you expect in the future. But choosing the most simple solution trumps attempting to future proof your code. It turns out that predicting the future is quite hard and there will be new feature requests that nobody had foreseen and code that has been made as generic as possible will not handle this well.
Spiderman says that with great power comes great responsibility. The converse also holds true. With great responsibility comes great power. You cannot just pick the easy part of TDD, be irresponsible and expect to have any power. The less easy parts of writing a test for every use case and of refactoring all the time make the practice of not attempting to guess the future possible. If you leave out the prerequisites the end result will not be so very pleasant.
The first is: Requirements change.
For many reasons. User needs change. Business processes change. Technology evolves.
If you're going to keep up, you must design code that is easy to change.
Code that is easy to change tends to be code that can work with lots of different code.
Code that can work with lots of different code tends to lean towards the general end of the spectrum, rather than being too specific.
Designing code that is modular does not mean that you need to over-engineer, and it doesn't even mean that it needs to be used more than once.
It should mean that the code has locality: The ability to understand the full effect of the code without also understanding the full context of the code around it or the full history and future life of every external variable it uses.
It may sound to new developers like I'm talking about something complicated, but it's the opposite:
Code that can be easily adapted to future use-cases tends to be more simple. It tends to know the least about it's environment. It tends to do only one thing, but do it so well as to be perhaps the optimal solution.
What follows from the first principle is perhaps the most important principle in software development. Remember this and you'll find yourself needing to do a small fraction of the work you once did to produce the same value:
A small change in requirements should lead to only a small change in implementation.
I was talking to a new guy that just joined my team yesterday on this subject. You really cannot predict the future.
You could also look at it from one other angle. If you are only building the bare minimum to satisfy the requirements, that is a lot less code you are writing. If you need to replace the system, that is a lot less work to go back and rework.
I dont like "used the simplest solution possible" advice, because people who I claimed doing that in real life tended to do unmaintainable spaghetti mess. It was "simple" in the sense of not having abstractions or nested function calls, but hard to read and understand big picture. Sometimes generic is a way to untangle such previous spaghetti mess. As in, it is second step on the road that requires 2 steps till it is really good.
Understand politics. A lot of those "future cases" are things that analysis or management requires initially or indirectly. They are also often result of trying to hit vague requirements - you dont know what customer really needs in enough detail, so you do it configurable in the hope of hitting the right place too. They are also situations in which people burned out in the past or special cases of special cases.
Otherwise said, the complicated thing is often requirement, initially. Someone had reason to ask for it or thought to have reason. It gets forgotten and ignored after a while in which case it is ok to cut it off.
Invest in very good testing, especially higher level integration / system testing.
Invest in a good dev/staging setup for your production environment, and also try to make rollouts and rollbacks automated and as painless as possible.
There will always be the need to change stuff, so get the pieces in place to make changes easier code, easier to test, easier to deploy, and easier to back the fuck up when you inevitably cause something to burn.
Of course the issue you run into is that the problem software is trying to solve changes over time and then the original abstraction which were correct become wrong over time. Then you should advocate refactoring to the new abstractions if possible.
Immediately. Never design for a future use case until it's a present use case and you're implementing it right then.
> How far should we go in making things generic ?
It depends on what it is. By not designing a thing to be generic up front, you have to figure out what n=2 looks like. Is that a function? A class? Copy a little bit of code? Then n=3. Once n=10, I feel like I have a good idea of the problem and how to make it generic, and it's rarely what I would have thought at the beginning.
Sometimes n never reaches 2. Then you've saved a lot of time. Also, you realize when you have to change things - maybe it's once a release, or very frequently. Things that are touched frequently probably need a refactor.
My rule of thumb is: never make tomorrow's possible problem today's complexity. If you design for future use cases, not only will it take you longer, but your code will inevitably be more complex than it needs to be, and therefore have more bugs at the very least due to that complexity.
Problems arise when people reach for the Cool Tools and start building Webscale things that can handle 100M operations every second. ...but the customer only needs the system to handle 4 users who type in everything crap by hand. But hey, it has a clustered database and Kafka and Kubernetes and looks REALLY good on your Resumé.
When the scale is determined, I personally like the mantra of "Make it work first, then make it pretty". First build an MVP (or an ugly Viable Product), that proves your plan actually works, then you can iterate over it to make the implementation cleaner or faster or have better UX.
If you get stuck making everything super-generic and able to handle all possible cases, you'll spend time bikeshedding and never get anything deployed. Just Repeat Yourself with wild abandon and copy/paste stuff all over until your Gizmo actually works. You can spend time figuring out which parts are actually possible to make generic later.
Ask yourself the following questions: * Are you an experienced developer on this particular problem domain? * Do you have good understanding of the future use cases, and the data structure when the new feature is added? * Will the new feature make business or technological sense?
Unless you have a concrete “yes” on all questions above, your design should go minimalistic. Without good understand of the new feature, the code base will likely need refactoring when the feature is implemented. Your effort of going the extra mile (which definitely should be applauded!) is better spent on making the code easier to read and refactor - better tests, better documentation, code review to propagate knowledge and catch bugs.
Another thing, extensibility is sometimes in conflict with readability! Here’s an hilarious example: https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...
In my experience following categories of people tend to fall into the trap easily (and it is really hard to convince them):
- Corporate people.
- Business school people.
The following can amplify it:
- Fund raising... you were successful in raising money so that means there is a market and people love your product ... well guess what? you are wrong.
If you are in such a situation, debating is not useful. Changing this mindset is only possible when you hit the wall at least once in your life. BUT ... you can accelerate the process of realizing it (before it is too late). Try to find a way to make people realize that all those bells and whistles and designs were not necessary cause nobody actually asked for and we are changing it anyway now.
* The nature of your "system" defines how much attention you should pay towards futureproofing. If it is an end-user app/feature only do what is required and nothing more. You are solving one instance of a (maybe) small class of problems. The future will dictate what needs to be added/modified/refactored when you reach that point and not before i.e. no predictions unless you know the domain well.
* If you are building an "architecture" component like OS/Framework/Library etc. then you need to pay attention to generality and extensibility and design with futureproofing in mind. Use standard best practices like data/api versioning, narrow module interfaces, information hiding etc.
* Always focus first on Readability and Maintainability to the exclusion of everything else. Only when you hit a wall with respect to aspects like Performance etc. do you go back and redo/refactor the code for the newly prioritized requirement.
Modify the code when the use cases change. In most of the times, open-closed principle is a trap.
However, over-engineering is not only a technical problem. It's more of a problem on project / people / finance, here are some examples but it would definitely not limited to these:
1. The margin is big and the team is big, so we need to keep these people busy.
2. The current stakeholder will cover the budget these iterations until we finish these features. after that, the maintenance costs maybe partially falls on us, or there's will be no budget at all. The less problems in the future, the better.
3. The current budget is for functionality A and someone pays it. And we plan to implement another feature B that is similar to these, and the budget comes from our own. Better make the solution generic so we can use it almost for free.
4. The list goes on...
Better fix those root problems first.
* A demo is a thing you can show. The first demo is usually
- the program will compile, run and print "hello world"
* Demos happen frequently. For a developer each day, for a team each week, etc.
Why does this help? A demo, actually showing something, is the only enforceable token that commits both part to the contract. In that way it is almost a currency. No stakeholder, whether manager, sales or developer can argue - either the demo was met, or not or was not clear.
This is much like sprints, test-driven, but a demo has the contract aspect.
Demo-driven reduces many of the causes/motives for over-engineering. * Is the over-engineering bit in the contract? Why not? * New function requires a new contract so no more last minute changes. (Been there done that). * Stakeholders gain experience designing demos. * Demos are adaptive. They provide tactile feedback.
My 2 cents
With modern higher-level languages and scalable cheap hardware, this motivation should have gone away and we should be writing code that is relatively easy to re-factor.
If I don't know that we will need this thing in the future, it doesn't go in. Simple. If 6 months down the line, we now need to add the new feature, I like to think that my code is largely maintainable enough to refactor it to add the feature.
The only exception I can think of is where something is designed to be extendible like e.g. Instagram filters where you might have 10 when you launch but you know that you will have more in the future so you write your code to allow additional filters to be plugged in relatively easily.
Things get over-engineered because too much time is allowed for the task.
The forcing function that's helped me is simply giving me and my teams small but meaningful time boxes to get a feature done. Sort of like "sprints", but with more teeth. E.g. An important feature is going to get announced in a newsletter ever 2 weeks. Sprints too often don't seem to focus on the shipping to customers part.
So we focus on shipping the smallest thing that could possibly work in an arbitrary time box. You know users need X. So you make X or X' or X'' - some version or whittled down version of X that'll relieve user pain. The time box does works wonders from keeping over complexifying from getting out of control.
Truth is most projects will be shelved before even they are complete.
You'll be taken off some projects because of financial or political BS.
So start with the assumption, there is limited time to deliver - now ask yourself - how can I maximize the effectiveness of that time? How can I do what truly matters without going into the rabbit hole of optimizations and using the best thing possible everywhere.
And once you've done that, you can always come back and improve the thing if the project is still around but most likely it will not be. Either you'd have switched the company or company would have switched you or the project has switched the company.
Architectural boundaries are hard to change later. Drawing the dependency graph and isolating nodes that can change independently is where the majority of our effort should go (imo). Even still, simple is less risky than complicated. Anytime there are more moving pieces than necessary, there is a risk of an unexpected requirement blowing a hole in a design. So identifying those dangling pieces and spending a lot of thought and energy in removing them is where I've found it to be rewarding to "overengineer".
In practice goals start to move after horizon/2. And a new document needs to be created to capture where the business is doubling down and where it is trimming.
Ex. if you have a problem to solve that goes like: Must create store with products that are blue only.
Then you'd break it up like:
Create store Filter products (Blue only) Create products in store
Then when you start coding you solve nothing more than what you put down as each task.
Ex. you don't do anything more than what's required. You might think you need to create a store and stores are businesses so you need to create some business wrapper etc. but nope. You don't need that until a client comes one day and requests it. Right now all you need is a store with blue products and that is all you're going to solve.
Often when you over-engineer something then you never need the extra abstractions you created.
You need to always ask yourself "What are the reasonable future use cases? How much will it cost to add infrastructure now to enable each of them, and how much will completing the feature in the future cost? How much will it cost to refactor my code later to add it if I don't make preparations for it now?" and only prepare now for things where not preparing would cost a lot more.
I'm finding the game of Go (Weiqi, Baduk) to be a great way to train in this skill, because it's all about seeing potential moves and deciding not to play them yet, and judging if a move should be played sooner or later and how much of a shape should be built now to enable it to be built later without wasting resources on building all of it now.
In my experience, keeping behavior static/hardcoded is the architectural equivalent of avoiding premature optimization.
Thus, we have every company it seems these days running around trying to implement microservices because “our architecture is like Netflix’s”. No, that LOB CRUD application isn’t like Netflix and you don’t need polyglot storage, Kubernetes, and microservices to capture input from a web form. However, one of the managers between the architect and the CEO (see: Peter Principle) is extremely impressed by this idea and can use it to advance their personal career.
My typical approach is to log all the commands to the file(s), and keep the system state in memory. When the server restarts, it simply replay most of the commands (skipping those intended to cause side effect)
This is like simplified event sourcing, or command sourcing.
I'm not relaying on any framework to build the backend, simple library handing rest and websocket API is enough.
I'll open source it soon but it's mostly as simple as it sounds, with some optimization to reduce disk space consumption after they becomes a problem in practice.
Here's one Kevlin Henney's lecture [1] that crystallizes it. It took me a long time to find it, so you are welcome.
Once you start naming things like this, adding future use cases becomes far less risky and thus you don't need to waste time on them.
Once you have a established a pattern of how the system is regularly extended, then you can use that to make predictions about the future. Keeping your codebase well-tested, small, and light will do far more to help you respond to change than guessing.
How do people “defend” that in big Corp?
Boat the technical roadmap and requirements with fantasy future features, so they can say “well that framework doesn’t support X, and our framework will support X”. It doesn’t matter whether X is useful in practice or not, since the managers don’t always have the power to make that call.
2: It's easier to refactor code with good automated tests, like unit tests, because you can push a button and know that it works.
3: Make sure that your startup order is well-defined. It's easier to debug a well-defined startup order than a startup order where everything is implicit.
4: Know the difference between a design pattern and a framework. Frameworks don't replace knowing how to use a design pattern correctly.
Don't do work before you need it, but also don't give up optionality unless you're getting commensurate benefits from doing so.
But there is a certain joy that comes when people ask “but what about XYZ” and you can respond “Yup, we thought of that!”
Granted, I work mostly on developer facing tools and services, which makes it much easier to anticipate the needs of my customer since I too am a developer.
And even with that caveat it’s not always a slam dunk... but it certainly is possible to anticipate requirements in a useful way.
The more time you spend thinking about the business and how what you're building will support it the better judgement you'll develop. You might not be able to articulate or argue it but you'll have an instinct on when an abstraction is going to be useful vs brittle.
If you build systems with these principles in mind, you can create systems that are extensible, without creating technical debt and YAGNIs.
It’s actually not that hard to refactor things when you do need a new feature... and who doesn’t love refactoring? It’s fun!
It’s also easier to refactor when your code is smaller and simpler because it’s only been coded to do the things you actually wanted it to do now.
If new requirements come in it's anyway lots of typing. You might as well spend the effort only when the full situation is known.
Accept the abstractions if: - the stack trace depth is never greater than double - the total code size is decreased - performance/memory hit is less than 10%
Thank you for bringing this up.
It is really hard to know what other people want or what they mean. If you can really understand what someone wants, perhaps you can avoid write 50% of a system you initially imagined.
Changing in the future costs something.
Designing for future changes up front makes sense if cost(future change) > cost(future proofing)
SAAS? Do virtually no future proofing.
IOT, do some.
Space probe? Do lots.
Also, if you haven't built quite a few relatively similar systems, don't do future proofing without talking to people who have.
2 - Write the simplest code to fulfill this specification;
3 - Improve on what you wrote so it will express better exactly the specification you have so far.
Other thought: good engineers can guess future cases more accurately, that's why they're good engineers.
A well-maintained, well-pruned test suite.
Writing tests forces you to clarify -- to yourself and to others -- precisely what this piece of software is and is not going to handle.
When I joined the attribute system had been build with a beautiful UI, and the backend was mostly working for managing attributes, but that was pretty much it. The first feature I was working on was showing products on the store, and for this the idea of attributes made sense. If you are selling a product in the "OLED TV" category you probably want a "Screen Size" attribute, and to be able to use it to compare against different products in that category. Through the platform we had maybe 500 product level attributes, with more being added all the time, so having them hard-coded wouldn't have been manageable. That was pretty much as well as it worked though.
Sellers needed to be able to manage their stock through the system, so on the warehouse entity there were attributes describing the number of products in stock, lead time, how often they restock, etc. The attributes didn't really have validations, but they had types which described what UI element should be displayed when they are entered. However all of the validations around that were at the whim of the front-end, and in some cases it would send what you would think should be a numerical type as a string (and then if you try to change it something would break as that expected it to be a string), so doing any kind of calculations or logic on the attributes was basically impossible. In the end I just added db-level fields to the stock entities, with validations in the backend to make sure these were as expected. The backend was a Rails app, so this took 10 minutes vs days trying to coerce the attribute system into doing what I needed.
As it was a Rails app we couldn't actually name the model of these Attribute, so had to give it another name, and whenever someone new joined (it was an early stage startup, so had high turnover) we had a 30 minute discussion explaining this. I never got the explanation of how the product owner expected logic to be attached to these attributes without a developer doing any work, but I'm sure they had an 'ingenious plan' for that too.
Needless to say, the startup burned through all it's funding without even launching, then managed to convince the investors to give them a little bit more, launched a half working product, and it turned out nobody wanted it.
Never make things generic apriori.
requirements -> tests -> specific code
if i reach the same code more than once, refactor and bother to generalize....
Once upon a time, I worked for a company who rents movies on DVD via kiosks. When I joined the team, pricing was hard coded everywhere as a one (1), because YAGNI. The code was not well factored, because iterations and velocity. The UI was poorly constructed via WinForms and the driver code for the custom robotics were housed inside of a black box with a Visual Basic 6 COM component fronting it. It was a TDD shop, and the tests had ossified the code base to the extent that even simple changes were slow and painful.
As always happens, the business, wanted more. Different price points! (OMG, you mean it won't always be a one (1)!!?) New products (OMG, you mean it won't always just be movies on DVD??!) And there were field operational challenges. The folks who stocked and maintained the machines sometimes had to wait for the hardware if it was performing certain kinds of maintenance tasks (customers too). Ideally, the machine would be able to switch between tasks at a hardware level "on the fly". Oh, and they wanted everything produced faster.
I managed to transform this mess. Technically, I would say it was (mostly) a success. Culturally and politically it was a nightmare. I suffered severe burnout afterwards. The lesson I learned is that doing things "right" often has an extremely high price to be paid, which is why it almost never happens.
On "over-engineering":
I find this trend fascinating, because I do not believe it to be an inherent issue. Rather, what has happened, is that "engineering" has moved ever closer to "the business", to the point of being embedded within it. What I mean by "embedding" here is structurally and culturally. [Aa]gile was the spark that started this madness.
Why does this matter? Engineering culture is distinct and there are lessons learned within we ought not ignore. However, when a group of engineers is subsumed into a business unit, their ability to operate as engineers with an engineering culture becomes vastly more difficult.
The primary lesson I feel we're losing in this madness is the distinction between capability enablement and the application of said abilities.
Think about hardware engineering: I do not necessarily know all of the ways you -- as the software engineer -- will apply the abilities I expose via my hardware. Look at the amazing things people have discovered about the Commodore 64 years after the hardware ceased production. Now, as Bob Ross would say, "Those are Happy Accidents." However, if I'm designing an IC, I need to think in terms of the abilities I expose as fundamental building blocks for the next layer up. Some of those abilities may never be used or rarely used, but it would be short sighted to not include them at all. I'm going to miss things, that's a given. My goal is to cover enough of the operational space of my component so it has a meaningful lifespan; not just one week. (N.B. This in no way implies I believe hardware engineers always produce good components. However, the mindset in play is the important take away.)
Obviously, the velocity of change of an IC is low because physics and economics. This leads everyone to assume that all software should be the opposite, but that's a flawed understanding. What happens today is we take C#, Java, Python, Ruby, etc. and start implementing business functionality at that level. To stretch my above hardware analogy, this is like we're taking a stock CPU/MCU off the shelf and writing the business functionality in assembly -- each and every time. Wait! What happened to all that stuff you learned in your CS undergrad!? Why not apply it?
The first thing to notice is that the "business requirements" are extremely volatile. Therefore, there must be a part of the system designed around the nature of that change delta. That part of the system will be at the highest, most abstract, level. Between, say the Java code, and that highest level, will be the "enablement layers" in service of that high velocity layer.
Next, notice how a hardware vendor doesn't care what you've built on top of their IC component? Your code, your problem. Those high-delta business requirements should be decoupled from software engineers. Give the business the tools they need to solve their own problems. This is going to be different for each business problem, but the pattern is always the same. The outcome of this design is that the Java/C#/whatever code now has a much lower change velocity and the requirements of it are future enablement in service of the tools and abstraction layer you've built for the business. Now they can have one week death march iterations all they want: changing colors, A/B testing, moving UI components around for no reason...whatever.
There are real-life examples of this pattern: Unity, Unreal Engine Blueprints, SAP, Salesforce. The point here isn't about the specifics of any one of these. Yes, a system like Blueprints has limits, but it's still impressive. We can argue that Unity is a crappy tool (poor implementation) but that doesn't invalidate the pattern. SAP suffers from age but the pattern is solid. The realization here is that the tool(s) for your business can be tailored and optimized for their specific use case.
Final thoughts
Never underestimate that the C3 project (where Extreme Programming was born) was written in Smalltalk, with a Gemstone database (persistent Smalltalk). One of the amazing traits of Smalltalk is that the entire environment itself is written in Smalltalk. Producing a system like I describe above, in Smalltalk, is so trivial one would not notice it. Unfortunately, most business applications are not written in environments nearly as flexible so the pattern is obscured. I've held the opinion for a long time that XP "worked" because of the skills of the individual team members and the unique development environment in use.
As I stated at the beginning, this path is fraught with heartache and dragons for human reasons.