Systems ideas that sound good and hard to make work
Tech Thought of the Day
Martin Casado posted this on X.
And as a matter of personal principle, I will not post on X.
However, the list interested me because the question's framing implied that these “never” work, and because they “never” work, they shouldn’t be attempted.
A framing of the question makes it almost sound - “These are bad ideas.”
And yet, I look at the list and think, These Things HaveWork, and When They Work, billions of dollars and industries and lives change.
DSLs—well, the greatest DSLs of all time were Fortran and COBOL. Heck, C++ was initially imagined as a DSL for systems programming. My personal favorite is SQL, and my least favorite is HTML. Then there are less famous ones, like the DSL used with FLex and YACC.
Live Migrating Process State - It depends on how you define a process, and VMware built a fortune around things like VMotion.
Anomaly Detection: The security industry would like to speak with Mr. Casado.
Control Systems Responding to Load - I worked on two of those. We called them schedulers.
Multi-master writes - There is one great example of such a system. It’s called AD.
p2p cache sharing—We had the NetCache product at NetApp, which allowed different caches to share cache state. It was not exactly a very successful product, but it did deliver business value.
Hybrid Parallelism—The systems I have worked on rely on some flavor of hybrid parallelism. Thread-based parallelism is a great idea within a single socket, but messages work better across sockets and machines. I even worked on a system that combined threads and messages, delivering stunning performance at very low CPU overhead.
Being clever vs over-provisioning: Depends on who is doing the cleverness. The entire basis of thin provisioning storage is that if you do the cleverness at the right layer, you don’t have to overprovision. And if you support live migration or storage migration, you can adjust on demand. The entire cloud infrastructure is predicated on the over-provisioning being done at a different layer.
My reaction to this list is that these are industry-defining ideas. And that implementing them correctly is hard. And if you don’t have the time, energy, and talent to implement them, then it’s easy to do them poorly.
So, perhaps a different way to frame these ideas is as system ideas that sound good and hard to make work.