Climbing the Ladder of Abstraction
Today's cloud infrastructure is fantastic. The richness and power of our cloud-native ecosystem around Kubernetes are easy to forget. It's hard to remember the world before containers and Kubernetes. How hard it was to get anything useful done. We have come a long way from the beginnings of cloud, multi-node, and multi-core development.
But this richness has come with a price. We are drowning in complexity, faced with too many products, decisions, and worries. What products to pick? How to use them individually? How to compose them into a single cohesive system? How to guarantee overall system correctness across product boundaries? How to provide observability of the system as a whole? How to evolve the system over time?
At the same time, users, competition, and new opportunities create new business requirements and a need to move and innovate faster while gracefully managing ever-increasing volumes of users and data—all at a faster pace. And we can't just move fast and break things; we have to do with predictability, repeatability, and reusability.
Serverless is very promising as a general developer experience (DX) for cloud and edge development— too important to be constrained to FaaS. Many cloud products, e.g., databases, message brokers, and caches, have embraced it and provide "serverless APIs." A step in the right direction but leaves developers with a complex integration project trying to compose them into a single functioning system.
Can we do better? Most definitely. Alfred North Whitehead famously said, "Civilization advances by extending the number of important operations which we can perform without thinking of them." Wisdom that very much applies to our industry.
We need to climb the ladder of abstraction. Reach a bit higher. But this requires that we, as developers, learn to let go of control and understand that we don't need every knob and that delegating means freeing oneself up to focus on more important things, building core business value. The recent trend of Platform Engineering that some companies are adopting as a way to manage complexity, shielding their developer teams from it through high-level APIs and abstractions.
As Timothy Keller said, "Freedom is not so much the absence of restrictions as finding the right ones, the liberating restrictions." We need to learn to embrace the constraints. But what are these liberating constraints, the liberating abstractions? There are three things we, as developers, can never delegate to a product or platform:
1. Data model: How do I model the business data and domain model? Including its structure, constraints, guarantees, storage, and query model.
2. API definition: How do I want the service to present itself to the outside world? How should it communicate and coordinate with other services? What data does it expose, and under what guarantees?
3. Business logic: The logic that drives the business value. How to act and operate on the data, store, query, transform, downsample, relay, mine intelligence, and trigger side-effects.
The first two, the data and API definitions, can be fully declaratively configured, leaving the developer with only the business logic—the function itself. The rest can and should, be fully delegated, managed, and automated by the underlying platform. In this model, all infrastructure product decisions (e.g., database, message broker, services mesh, cache, API gateway), data storage/query models, distributed systems patterns around orchestration, resilience, replication, sharding, caching, scheduling, communication, back-pressure, and much more, are inferred from the code, leaving the developer to focus solely on delivering business value.
This is not just wishful futuristic mumbo jumbo but a new DX for the cloud that is already here and available today through Kalix. Kalix is a developer PaaS for low-latency, scalable, resilient, event-driven cloud-native APIs and applications. It has a high-velocity, polyglot, ZeroOps DX with SDKs in the most popular languages (e.g., JavaScript, Typescript, Java, Scala, Go, Rust, C#, Python) and integration with standard cloud workflows. Try it out and let us know what you think.
