There are two forces that are driving the abstraction of infrastructure:
- Before, on-prem applications were not architected so that you could figure out exactly what cloud infrastructure the application needed. Now, greenfield applications are cloud-native, built on cloud APIs from scratch. While there's still the need for manual translation (what infrastructure does this container need?), it's becoming easier and easier to define it or guess.
- Cloud APIs are becoming more mature. While there's no single abstraction across all cloud providers, there are enough companies putting serious work into maintaining each separate set of libraries (e.g., HashiCorp/Terraform, Pulumi, Minio, etc.).
Developers would love to have a zero-added-effort abstraction for infrastructure – write your application and automatically figure it out. But there's no silver bullet (yet?).
A few different ways to "magically" abstract the infrastructure, each operating at a different level in the stack:
- The Decorator Approach – Decorate your functions with an annotation that configures the cloud resources that the function or program needs. This is most popular in Python (where you often have to burst into the cloud for some workflow that might require special hardware). This is how Modal works. It's also how the now archived Metaparticle worked for Kubernetes (in Java, C#, JS, and Python). I even wrote a version of this for machine learning pipelines with kubeflow/fairing.
- The PaaS/Framework Approach – Output build artifacts according to a specific API that can be translated to cloud resources. In theory, this is how NextJS's v3 build output API should work (although Vercel is the only service that plausibly consumes it). Other PaaS approaches require you to follow a specific format in how you package your code (e.g., containers) or implement a certain entrypoint.
- The Library Approach – Abstract the cloud resource at the library level. Instead of initializing a specific client to call into the infrastructure, you call a generic API. The Go Cloud CDK is a good example of this. It doesn't actually instantiate or manage the infrastructure, but it lets you, in theory, switch it out without noticing much difference. In practice, there are always implementation-specific features and behaviors, even for the most basic APIs (like S3 or queues).
- The CDK/SDK Approach – Manage your infrastructure with code in TypeScript or whatever language you write your application in. This is the complement to abstracting infrastructure in the library approach, as you might be able to share some code or at least project scaffolding with the infrastructure as code. I think Dagger is onto something here by noticing the deep connection between build, deploy, and infrastructure – your infrastructure-as-code needs to know about your builds, so how do you piece them together?
- The Language Approach – whether it's a DSL or a general-purpose programming language, treating infrastructure and code exactly the same can be useful. You can think of this as something that combines both the Library approach and the CDK approach in one. However, there's a tough cold start problem with DSLs. Who will be the end-user of infrastructure-as-code? DevOps? Platform Engineers? Application developers?