Raising the question
In recent years, a serverless style of architecture design has become increasingly popular within AWS. When a service’s infrastructure becomes sufficiently complex, AWS CloudFormation tends to reveal all kinds of issues in practice: dependencies across multiple pipelines; updates of some stacks must depend on new or old versions of other stacks; complex dependencies can inevitably lead to conflicts; after some resources update successfully, subsequent steps fail and the system enters an intermediate state; the rollback process itself can also fail (UPDATE_ROLLBACK_FAILED). In the end, the entire system gets stuck in a state that is neither fully old nor fully new. Manual remediation becomes extremely complex and time-consuming, and frankly miserable. This kind of poor user experience easily leads to a certain “I’m thinking”: Could CloudFormation possibly provide full ACID semantics, like a database? If we could require these control plane operations to either all succeed or not happen at all, then the pain points of complex dependencies, failed updates, and failed rollbacks would seem to be greatly alleviated. After all, formally speaking, control plane APIs are only responsible for managing user resources; in essence, they are operating on a set of database objects and could be transactional.
The classic approach: 2PC
Following this line of thought, the first step is often to analogize to two-phase commit: since a stack update involves multiple resources, multiple services, and multiple control planes, it seems plausible to have each participant first enter a prepare state, confirming it can complete the intended change; once all participants are ready, commit in a unified manner; if some step fails midway, abort in a unified manner. From this perspective, what CloudFormation seems to lack is simply stronger distributed ACID transaction support. The first practical reaction to this idea is: prepare likely implies resource reservation—names must be taken first, capacity must be reserved first, IP addresses must be locked so they cannot be allocated to others. This affects other users’ requests and reduces overall throughput and fairness. But this concern is really an engineering trade-off, not a fundamental impossibility. Locks in databases also block concurrent access, and long transactions (such as full table scans) also block other write transactions. So merely pointing out that prepare locks resources is not enough to show that infrastructure systems cannot be transactional. It only introduces unrealistic cost, without yet touching the boundary of capability itself.
What kind of commitment do distributed transactions require?
Pushing the analysis further, the question shifts from “can we do prepare” to “what strength of guarantee must prepare provide”. If prepare is only a strict pre-check—only verifying that under current conditions an operation appears feasible—then this is usually not difficult, and it does have engineering value. Many resources can perform fairly rigorous validation. Whether names conflict, whether configuration is valid, whether permissions are sufficient, whether dependencies exist, whether capacity appears available at the current moment—these can all be checked before commit. But such a prepare is closer to a preflight check than to prepare in database transactions. It expresses “it seems executable right now”, and does not guarantee “the subsequent commit will definitely succeed”. True strong transactional semantics imply a commitment: once a participant enters prepare, as long as it eventually receives commit, it should be able to complete the operation, and not fail due to changes in external conditions. This is where we run head-first into the essential wall.
In infrastructure systems, the success of many operations does not depend only on updates to database objects. Take resources like a target group as an example: prepare can do many things—validate that targets exist, rules do not conflict, configuration is valid, current health status looks normal. These can be checked in advance, even quite strictly. But whether commit ultimately succeeds must still depend on another set of constraints of a completely different nature: whether the targets are still healthy at commit time, whether the underlying instances continue running, whether the network path is still available, whether connections and traffic in the data plane still have capacity, and whether external requests trigger new state changes during the process. These conditions share common traits—they continuously change; they are not fully controlled by the control plane; and they cannot truly be frozen. At this point, the prepare problem is no longer “can we check early”, but “which states that determine success can the control plane actually commit to”. If success depends on runtime state over some future period, then prepare can only validate the current state, and cannot promise the future state. This is precisely where transactional guarantees start to break down: they cannot fully control the physical world—you can never answer the question: if the machine goes down, how can you still commit an operation you previously promised would succeed?
From this angle, the reason CloudFormation cannot support true ACID is that key correctness conditions in infrastructure systems partly depend on runtime state that cannot be frozen and cannot be fully transactionalized to provide reliable global execution guarantees. Machines can fail at any time; health status can change outside the control plane; network and connection state fluctuates continuously; real traffic does not pause just because the system is executing prepare; and some external side effects, once they happen, cannot be fully undone. The key here is not that “operations can fail”, because database systems also run in a world with concurrency, contention, and crashes. The difference is that a database’s key state is mostly internal to the system, and can be locked, snapshotted, rolled back, and persisted; whereas in infrastructure systems, key state is partially externalized into the runtime environment, and not all conditions that correctness depends on live inside the control plane.
Thinking further, CloudFormation spans a large number of AWS services. Each service has an independent control plane and distinct resource types. This not only makes transaction coordination more complex from an engineering standpoint; more importantly, the semantics across services are not consistent: some purely resource-management services could, in theory, support transactional semantics, such as a stronger prepare or state reservation; while other services that depend on runtime behavior cannot provide similar guarantees. This brings a more fundamental limitation: transactional semantics cannot hold locally within the system. Once some step depends on runtime state that cannot be frozen and cannot provide a reliable commit guarantee, that uncertainty propagates along dependency relationships, preventing the overall operation from maintaining ACID. In other words, even if some control planes could be transactionalized, the non-transactional parts will “infect” the whole system, making it impossible for CloudFormation to provide a unified transactional semantics overall. Therefore, AWS’s multiple control planes not only increase coordination complexity; they also propagate this inconsistency in resource-management semantics, making it harder to establish ACID guarantees at a global level.
The abstraction boundary of IaC
We know that CloudFormation is a concrete and mature implementation of Infrastructure as Code in the AWS ecosystem. Since we are now confident that the correctness of infrastructure systems themselves depends on runtime state that cannot be frozen and cannot be fully transactionalized to provide reliable global execution guarantees, then this limitation is not merely a shortcoming of CloudFormation; it is inherent in the IaC abstraction itself. When we use IaC to describe and drive systems, the correctness guarantees it can provide are limited by the scope of state that the control plane can cover. Once system correctness depends on dynamic runtime behavior outside the control plane, this abstraction inevitably starts to fail.
IaC’s original design goal is actually quite clear. It excels at describing structural things: network topology, access control, static dependencies, resource provisioning, runtime capacity. These are structural states in the control-plane sense, well-suited to declarative configuration, version control, and automated execution. In the era of traditional virtual machines and containers, the boundary between application code and infrastructure was clear; IaC was very successful in these areas and did significantly improve infrastructure maintainability and traceability. In this context, looking back at modern serverless systems that are built almost entirely on IaC, we find that they gradually bring parts that used to belong to system behavior into infrastructure descriptions, thereby approaching or even exceeding this abstraction boundary: resource definitions, event-driven behavior, permission relationships, execution state machines, fault tolerance and retries are all expressed through infrastructure descriptions. But this does not naturally belong to the abstraction layer IaC is good at. When system behavior is indirectly encoded through infrastructure, IaC’s abstraction boundary starts to overlap with the system behavior boundary. At that point, IaC no longer merely describes the system’s static structure; it begins to carry the system’s behavior itself. The “stack update failure hell” we encounter cannot simply be attributed to unreliable engineering implementation; it is a systemic consequence of breaking through the abstraction boundary.
This is also why complex serverless systems often become hard to control in practice. Lambda, Step Functions, EventBridge, SNS/SQS, IAM, API Gateway—mix them together and, on the surface, it is still an infrastructure definition; in reality, it is already expressing a behavior model of a distributed system. Application deployment and infrastructure deployment almost overlap; application complexity maps directly into infrastructure complexity; and IaC tools are forced to take on a role they are not suited for, effectively acting as an implicit orchestration engine. The difficulty here is not only that there are too many stacks, dependencies are too complex, or update order is too hard to reason about; it is that the system’s behavior itself has been encoded into resource dependency relationships. Switching tools—for example, Terraform, CDK, or other frameworks—can change syntax and local abstractions, but usually cannot eliminate this part of the complexity imposed by the underlying resource model.
Kubernetes: modeling a dynamic world
After the discussion above, it becomes easier to understand Kubernetes’ design philosophy. Kubernetes also appears to give up stronger transactional semantics: it allows inconsistent intermediate states, allows operations to fail, and allows the system to deviate from the desired state for some period. But it adopts a modeling approach that is better suited to a dynamic world. It does not require each step to succeed in one shot, nor does it try to guarantee that a system that cannot fundamentally be frozen is perfectly correct at all times. It declares the desired state, continuously compares actual state with desired state through a reconciliation loop, and keeps correcting deviations. What truly matters here is not that it uses Raft or has strongly consistent control plane state storage, although those mechanisms are of course indispensable; the more critical idea is that the system accepts the reality that drift will continually occur, and treats continuously correcting drift as the primary design goal.
From this perspective, the difference between transaction and reconciliation is not merely a difference in implementation techniques, but a difference in how the problem is modeled. The latter acknowledges that in such systems, ACID transactional semantics are no longer an appropriate modeling approach. The transactional model seeks to prevent inconsistency, aiming for each operation to complete successfully or roll back within strict boundaries. The reconciliation model admits that inconsistency will keep happening, and designs the system as a self-correcting process. For a system where runtime state changes continuously, physical health and traffic cannot be frozen, and external side effects are difficult to fully retract, the latter is usually closer to reality. It seems weaker because it does not promise consistent correctness at all times; but it is more reliable in engineering terms because it treats failure and drift as normal by design.
Summary
Back to the initial question: why can’t CloudFormation support ACID like a database? A more concise answer is: what CloudFormation faces is not a purely resource-state storage problem. The correctness of what it manages partly depends on the continuously changing runtime world outside the control plane, and that portion of state cannot be frozen, cannot be fully snapshotted, and cannot be completely enclosed by transaction boundaries. As a result, failure states in complex deployments cannot be eliminated entirely.
Therefore, tools like IaC are very effective for structural problems, but naturally have boundaries for behavioral problems. The reason complex serverless systems so easily become hard to control is essentially that application logic, asynchronous behavior, and infrastructure resources are coupled too tightly, causing infrastructure definitions to carry system behavior itself. Under this premise, the focus of engineering practice is no longer merely to pursue stronger guarantees that apply will succeed; it will gradually shift toward constraining IaC’s responsibility boundary, peeling behavioral complexity out of the resource graph, and handling runtime uncertainty with methods better suited to dynamic systems.
From this angle, Infrastructure as Code is better understood as an engineering approach that is effective within specific problem boundaries. It is good at describing system structure and static topology, making control plane state clearer and more traceable, but it is hard for it alone to model and control dynamic system behavior. The layered architectures, platformization, and hybrid architectures that industry has gradually evolved are, in essence, separating structure from behavior, decoupling the control plane from runtime, so as to avoid compressing both into the same abstraction.
A one-sentence summary:
Transaction tries to guarantee determinism by “freezing time”, while infrastructure systems run in a world that is always changing; this is the root cause of the former’s failure.