What happens when Terraform’s state drifts from real infrastructure, and how do you design around it?
Terraform keeps a state file that maps each resource block in your config to a real resource ID in the cloud. Drift happens when reality changes without Terraform knowing, someone edits a firewall rule directly in the Azure Portal, or an autoscaler changes an instance count outside Terraform’s control. On the next terraform plan, Terraform compares its state against a fresh read of real infrastructure and shows a diff you didn’t expect, or worse, silently plans to revert a manual emergency fix someone made during an incident.
How to design around it
Treat the cloud console as read-only in practice, enforce it with IAM policies that block manual writes to Terraform-managed resource groups except for a small break-glass role. Run terraform plan on a schedule (not just before applies) purely to detect drift and alert on it, even if you don’t auto-apply. For resources that autoscalers or other systems legitimately need to modify (like instance counts), use ignore_changes in a lifecycle block so Terraform stops fighting a system that’s supposed to own that one field.
Best practice
Keep state in a remote backend with locking (Azure Storage with blob leasing, or Terraform Cloud), never local state files, so two engineers running apply at once can’t corrupt it. Split state into multiple smaller state files per environment or service boundary rather than one giant state file for the whole account, so a mistake or a slow plan in one area doesn’t block or risk everything else.
Edge case interviewers probe for
Ask what happens if someone deletes a resource manually. Terraform’s next plan will want to recreate it, since state still says it should exist. If that’s actually the intended change, you fix it by removing the resource from state (terraform state rm) or updating the config to match, not by letting Terraform “helpfully” recreate something someone deliberately removed.
Common mistake
Responding to drift by running terraform apply immediately to “fix” it without first running plan and actually reading the diff. If the manual change was a legitimate incident fix, blindly applying can revert exactly the fix that resolved the outage.
What the interviewer is checking
Whether you understand infrastructure as code as an ongoing discipline (drift detection, state hygiene, access control) rather than a one-time “write some HCL and apply it” task.
Imagine Terraform as a very detailed shopping list plus a photo of what your kitchen should look like. Drift is what happens when someone raids the fridge and rearranges things without updating the photo. Next time you check the photo against the actual fridge, they don’t match, and if you’re not careful, “fixing” it might mean throwing away food someone put there for a good reason, like leftovers from last night’s dinner that saved you from cooking again.
The fix isn’t to never let anyone touch the fridge. It’s to put a note on it saying “please don’t rearrange without updating the photo,” check the fridge against the photo regularly so surprises get caught early, and for the shelf where the kids always leave snacks (something that’s supposed to change on its own), just stop including that shelf in the photo comparison entirely.
Why interviewers ask this
Anyone can write Terraform that works on day one. This checks whether you’ve actually operated it long enough to hit the real failure mode: reality and state disagreeing.
What a strong answer signals
That you think about IaC as a living system needing guardrails (locking, access control, drift detection), not a script you run once.
Common follow-ups
- How would you split a monolithic state file safely?
- What does state locking actually prevent?
- How do you handle secrets that shouldn’t live in state?
Advanced variation
“How would you import an existing, manually-created resource into Terraform management without recreating it,” which expects knowledge of terraform import and writing matching config carefully before running any plan.
A platform team managing shared Azure infrastructure hit repeated incidents where an on-call engineer would manually bump a VM scale set’s instance count during a traffic spike, then the next scheduled Terraform apply would silently scale it back down to the config’s static value, undoing the fix hours later. Adding lifecycle { ignore_changes = [instance_count] } to that resource, and moving scaling decisions to an autoscaler policy instead of a static Terraform value, stopped Terraform from fighting the autoscaler while keeping everything else in that resource under normal IaC management.
resource "azurerm_linux_virtual_machine_scale_set" "app" {
name = "app-vmss"
instances = 3 # static baseline; autoscaler owns real-time count
lifecycle {
# Terraform stops trying to "correct" a value the autoscaler
# legitimately changes outside of Terraform's control.
ignore_changes = [instances]
}
}
resource "azurerm_monitor_autoscale_setting" "app" {
# The autoscaler, not a hardcoded number, owns scaling decisions.
target_resource_id = azurerm_linux_virtual_machine_scale_set.app.id
profile {
name = "default"
capacity {
minimum = "2"
maximum = "10"
default = "3"
}
}
}- 1Drift happens when real infrastructure changes outside Terraform’s knowledge, causing state and reality to disagree.
- 2Run scheduled plans purely to detect drift early, even without auto-applying.
- 3Use lifecycle ignore_changes for fields legitimately owned by another system, like an autoscaler.
- 4Always read the plan diff before applying after drift is detected; blind applies can revert legitimate manual fixes.
- 5Use a locked remote backend and split state by boundary so mistakes stay contained.