What’s the difference between smoke, sanity, and regression testing, and when do you run each in a CI pipeline?
All three answer different questions at different points in the pipeline. Smoke testing asks “did the build even come up correctly,” a fast, shallow pass over the most critical paths (can you log in, does the homepage load, does the checkout button exist), run immediately after every deploy before anyone bothers testing anything deeper. Sanity testing asks “did this specific bug fix or small change work, without obviously breaking its immediate surroundings,” a narrow, focused check right after a fix, not a full pass. Regression testing asks “did this change break anything elsewhere in the system that used to work,” and is the broadest and slowest of the three, re-running a large suite covering previously working functionality.
Where each fits in a pipeline
Smoke tests run on every deploy to every environment, including production, and should complete in under a couple of minutes, if they fail, you roll back immediately without wasting time on anything else. Sanity tests run after a targeted bug fix, before it merges, focused only on the affected area. Full regression suites are the most expensive and usually run on a schedule or before a release cut, not on every single commit, because running thousands of end-to-end tests on every push would make the pipeline unusably slow.
Best practice
Structure your test suite as a pyramid: many fast unit tests, fewer integration tests, and a small number of true end-to-end regression tests, since end-to-end tests are slow and flaky compared to unit tests. Automate smoke tests fully since they gate every deploy; regression suites are also worth automating, but accept that a full regression pass is expensive and shouldn’t block every single commit.
Edge case interviewers probe for
Ask what you’d do if the automated regression suite takes 3 hours and the team wants to deploy multiple times a day. The answer involves splitting regression into a smaller “critical path” subset that runs on every merge, with the full suite running nightly or before a release, rather than either blocking every deploy for 3 hours or skipping regression entirely.
Common mistake
Treating “regression testing” and “re-running all my tests” as the same thing without distinguishing scope. A true regression suite specifically targets areas that could plausibly be affected by the change, not blindly rerunning everything every time regardless of relevance, which wastes time and obscures which failures actually matter.
What the interviewer is checking
Whether you think about testing strategically, matching the right test type’s cost and depth to the right moment in the pipeline, instead of treating “testing” as one undifferentiated activity.
Imagine you just got a new car back from the shop. Smoke testing is turning the key and checking the engine starts, the lights work, and the brakes respond, thirty seconds, just confirming the car isn’t obviously broken before you even leave the parking lot.
Sanity testing is when the mechanic specifically fixed your air conditioning, so you just check the AC works and the dashboard near it still looks fine. You’re not testing the whole car again, just the one thing that got touched and its immediate neighbors.
Regression testing is taking the car on the full test track you always use: hills, sharp turns, highway speed, parallel parking, to make sure that fixing the AC didn’t somehow mess up the brakes or the steering, things that should have nothing to do with the AC but you’re checking anyway because cars (and code) sometimes surprise you with unrelated side effects.
Why interviewers ask this
These terms get used loosely on real teams. This checks whether you actually understand the distinct purpose and cost of each, not just the vocabulary.
What a strong answer signals
That you think about testing cost versus coverage tradeoffs, and can design a pipeline that catches problems fast without making every deploy painfully slow.
Common follow-ups
- What’s the testing pyramid and why does shape matter?
- How do you handle flaky end-to-end tests in a regression suite?
- What would you automate first if starting from zero test coverage?
Advanced variation
“How would you decide which regression tests to run on every commit versus nightly,” which expects a risk-based approach: run tests covering recently-changed or historically-fragile areas on every commit, the rest on a schedule.
An e-commerce platform’s CI pipeline ran a 5-minute smoke suite (login, search, add to cart, checkout page loads) on every deploy to any environment, blocking the deploy entirely on failure. A separate, larger regression suite covering payment edge cases, discount code combinations, and inventory edge cases ran nightly against staging and before every release tag, taking around 90 minutes. When a discount code bug was fixed, the developer ran just the discount-related sanity subset (about 15 tests) locally before opening the pull request, rather than waiting for the full nightly regression run to confirm the immediate fix worked.
name: test-pipeline
on: [push, pull_request]
jobs:
smoke:
# Fast, blocks every deploy. Fails fast on obvious breakage.
runs-on: ubuntu-latest
steps:
- run: npm run test:smoke # ~2 min: login, homepage, checkout render
regression-critical-path:
# Runs on every push; a targeted subset, not the full suite.
needs: smoke
runs-on: ubuntu-latest
steps:
- run: npm run test:regression -- --tag=critical-path
regression-full:
# Full suite, nightly only, not on every commit.
if: github.event.schedule == '0 2 * * *'
runs-on: ubuntu-latest
steps:
- run: npm run test:regression -- --tag=full- 1Smoke testing is a fast, shallow check that the build isn’t obviously broken; it gates every deploy.
- 2Sanity testing is a narrow check that a specific fix worked without breaking its immediate area.
- 3Regression testing is the broadest, slowest check that a change didn’t break previously working functionality elsewhere.
- 4Match test depth to pipeline stage: smoke on every deploy, sanity after fixes, full regression on a schedule or before release.
- 5Split regression into a fast critical-path subset for every commit and a full suite for nightly or release runs.