./blog/eks-rds-zero-downtime-scaling
RDS storage growth doesn't have to mean a maintenance window — but "zero downtime" and "zero risk" aren't the same claim, and I learned the difference the hard way.
Scaling RDS storage online with zero downtime
And what I got wrong the first time
A service backed by RDS for PostgreSQL was quietly eating through its allocated storage faster than anyone expected, and the "fix" was supposed to be boring: bump the storage, let RDS do it online, move on. It mostly was boring — right up until autoscaling tried to fire a second time inside its own cooldown window and couldn't. Here's how online storage modification actually behaves under the hood, and the two things I'd tell myself to check before doing this in production again.
Modifying RDS storage — size, IOPS, or throughput on gp3 — applies online with no reboot and essentially no downtime in the traditional sense. But the instance enters a
storage-optimization state afterward with measurably elevated latency, autoscaling
is frozen for at least 6 hours (or until optimization finishes, whichever is longer) and capped
at four modifications per rolling 24 hours, and storage only ever scales up. Set your
thresholds with that cooldown in mind, or a genuine traffic spike can land you in
storage-full with no lever left to pull.
how-it-actually-works --gp3-online-modify
Since RDS moved General Purpose storage to gp3, storage size, provisioned IOPS, and throughput are three independent dials — you're no longer forced to over-provision size just to buy more IOPS the way gp2 required. Modifying any of them is an online operation: no reboot, no failover, no maintenance window required. Amazon's own storage autoscaling feature works the same way — turning it on for an existing instance takes effect immediately, with no restart and no downtime.
What "online" doesn't mean is "invisible." Once a modification starts, the instance sits in
a storage-optimization status while the underlying volume rebalances, and AWS
is explicit that you should expect elevated latency — single-digit milliseconds — during
that window. On a service with a tight p99 budget, that's very noticeable even though
nothing "went down."
what-went-wrong --the-first-time
Storage autoscaling was on, with allocated storage at 500 GiB and a max threshold set at
2 TiB — comfortable-looking headroom on paper. A backfill job that weekend wrote faster
than expected, autoscaling correctly kicked in and grew the volume once, and everything
looked fine. Then the backfill kept going, crossed the free-space threshold a second time
a couple of hours later, and autoscaling didn't fire. It couldn't: RDS won't scale storage
again until at least 6 hours have passed since the last modification, or until
storage-optimization finishes, whichever is later — and separately, it caps you
at four storage modifications in any rolling 24-hour window. The instance sat at
storage-full for the rest of that window while the backfill job errored out
trying to write.
Two details made this worse than it needed to be. First, each autoscaling step only grows storage by the greatest of 10 GiB, 10% of current allocated storage, or RDS's own predicted 7-hour growth — which is conservative by design, but means a genuinely bursty write pattern can outrun the increment size even when the ceiling looks generous. Second, storage autoscaling only ever scales up. There's no shrink-back once the volume grows, so every panic-driven jump becomes a permanent line item on the bill — which is its own argument for getting the threshold right up front instead of relying on autoscaling as the whole plan.
runbook --what-id-check-next-time
Size the threshold for the cooldown, not the average
Set the free-space threshold high enough to absorb your worst realistic write burst over a 6+ hour window, not your average daily growth rate.
Alarm on FreeStorageSpace directly
Don't rely on autoscaling as your only signal. A CloudWatch alarm on low free space, separate from the autoscaling trigger, buys a human a chance to intervene before storage-full.
Know your modification budget
Four storage changes per rolling 24 hours, across manual and automatic modifications combined. A manual bump during an incident can eat the budget autoscaling needed.
Expect the latency, don't debug it live
storage-optimization causes real, expected single-digit-millisecond latency increases. Know what that looks like on your dashboards before it happens at 2am.
Tune gp3 IOPS/throughput independently
If the real problem is I/O, not capacity, raising provisioned IOPS or throughput on gp3 directly is often the cheaper, more targeted fix than growing size to chase IOPS the way gp2 required.
Remember scaling only goes up
There's no autoscaling shrink path. Treat every scale-up as permanent, and size deliberately rather than letting bursts set your bill.