Spot Instance Interrupted at 2am: How to Save Your Checkpoint

Poll http://169.254.169.254/latest/meta-data/spot/termination-time, trap SIGTERM, and write checkpoints to S3 or EFS every 5–15 minutes. The two-minute notice only works if the fast checkpoint path, the IAM role and the launch template are already in place before the job starts.

The metadata endpoint returns a UTC timestamp that sits almost exactly 120 seconds ahead of the kill. AWS has published it that way since 2021, and the EventBridge variant fires even earlier, at roughly 30 seconds before termination. Treat the timestamp as a deadline for a checkpoint write you have already rehearsed, not as a signal to start designing one.

What catches people is arithmetic they never ran. If a checkpoint takes 45 seconds to reach S3, a 15-minute interval leaves you at risk twice over: a slow write plus a queue drain can eat the whole window. At that write speed, 10 minutes is the honest ceiling.

Instance store and attached EBS volumes vanish on termination by default, so a checkpoint that lives on the instance was never a checkpoint. And the notice only arrives if the instance can reach IMDS — block port 169.254.169.254 in a hardened security group or run with IMDS disabled, and your handler stays silent until the kernel kills the process.

• Two-minute warning, not a grace period: the Spot interruption notice gives exactly 120 seconds, delivered through instance metadata and mirrored by a roughly 30-second EventBridge event.
• Match interval to write speed: a 45-second checkpoint write to S3 supports a maximum safe interval of about 10 minutes, against the 15-minute ceiling most teams adopt.
• Durable storage only: use Amazon S3 or Amazon EFS, because instance store and EBS volumes attached to a terminated Spot Instance are destroyed by default.
• Pre-wire the permissions: put an IAM role with write access to the checkpoint bucket in the launch template so the SIGTERM handler can flush without a credential lookup.
• CloudWatch gives you less time: the EventBridge notice arrives at roughly 30 seconds, so any recovery path that depends on it must be faster than the metadata-driven one.

What exactly happens during a spot instance interruption?

AWS does not kill a spot instance without saying so first. When capacity is reclaimed, the Instance Metadata Service starts returning an RFC 1123 timestamp at http://169.254.169.254/latest/meta-data/spot/termination-time, and only after that does the operating system deliver SIGTERM to PID 1 and, on most AMIs, to running processes. The gap between the two is the whole game: you get 2 minutes from the earliest of those signals, and 120 seconds is also the hard SIGTERM grace period. If your process is still alive at the end of it, AWS force-terminates the instance. No SIGKILL, no flush, no last write. The EBS volumes go away with it unless you set DeleteOnTermination=false.

Polling the metadata endpoint is the only signal available on every instance type and every AMI, which is why it is the one to build on. Hit 169.254.169.254 every 5 seconds from a sidecar or a daemon thread; IMDSv2 requires a PUT for a session token first, so budget for two requests per poll, not one. On the first non-404 response you have up to 120 seconds to stop accepting new work, finish the in-flight step, and write state. A 10 GB checkpoint to Amazon S3 over a 1 Gbps link takes roughly 45 seconds, which fits — but only if the write starts immediately and the job is not midway through a batch that cannot be abandoned.

EventBridge gives you a second path, and it is easier to wire into automation but strictly worse on timing. The EC2 Spot Instance Interruption Warning event lands in your default event bus with about a 30-second delay from the moment AWS decides to reclaim. That is 30 seconds of your 120 already spent before a Lambda function even starts. Use EventBridge for bookkeeping — marking the node drained, notifying a queue, updating a job registry — and use IMDS for the actual shutdown sequence. If you are on Kubernetes with the AWS node termination handler, or on AWS Batch, this plumbing already exists; verify it is running rather than assuming it is. The handler that ships with a managed node group is not the same as the one you get from the EKS add-on, and a missing DaemonSet is a silent failure until the first 2am interruption.

The part that actually bites

Two things go wrong in practice. The first is a job that checks the metadata endpoint but blocks on a step that cannot be interrupted — a 40-minute hyperparameter sweep, a multi-stage Spark shuffle, a database transaction that must commit or roll back. SIGTERM arrives, your handler sets a flag, and the flag is never read because the worker thread is inside a library that does not poll it. The fix is to make steps short and idempotent, not to make the handler cleverer. The second is an IAM role that lets the instance read the metadata endpoint but not write to the S3 bucket holding checkpoints, which you discover exactly once. Test both with AWS Fault Injection Simulator, which can send a real spot interruption to a running instance on demand. Doing that on a Tuesday afternoon beats discovering it at 2am, which is the entire point of preparing before launch.

How to detect the 2-minute termination notice in code

This procedure applies to any long-running process on EC2 Spot Instances where the work between checkpoints is worth more than the seconds you spend polling. It needs an IAM role attached to the instance that can read the metadata service, a writable destination outside the instance (Amazon S3 or Amazon EFS), and a checkpoint routine that can be called from a signal handler rather than only from the training loop. Do this before launch. The whole setup is maybe twenty minutes of work, and AWS Fault Injection Simulator will test it for you for a couple of dollars.

1. Poll the termination-time endpoint from a sidecar, not the training loop. A shell loop or tiny Python thread hits http://169.254.169.254/latest/meta-data/spot/termination-time every 5 seconds with a 1-second timeout. The endpoint returns an ISO 8601 timestamp when a notice is active and HTTP 404 when it is not. Five seconds costs nothing measurable and leaves plenty of slack inside the 120-second SIGTERM grace period. Run this in its own thread or separate process so a busy GPU loop cannot starve it.
2. Use IMDSv2, which since 2024 is the default on new AWS accounts. That means a PUT to /latest/api/token with X-aws-ec2-metadata-token-ttl-seconds: 21600, then a GET with the returned token in the X-aws-ec2-metadata-token header. One-liner curl against the v1 endpoint will return 401 and you will sit there wondering why your detector never fires.
3. On a 200 response, immediately call your checkpoint function and then trigger your own graceful shutdown. Do not wait for SIGTERM. You have the 2 minutes from the notice, and using the first 10 seconds of it to write state means the remaining time is genuinely yours.
4. Handle SIGTERM explicitly, because you will not always get the metadata notice. The two-minute warning is delivered best-effort, and you can also be interrupted by a manual stop, a Spot rebalance recommendation, or an ASG scale-in. Python needs signal.signal(signal.SIGTERM, handler); Java needs Runtime.getRuntime().addShutdownHook(...) or a JVM signal handler; Go needs a signal.Notify channel. The handler does one thing: flush the current checkpoint synchronously and exit with a non-zero code.
5. Kubernetes and AWS Batch sit one layer above this. On EKS, a Spot interruption arrives as a node termination event and the pod gets SIGTERM after the kubelet drains, still within the 120-second window. AWS Batch since 2024 passes the spot interruption notice to the job as an environment variable, which is easier than polling. Either way, the handler in your application is the thing that saves you.
6. EventBridge is the other mechanism, and it is worth having for orchestration rather than for the instance itself. CloudWatch Events and EventBridge emit a spot interruption warning 30 seconds before the two-minute metadata notice appears. That is more than 30 seconds earlier warning than the metadata endpoint, but it is delivered to a Lambda or SQS queue, not to the dying instance, so use it to update your job tracker, cancel downstream work, or drain a queue. Do not design your checkpointing around it: the instance still needs to save itself.
7. Test the whole path with AWS Fault Injection Simulator. A aws:ec2:send-spot-instance-interruptions experiment gives you a real interruption message and a real SIGTERM with the real grace period. Run it against a canary instance before you point it at a week-long training job. Fifteen minutes of setup, no production impact.
8. Verify the exit path. After the process dies, confirm the checkpoint landed in S3 with a head-object call and that your launcher or orchestration layer sees the non-zero exit code and resubmits. If the resubmission is manual, you have not finished the job.

The failure mode. The detection code fires, the handler runs, the process starts writing a 10 GB checkpoint to S3, and the instance is terminated 47 seconds later mid-write. This is the case most articles get wrong. A 10 GB checkpoint over a 1 Gbps link takes roughly 45 seconds on a good day, and Spot network bandwidth is not guaranteed on all instance families. If your checkpoint takes 90 seconds, no polling interval and no signal handler can save you, because the body of work you are trying to save does not fit in the time you have. Make the checkpoint faster before you make the detector smarter.

The two levers that actually work: shard the checkpoint across multiple S3 multipart uploads in parallel, which on a 10 Gbps instance turns that 45 seconds into under 10; or checkpoint less data, by saving optimizer state in fp16 or by storing only the delta from the last full checkpoint. A 5–15 minute checkpoint interval is the usual recommendation, but the number that matters is the ratio of write time to interval. If your write is 45 seconds and your interval is 5 minutes, you spend 15% of your compute on checkpointing and you can always finish a write inside the window. If your write is 90 seconds and your interval is 5 minutes, you spend 30% and you are one bad network day away from losing everything since the previous interval. Fix the write first. At $0.004 per GB-month for S3 Standard in us-east-1, a 10 GB checkpoint costs about four cents a month to keep, which is not the constraint.

Where should you write checkpoints? A comparison of storage options

Checkpoint storage has one hard requirement that ordinary training infrastructure does not: the bytes must be readable by a completely different machine, in a different availability zone, after the current instance is gone. Instance store and EBS both fail that test. EBS volumes can survive termination if you set DeleteOnTermination=false, but the replacement instance must be in the same AZ and must wait for the volume to detach and reattach, which routinely takes longer than the 120 seconds you have.

Latency is the second constraint, and it is where people misjudge things. A 10 GB checkpoint written from a g5.xlarge over a 1 Gbps path lands in S3 in roughly 45 seconds — fine inside the 2-minute SIGTERM window, but only if the write starts immediately and the process is not still flushing gradients. If your checkpoint is 80 GB and your write path is slow, the window closes before the upload finishes and you are left with a truncated shard. Pick storage whose write throughput you have actually measured at your checkpoint size, not at 1 MB test files.

S3 wins for most batch work because $0.004 per GB-month is roughly 75× cheaper than EFS Standard, and because a job that writes one 10 GB checkpoint every 10 minutes does not benefit from POSIX semantics. The case that flips is many small writes: if a distributed job writes 5,000 files of 2 MB each per checkpoint, per-request latency dominates and S3 PUT overhead pushes a checkpoint past the 2-minute budget, while EFS General Purpose absorbs the same load at 5–20 ms per file. FSx for Lustre only pays off above roughly 50 GB per checkpoint, where its per-TiB throughput beats what a single S3 multipart upload can sustain from one node.

How often should you checkpoint to survive a 2-minute warning?

The interval is arithmetic, not folklore. AWS gives you 120 seconds of SIGTERM grace after the interruption notice appears at 169.254.169.254/latest/meta-data/spot/termination-time, and the termination can arrive at any point inside that window. You want at least two checkpoint boundaries inside the window: one for the work that is about to be lost when the signal lands, and one to guarantee the job can restart from a state that was written after the most recent training step you care about. So the formula is (120 − checkpoint_time) / 2 seconds as your hard ceiling, then you halve it again for comfort.

• Measure your checkpoint_time first, from the job, not from a laptop. Time the write from the first byte flushed to the last byte acknowledged on the target store. A 10 GB PyTorch state dict going to Amazon S3 over a 1 Gbps link lands around 45 seconds in practice; the same file to Amazon EFS in the same AZ is usually under 10 seconds. If you have never measured it, you do not have a formula, you have a guess.
• Apply the formula and round down. At 45 seconds per checkpoint the ceiling is (120 − 45) / 2 = 37.5 seconds, which is obviously unusable for training. That is the point: the formula tells you a 45-second write cannot survive a two-minute window on its own, so the interval has to be driven by acceptable lost work, and the write has to be made shorter. For an 8-second write the ceiling is 56 seconds, still tight, so you fall back to job economics.
• Default to 5–15 minutes for typical deep learning jobs, and set it from your step time. If a step takes 400 ms, a 10-minute interval costs you at most 1,500 steps of rework, which on a 3-hour job is under 6% of the run. The 5-minute end suits jobs with expensive steps or non-deterministic data loaders that are painful to replay; 15 minutes suits cheap, idempotent steps where the write itself is a meaningful tax on throughput.
• Budget the checkpoint tax explicitly. A 45-second write every 10 minutes is 7.5% of wall clock spent writing. Reduce the interval to 5 minutes and that becomes 15%. Spot's discount — up to 90% versus On-Demand — still makes that a good trade, but only if your job actually restarts from the last checkpoint instead of the beginning.
• For models above roughly 20 GB, stop writing whole state dicts. Switch to sharded writes (one file per rank, as PyTorch's distributed checkpoint API and DeepSpeed both support) or incremental deltas that only persist changed parameter groups and optimizer state. Sharding turns a single 45-second serialized write into N parallel writes of 45/N seconds, which moves the formula's ceiling from seconds to minutes.
• Use the shorter 30-second signal as a backstop, not as your interval driver. EventBridge spot interruption warnings arrive up to 30 seconds earlier than the IMDS notice in many configurations, and CloudWatch metrics give you breadth across a fleet. Neither is guaranteed to beat the IMDS path, so do not size your interval assuming you will get 150 seconds.
• Write atomically, or your interval is a lie. A checkpoint that is half-written when SIGTERM fires is a corrupt checkpoint. Write to a temporary key, fsync, then rename or copy into the canonical path, and have the loader validate a checksum or a completion marker before it trusts the file. This is where most teams lose a night, not in the interval math.

The step people get wrong is treating the checkpoint write as free and optimizing the interval alone. A 45-second write with a 15-minute interval looks fine on paper and still loses 45 seconds of every 15 minutes to I/O that produces nothing, and — worse — it cannot finish inside a two-minute window if the job happens to start a checkpoint at second 100. Shorten the write before you shorten the interval. Shard it, move it to an EFS mount in the same AZ for the hot copy and life-cycle it to S3, or checkpoint only the layers that changed since the last write. The interval is the easy dial; the write path is what actually decides whether the 2am page is a footnote or a rebuild.

What to configure before the first kill: IAM roles, launch templates, and spot fleet

Everything a Spot instance needs to die gracefully has to exist before the instance boots. If you attach permissions or shutdown behaviour after the fact, you are rebuilding the same instance by hand every time it gets reclaimed, which is exactly the manual recovery you were trying to avoid. The settings below are the ones that turn a 2-minute notice into a routine event.

• IAM role with scoped S3 permissions. The instance profile needs s3:PutObject and s3:GetObject on the checkpoint bucket, plus s3:ListBucket on the bucket itself. Scope the resource to arn:aws:s3:::your-checkpoint-bucket/jobs/${aws:PrincipalTag/job-id}/* rather than * — a wildcard role on a shared account is how one bad job overwrites another team's checkpoints.
• Launch template with the profile attached. Create the role, attach it to an instance profile, and reference that profile in the launch template. User data installs dependencies: CUDA drivers, the correct torch or tensorflow wheel, and your code pulled from a known commit SHA. Never pull main in user data, because an instance launched at 03:00 gets whatever was pushed at 02:55.
• Instance initiated shutdown behaviour set to terminate. The default is stop, which is wrong for Spot. A stopped Spot instance keeps its EBS volumes and its placement, you keep paying for the volumes, and the instance may never come back at a useful price. Terminate cleans up.
• Termination protection off. Enable it and the EC2 service cannot reclaim the instance, so you sit in a zombie state until the two-minute window expires and the instance is force-terminated anyway — with less time to checkpoint than if you had left it alone.
• Spot Fleet or a mixed-instances ASG rather than a single instance request. A fleet across three or four instance types and at least two availability zones cuts your interruption rate substantially. A c5.2xlarge in one AZ gets reclaimed far more often than an equivalent pool shuffled across c5, c6i, m5, and m6i. The 90% discount is only useful if the instance is actually running.
• Instance Metadata Service v2 enabled and hop limit set. Set the IMDS hop limit to 2 if the container needs to reach 169.254.169.254/latest/meta-data/spot/termination-time from inside a pod; the default of 1 blocks it and your poller silently returns 404 forever.
• EventBridge rule for the 30-second spot warning. Route EC2 Spot Instance Interruption Warning to a Lambda that flips a flag in DynamoDB or publishes to an SNS topic your job subscribes to. This fires earlier than IMDS and gives you a second, independent signal — useful when the poller itself is stuck.

The one that gets missed most often is the IMDS hop limit. Teams containerise the job, the poller works fine on a bare EC2 host during testing, and then in production the metadata call returns 401 or 404 because the request is crossing a network namespace. The failure is silent if the code treats a non-200 as "no interruption pending." Test it by running an AWS Fault Injection Simulator experiment with the aws:ec2:send-spot-instance-interruptions action against a staging fleet — it sends a real two-minute notice without you having to gamble on the spot market, and it is the only reliable way to confirm the whole chain from IMDS read to SIGTERM handler to S3 upload actually works.

How do you automate job restart after a spot interruption?

The restart itself should be somebody else's problem. On AWS Batch, set a retry strategy on the job definition with attempts of at least 3, and the service reschedules the container onto fresh capacity after a spot reclaim without you touching anything. The important part is what the job does on attempt two: read the checkpoint prefix from an environment variable or from the queue's job parameters, find the highest-numbered checkpoint present, and resume from it. If your training script exits with code 1 on interrupt, Batch treats that as a failed attempt and retries. If it exits 0, Batch assumes success and the job is marked complete even though the model never finished. Always exit non-zero when a SIGTERM arrives and you have not reached the final step.

On Kubernetes the equivalent is a managed node group of spot capacity plus a controller that will not let a pod sit Pending forever. Karpenter provisions replacement nodes within roughly 45-90 seconds of a reclaim in us-east-1, which is fast enough that a restart lands inside your normal checkpoint interval. Self-managed Cluster Autoscaler works too but reacts more slowly, often two to four minutes, because it scales on pending pods rather than on the interruption notice itself. Set a PodDisruptionBudget with maxUnavailable: 0 on anything stateful and graceful termination of at least 120 seconds so the container actually receives the SIGTERM before the kubelet pulls the plug. For jobs that can run as a single pod, a Job with backoffLimit: 6 and a restart policy of OnFailure is enough; the pod is recreated on a new node and reads its checkpoint on startup.

What makes the whole loop work is a single source of truth for "where did we get to". Write a small pointer object to S3 — something like s3://my-bucket/runs/<run-id>/latest.json containing the step number, the checkpoint key, and a timestamp — and update it after every successful checkpoint write. At roughly $0.004 per GB-month for S3 Standard in us-east-1, the cost of that pointer is effectively zero, and the 99.9% durability SLA means you will not lose it to a bad AZ. DynamoDB works as well and gives you a cheap conditional write to prevent two racing attempts from clobbering each other, which matters the moment you have more than one job per run ID. Amazon EFS is the wrong choice for the pointer because it does not survive instance termination cleanly if the mount is not detached, and the whole point is to survive termination. On startup the job reads the pointer, downloads the checkpoint, and continues. No human decides anything.

Test the failure path before you trust it. AWS Fault Injection Simulator can send an actual spot interruption notice to a running instance, and running that once per quarter against a staging job will catch the two failure modes that bite hardest: a script that swallows SIGTERM and exits 0, and a checkpoint pointer that gets written before the checkpoint bytes have finished uploading. The first turns a retry into a silent partial run; the second turns a resume into a corrupt model file. Both are cheap to prevent and expensive to discover at 2am.

How to test your checkpointing before you actually get interrupted

Run this procedure once per job type, on a disposable instance, before you trust any of the automation you built in the previous sections. It needs an IAM principal that can create FIS experiments (or, for the manual path, ec2:ModifyInstanceMetadataOptions and permission to write to IMDS), a job that has already written at least one checkpoint to S3 or EFS, and roughly 20 minutes of wall-clock time. Cost is trivial: a single m5.xlarge spot instance for 20 minutes runs well under $0.05 at the current up-to-90% spot discount, plus a few cents of S3 PUT requests. The 2-minute interruption notice is not configurable, so everything you test here is a rehearsal for an event that will arrive with exactly 120 seconds of SIGTERM grace period and no more.

1. Confirm the metadata endpoint actually answers. Before injecting anything, hit http://169.254.169.254/latest/meta-data/spot/termination-time from the instance and accept the HTTP 404. A 404 means IMDS is up and no interruption is pending; a timeout means your IMDSv2 hop limit is set to 1 and you will never see the notice. Fix the launch template before you go further. This takes 30 seconds and catches the single most common silent failure.
2. Do not use FIS to terminate the instance — use it to send the interruption notice. In the AWS Fault Injection Simulator console, create an experiment with the action aws:ec2:send-spot-instance-interruptions, target your instance by ID, and set durationBeforeInterruption to PT2M. FIS requires the instance to be a genuine Spot Instance with the aws:ec2:spot-instance resource type. Schedule the experiment for now plus 5 minutes so you have time to attach a terminal. This is the step people botch: they reach for aws:ec2:terminate-instances instead, which kills the box instantly and tests nothing about your notice handling.
3. Watch for the event. On a 2-minute duration, FIS publishes the metadata value and your poller should log something like spot/termination-time: 2026-09-14T02:00:00Z within a few seconds. If you are also consuming the older CloudWatch Events / EventBridge spot interruption warning, remember that path fires only 30 seconds before termination, not 120. Do not rely on it alone.
4. Verify the checkpoint actually lands. When your SIGTERM handler fires, it should flush the current step's state and write to S3. Budget for it: a 10 GB checkpoint over a 1 Gbps link takes roughly 45 seconds, which leaves only ~75 seconds of margin. If your write exceeds 90 seconds, the process is killed mid-PUT and you have lost the interval. Time this explicitly with time aws s3 cp on a real checkpoint file, not a 10 MB test fixture.
5. Let the instance die, then relaunch. Check that the replacement process reads the newest checkpoint, not the one before it. Grep the job logs for your own resume message and the checkpoint timestamp — if the timestamp is older than the moment of the SIGTERM, your handler is writing to a buffer that never flushed or to a path the new instance cannot see.
6. Repeat the experiment once more with the safest realistic interval. If your job checkpoints every 5 minutes, the worst case is losing 5 minutes of compute, which at spot prices is close to free. If it checkpoints every 15, that's 15 minutes of GPU time gone per interruption, and on an p4d.24xlarge at spot rates that is real money.
7. Clean up. Delete the FIS experiment template, terminate the test instance, and remove any temporary IAM policy you attached for the injection. FIS experiments left enabled will fire on their schedule against whatever currently matches the target selector.

The failure mode this procedure exists to catch: a checkpointing path that works when you test it manually with kill -TERM because you have already stopped writing, and fails under a real spot notice because the handler runs while the training loop is mid-batch and the state is inconsistent. If your resume produces a loss spike or a nondeterministic dataloader position, you will only find out here, on a throwaway instance, rather than at 2am on the run that mattered.

Frequently Asked Questions

What is the spot instance termination notice URL?

Query http://169.254.169.254/latest/meta-data/spot/termination-time from inside the instance. It returns the termination timestamp in ISO 8601 UTC, for example 2026-09-14T02:00:00Z. The endpoint is part of IMDS and needs no credentials, but IMDSv2 rejects unauthenticated GETs, so put a session token in the X-aws-ec2-metadata-token header. Poll it every 5 seconds or listen for the same signal through EventBridge.

Can I extend the 2-minute spot interruption notice?

No. The two-minute window is fixed by EC2 and there is no support ticket, quota or account setting that lengthens it. What you can do is start earlier: the EC2 Spot Instance Interruption Warning event reaches Amazon EventBridge roughly 30 seconds before the instance-level notice, so a Lambda target on that rule buys you about 30 extra seconds. Anything that cannot finish in 120 seconds should be checkpointing continuously rather than racing the clock.

How do I checkpoint a PyTorch training job on spot instances?

Call torch.save every N epochs and write to S3 or EFS, not to instance store, saving model.state_dict(), optimizer.state_dict(), the epoch number and the RNG state in one dict. Register a handler with signal.signal(signal.SIGTERM, handler) so the final write fires when the notice arrives. Save to a temp file and rename atomically, otherwise a job killed mid-write leaves a truncated checkpoint that torch.load refuses.

What happens to my EBS volume when a spot instance is terminated?

The root volume is deleted by default, and any additional EBS volume you attached is deleted too unless its DeleteOnTermination flag is set to false. Instances launched through a launch template or an Auto Scaling group inherit whatever that template says, so a template built with defaults will take your checkpoint volume with it. Check with describe-volumes before you rely on it.

How much does it cost to store checkpoints in S3?

S3 Standard is about $0.023 per GB-month in us-east-1 as of September 2026, so 500 GB of checkpoints runs roughly $11.50 a month. Keep the last two or three checkpoints in Standard and move anything older to S3 Intelligent-Tiering or Glacier Instant Retrieval, which starts around $0.004 per GB-month. You also pay $0.005 per 1,000 PUT requests, which matters if you checkpoint every few minutes.

Does Kubernetes automatically handle spot instance interruptions?

Kubernetes reschedules the pod onto another node, but it has no idea what was in memory, so the training run restarts from the last checkpoint it can find. Durability is still your job. Run the AWS Node Termination Handler or the Karpenter interruption queue to cordon and drain the node before the two-minute notice expires, then let your SIGTERM handler write the final checkpoint. Without a termination handler, pods often die before the handler runs.