Preemption
Discounted spot sandboxes that Daytona can reclaim when it needs capacity.
Daytona sandboxes run on reserved capacity by default: once placed, a sandbox keeps its resources until you delete it. A spot sandbox is preemptible: it is billed at a discount, and Daytona can reclaim it at any time to free capacity for full-price sandboxes. The tier is set when the sandbox is created and cannot be changed afterwards.
A reclaimed ephemeral Linux spot sandbox is deleted, with its filesystem. A reclaimed persistent Linux spot sandbox, such as a fork of a spot sandbox, is paused, and a resume continues it. To continue the work of an ephemeral sandbox, create a new sandbox and keep results in storage outside the sandbox, such as your own object storage. A reclaimed Windows spot sandbox is stopped, with the disk kept, and a start boots Windows from the preserved disk.
Use cases
Spot sandboxes are used for work that tolerates an interruption in exchange for a lower price.
- Batch jobs: data processing, builds, and test runs start again from the beginning when a sandbox is reclaimed. Spot multiplies the CPU, memory, and disk price by 0.7.
- Evaluation and reinforcement learning (RL): many short test cases or episodes run on spot sandboxes. A reclaimed test case or episode runs again in a new sandbox.
- CI runs: build and test jobs run on Linux or Windows spot sandboxes. A reclaim costs one rerun of the job.
- Stateless workers: workers that keep no state in the sandbox lose nothing on a reclaim. A new spot sandbox replaces the reclaimed one.
- Results outside the sandbox: a spot sandbox writes its output to storage outside the sandbox, such as your own object storage, so a reclaim loses only the work in progress.
- Development and experiments: sandboxes whose loss costs only a restart run at the spot price.
Spot sandboxes
A spot sandbox differs from an on-demand sandbox only in price and reclaim. In every other respect it is an ordinary sandbox: it supports the same images, resources, and lifecycle operations as an on-demand sandbox of its type. Spot multiplies the CPU, memory, and disk price by 0.7; see pricing dimensions.
Linux and Windows sandboxes can be spot. GPU sandboxes cannot.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.create(spot=True)| Parameter | Type | Description |
|---|---|---|
spot | bool | Places the sandbox on discounted, reclaimable capacity. The default is False. The value is fixed at create. |
Reclaim policy
Daytona tracks the CPU, memory, and disk reserved on each host. When the reservation on a host crosses a threshold on any of the three, Daytona reclaims running spot sandboxes on that host until the projected reservation drops below it. Newer spot sandboxes are reclaimed before older ones, interrupting the least accumulated work first. On-demand sandboxes are never reclaimed.
A reclaim is immediate. Daytona does not notify the sandbox or your application in advance, and there is no grace period. Daytona does not replace a reclaimed sandbox on its own when capacity frees up: creating a new one is up to you.
A reclaimed sandbox is billed like any deleted sandbox.
Reclaim outcomes
Daytona records the reclaim time, then tears an ephemeral sandbox down like a user request: the sandbox moves to deleting, then deleted, and only data in storage outside the sandbox remains. A persistent Linux sandbox is paused instead. A Windows sandbox is stopped, since Windows cannot pause, and keeps its disk.
For the full state model, see lifecycle.
Reclaim detection
Reclaim detection tells a reclaim apart from a delete you requested. Every reclaimed sandbox carries the time of the reclaim. A sandbox that was never reclaimed carries no reclaim time.
from daytona import Daytona
daytona = Daytona()
sandbox = daytona.v2.get("{sandboxId}")
# None unless the platform reclaimed this spot sandbox
if sandbox.spot_evicted_at:
print(f"reclaimed at {sandbox.spot_evicted_at}, state: {sandbox.state}")
# Find spot sandboxes
page = daytona.v2.search(spot=True)
for sandbox in page.items:
print(sandbox.id, sandbox.state, sandbox.spot_evicted_at)| Parameter | Type | Description |
|---|---|---|
spot | bool | Filters by capacity type. True returns only spot sandboxes, and False returns only on-demand sandboxes. The default is None, which returns both. |
Reclaim recovery
A reclaimed ephemeral sandbox is gone; continue in a new sandbox from data stored outside the sandbox.
from daytona import Daytona
daytona = Daytona()
previous = daytona.v2.get("{sandboxId}")
# A reclaimed sandbox is deleted; only data outside it remains
if previous.spot_evicted_at:
sandbox = daytona.v2.create(spot=True)
# Continue from the last checkpoint the previous sandbox wrote to your own storage
sandbox.process.run("python3 train.py --resume-from s3://my-bucket/checkpoint")Opting in to preemption
Spot suits work that tolerates interruption and can be restarted or continued: batch jobs, evaluation and reinforcement learning rollouts, CI runs, and stateless workers. Keep spot off sandboxes that serve live traffic or hold state that exists nowhere else.
- Checkpoint to storage outside the sandbox from spot sandboxes, so a replacement sandbox can continue where the reclaimed one stopped.
- Check after interruptions: when a sandbox stops responding or changes state unexpectedly, read its reclaim time to tell a reclaim apart from a failure.