Why do jobs keep failing or restarting?

If container jobs regularly stop, restart or last only a few minutes, and the app shows no Degraded notification, the cause is almost always on the customer's side rather than yours. You're paid for the time each job runs, and the machine goes straight back into the queue.

What's usually happening

• The customer ended or moved the job. Customers scale their workloads up and down all day, and they can reallocate a job to another machine at any time. Short jobs can be expected on the network.
• The job itself has a problem. A customer still testing a new workload may ship one that crashes or fails its own health check; it restarts a few times and is then withdrawn. Salad tells the customer automatically. Nothing for you to do.
• It was a batch job - a job meant to run for a bounded time and finish.

When it is your machine

If restarts come with a notification, follow it: Degraded State. Restarts can also come from a connection that drops (the job loses contact and is moved), from the PC sleeping, or from another program grabbing the GPU mid-job. A wired connection and leaving the PC alone fix most of these: Improve your internet connection, Disable sleep mode.

If every job on your machine lasts under a few minutes for a whole day, send Salad Support your machine ID and your logs folder zipped (how to find it); the logs show whether the job exited on its own or was stopped from outside.