Serverless GPU in Databricks: Where It Makes Sense and Where It Doesn't
Serverless GPU availability in Databricks was one of the quieter DAIS announcements, but it's worth talking through because the "serverless = cheaper and easier" assumption that applies to SQL warehouses and notebook compute doesn't hold the same way for GPU workloads. The economics are meaningfully different.
What Serverless GPU Gives You
Zero-infrastructure GPU access for fine-tuning, inference, and model evaluation. You don't provision a cluster, choose an instance type, or manage driver/library versions for the GPU stack. Spin up, run, spin down. For teams that do GPU work occasionally — quarterly fine-tune runs, one-off model evaluations, experimental inference testing — that's a real operational improvement.
The sweet spot matches the serverless SQL pattern: short to medium duration jobs where the overhead of cluster management exceeds the overhead of per-job startup cost.
Where the Economics Break Down
Long-running GPU training jobs. Continuous inference serving. Anything that runs for hours or days — the serverless premium per GPU-hour gets expensive compared to reserved or spot instance clusters for sustained workloads. The same five-minute threshold that applies to serverless SQL applies here, but the cost delta is larger because GPU hours are expensive to begin with.
Also: memory-intensive models that require specific GPU configurations — large context windows, multi-GPU tensor parallelism, high VRAM requirements — may not fit the instance types available in the serverless pool. Check the spec sheet before you assume your fine-tune job will run as-is.
The Practical Decision
Use serverless GPU for experimentation, evaluation, and infrequent fine-tunes. Use provisioned GPU clusters for production inference serving, long training runs, and anything with specific hardware requirements. The same logic you already apply to CPU serverless vs. provisioned compute applies here — just with a larger cost multiple when you get it wrong. I'm here to help work through the right configuration for your specific workload.