Cache Model Weights

API: modelplane.ai/v1alpha1 · ModelCache

A ModelCache stages a model’s weights on shared workload-cluster storage, fetched once from the configured source rather than downloaded again on every pod start. ModelDeployments reference a cache via spec.template.spec.modelCacheRef.name, and Modelplane mounts it at /mnt/models in every serving pod, shared across the pods of a multi-node engine. The engine reads weights locally from the mount.

ModelCache is recommended for multi-node deployments and optional for single-node cold-start optimization.

What to cache

The required source enum names the kind, with the matching source object set alongside it. Setting source: HuggingFace selects spec.huggingFace, which carries the repo to fetch, an optional revision (branch, tag, or commit), and sizeGiB, how much storage the weights get on each cluster. HuggingFace is the only source today.

The cache downloads every file in the repository unless you narrow it. Many repositories publish the same weights more than once, such as one large checkpoint beside the same weights split across many files, or one copy per format. include and exclude take glob patterns, passed to hf download as --include and --exclude. Size the cache to the files it downloads.

This repository holds 90 GiB of weights in two formats. The cache downloads only the 45 GiB in Mistral’s own format, so the engine must load that format. For vLLM, pass --tokenizer-mode mistral --config-format mistral --load-format mistral, as the model’s card does:

spec:
  source: HuggingFace
  huggingFace:
    repo: mistralai/Mistral-Small-3.2-24B-Instruct-2506
    include: ["consolidated.safetensors", "params.json", "tekken.json"]
    sizeGiB: 50

Make sure the cache contains every file the engine reads at startup, such as the tokenizer and the chat template. The engine downloads a file the cache left out, or fails to start if it runs offline.

The cache downloads once. The API server rejects a change to repo, revision, include or exclude, because a change would mean downloading again and restarting every engine that reads the cache. To change what a deployment reads, create a ModelCache with a new name and point the deployment’s modelCacheRef at it. The engines keep the old cache until the new one is ready on their cluster. Delete the old cache after they move. A cache recreated under its old name gets its old claim’s name, and it can’t stage while engines still mount that claim.

The engine’s args name the model the same way with or without a cache. A HuggingFace source stages into HuggingFace’s own cache layout on the mount, and Modelplane sets HF_HUB_CACHE on every consuming pod, so --model=<repo> resolves to the staged weights instead of pulling them. Adding or removing a cache doesn’t change the engine command. Modelplane never injects --model itself: naming the model belongs to the engine command, like every other flag.

Name the same revision the cache staged. A bare repository ID resolves at the default branch, which finds a cache staged without a revision or with revision: main. A cache pinned to a commit or tag needs the engine to pass that revision too (--revision for vLLM). If the engine uses the default branch instead, it finds nothing staged there and downloads the model a second time.

Waiting for the cache

A replica doesn’t start its engines until the cache is ready on its cluster. Until then the ModelReplica reports ModelAccepted as False with reason WaitingForCache, and a message from the cache that says what it’s waiting for. The ModelCache gives the same message for each cluster:

kubectl get modelcache qwen -n ml-team -o jsonpath='{.status.clusters}'

The message gives the claim that hasn’t bound and the node it’s waiting on, the repository the hydration Job is downloading (Downloading <repo> into PersistentVolumeClaim <claim>), or the Job that failed and why. A replica checks its cache about once a minute. Its engines start up to about a minute after the cache turns Ready.

A replica whose engines already run doesn’t wait. Until its cache is ready on its cluster, the engines keep the cache they mount now, or run without one if they mount none. This holds when the cache stops being ready, when someone deletes it, and when you add a cache to the deployment or switch it to another one. The engines restart onto the new cache once it’s ready.

Running without contacting HuggingFace

An engine that loads a cached model still checks HuggingFace when it starts, unless you tell it not to. To start without contacting HuggingFace, set HF_HUB_OFFLINE=1 in the engine container’s env:

env:
- name: HF_HUB_OFFLINE
  value: "1"

The engine then reads only the cache, so the cache must contain every file it needs at the revision the engine uses. Modelplane doesn’t set the variable itself, because it breaks an engine that fetches a second repository at startup, such as a separate tokenizer. A ModelCache can’t stage a second repository for one deployment yet (#281 ).

Authenticating

A gated or private model needs a credential to fetch. When a cache stages the weights, the credential goes on the cache: set authSecret to name a Secret in the cache’s namespace, and Modelplane propagates it to every cluster the cache stages to, for the hydration to read.

Create the Secret once on the control plane, then reference it:

kubectl create secret generic hf-token \
  --namespace ml-team \
  --from-literal=HF_TOKEN=hf_xxxxxxxx

spec:
  source: HuggingFace
  huggingFace:
    repo: Qwen/Qwen3-Coder-480B-A35B-Instruct
    authSecret:
      name: hf-token         # a Secret in this ModelCache's namespace
      key: HF_TOKEN          # defaults to HF_TOKEN
    sizeGiB: 1100

Without a cache, the engine fetches the model itself at startup, so the credential goes on the ModelDeployment instead, as HF_TOKEN in the engine container’s env.

Where to cache

An optional clusterSelector scopes where the cache is staged. Omitting it stages the cache on every cluster in the fleet; setting matchLabels restricts it to clusters with those labels. Either way, a cluster with no cache storage is skipped. A ModelDeployment that references the cache places replicas only on clusters the cache stages to, and moves a replica off a cluster the cache stops staging to, since the cache’s volume is removed from it.

Loading from cache

A cache only shortens cold starts if the engine reads from it quickly. With its default loader an engine can read a large model from shared storage slowly enough that the cache makes cold starts worse than fetching the model directly, since you pay to hydrate the cache and then wait on a slow read. Choose a fast loader with your engine flags.

For vLLM on EKS, --load-format=runai_streamer reads from the EFS-backed cache dramatically faster than the default loader (minutes rather than tens of minutes for a large model), tuned further with --model-loader-extra-config:

args:
- --model=RedHatAI/Kimi-K2-Instruct-quantized.w4a16
- --load-format=runai_streamer
- --model-loader-extra-config={"concurrency":16,"distributed":true}

The right loader and settings depend on the engine and the storage backend, so treat these as a starting point and measure your own cold-start time.

Accelerating with ModelExpress

A cache’s weights always live on its own PVC, portable across every cluster. On a Dynamo cluster the serving stack also runs a ModelExpress server, and Modelplane injects ModelExpress env into every engine pod that references a cache. An engine opts in with --load-format modelexpress: the first replica loads from its PVC seed and publishes itself as a source, and later replicas pull from a peer over RDMA rather than reading storage again. A replica that finds no compatible peer, or no fabric connecting it to one, falls back to the PVC, so the cache still has to be sized and kept for every replica. Because the env is inert unless the engine opts in, a cache still works unchanged on a Standard cluster and a deployment is portable between the two.

Modelplane injects no --load-format flag: the ML team’s engine command decides whether to use ModelExpress’s loader, the same as it decides --load-format=runai_streamer above. Write it yourself, verbatim:

command: ["/bin/sh", "-c"]
args:
- >-
  pip install --index-url https://pypi.nvidia.com modelexpress &&
  exec vllm serve Qwen/Qwen2.5-7B-Instruct --load-format modelexpress

Storage prerequisites

The cache PVC needs a ReadWriteMany (RWX) StorageClass on the workload cluster. Some cluster sources provide one automatically; on others the platform admin sets one up. See Register a Cluster for what each source provides and how to bring your own backend.

Your ModelCache and ModelDeployment specs are the same on every cluster source.

Example

# A ModelCache stages a model artifact on workload-cluster storage as a
# first-class resource. Modelplane composes a ReadWriteMany PVC on each matched
# cluster and hydrates it once from the configured source. A ModelDeployment
# references it via spec.modelCacheRef; the PVC mounts at /mnt/models read-write
# into every serving pod, so the engine reads weights locally instead of
# fetching them at boot.
#
# The engine command doesn't change when a model is cached. A HuggingFace source
# stages in HuggingFace's own cache layout, and Modelplane points the engine's
# HF_HUB_CACHE at the mount, so naming the model by its repo id
# (--model=Qwen/Qwen3-Coder-480B-A35B-Instruct) resolves to the staged weights
# rather than pulling them.
apiVersion: modelplane.ai/v1alpha1
kind: ModelCache
metadata:
  name: qwen3-coder
  namespace: ml-team
spec:
  source: HuggingFace
  huggingFace:
    repo: Qwen/Qwen3-Coder-480B-A35B-Instruct
    # Gated repo, so a Hugging Face token is needed. Create this Secret once in
    # the ModelCache's namespace on the control plane; Modelplane propagates it
    # to each matched cluster.
    authSecret:
      name: hf-token
      key: HF_TOKEN
    sizeGiB: 1100
  # Optional: stage only on clusters matching these labels. Omit to stage on
  # every matched cluster. Narrowing this also narrows where a referencing
  # ModelDeployment can place new replicas.
  # clusterSelector:
  #   matchLabels:
  #     modelplane.ai/tier: frontier