Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 32 additions & 21 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -471,15 +471,15 @@ Distributed backend selection lives outside the YAML, in env vars: `MX_METADATA_

### ModelRegistryBackend (Redis and Kubernetes CRD)

**Redis backend**: single Redis Hash per cached model at `mx:model:{name}` with fields `provider`, `status`, `created_at` (RFC3339), `last_used_at` (RFC3339), and optional `message`. No secondary indexes — LRU ordering and status counts are computed on demand by `SCAN` + pipelined `HGETALL`/`HGET`. Claim, retry, and `set_status` updates use Lua scripts so concurrent readers see either the pre-update record or the complete post-update record, never a partially-written hash.
**Redis backend**: single Redis Hash per cached model at `mx:model:{provider}:{name}` with fields `provider`, `status`, `created_at` (RFC3339), `last_used_at` (RFC3339), and optional `message`. No secondary indexes — LRU ordering and status counts are computed on demand by `SCAN` + pipelined `HGETALL`/`HGET`. Claim, retry, and `set_status` updates use Lua scripts so concurrent readers see either the pre-update record or the complete post-update record, never a partially-written hash.

**Kubernetes CRD backend**: one `ModelCacheEntry` CR per cached model in the server's namespace. `spec.modelName` + `spec.provider` are immutable; `status.{phase,createdAt,lastUsedAt,message}` are patched via the status subresource. Atomicity on the claim path comes from etcd's name-uniqueness on `create` (409 Conflict on the loser). CR names use the shared `sanitize_model_name` with an `mx-cache-` prefix to stay distinct from the P2P `ModelMetadata` CRs.
**Kubernetes CRD backend**: one `ModelCacheEntry` CR per cached model in the server's namespace. `spec.modelName` + `spec.provider` are immutable; `status.{phase,createdAt,lastUsedAt,message}` are patched via the status subresource. Atomicity on the claim path comes from etcd's name-uniqueness on `create` (409 Conflict on the loser). CR names are `mx-cache-` followed by `sanitize_registry_name("{provider}/{model_name}")`, which lowercases, maps `/` to `--`, and appends a sha256 suffix over the original name. The P2P `ModelMetadata` CRs use their own `sanitize_model_name`; the two are separate implementations, not a shared helper.

Key operations on the async `RegistryBackend` trait:

| Method | Redis implementation |
|--------|----------------------|
| `get_status(name)` | `HGET mx:model:{name} status` |
| `get_status(name)` | `HGET mx:model:{provider}:{name} status` (falls back to the legacy name-only key so pre-0.5.0 records still resolve) |
| `set_status(name, provider, status, msg)` | Lua `EVAL` updates status/provider/last_used_at/message atomically and `HSETNX`s `created_at` to preserve the first-write timestamp |
| `try_claim_for_download(name, provider)` | `HSETNX status DOWNLOADING`; winner populates remaining fields without contention |
| `touch_model(name)` | `HSET last_used_at {now}` (gated on `EXISTS` so touch is update-only, never create) |
Expand Down Expand Up @@ -541,18 +541,25 @@ The `Client` struct in `modelexpress_client/src/lib.rs` wraps gRPC connections:

| Method | Purpose |
|--------|---------|
| `new(config)` | Create client with config |
| `health_check()` | Call HealthService |
| `ping()` | Call ApiService with "ping" |
| `download_model(name, provider)` | Trigger download via streaming RPC |
| `download_model_direct(name, provider, cache_dir)` | Download directly from provider |
| `ensure_model(name, provider, strategy)` | Smart download with fallback strategy |
| `stream_model_files(name)` | Stream model files from server |
| `list_model_files(name)` | List model files on server |
| `delete_model(name, cache_dir)` | Delete cached model |
| `validate_model(name, cache_dir)` | Validate cached model integrity |

Each download entry point has a `_revision` variant — `request_model_revision`, `request_model_on_server_revision`, `request_model_with_smart_fallback_revision` — that takes an optional branch, tag, or commit SHA and returns a `ModelDownloadResult { path, resolved_revision }`. The non-`_revision` methods keep their existing signatures and delegate with no revision pinned.
| `new(config)` | Create a client with the given configuration |
| `new_with_cache(config, cache_config)` | Create a client with an explicit cache configuration |
| `get_cache_config()` | Get the client's cache configuration, if any |
| `set_cache_config(cache_config)` | Set the client's cache configuration |
| `list_cached_models()` | List locally cached models as `CacheStats` |
| `clear_cached_model(name, provider)` | Remove a model's local files for a given provider |
| `clear_all_cached_models()` | Clear the entire local cache |
| `delete_model_on_server(name, provider)` | Delete the model's record from the server-side registry, so a cleared model leaves no stale `DOWNLOADED` record |
| `get_model_path(name, provider)` | Resolve the local cache path for a model through its provider |
| `health_check()` | Call HealthService and return the server `Status` |
| `send_request(action, payload)` | Send a generic ApiService request and deserialize the response |
| `request_model_on_server(name, provider)` | Request a download on the server at the provider's default revision |
| `request_model_on_server_revision(name, provider, revision)` | Same, pinned to a branch, tag, or commit SHA; returns the resolved revision |
| `request_model(name, provider)` | Request a model using the server as source of truth, streaming files locally when shared storage is disabled |
| `request_model_revision(name, provider, revision)` | Same, pinned to a revision; returns the snapshot path and the revision it resolved to |
| `request_model_with_smart_fallback(name, provider, ...)` | Request via the server, falling back to a direct provider download when the connection cannot be established |
| `request_model_with_smart_fallback_revision(name, provider, revision, ...)` | Same, with the direct-download fallback honouring the same pinned revision |

The `_revision` variants all take an optional branch, tag, or commit SHA, but they do not share a return type. `request_model_revision` and `request_model_with_smart_fallback_revision` return a `ModelDownloadResult { path, resolved_revision }`. `request_model_on_server_revision` only asks the server to fetch the model, so it returns the resolved revision on its own as an `Option<String>`. The non-`_revision` methods delegate to them with no revision pinned and discard the result.

### Download Strategies

Expand All @@ -568,12 +575,16 @@ The `Cli` struct in `args.rs` embeds `ClientArgs` via `#[command(flatten)]`. Com

| Command | Purpose |
|---------|---------|
| `health` | Check server health |
| `ping` | Ping server |
| `model download <name>` | Download a model. `--revision <branch\|tag\|sha>` pins a revision; the resolved commit SHA is reported back |
| `model list-files <name>` | List model files |
| `model clear <name>` | Delete cached model |
| `model validate <name>` | Validate model integrity |
| `health` | Check server health and status |
| `model download <name>` | Download a model with various strategies (automatically cached). `--revision <branch\|tag\|sha>` pins a revision; the resolved commit SHA is reported back |
| `model init` | Initialize model storage configuration |
| `model list` | List downloaded models |
| `model status` | Show model storage status and usage |
| `model clear <name>` | Clear a specific model from storage |
| `model clear-all` | Clear all models from storage |
| `model validate [name]` | Validate model integrity |
| `model stats` | Show model storage statistics |
| `api send <action>` | Send a custom API request |

Output formats: `--format human` (default), `--format json`, `--format json-pretty`.

Expand Down
13 changes: 10 additions & 3 deletions docs/CLI.md
Original file line number Diff line number Diff line change
Expand Up @@ -398,15 +398,22 @@ modelexpress-cli model status

### Configuration File Support

The CLI loads a YAML configuration file via `-c, --config <FILE>` (default
`~/.model-express/config.yaml`). Values are merged in this order of precedence,
highest first:
The CLI loads a YAML configuration file via `-c, --config <FILE>`. With no
`--config`, it searches these paths in order and uses the first that exists:
`model-express.yaml` and `model-express.yml` in the working directory, then
`/etc/model-express/config.yaml` and `/etc/model-express/config.yml`. Values are
merged in this order of precedence, highest first:

1. Command line arguments
2. Environment variables (`MODEL_EXPRESS_*`)
3. Configuration file
4. Built-in defaults

This client configuration is separate from the cache configuration written by
`modelexpress-cli model init`, which lives at `~/.model-express/config.yaml` and
holds the local storage path and server endpoint used by the cache-management
commands.

```bash
modelexpress-cli --config /etc/modelexpress/client.yaml health
```
Expand Down
1 change: 0 additions & 1 deletion docs/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -600,7 +600,6 @@ See [`K8S_SERVICE_BACKEND.md`](K8S_SERVICE_BACKEND.md) for the design rationale,
| `MX_K8S_SERVICE_PATTERN` | `mx-sources` | DNS template for the `k8s-service` backend. `{rank}` is substituted with the worker's own rank. If the resolved pattern has no `:port`, the client auto-appends `:{MX_WORKER_GRPC_PORT + rank}` (multi-GPU-per-pod shape); if it has an explicit port, that port is used verbatim (1-GPU-per-pod shape). |
| `MX_K8S_SOURCE_RETRIES` | `5` | `k8s-service` backend: max retries on `FAILED_PRECONDITION` (revision mismatch during rolling updates). Each retry opens a fresh gRPC channel so kube-proxy re-picks a backend. |
| `MX_K8S_SOURCE_BACKOFF_SECONDS` | `0.5` | `k8s-service` backend: sleep between retry attempts. |
| `MX_STATUS_TTL_SECS` | `3600` | TTL for Redis metadata keys (seconds) |
| `REDIS_URL` | `redis://localhost:6379` | Redis connection URL (Redis backend only) |
| `MX_METADATA_NAMESPACE` | `default` | K8s namespace for CRD backend |
| `VLLM_RPC_TIMEOUT` | `7200000` | vLLM RPC timeout in ms (2 hours for large models) |
Expand Down
8 changes: 5 additions & 3 deletions docs/metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -354,7 +354,9 @@ without a Pod owner reference. This preserves behavior for older clients and
non-Kubernetes environments; the server-side stale metadata reaper remains the
cleanup path in those cases.

**Model lifecycle CRD name format**: `mx-cache-{sanitized-model-name}-{hash}`
**Model lifecycle CRD name format**: `mx-cache-{provider}--{sanitized-model-name}-{hash}`

The name is `mx-cache-` followed by `sanitize("{provider}/{model_name}")`, so each `/` becomes `--` and the sha256 suffix binds the pair `(provider, name)`. For provider `HuggingFace` and model `deepseek-ai/DeepSeek-V3` the CR is named `mx-cache-huggingface--deepseek-ai--deepseek-v3-{hash}`. Pre-0.5.0 deployments carry name-only CRs (`mx-cache-{sanitized-model-name}-{hash}`), which the server still looks up so those records migrate.

`ModelCacheEntry.spec.modelName` preserves the original model name while `status.phase`, `status.createdAt`, `status.lastUsedAt`, and `status.message` track the same lifecycle fields as the Redis `mx:model:*` hash.

Expand Down Expand Up @@ -439,14 +441,14 @@ status:

```bash
kubectl get modelcacheentries -n <namespace>
kubectl get modelcacheentry mx-cache-deepseek-ai--deepseek-v3-<hash> -n <namespace> -o yaml
kubectl get modelcacheentry mx-cache-huggingface--deepseek-ai--deepseek-v3-<hash> -n <namespace> -o yaml
```

```yaml
apiVersion: modelexpress.nvidia.com/v1alpha1
kind: ModelCacheEntry
metadata:
name: mx-cache-deepseek-ai--deepseek-v3-<hash>
name: mx-cache-huggingface--deepseek-ai--deepseek-v3-<hash>
spec:
modelName: deepseek-ai/DeepSeek-V3
provider: HuggingFace
Expand Down
2 changes: 1 addition & 1 deletion examples/k8s_service_sources/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ The Service's Endpoints object is the source list, maintained by Kubernetes base
## Prerequisites

1. Kubernetes cluster with GPU nodes. The YAMLs request `rdma/shared_ib` resources for InfiniBand/RoCE - the fast path and the configuration production should run on. Without RDMA, UCX/NIXL falls back to plain TCP at significant throughput cost; drop the `rdma/shared_ib` resource requests from the manifests to run without it.
2. A path for weights to reach the pods. Any of: pre-downloaded to a shared PVC, streamed from S3 (set `MX_S3_URI`), or downloaded from HuggingFace at pod startup. For the HuggingFace option, create the token secret with `kubectl create secret generic hf-token-secret --from-literal=HF_TOKEN=<token>`.
2. A path for weights to reach the pods. Any of: pre-downloaded to a shared PVC, streamed from S3 (set `MX_MODEL_URI`), or downloaded from HuggingFace at pod startup. For the HuggingFace option, create the token secret with `kubectl create secret generic hf-token-secret --from-literal=HF_TOKEN=<token>`.
3. A model revision you trust. Pin it via `MX_MODEL_REVISION=<commit_sha>` (or set `model_config.revision` in vLLM) so `mx_source_id` is content-addressed.

## Deploying
Expand Down
2 changes: 1 addition & 1 deletion helm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ The following table lists the configurable parameters of the ModelExpress chart
| `persistence.mountPath` | Mount path | `/root` |
| `env.MODEL_EXPRESS_SERVER_PORT` | Server port | `8001` |
| `env.MODEL_EXPRESS_LOGGING_LEVEL` | Logging level | `info` |
| `env.MODEL_EXPRESS_CACHE_DIRECTORY` | Cache directory | `/app/cache` |
| `env.MODEL_EXPRESS_CACHE_DIRECTORY` | Cache directory | `/root` |
| `env.MX_METADATA_BACKEND` | Distributed backend (`redis` or `kubernetes`). Server fails to start without this. | `<required>` |
| `env.REDIS_URL` | Redis connection URL; required when backend is `redis`. Chart does not bundle Redis. | `<required when backend=redis>` |
| `livenessProbe.enabled` | Enable liveness probe | `true` |
Expand Down
Loading