LocalAI v4.3.0 — Release Notes & Engineering Impact

# 🎉 LocalAI 4.3.0 Release! 🚀 <h1 align="center"> <br> <img height="300" src="https://raw.githubusercontent.com/mudler/LocalAI/refs/heads/master/core/http/static/logo.png"> <br> <br> </h1> LocalAI 4.3.0 is out! This release **hardens the trust boundary** and **improves defaults for speed**. Backend OCI images now ship with **keyless cosign signatures** and a per-gallery `verification:` policy, with an opt-in strict mode that fails closed. The `llama-cpp` server-side prompt cache works **by default**: repeated system prompts (agents, OpenAI/Anthropic-compatible CLIs, coding assistants) collapse from **minutes to seconds** without touching YAML. Distributed mode gets rounds of optimizations. Usage tracking grows a **per-API-key + per-user Sources view** so admins can finally answer "who is burning the GPU?". And, for everyone on a Jetson/DGX box, the **L4T13 (cu130/aarch64) backends are back**. --- ## 📌 TL;DR | Feature | Summary | |---------|---------| | 🔐 **Signed Backends** | Keyless cosign + sigstore-go verification for backend OCI images, OCI 1.1 referrers, `not_before` revocation, opt-in strict mode. | | ⚡ **Prompt Cache by Default** | `llama-cpp` server-side prompt cache works out of the box. Repeated system prompts go from 5-8 min to seconds. | | 📊 **Usage per API Key** | New Sources tab attributes traffic to keys and users. Revoked keys stay readable in history. | | 🛰️ **Distributed v3** | Per-request replica routing, cached `probeHealth`, async per-node installs with streaming progress, unified backend-logs entry point. | | 🩺 **Traces UI Stays Snappy** | `LOCALAI_TRACING_MAX_BODY_BYTES` caps API + backend trace payloads. Admin Traces page stops drowning in 40 MB embeddings. | | 🧊 **Nix Flake** | Dockerless setup for NixOS users via `flake.nix` + dev shell. | | 🦾 **Jetson Thor Restored** | `vllm` / `sglang` / `vllm-omni` L4T13 backends switched to PyPI aarch64+cu130 wheels (torch 2.10 ABI fix). | --- ## 🚀 New Features & Major Enhancements ### 🔐 Signed Backends with Keyless Cosign LocalAI now verifies that backend OCI images came from our CI, not a compromised registry or MITM. This closes a real trust gap: the gallery YAML told LocalAI which image to pull, but nothing checked the bytes. The producer side (`.github/workflows/backend_merge.yml`) signs every merged backend image (and every per-arch entry under the manifest list) with **sigstore/cosign** keyless via Fulcio + Rekor, using OCI 1.1 referrers (no legacy `:tag.sig`). The consumer side (`pkg/oci/cosignverify`, built on **sigstore-go**) verifies signatures against a per-gallery `verification:` policy: ```yaml verification: issuer_regex: "^https://token\\.actions\\.githubusercontent\\.com$" identity_regex: "^https://github\\.com/mudler/LocalAI/\\.github/workflows/backend_merge\\.yml@.*$" not_before: "2026-05-22T00:00:00Z" ``` - **TUF trusted root** cached process-wide, so N backends from one gallery do **1** fetch, not N. - **`not_before` is the revocation lever**: keyless Fulcio certs are ephemeral, so revocation is policy-side. Advance the date in the gallery YAML and every signature predating the cutoff is invalidated. - **Digest pinning** closes the TOCTOU window between verify and pull. - **Strict mode**: `--require-backend-integrity` (or `LOCALAI_REQUIRE_BACKEND_INTEGRITY=true`) escalates missing policy / empty SHA256 from warn to **hard-fail**. Rollout is backward-compatible: until a gallery ships a `verification:` block, installs proceed with a warning. The default `backend/index.yaml` will be populated next, and strict mode is opt-in. See [.agents/backend-signing.md](.agents/backend-signing.md) for the full producer + consumer story. > 🔗 PRs: #9823 (consumer + producer + plumbing), #9957 (fix for current cosign releases). --- ### ⚡ Prompt Cache: On by Default `llama-cpp` ships with a server-side prompt cache, but until now LocalAI was not enabling it by default. Repeated system prompts (agents, Claude-Code-style coding assistants, OpenAI-compatible CLIs with long instructions) were re-prefilled on every call. With this release, the same workload collapses to **seconds** without no specific configuration on your side. Two changes, one default flip each: 1. **`kv_unified=true` by default** in `grpc-server.cpp`. The previous `false` was silently force-disabling `cache_idle_slots` at server init (the host prompt cache was being allocated but never written across requests). 2. **`prompt_cache_all` defaults to `true`** at the YAML layer, matching upstream `llama.cpp`'s own `common.h` default. The per-request `cache_prompt` knob is now on out of the box. You can still opt out with `options: ["kv_unified:false"]` or `prompt_cache_all: false`, and there are new option keys (`cache_idle_slots`, `checkpoint_every_nt`) for tuning. Docs in `docs/content/advanced/model-configuration.md` got a worked example for the repeated-system-prompt workload and a proper explanation of how `kv_unified`, `cache_ram`, and `cache_idle_slots` interact. > 🔗 PRs: #9925 (kv_unified + cache_idle_slots defaults + docs), #9951 (prompt_cache_all tristate default). --- ### 📊 Per-API-Key Usage Tracking Closes #9862. The usage page now answers "**who** spent these tokens?", not just "how many tokens were spent". - `usage_records` gained `Source` (`apikey` / `web` / `legacy`), `APIKeyID`, `APIKeyName`, plus an idempotent backfill of pre-feature rows on `InitDB`. - Auth middleware plumbs the resolved `*UserAPIKey` and the request source through the Echo context. Usage middleware snapshots the key id + name, so **revoked keys stay readable in history** (rendered as `(revoked)`). - New endpoints: `GET /api/auth/usage/sources` (self, no legacy) and `GET /api/auth/admin/usage/sources` (admin, with `user_id` / `api_key_id` filters, 200-key truncation). - React Usage page gains a **Sources tab** with a source-mix ribbon, a top-7 + Other time chart, and a searchable/sortable table with drill-in chip. - Admin view (follow-up in #9935) also rolls up `(source, user_id, user_name)` so Web UI session traffic is split per user instead of lumped into one global "Web UI" row, and every named-key row shows the owning account. Docs: `features/authentication.md` gained a full Usage Tracking section with the new tab, endpoints, response shape and migration notes. > 🔗 PRs: #9920 (core + Sources tab), #9935 (per-user attribution in admin view). --- ### 🛰️ Distributed Mode v3 Distributed mode keeps hardening. This release fixes the two things that bit operators hardest in practice and lays the groundwork for the next round of UX. **Per-request routing across replicas (#9968)** restores cross-node load balancing. The bug: `ModelLoader.Load` cached a `*Model` whose embedded `InFlightTrackingClient` was bound to a single `(nodeID, replicaIndex)`. After the first request, every subsequent call reused that wrapper and **pinned to whichever node won the first pick**, even after the reconciler scaled the model out. The reproducer from the report: ``` dgx-spark1 loaded in_flight=6 nvidia-thor1 loaded in_flight=0 (← idle, never gets traffic) ``` Now `SmartRouter.Route` runs per request, the existing `in_flight ASC, last_used ASC, available_vram DESC` round-robin actually fires, and the replica-selection rule lives in **one place** (`PickBestReplica`) with a mirror spec asserting the SQL `ORDER BY` and the Go picker agree on a seeded dataset. `probeHealth` is now memoized per `(nodeID, addr)` with a 30s TTL and `singleflight` coalescing, so a burst of new requests doesn't stall on a `HealthCheck` that llama.cpp serializes against in-flight `Predict`. **Async per-node installs via the gallery job queue (#9928).** `POST /api/nodes/:id/backends/install` used to block the request for up to 3 minutes while the worker pulled the image, freezing the React UI's Backends picker. It now returns **HTTP 202 + `jobID`** immediately, scoped to a one-element `targetNodeIDs` allowlist, with a node-scoped opcache row so concurrent installs on different nodes don't collide. The Operations panel surfaces a `nodeID` field for attribution. **Resilient backend installs with streaming progress (#9958).** Two phases: - *Phase 1*: `LOCALAI_NATS_BACKEND_INSTALL_TIMEOUT` / `LOCALAI_NATS_BACKEND_UPGRADE_TIMEOUT` env vars (default **15m**, previously hardcoded 3m). A NATS round-trip timeout while the worker is still pulling no longer reports as a hard failure: per-node status becomes `running_on_worker`, the queue row stays alive without bumping `Attempts`, and `ListBackends` proactively clears install rows whose intent is satisfied (so the UI updates instantly instead of waiting up to 15m for the next reconciler tick). - *Phase 2*: workers publish debounced (~250ms) `BackendInstallProgressEvent` values on a transient `nodes.<nodeID>.backend.install.<opID>.progress` subject. The master subscribes for the duration of the request and forwards each event into `OpStatus.UpdateStatus`, so the admin UI gets **per-byte progress for distributed installs** the same way local-mode does, with no UI changes. Backward compatible: old workers stay silent, new masters tolerate silence. **Unified backend-logs entry point (#9949).** `/app/backend-logs/:modelId` is now a single, mode-aware route. In standalone it's the local WebSocket view, unchanged. In distributed it probes `nodesApi.getModels`, filters by `model_name`, then routes: 0 hits → empty state with a link to Nodes; 1 hit → `<Navigate replace>` to the per-node logs URL preserving the `?from=` deep-link timestamp; N hits → a picker listing each hosting worker with node id, replica index and load state. Every view that links to backend logs now points at the same URL. **Bug-hunt harness.** A new distributed test harness landed in `tests/distributed/` to catch the kind of regressions the #9968 reproducer surfaced. > 🔗 PRs: #9968, #9928, #9958, #9949, plus the `tests(distributed): add bug-hunt harness` commit. --- ### 🩺 Admin Traces UI: Stays Responsive Under Load Two complementary caps fix the symptom where the admin Traces page sat in "loading" forever on a chatty agent-pool RAG deployment. - **API-side (#9946)**: `LOCALAI_TRACING_MAX_BODY_BYTES` (default 64 KiB) caps each captured request/response body in the trace middleware. The full payload still flows to the real client; only the trace copy is bounded. `body_truncated` + original `body_bytes` are recorded so the dashboard can surface that truncation happened. Observed before the fix on a live deployment: `/api/traces` returned **44.6 MB** (466 traces, 447 `/embeddings`, top body 1.38 MB). The Traces UI Clear button is also kept enabled during loading, which is exactly when you need it. - **Backend-side (#9960)**: `RecordBackendTrace` walks the `Data` map and replaces any string value larger than the cap with `<truncated: N bytes>`. Producers (`core/backend/llm.go`, `core/trace/audio_snippet.go`) apply head-preserving truncation upstream so the UI still shows useful leading content. TTS / `audio_transform` traces drop the base64 snippet when the encoded blob exceeds the cap (truncated base64 is undecodable; the React `WaveformPlayer` already no-ops without it). Both knobs are live-tunable from the Traces settings panel. > 🔗 PRs: #9946 (API side), #9960 (backend side). --- ### 🧊 Nix Flake for NixOS Users New `flake.nix` + `flake.lock` ship a reproducible, dockerless setup for NixOS, plus a dev shell for hacking on LocalAI without a container. > 🔗 PRs: #9851 (initial flake), #9894 (correct src path + dev shell). --- ### 🦾 Jetson Thor (L4T13) Backends Restored The `cuda13-nvidia-l4t-arm64-vllm` / `sglang` / `vllm-omni` backends crashed at import with an undefined `c10::MessageLogger` symbol after the `pypi.jetson-ai-lab.io/sbsa/cu130` mirror started shipping torch 2.11 next to vllm/sglang wheels built against torch 2.10. Per the [PyTorch April 2026 announcement](https://pytorch.org/blog/vllm-and-pytorch-work-together-to-improve-the-developer-experience-on-aarch64/), all three backends now pull from **PyPI's official aarch64 + cu130 wheels** instead, with the L4T13 `pyproject.toml` retired in favor of the standard `requirements-${profile}.txt` pattern used everywhere else. > 🔗 PR: #9950. --- ### 📎 Chat: File Attachments + Stream Usage + Selection Three independent fixes that together make the chat experience visibly better: - **Text-file attachments actually reach the model (regression from react-ui port) (#9896).** `.txt`, `.md`, `.csv`, `.json` content was silently dropped in `useChat.js` (only image_url and audio_url branches added content; the `else` branch only pushed metadata). `Home.jsx` also never called `file.text()` for files attached from the home screen. Both fixed. PDF files still need a parser (PDF.js or server-side extraction) and are flagged as a known limitation. - **Stream `include_usage` returns non-zero with tools (#9941).** Fixes #9927. `processTools` discarded the cumulative `TokenUsage` from `ComputeChoices`, so the streaming trailer reported `{0, 0, 0}` whenever a `tools` array was present. The fix forwards the authoritative final usage via a sentinel chunk before `close(responses)`, with the outer loop updated to capture before the empty-Choices skip. The OpenAI streaming spec contract is preserved (intermediate chunks still carry no `usage`). - **Chat selection stops getting wiped every second (#9917).** React 19 dropped the old `lastHtml === nextHtml` short-circuit in its DOM diff, so the 1s `/api/operations` poll re-assigning `setOperations` with a fresh array reference was collapsing text selection on every assistant message. Now JSON-compared and short-circuited. Bonus: the per-message copy button works over plain HTTP via a hidden-textarea + `execCommand('copy')` fallback when `navigator.clipboard` is unavailable. --- ### 🔧 llama.cpp Stability + Refactors - **`tensor_buft_overrides` sentinel terminator (#9919).** Mirror upstream `common/arg.cpp:645-658`: pad placeholders at the end of the main vector so `back().pattern == nullptr` holds, and append a single `{nullptr, nullptr}` to the draft vector when non-empty. - **`refactor(agents): bump skillserver, drop redundant Name` (#9916).** `list_skills` and `search_skills` now return the same shape (only `id`, no duplicated `name`). Adds a Ginkgo regression that drives the LocalAI `FilesystemManager` through an in-process MCP session. - **Swagger refreshes (#9872, #9962)** keep the OpenAPI surface in sync with the routes added this cycle. --- ### 🛠️ CI & Image Plumbing - **Chronologically-orderable master tags.** Master images are now also tagged `master-<epoch>-<sha>` so they sort by build time. The pre-existing `master` tag still moves with `HEAD`. - **Backend signing CI**: `COSIGN_EXPERIMENTAL=1` is set for the oci-1-1 referrers mode in the backend-signing job to keep current cosign versions happy. --- ## 🐛 Bug Fixes (recap) - 🔁 `fix(distributed): route per request across loaded replicas + cache probeHealth` - #9968 - 🧰 `fix(distributed): make admin backend installs resilient and observable` - #9958 - ⏳ `fix(nodes): make per-node backend install async via gallery job queue` - #9928 - 🧾 `fix(traces): cap backend trace Data to keep admin UI responsive` - #9960 - 🧾 `fix(traces): cap captured body size to keep admin Traces UI responsive` - #9946 - 🪟 `fix(react-ui): unify backend-logs entry point for distributed mode` - #9949 - 📎 `fix: inject text-file content into chat completions messages` - #9896 - 🧮 `fix(openai): stream usage non-zero when tools are enabled` - #9941 - 🧷 `fix(react-ui/chat): stop wiping selection on every /api/operations poll` - #9917 - 🧱 `fix(llama-cpp): terminate tensor_buft_overrides with sentinel` - #9919 - 🦾 `fix(L4T13 backends): switch vllm/sglang/vllm-omni to PyPI aarch64+cu130 wheels` - #9950 - 🧊 `fix(nix): correct flake src path and add dev shell` - #9894 - 🔐 `Fix backend manifest merge signing on current cosign releases` - #9957 - 🧯 `[utils] Fail immediately on extraction errors` - #9926 --- ## 👒 Dependencies Heavy bump cycle across submodules and Go/Python deps: - **`ggml-org/llama.cpp`**: 7 bumps (#9855, #9876, #9897, #9915, #9934, #9952, #9963) - **`ggml-org/whisper.cpp`**: 4 bumps (#9877, #9898, #9929, #9954) - **`ikawrakow/ik_llama.cpp`**: 7 bumps (#9866, #9875, #9899, #9914, #9930, #9953, #9966) - **`leejet/stable-diffusion.cpp`**: 4 bumps (#9907, #9933, #9955, #9965) - **`antirez/ds4`**: 5 bumps (#9864, #9900, #9909, #9932, #9964) - **`ace-step/acestep.cpp`**: bumped to `ed53caf` with wrapper API adapted (#9908, #9913) - **Model gallery**: checksum refreshes (#9901, #9910) - **Go modules**: `alecthomas/kong` 1.14→1.15 (#9881), `aws/aws-sdk-go-v2` 1.41.6→1.41.7 (#9892), `onsi/ginkgo/v2` 2.28.2→2.29.0 (#9882), `golang.org/x/crypto` 0.50→0.51 (#9886) - **Python**: `transformers` ≥5.8.1 (#9883), `sentence-transformers` 5.4→5.5 (#9888) --- ## 📖 Documentation - `docs: :arrow_up: update docs version mudler/LocalAI` - #9863 - Plus inline docs updates folded into the feature PRs above (prompt-cache explainer, authentication / usage tracking section, backend signing guide). --- ## 🙌 New Contributors * @Azteczek made their first contribution in https://github.com/mudler/LocalAI/pull/9851 * @inquam made their first contribution in https://github.com/mudler/LocalAI/pull/9896 * @RinZ27 made their first contribution in https://github.com/mudler/LocalAI/pull/9926 Enjoy! --- **Full Changelog**: https://github.com/mudler/LocalAI/compare/v4.2.6...v4.3.0