hermes/dsh
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

dsh 0.2.0-rc.2: in-container update (experimental channel), path-agnostic compose, upstream patches adapted to 0.1.x/0.2.x

Browse Source
This commit is contained in:
hermes committed 2026-10-01 19:21:19 +00:00
commit e6dcef9bf5
302 files changed
+75178
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.env.example
+43
View File
@@ -0,0 +1,43 @@
# ── Обязательное ────────────────────────────────────────────────────────────
# Ключ DeepSeek API (platform.deepseek.com). Без него dsh не сможет работать с моделью.
DEEPSEEK_API_KEY=
# ── Образ ───────────────────────────────────────────────────────────────────
# Свой реестр (обновляется: docker compose pull && docker compose -p dsh up -d)
DSH_IMAGE=gitea.archeon.io/archeon/dsh:latest
# Локальная сборка из этого репо (см. scripts/build-image.sh):
# DSH_IMAGE=dsh-local:0.2.0-rc.2
# ── Пути (path-агностичность) ───────────────────────────────────────────────
# Каталог данных: по умолчанию ./data рядом с compose-файлом.
# Можно вынести куда угодно — абсолютный путь или другой каталог:
# DSH_DATA_DIR=/var/lib/dsh
DSH_DATA_DIR=./data
# Каталог проектов (воркспейс агента):
# DSH_WORKSPACE_DIR=/home/myuser/projects
DSH_WORKSPACE_DIR=./workspace
# ── Сеть/порты ──────────────────────────────────────────────────────────────
# 127.0.0.1 — доступ только с этого хоста (за реверс-прокси/TLS).
# Для доступа из LAN впишите IP хоста или 0.0.0.0 и настройте DSH_TRUSTED_HOSTS.
DSH_LISTEN_ADDRESS=127.0.0.1
DSH_PORT=3080
# Порт code-server редактора (должен совпадать с настройкой плагина в UI!)
DSH_EDITOR_PORT=18500
# DSH_TRUSTED_HOSTS=192.168.1.50,dsh.example.lan
# ── dsh ─────────────────────────────────────────────────────────────────────
DSH_DEFAULT_WORKSPACE=/workspace
# true — дать агенту безлимитный root в песочницах (ОСОЗНАННО):
DSH_SUDO_ENABLED=false
DSH_TELEMETRY_DISABLED=true
# Секунды пробации experimental-обновления до автоподтверждения
DSH_EXPERIMENTAL_PROBATION_SECONDS=120
# ── Имена контейнеров (менять при конфликтах) ──────────────────────────────
DSH_CONTAINER_NAME=dsh
DSH_DIND_NAME=dsh-dind
# ── Опционально: память агентов OpenViking ─────────────────────────────────
# OPENVIKING_URL=https://ov.example
# OPENVIKING_API_KEY=
.gitignore
+5
View File
@@ -0,0 +1,5 @@
# локальные данные стенда и секреты — не коммитятся
.env
data/
workspace/
*.log
LICENSE
+26
View File
@@ -0,0 +1,26 @@
MIT License
Copyright (c) 2026 archeon / Pavel Zhukov (compose, scripts, docs)
Container source in `upstream/`: Copyright (c) 2026 yjrszcq — MIT
(https://github.com/yjrszcq/dsh-docker, см. upstream/LICENSE)
Данный репозиторий распространяется по лицензии MIT; вендоренная копия
`upstream/` сохраняет своё исходное уведомление MIT.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
README.md
+83
View File
@@ -0,0 +1,83 @@
# dsh (DeepSeek Harness) — сборка archeon
Самодостаточный репозиторий: Docker-образ [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness)
(`@deepseek-ai/dsh`) + **обновление кода dsh прямо из работающего контейнера**
(как плагины, без пересборки) + **path/user-агностичный docker-compose** как
пример для разработчиков.
## Что внутри
| Путь | Что это |
|---|---|
| `upstream/` | вендоренный исходник Docker-образа ([yjrszcq/dsh-docker](https://github.com/yjrszcq/dsh-docker), MIT; см. `upstream/PROVENANCE.md`) |
| `docker-compose.yml` | path/user-агностичный стенд: dsh + вложенный dind + воркспейс |
| `.env.example` | все настройки в одном месте (пути, порты, ключи) |
| `scripts/update-dsh.sh` | **обновление dsh изнутри контейнера** до npm latest (пробация + автокат) |
| `scripts/update-plugins.sh` | установка/обновление плагинов профиля web + фикс code-server |
| `scripts/build-image.sh` | пересборка образа под конкретную версию dsh |
| `docs/UPDATE-MECHANISM.md` | как работает обновление изнутри (анатомия платформы) |
## Быстрый старт
```sh
cp .env.example .env # вписать DEEPSEEK_API_KEY
docker compose -p dsh up -d
docker compose -p dsh exec dsh dsh-platform access generate-key # одноразовый ключ админа
xdg-open http://127.0.0.1:3080
```
Все пути по умолчанию относительные: данные — `./data/`, проекты — `./workspace/`
(рядом с compose-файлом). Перенос на другой хост/пользователя: `cp -r` всего
каталога (или переопределите `DSH_DATA_DIR`/`DSH_WORKSPACE_DIR` в `.env`).
Контейнер работает под uid 1000; если ваш uid другой:
`docker compose -p dsh --profile fix-perms run --rm fixperms`.
## Обновление dsh «как плагины» (главная фича)
```sh
scripts/update-dsh.sh # до npm latest: download → probation → commit
scripts/update-dsh.sh check # только посмотреть installed / available
```
- Ставится в персистентный volume `/data/platform` → переживает `down/up` и
пересоздание контейнера.
- **Важно**: механизм обновления живёт в stage0/management из **образа**, поэтому
работает только у живого контейнера (обновлять «выключенный» контейнер нельзя —
сначала `up -d`). При переходе на существенно новый образ заново проверьте
патчи окружения (`upstream/container/environment/resources/patches/`) против
новой версии dsh: образ собирается с dev-патчами, которые якорятся на
внутренности `bin.js`/`profile-boot` (в upstream пин версий стоит именно
поэтому). Наши патчи толерантны к 0.1.x и 0.2.x — проверены на обеих ветках.
- Experimental-канал платформы качает **официальный tarball с npm** (canonical
URL, sha256, без редиректов) и активирует новую версию с **пробацией**
(`DSH_EXPERIMENTAL_PROBATION_SECONDS`, по умолчанию 120 c) и **автооткатом**
при провале здоровья.
- Тот же механизм в UI: Management Console `/_dsh_platform/console/`
(Update → channel experimental → Update).
- Плагины профиля web живут в `/data/dsh` и обновляются отдельно
(`scripts/update-plugins.sh`).
### Пересборка образа под конкретную версию
Community-пин `latestSupportedDsh` (в upstream-исходнике = `0.1.5-rc.2`) в
dev-режиме сборки не проверяется — версию можно задать напрямую:
```sh
scripts/build-image.sh 0.2.0-rc.2 --push # тегирует <repo>:<версия> + :latest и пушит
```
## Почему не официальный образ
Официального образа у DeepSeek Harness нет; community-образ `szcq/deepseek-harness`
застрял на 0.1.5-rc.2 (пин автора). Этот репозиторий собирает свой образ из
того же исходника с актуальной версией и держит механизм обновления изнутри,
не дожидаясь пересборки.
## Изоляция песочниц агента
Контейнеры, которые запускает агент dsh, живут во **вложенном dind**
(`DOCKER_HOST=tcp://dind:2375`) — отдельный демон, отдельные образы/тома
(`./data/dind/`). Каталог проектов смонтирован в dind под тем же путём
`/workspace`, поэтому агент может бинд-маунтить свои песочницы на файлы
воркспейса. Порт web UI по умолчанию слушает только `127.0.0.1` — для доступа
из LAN настройте `DSH_LISTEN_ADDRESS` + `DSH_TRUSTED_HOSTS`.
docker-compose.yml
+105
View File
@@ -0,0 +1,105 @@
# ════════════════════════════════════════════════════════════════════════════
# DeepSeek Harness (dsh) + вложенный dind для песочниц агента
# ════════════════════════════════════════════════════════════════════════════
# Path/user-агностично: ВСЕ пути хоста — относительные (рядом с этим файлом)
# и переопределяются через .env. Работает у любого пользователя на любом хосте.
#
# Быстрый старт:
# cp .env.example .env # и вписать DEEPSEEK_API_KEY
# docker compose -p dsh up -d
# docker compose -p dsh exec dsh dsh-platform access generate-key # ключ первого входа
# UI: http://127.0.0.1:${DSH_PORT:-3080}
#
# Обновление dsh изнутри контейнера (как плагины):
# scripts/update-dsh.sh # до npm latest, с пробацией и откатом
# Management Console: /_dsh_platform/console/
#
# Права на каталогах данных: контейнер работает под uid 1000 (node).
# Если ваш uid ≠ 1000: docker compose -p dsh --profile fix-perms run --rm fixperms
# ════════════════════════════════════════════════════════════════════════════
services:
dsh:
image: ${DSH_IMAGE:-gitea.archeon.io/archeon/dsh:latest}
# build: альтернатива — собрать прямо из этого репо:
# docker build --build-arg DSH_VERSION=<версия|latest> -t dsh-local upstream/
# # затем DSH_IMAGE=dsh-local в .env
container_name: ${DSH_CONTAINER_NAME:-dsh}
restart: unless-stopped
depends_on:
dind:
condition: service_started
ports:
- "${DSH_LISTEN_ADDRESS:-127.0.0.1}:${DSH_PORT:-3080}:3080" # web UI + management console
- "${DSH_LISTEN_ADDRESS:-127.0.0.1}:${DSH_EDITOR_PORT:-18500}:18500" # code-server редактора
environment:
DEEPSEEK_API_KEY: ${DEEPSEEK_API_KEY:-}
DSH_HOME: /data/dsh
DSH_PLATFORM_DATA: /data/platform
DSH_DEFAULT_WORKSPACE: ${DSH_DEFAULT_WORKSPACE:-/workspace}
DSH_TRUSTED_HOSTS: ${DSH_TRUSTED_HOSTS:-}
DSH_TELEMETRY_DISABLED: ${DSH_TELEMETRY_DISABLED:-true}
DSH_EXPERIMENTAL_PROBATION_SECONDS: ${DSH_EXPERIMENTAL_PROBATION_SECONDS:-120}
DOCKER_HOST: tcp://dind:2375 # песочницы агента — во вложенный dind
# Опциональная память агентов (раскомментировать при необходимости):
# OPENVIKING_URL: ${OPENVIKING_URL}
# OPENVIKING_API_KEY: ${OPENVIKING_API_KEY}
group_add:
- "dsh-sudo-${DSH_SUDO_ENABLED:-false}"
volumes:
# Данные — относительные каталоги рядом с compose-файлом (переопределяются .env):
- ${DSH_DATA_DIR:-./data}/home:/home/node # HOME: .dsh-editor, кэши, user-data
- ${DSH_DATA_DIR:-./data}/dsh:/data/dsh # DSH_HOME: профили, ПЛАГИНЫ, settings.yaml
- ${DSH_DATA_DIR:-./data}/platform:/data/platform # состояние платформы: ОБНОВЛЕНИЯ, откаты
- ${DSH_WORKSPACE_DIR:-./workspace}:/workspace # проекты (= DSH_DEFAULT_WORKSPACE)
# Глобальные правила агента (опционально): раскомментировать и положить AGENTS.md
# рядом с compose-файлом:
# - ./AGENTS.md:/data/dsh/AGENTS.md:ro
networks: [dshnet]
healthcheck:
test: ["CMD", "curl", "--fail", "--silent", "--noproxy", "*", "http://127.0.0.1:3080/_dsh_gateway/health"]
interval: 30s
timeout: 5s
start_period: 60s
retries: 3
# Вложенный docker-демон: контейнеры, которые запускает агент dsh.
dind:
image: docker:29-dind
container_name: ${DSH_DIND_NAME:-dsh-dind}
restart: unless-stopped
privileged: true
environment:
DOCKER_TLS_CERTDIR: "" # plain tcp://2375 — только внутри сети dshnet
command:
- --host=tcp://0.0.0.0:2375
- --default-address-pool=base=10.200.0.0/16,size=24
volumes:
- ${DSH_DATA_DIR:-./data}/dind:/var/lib/docker # образы/тома песочниц — персистентны
- ${DSH_WORKSPACE_DIR:-./workspace}:/workspace # ТЕ ЖЕ проекты под ТЕМ ЖЕ путём:
# агент монтирует /workspace/... в песочницы
networks: [dshnet]
# Разовый помощник: чинит владельца каталогов данных под uid 1000 (контейнерный node).
# Запуск: docker compose -p dsh --profile fix-perms run --rm fixperms
fixperms:
image: alpine:3.20
profiles: ["fix-perms"]
user: "0:0"
entrypoint: ["sh", "-c"]
command:
- |
echo "Fixing ownership of data dirs to uid 1000 (container user 'node')...";
chown -R 1000:1000 /fix/home /fix/dsh /fix/platform /fix/workspace /fix/dind 2>/dev/null || true;
chgrp -R 1000 /fix/home /fix/dsh /fix/platform /fix/workspace /fix/dind 2>/dev/null || true;
echo "Done."
volumes:
- ${DSH_DATA_DIR:-./data}/home:/fix/home
- ${DSH_DATA_DIR:-./data}/dsh:/fix/dsh
- ${DSH_DATA_DIR:-./data}/platform:/fix/platform
- ${DSH_DATA_DIR:-./data}/dind:/fix/dind
- ${DSH_WORKSPACE_DIR:-./workspace}:/fix/workspace
networks:
dshnet:
driver: bridge
docs/UPDATE-MECHANISM.md
+74
View File
@@ -0,0 +1,74 @@
# Анатомия обновления dsh изнутри контейнера
Зачем: у `dsh` CLI нет команды update, а community-образ пересобирает автор по
своему пину (это и есть причина «показывает 0.1.5, когда на npm уже 0.2.x»).
Контейнер платформы решает это собственным механизмом обновления — им и пользуемся.
## Кто есть кто в контейнере
- **stage0** (PID 1, root) — супервизор платформы: trust-граница, bootstrap,
management-сервис, gateway; живёт в образе (`/opt/dsh-platform/runtime`).
- **dsh runtime** — сам агент; работает под `node` (uid 1000) через
`/run/dsh-platform/views/runtime/bin/dsh` (view поверх управляемого деплоя).
- `/usr/local/bin/dsh` и `/usr/local/bin/dsh-platform` — **шимы** (не бинарники):
root-вызовы дропают привилегии в node и пробрасывают переменные окружения.
- **management console** (`/_dsh_platform/console/`) + CLI `dsh-platform` —
управление: статус, каналы, update, rollback, логи.
## Где живут данные (и почему обновление персистентно)
| Путь контейнера | Что там | Volume |
|---|---|---|
| `/data/dsh` | DSH_HOME: профили, **плагины**, settings.yaml, сессии | да |
| `/data/platform` | **деплой платформы: runtime-слоты, pristine-копии dsh, snapshots, журнал обновлений** | да |
| `/home/node` | HOME: `.dsh-editor` (code-server), кэши | да |
| `/opt/dsh-platform` | код платформы из образа | нет (образ) |
Experimental-обновление активирует новую версию dsh **в `/data/platform`** →
переживает пересоздание контейнера. Откат — тоже (предыдущий runtime-слот).
## Два канала обновления
- **stable** — подписанные манифесты автора образа (release-channel на GitHub).
Качает bootstrap+environment+dsh. Работает только когда автор выпустил релиз
(для 0.2.x его ещё нет — и потому stable-канал нам не подходит).
- **experimental** — качает **напрямую с npm** (`OfficialDshDownloader`:
canonical `https://registry.npmjs.org/@deepseek-ai/dsh/-/dsh-<ver>.tgz`,
запрет редиректов, контроль sha256/размера, лимиты), поверх текущего
bootstrap/environment. Пин `latestSupportedDsh` здесь не блокирует: он
автовыдвигается (`advanceSupportedDsh`), а gate для experimental — только
«версия больше текущей» (`NpmRegistryClient.latest`).
## Командный путь обновления (что делает `scripts/update-dsh.sh`)
1. `dsh-platform channel experimental` — переключить канал (PUT `/channel`).
2. `dsh-platform update --wait` — coordinator: скачать tarball с npm →
«candidate deployment» → пауза dsh-runtime → активация → resume.
3. Пробация `DSH_EXPERIMENTAL_PROBATION_SECONDS` (по умолчанию 120 c):
платформа мониторит здоровье dsh; провал → **автооткат** к предыдущему
деплою; успех → candidate коммитится (`deployments/candidate/commit`).
4. Проверка: `dsh --version` == npm latest.
Автопроверка обновлений платформой: раз в ~6 ч (21600 c), отключается/настраивается
в Management Console (Automatic checks).
## Если что-то пошло не так
```sh
docker exec dsh dsh-platform status
docker exec dsh-platform logs --source updater --limit 100
docker exec dsh dsh-platform rollback # вернуться к предыдущему деплою
```
Coordinator сам восстанавливается после прерванного обновления (journal +
snapshots + `recover`). Кнопка «return stable» в консоли откатывает и
данные-снапшот (требует интерактивного подтверждения — это осознанная потеря
данных новее снапшота).
## Известные ограничения
- Experimental ставит **только npm dist-tag `latest`** (политика платформы).
Конкретная версия — путь `scripts/build-image.sh <version>` + новая `DSH_IMAGE`.
- Плагины и профили (`/data/dsh`) обновлением dsh не трогаются; при мажорном
скачке возможна несовместимость плагинов — лечение: `dsh plugin --profile web update`.
- Stage0/bootstrap/environment остаются от образа; обновляется только слой dsh.
scripts/build-image.sh
+41
View File
@@ -0,0 +1,41 @@
#!/bin/sh
# ════════════════════════════════════════════════════════════════════════════
# Сборка образа dsh из вендоренного исходника (upstream/) в локальный Docker.
# Пин сообщества (latestSupportedDsh) в dev-режиме сборки не действует — версию
# задаём явно. Результат тегируется в реестр archeon.
#
# Использование:
# scripts/build-image.sh # версия = npm dist-tag latest
# scripts/build-image.sh 0.2.0-rc.2 # конкретная версия
# scripts/build-image.sh 0.2.0-rc.2 --push # + docker push в реестр
# ════════════════════════════════════════════════════════════════════════════
set -eu
cd "$(dirname "$0")/.."
VERSION=${1:-}
PUSH=0
[ "${2:-}" = --push ] && PUSH=1
if [ -z "$VERSION" ] || [ "$VERSION" = latest ]; then
VERSION=$(curl -fsS https://registry.npmjs.org/-/package/@deepseek-ai%2Fdsh/dist-tags | sed -n 's/.*"latest":"\([^"]*\)".*/\1/p')
[ -n "$VERSION" ] || { echo "не удалось определить npm latest" >&2; exit 1; }
fi
IMAGE=${DSH_IMAGE:-gitea.archeon.io/archeon/dsh}
REPO=${IMAGE%%:*}
echo "=== Сборка $REPO:$VERSION (DSH_VERSION=$VERSION)"
DOCKER_BUILDKIT=1 docker build \
--build-arg DSH_VERSION="$VERSION" \
-t "$REPO:$VERSION" \
-t "$REPO:latest" \
upstream/
echo "=== Проверка: версия внутри образа"
docker run --rm --entrypoint /bin/sh "$REPO:$VERSION" \
-c "grep -o '\"version\":\"[^\"]*\"' /opt/dsh-platform/seed/pristine/*/package.json | head -1"
if [ "$PUSH" = 1 ]; then
echo "=== Push в реестр"
docker push "$REPO:$VERSION"
docker push "$REPO:latest"
fi
echo "Готово: $REPO:$VERSION (+ latest). Обновить стенд: docker compose -p dsh pull && docker compose -p dsh up -d"
scripts/update-dsh.sh
+77
View File
@@ -0,0 +1,77 @@
#!/bin/sh
# ════════════════════════════════════════════════════════════════════════════
# Обновление dsh внутри работающего контейнера — «как плагины», без пересборки.
#
# Механизм: management console платформы, канал experimental — ставит npm-версию
# @deepseek-ai/dsh в /data/platform ( персистентный volume), с пробацией и
# автоматическим откатом при неудаче. Код dsh переживает пересоздание контейнера.
#
# Использование:
# scripts/update-dsh.sh # до npm dist-tag latest
# scripts/update-dsh.sh check # только показать текущую/доступную версии
# scripts/update-dsh.sh --container X # другое имя контейнера (по умолчанию dsh)
#
# Примечания:
# - experimental ставит ТОЛЬКО npm dist-tag latest (вверх). Конкретная версия —
# scripts/build-image.sh <версия> + смена DSH_IMAGE в .env.
# - Плагины профиля web (/data/dsh) не трогаются и переживают обновление.
# ════════════════════════════════════════════════════════════════════════════
set -eu
CONTAINER=dsh
ACTION=update
while [ $# -gt 0 ]; do
case "$1" in
--container) CONTAINER="$2"; shift 2 ;;
check) ACTION=check; shift ;;
*) echo "unknown arg: $1" >&2; exit 64 ;;
esac
done
say() { printf '\n=== %s\n' "$*"; }
docker inspect -f '{{.State.Running}}' "$CONTAINER" >/dev/null 2>&1 || {
echo "Контейнер '$CONTAINER' не запущен. Сначала: docker compose -p dsh up -d" >&2; exit 1; }
say "Текущая версия"
CURRENT=$(docker exec "$CONTAINER" dsh --version)
echo "installed: $CURRENT"
say "Доступно на npm"
TARGET=$(docker exec "$CONTAINER" node -e "fetch('https://registry.npmjs.org/-/package/@deepseek-ai%2Fdsh/dist-tags').then(r=>r.json()).then(d=>console.log(d.latest)).catch(e=>{console.error(e.message);process.exit(1)})")
echo "npm latest: $TARGET"
[ "$ACTION" = check ] && exit 0
if [ "$CURRENT" = "$TARGET" ]; then
echo "Уже актуально ($CURRENT). Выход."; exit 0
fi
say "Переключаю канал на experimental"
docker exec "$CONTAINER" dsh-platform channel experimental
say "Запускаю обновление (скачивание+установка+рестарт рантайма; это минуты)"
if ! docker exec "$CONTAINER" dsh-platform update --wait; then
echo "ОШИБКА: обновление не прошло. Диагностика:" >&2
docker exec "$CONTAINER" dsh-platform status || true
docker exec "$CONTAINER" dsh-platform logs --source updater --limit 50 || true
echo "Платформа обычно сама откатилась к $CURRENT. Повтор: scripts/update-dsh.sh" >&2
exit 1
fi
PROBATION=$(docker exec "$CONTAINER" printenv DSH_EXPERIMENTAL_PROBATION_SECONDS || echo 120)
say "Пробация ${PROBATION}s — жду автоподтверждение"
WAIT=$((PROBATION + 180)); I=0
while [ "$I" -lt "$WAIT" ]; do
NEW=$(docker exec "$CONTAINER" dsh --version 2>/dev/null || echo '?')
if [ "$NEW" = "$TARGET" ]; then
say "Готово: dsh $NEW (было $CURRENT)"
docker exec "$CONTAINER" dsh-platform status || true
exit 0
fi
sleep 15; I=$((I + 15))
done
echo "Таймаут ожидания. Текущая версия: $(docker exec "$CONTAINER" dsh --version 2>/dev/null || echo '?')." >&2
echo "Проверьте вручную: docker exec $CONTAINER dsh-platform status" >&2
exit 1
scripts/update-plugins.sh
+41
View File
@@ -0,0 +1,41 @@
#!/bin/sh
# ════════════════════════════════════════════════════════════════════════════
# Обновление плагинов профиля web (и фикс bind-адреса code-server после этого).
# CLI-апдейт dsh НЕ трогает плагины — они обновляются отдельно.
#
# Использование:
# scripts/update-plugins.sh [--add github:<owner>/<repo>] [--no-restart]
# ════════════════════════════════════════════════════════════════════════════
set -eu
CONTAINER=${DSH_CONTAINER_NAME:-dsh}
ADD=""
RESTART=1
while [ $# -gt 0 ]; do
case "$1" in
--add) ADD="$2"; shift 2 ;;
--no-restart) RESTART=0; shift ;;
*) echo "unknown arg: $1" >&2; exit 64 ;;
esac
done
docker inspect -f '{{.State.Running}}' "$CONTAINER" >/dev/null 2>&1 || {
echo "Контейнер '$CONTAINER' не запущен" >&2; exit 1; }
# Внутренний редирект: без него docker exec может виснуть на фоновом потомке pnpm.
if [ -n "$ADD" ]; then
echo "=== Установка плагина: $ADD"
docker exec "$CONTAINER" sh -c "dsh plugin --profile web add '$ADD' >/tmp/dsh-plugin.log 2>&1; tail -5 /tmp/dsh-plugin.log"
fi
echo "=== Обновление всех плагинов профиля web"
docker exec "$CONTAINER" sh -c "dsh plugin --profile web update >/tmp/dsh-plugin.log 2>&1; tail -20 /tmp/dsh-plugin.log"
echo "=== Фикс bind-адреса code-server (127.0.0.1 -> 0.0.0.0), иначе порт 18500 снаружи refused"
docker exec "$CONTAINER" sh -c \
"sed -i \"s/'--bind-addr', '127.0.0.1:' + port/'--bind-addr', '0.0.0.0:' + port/\" /data/dsh/profiles/web/node_modules/dsh-vsceditor/lib/host.js 2>/dev/null || echo '(dsh-vsceditor не установлен — пропуск)'"
if [ "$RESTART" = 1 ]; then
echo "=== Рестарт контейнера (pnpm пересобрал граф node_modules под живой процесс)"
docker restart "$CONTAINER"
echo "ОК. Дождитесь healthy: docker inspect -f '{{.State.Health.Status}}' $CONTAINER"
fi
upstream/.dockerignore
+16
View File
@@ -0,0 +1,16 @@
.git
.github
.checkpoint-karpathy
.env
temp
data
**/node_modules
container/**/test
container/**/test/**
container/**/tests
container/**/tests/**
container/test
workspace
README.md
README_CN.md
docker-compose.yaml
upstream/.env.example
+33
View File
@@ -0,0 +1,33 @@
# Docker image tag
DSH_IMAGE_TAG=latest
# Host port binding
DSH_LISTEN_ADDRESS=127.0.0.1
DSH_PORT=3080
# Local directory mounted as /workspace
DSH_WORKSPACE=./workspace
# Set true only to grant DSH and its Agent unrestricted passwordless root access
DSH_SUDO_ENABLED=false
# Initial workspace directory shown inside the container
DSH_DEFAULT_WORKSPACE=/workspace
# true disables telemetry; false enables it
DSH_TELEMETRY_DISABLED=true
# Inject a crypto.randomUUID compatibility shim when the browser lacks it
DSH_PROXY_POLYFILL=true
# Required for access through LAN IPs or domains. Use comma-separated host[:port]
# values without schemes or paths. Empty accepts loopback Hosts only; * accepts any Host.
DSH_TRUSTED_HOSTS=
# Platform update activation
DSH_ACTIVATION_TIMEOUT_SECONDS=60
DSH_EXPERIMENTAL_PROBATION_SECONDS=120
# Unified platform JSONL log retention
DSH_LOG_MAX_BYTES=104857600
DSH_LOG_RETENTION_DAYS=14
upstream/.github/workflows/docker.yaml
Vendored
+331
View File
@@ -0,0 +1,331 @@
name: Publish Container Images
on:
workflow_run:
workflows:
- Publish Supported Platform Target
types:
- completed
workflow_dispatch:
inputs:
source_commit:
description: Source commit to build
required: true
default: main
type: string
channel_commit:
description: release-channel ref to search for the matching signed target
required: false
default: release-channel
type: string
concurrency:
group: production-container-images
cancel-in-progress: false
env:
DOCKERHUB_IMAGE: szcq/deepseek-harness
GHCR_IMAGE: ghcr.io/${{ github.repository }}
jobs:
build:
name: Verify, test, and publish images
if: >-
(github.event_name == 'workflow_run' &&
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.head_branch == 'main') ||
github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
timeout-minutes: 90
environment: production-image
permissions:
contents: read
packages: write
steps:
- name: Checkout reviewed source
uses: actions/checkout@v6
with:
ref: ${{ github.event_name == 'workflow_run' && github.event.workflow_run.head_sha || inputs.source_commit }}
path: source
- name: Checkout signed channel history
uses: actions/checkout@v6
with:
ref: ${{ github.event_name == 'workflow_dispatch' && inputs.channel_commit || 'release-channel' }}
fetch-depth: 0
path: channel
- name: Resolve immutable signed channel state
id: channel
run: |
source_commit="$(git -C source rev-parse HEAD)"
channel_commit=
while IFS= read -r candidate; do
candidate_source="$(git -C channel show "${candidate}:channel.json" | jq -er .sourceCommit)"
if test "$candidate_source" = "$source_commit"; then
channel_commit="$candidate"
break
fi
done < <(git -C channel log --format=%H -- channel.json)
if test -z "$channel_commit"; then
echo "No signed release-channel target matches source commit $source_commit" >&2
exit 1
fi
git -C channel checkout --detach "$channel_commit"
{
printf 'commit=%s\n' "$channel_commit"
printf 'source_commit=%s\n' "$source_commit"
} >> "$GITHUB_OUTPUT"
# shellcheck disable=SC2016
printf 'Using signed channel commit `%s` for source commit `%s`.\n' \
"$channel_commit" "$source_commit" >> "$GITHUB_STEP_SUMMARY"
- name: Set up Node.js
uses: actions/setup-node@v6
with:
node-version: 24
cache: npm
cache-dependency-path: |
source/container/control-plane/services/management/package-lock.json
source/container/environment/resources/plugins/platform-management/package/package-lock.json
- name: Install Management runtime dependencies
working-directory: source
run: |
npm ci --omit=dev --ignore-scripts --no-audit --no-fund --prefix container/control-plane/services/management
npm ci --omit=dev --ignore-scripts --no-audit --no-fund --legacy-peer-deps --prefix container/environment/resources/plugins/platform-management/package
- name: Verify gateway and Compose configuration
working-directory: source
run: |
npm test --prefix container/control-plane/services/gateway
npm test --prefix container/platform
node container/test/compose-config.mjs
sh -n container/test/container-smoke.sh
sh -n container/test/devtools-smoke.sh
- name: Verify signed Supported Target
id: signed-target
working-directory: source
env:
RECOVERY_ROOT_PUBLIC_KEY: ${{ secrets.DSH_RECOVERY_ROOT_PUBLIC_KEY }}
run: |
test -n "$RECOVERY_ROOT_PUBLIC_KEY"
source_commit="$(git rev-parse HEAD)"
test "$(jq -er .sourceCommit ../channel/channel.json)" = "$source_commit"
sequence="$(jq -er .targetSequence ../channel/stable.json)"
test "$(jq -er .targetSequence ../channel/channel.json)" = "$sequence"
release_dir="../channel/targets/$sequence"
test -d "$release_dir"
for name in stable.json stable.sig.json keyring.json keyring.sig.json; do
cmp "../channel/$name" "$release_dir/$name"
done
image_input=container/platform/image-input
mkdir -p "$image_input/release"
cp -a "$release_dir/." "$image_input/release/"
printf '%s\n' "$RECOVERY_ROOT_PUBLIC_KEY" > "$image_input/recovery-root.spki.base64"
cp release/supported-target.json "$image_input/supported-target.json"
cp container/environment/definition.json "$image_input/environment-definition.json"
version="$(jq -er '.latestSupportedDsh' release/supported-target.json)"
npm_input="$RUNNER_TEMP/image-npm-input"
mkdir -p "$npm_input"
npm pack "@deepseek-ai/dsh@${version}" --pack-destination "$npm_input" >/dev/null
tarball="$(find "$npm_input" -maxdepth 1 -type f -name '*.tgz' -print -quit)"
test -n "$tarball"
cp "$tarball" "$image_input/dsh.tgz"
verified="$(node container/platform/tools/verify-image-release.mjs \
"$image_input/release" "$image_input/recovery-root.spki.base64" "$image_input/dsh.tgz" \
"$image_input/supported-target.json" "$image_input/environment-definition.json")"
test "$(jq -er .dshVersion <<<"$verified")" = "$version"
test "$(jq -er .targetSequence <<<"$verified")" = "$sequence"
environment_version="$(jq -er .environmentVersion <<<"$verified")"
test "$environment_version" = "$(jq -er .version container/environment/definition.json)"
{
printf 'version=%s\n' "$version"
printf 'environment_version=%s\n' "$environment_version"
printf 'sequence=%s\n' "$sequence"
} >> "$GITHUB_OUTPUT"
cp "$image_input/recovery-root.spki.base64" container/platform/seed/trust/recovery-root.spki.base64
cp "$release_dir/keyring.json" container/platform/seed/trust/keyring.json
cp "$release_dir/keyring.sig.json" container/platform/seed/trust/keyring.sig.json
rm container/platform/seed/trust/DEVELOPMENT_FIXTURE
test ! -e container/platform/seed/trust/DEVELOPMENT_FIXTURE
printf 'Building signed DSH %s with Environment %s (sequence %s).\n' \
"$version" "$environment_version" "$sequence" >> "$GITHUB_STEP_SUMMARY"
- name: Generate registry tags
id: tags
env:
DSH_VERSION: ${{ steps.signed-target.outputs.version }}
ENVIRONMENT_VERSION: ${{ steps.signed-target.outputs.environment_version }}
run: |
[[ "$ENVIRONMENT_VERSION" =~ ^([0-9]+)\.([0-9]+)\.([0-9]+)$ ]]
major="${BASH_REMATCH[1]}"
minor="${BASH_REMATCH[2]}"
{
echo 'standard<<EOF'
echo "${DOCKERHUB_IMAGE}:${DSH_VERSION}"
echo "${GHCR_IMAGE}:latest"
echo "${GHCR_IMAGE}:${ENVIRONMENT_VERSION}"
echo "${GHCR_IMAGE}:${major}.${minor}"
echo "${GHCR_IMAGE}:${major}"
echo "${GHCR_IMAGE}:dsh-${DSH_VERSION}"
echo 'EOF'
echo 'devtools<<EOF'
echo "${DOCKERHUB_IMAGE}:${DSH_VERSION}-devtools"
echo 'EOF'
} >> "$GITHUB_OUTPUT"
- name: Set up QEMU
uses: docker/setup-qemu-action@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Build standard smoke-test image
uses: docker/build-push-action@v7
with:
context: ./source
file: ./source/Dockerfile
platforms: linux/amd64
load: true
provenance: false
build-args: |
SIGNED_IMAGE_INPUT=true
DSH_VERSION=${{ steps.signed-target.outputs.version }}
ENVIRONMENT_VERSION=${{ steps.signed-target.outputs.environment_version }}
TARGET_SEQUENCE=${{ steps.signed-target.outputs.sequence }}
PLATFORM_REVISION=${{ steps.channel.outputs.source_commit }}
tags: dsh-gateway-smoke:${{ github.run_id }}
cache-from: type=gha,scope=standard
cache-to: type=gha,mode=max,scope=standard
- name: Run standard container smoke checks
working-directory: source
run: container/test/container-smoke.sh "dsh-gateway-smoke:${{ github.run_id }}"
- name: Build devtools smoke-test image
uses: docker/build-push-action@v7
with:
context: ./source
file: ./source/Dockerfile
platforms: linux/amd64
load: true
provenance: false
build-args: |
SIGNED_IMAGE_INPUT=true
DSH_VERSION=${{ steps.signed-target.outputs.version }}
ENVIRONMENT_VERSION=${{ steps.signed-target.outputs.environment_version }}
TARGET_SEQUENCE=${{ steps.signed-target.outputs.sequence }}
PLATFORM_REVISION=${{ steps.channel.outputs.source_commit }}
INSTALL_DEVTOOLS=true
tags: dsh-gateway-devtools-smoke:${{ github.run_id }}
cache-from: type=gha,scope=devtools
cache-to: type=gha,mode=max,scope=devtools
- name: Run devtools smoke checks
working-directory: source
run: container/test/devtools-smoke.sh "dsh-gateway-devtools-smoke:${{ github.run_id }}"
- name: Log in to Docker Hub
uses: docker/login-action@v4
with:
username: szcq
password: ${{ secrets.DOCKER_TOKEN }}
- name: Log in to GitHub Container Registry
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ github.token }}
- name: Push versioned devtools image to Docker Hub
uses: docker/build-push-action@v7
with:
context: ./source
file: ./source/Dockerfile
platforms: linux/amd64,linux/arm64
push: true
provenance: false
build-args: |
SIGNED_IMAGE_INPUT=true
DSH_VERSION=${{ steps.signed-target.outputs.version }}
ENVIRONMENT_VERSION=${{ steps.signed-target.outputs.environment_version }}
TARGET_SEQUENCE=${{ steps.signed-target.outputs.sequence }}
PLATFORM_REVISION=${{ steps.channel.outputs.source_commit }}
INSTALL_DEVTOOLS=true
tags: ${{ steps.tags.outputs.devtools }}
cache-from: type=gha,scope=devtools
cache-to: type=gha,mode=max,scope=devtools
- name: Push versioned standard image to Docker Hub and GHCR
uses: docker/build-push-action@v7
with:
context: ./source
file: ./source/Dockerfile
platforms: linux/amd64,linux/arm64
push: true
provenance: false
build-args: |
SIGNED_IMAGE_INPUT=true
DSH_VERSION=${{ steps.signed-target.outputs.version }}
ENVIRONMENT_VERSION=${{ steps.signed-target.outputs.environment_version }}
TARGET_SEQUENCE=${{ steps.signed-target.outputs.sequence }}
PLATFORM_REVISION=${{ steps.channel.outputs.source_commit }}
tags: ${{ steps.tags.outputs.standard }}
cache-from: type=gha,scope=standard
cache-to: type=gha,mode=max,scope=standard
- name: Promote Docker Hub rolling tags
env:
DSH_VERSION: ${{ steps.signed-target.outputs.version }}
run: |
docker buildx imagetools create \
--tag "${DOCKERHUB_IMAGE}:latest-devtools" \
"${DOCKERHUB_IMAGE}:${DSH_VERSION}-devtools"
docker buildx imagetools create \
--tag "${DOCKERHUB_IMAGE}:latest" \
"${DOCKERHUB_IMAGE}:${DSH_VERSION}"
- name: Verify published registry manifests
env:
DSH_VERSION: ${{ steps.signed-target.outputs.version }}
ENVIRONMENT_VERSION: ${{ steps.signed-target.outputs.environment_version }}
run: |
docker buildx imagetools inspect "${DOCKERHUB_IMAGE}:${DSH_VERSION}"
docker buildx imagetools inspect "${DOCKERHUB_IMAGE}:${DSH_VERSION}-devtools"
docker buildx imagetools inspect "${DOCKERHUB_IMAGE}:latest"
docker buildx imagetools inspect "${DOCKERHUB_IMAGE}:latest-devtools"
docker buildx imagetools inspect "${GHCR_IMAGE}:${ENVIRONMENT_VERSION}"
docker buildx imagetools inspect "${GHCR_IMAGE}:dsh-${DSH_VERSION}"
outputs:
dsh_version: ${{ steps.signed-target.outputs.version }}
environment_version: ${{ steps.signed-target.outputs.environment_version }}
sequence: ${{ steps.signed-target.outputs.sequence }}
notify-result:
name: Notify container image publication result
needs: build
if: >-
always() &&
(needs.build.result == 'success' || needs.build.result == 'failure')
uses: yjrszcq/github-workflows/.github/workflows/gotify-notify.yml@c4c7fe9e17854cd4962913ac2792513eab0be988
with:
title: >-
${{
needs.build.result == 'success' &&
format('DSH {0} 镜像已发布', needs.build.outputs.dsh_version) ||
'DSH 镜像发布失败'
}}
message: >-
${{
needs.build.result == 'success' &&
format('DSH {0} 与 Environment {1} 的标准镜像已同步到 Docker Hub 和 GHCR，Devtools 镜像已发布到 Docker Hub；签名目标序列为 {2}。', needs.build.outputs.dsh_version, needs.build.outputs.environment_version, needs.build.outputs.sequence) ||
format('Docker Hub 或 GHCR 镜像的构建、测试、推送或验证失败。请查看 GitHub Actions：{0}/{1}/actions/runs/{2}', github.server_url, github.repository, github.run_id)
}}
priority: ${{ needs.build.result == 'failure' && 8 || 7 }}
secrets:
gotify_url: ${{ secrets.GOTIFY_URL }}
gotify_token: ${{ secrets.GOTIFY_TOKEN }}
upstream/.github/workflows/dsh-candidate-validation.yml
Vendored
+133
View File
@@ -0,0 +1,133 @@
name: DSH Candidate Validation
on:
workflow_call:
inputs:
ref:
description: Candidate commit to validate
required: true
type: string
workflow_dispatch:
inputs:
ref:
description: Branch, tag, or commit to validate (defaults to the selected workflow ref)
required: false
type: string
pull_request_target:
branches:
- main
- dev
paths:
- release/supported-target.json
- container/**
- Dockerfile
- docker-compose.yaml
- .github/workflows/dsh-candidate-validation.yml
concurrency:
group: dsh-candidate-${{ inputs.ref || github.event.pull_request.head.sha || github.sha }}
cancel-in-progress: true
permissions:
contents: read
jobs:
validate:
name: Build candidate with current Environment
if: >-
github.event_name != 'pull_request_target' ||
github.event.pull_request.head.ref != 'automation/dsh-upstream-candidate'
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- name: Checkout candidate
uses: actions/checkout@v6
with:
ref: ${{ inputs.ref || github.event.pull_request.head.sha || github.sha }}
persist-credentials: false
- name: Set up Node.js
uses: actions/setup-node@v6
with:
node-version: 24
cache: npm
cache-dependency-path: |
container/control-plane/services/management/package-lock.json
container/environment/resources/plugins/platform-management/package/package-lock.json
- name: Install Management runtime dependencies
run: |
npm ci --omit=dev --ignore-scripts --no-audit --no-fund --prefix container/control-plane/services/management
npm ci --omit=dev --ignore-scripts --no-audit --no-fund --legacy-peer-deps --prefix container/environment/resources/plugins/platform-management/package
- name: Resolve and verify candidate
id: candidate
run: |
result="$(node container/platform/tools/supported-target.mjs validate \
release/supported-target.json container/environment/definition.json)"
version="$(jq -er '.target.latestSupportedDsh' <<<"$result")"
environment="$(jq -er '.target.environment' <<<"$result")"
registry_version="$(npm view "@deepseek-ai/dsh@${version}" version --json | jq -er 'if type == "string" then . else error("expected one version") end')"
test "$registry_version" = "$version"
printf 'version=%s\n' "$version" >> "$GITHUB_OUTPUT"
printf 'environment=%s\n' "$environment" >> "$GITHUB_OUTPUT"
# shellcheck disable=SC2016
{
printf '### Candidate Runtime\n\n'
printf -- '- DSH: `%s`\n' "$version"
printf -- '- Environment: `%s`\n' "$environment"
} >> "$GITHUB_STEP_SUMMARY"
- name: Verify upstream npm Artifact integrity
env:
DSH_VERSION: ${{ steps.candidate.outputs.version }}
run: |
mkdir "$RUNNER_TEMP/dsh-package"
expected="$(npm view "@deepseek-ai/dsh@${DSH_VERSION}" dist.integrity --json | jq -er 'if type == "string" and startswith("sha512-") then . else error("missing SHA-512 integrity") end')"
npm pack "@deepseek-ai/dsh@${DSH_VERSION}" --pack-destination "$RUNNER_TEMP/dsh-package" >/dev/null
tarball="$(find "$RUNNER_TEMP/dsh-package" -maxdepth 1 -type f -name '*.tgz' -print -quit)"
test -n "$tarball"
actual="sha512-$(openssl dgst -sha512 -binary "$tarball" | base64 -w 0)"
test "$actual" = "$expected"
- name: Run static compatibility checks
run: |
npm test --prefix container/control-plane/services/gateway
npm test --prefix container/platform
node container/test/compose-config.mjs
sh -n container/test/container-smoke.sh
sh -n container/test/devtools-smoke.sh
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Build candidate smoke image
uses: docker/build-push-action@v7
with:
context: .
file: ./Dockerfile
load: true
platforms: linux/amd64
provenance: false
build-args: |
DSH_VERSION=${{ steps.candidate.outputs.version }}
tags: dsh-candidate:${{ github.run_id }}
- name: Run candidate container smoke checks
run: container/test/container-smoke.sh "dsh-candidate:${{ github.run_id }}"
- name: Build candidate devtools image
uses: docker/build-push-action@v7
with:
context: .
file: ./Dockerfile
load: true
platforms: linux/amd64
provenance: false
build-args: |
DSH_VERSION=${{ steps.candidate.outputs.version }}
INSTALL_DEVTOOLS=true
tags: dsh-candidate-devtools:${{ github.run_id }}
- name: Run candidate devtools smoke checks
run: container/test/devtools-smoke.sh "dsh-candidate-devtools:${{ github.run_id }}"
upstream/.github/workflows/dsh-upstream-update.yml
Vendored
+237
View File
@@ -0,0 +1,237 @@
name: DSH Upstream Update
on:
schedule:
- cron: "0 4 * * *"
workflow_dispatch:
concurrency:
group: dsh-upstream-update
cancel-in-progress: false
permissions:
contents: read
env:
BASE_BRANCH: dev
CANDIDATE_BRANCH: automation/dsh-upstream-candidate
DSH_PACKAGE: "@deepseek-ai/dsh"
SUPPORTED_TARGET: release/supported-target.json
ENVIRONMENT_DEFINITION: container/environment/definition.json
jobs:
discover:
name: Discover and propose upstream DSH
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
outputs:
changed: ${{ steps.candidate.outputs.changed }}
current: ${{ steps.candidate.outputs.current }}
upstream: ${{ steps.candidate.outputs.upstream }}
environment: ${{ steps.candidate.outputs.environment }}
pull_request_url: ${{ steps.propose.outputs.pull_request_url }}
pull_request_number: ${{ steps.propose.outputs.pull_request_number }}
head_sha: ${{ steps.propose.outputs.head_sha }}
steps:
- name: Checkout protected branch
uses: actions/checkout@v6
with:
ref: ${{ env.BASE_BRANCH }}
fetch-depth: 0
- name: Set up Node.js
uses: actions/setup-node@v6
with:
node-version: 24
- name: Resolve latest official DSH
id: upstream
run: |
npm view "${DSH_PACKAGE}@latest" version --json > "$RUNNER_TEMP/dsh-version.json"
version="$(jq -er 'if type == "string" then . else error("expected one version") end' "$RUNNER_TEMP/dsh-version.json")"
printf 'version=%s\n' "$version" >> "$GITHUB_OUTPUT"
- name: Advance candidate Supported Target
id: candidate
env:
GH_TOKEN: ${{ github.token }}
UPSTREAM_VERSION: ${{ steps.upstream.outputs.version }}
run: |
base_sha="$(git rev-parse HEAD)"
current="$(jq -er '.latestSupportedDsh' "$SUPPORTED_TARGET")"
trusted_platform="$RUNNER_TEMP/trusted-platform"
mkdir -p "$trusted_platform/tools"
cp -R container/platform/lib "$trusted_platform/lib"
cp container/platform/tools/supported-target.mjs "$trusted_platform/tools/supported-target.mjs"
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
candidate_remote_sha="$(git ls-remote --heads origin "$CANDIDATE_BRANCH" | cut -f 1)"
git switch --create "$CANDIDATE_BRANCH" "$base_sha"
result="$(node "$trusted_platform/tools/supported-target.mjs" advance \
"$SUPPORTED_TARGET" "$ENVIRONMENT_DEFINITION" "$UPSTREAM_VERSION")"
changed="$(jq -r 'if (.changed | type) == "boolean" then .changed else error("expected boolean changed") end' <<<"$result")"
environment="$(jq -er '.target.environment' <<<"$result")"
{
printf 'changed=%s\n' "$changed"
printf 'current=%s\n' "$current"
printf 'upstream=%s\n' "$UPSTREAM_VERSION"
printf 'environment=%s\n' "$environment"
printf 'candidate_remote_sha=%s\n' "$candidate_remote_sha"
} >> "$GITHUB_OUTPUT"
# shellcheck disable=SC2016
{
printf '### DSH upstream discovery\n\n'
printf -- '- Latest Supported: `%s`\n' "$current"
printf -- '- Latest Official: `%s`\n' "$UPSTREAM_VERSION"
printf -- '- Environment: `%s`\n' "$environment"
printf -- '- Candidate required: `%s`\n' "$changed"
} >> "$GITHUB_STEP_SUMMARY"
- name: Create or update candidate pull request
id: propose
if: steps.candidate.outputs.changed == 'true'
env:
GH_TOKEN: ${{ github.token }}
CURRENT_VERSION: ${{ steps.candidate.outputs.current }}
UPSTREAM_VERSION: ${{ steps.candidate.outputs.upstream }}
ENVIRONMENT_VERSION: ${{ steps.candidate.outputs.environment }}
run: |
git add -- "$SUPPORTED_TARGET"
git commit -m "chore(dsh): validate ${UPSTREAM_VERSION}"
candidate_remote_sha="${{ steps.candidate.outputs.candidate_remote_sha }}"
if [ -n "$candidate_remote_sha" ]; then
git push --force-with-lease="refs/heads/$CANDIDATE_BRANCH:$candidate_remote_sha" \
origin "HEAD:refs/heads/$CANDIDATE_BRANCH"
else
git push origin "HEAD:refs/heads/$CANDIDATE_BRANCH"
fi
body="$RUNNER_TEMP/dsh-candidate-pr.md"
# shellcheck disable=SC2016
{
printf '## DSH candidate validation\n\n'
printf '| State | Version |\n|---|---|\n'
printf '| Latest Supported | `%s` |\n' "$CURRENT_VERSION"
printf '| Latest Official candidate | `%s` |\n' "$UPSTREAM_VERSION"
printf '| Current Environment | `%s` |\n\n' "$ENVIRONMENT_VERSION"
printf 'This PR changes only the candidate Supported Target. Compatibility CI validates the new DSH with the current Environment. Merge into dev records the validated candidate; promotion from dev to main remains the production publication gate.\n'
} > "$body"
pr_number="$(gh pr list --state open --head "$CANDIDATE_BRANCH" --json number --jq '.[0].number // empty')"
if [ -z "$pr_number" ]; then
pr_url="$(gh pr create \
--base "$BASE_BRANCH" \
--head "$CANDIDATE_BRANCH" \
--title "chore(dsh): validate ${UPSTREAM_VERSION}" \
--body-file "$body")"
pr_number="$(gh pr view "$pr_url" --json number --jq .number)"
else
gh pr edit "$pr_number" \
--base "$BASE_BRANCH" \
--title "chore(dsh): validate ${UPSTREAM_VERSION}" \
--body-file "$body"
pr_url="$(gh pr view "$pr_number" --json url --jq .url)"
fi
{
printf 'pull_request_url=%s\n' "$pr_url"
printf 'pull_request_number=%s\n' "$pr_number"
printf 'head_sha=%s\n' "$(git rev-parse HEAD)"
} >> "$GITHUB_OUTPUT"
printf 'Candidate PR: %s\n' "$pr_url" >> "$GITHUB_STEP_SUMMARY"
validate-candidate:
name: Validate candidate Runtime
needs: discover
if: needs.discover.outputs.changed == 'true'
permissions:
contents: read
uses: ./.github/workflows/dsh-candidate-validation.yml
with:
ref: ${{ needs.discover.outputs.head_sha }}
report-validation:
name: Report candidate validation
needs: [discover, validate-candidate]
if: always() && needs.discover.outputs.changed == 'true'
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
statuses: write
steps:
- name: Publish commit status and PR report
env:
GH_TOKEN: ${{ github.token }}
HEAD_SHA: ${{ needs.discover.outputs.head_sha }}
PR_NUMBER: ${{ needs.discover.outputs.pull_request_number }}
VALIDATION_RESULT: ${{ needs.validate-candidate.result }}
UPSTREAM_VERSION: ${{ needs.discover.outputs.upstream }}
ENVIRONMENT_VERSION: ${{ needs.discover.outputs.environment }}
run: |
if [ "$VALIDATION_RESULT" = success ]; then
state=success
label=passed
description="Candidate Runtime validation passed"
else
state=failure
label=failed
description="Candidate Runtime validation failed"
fi
run_url="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
gh api --method POST "/repos/${GITHUB_REPOSITORY}/statuses/${HEAD_SHA}" \
-f state="$state" \
-f context='DSH candidate Runtime' \
-f description="$description" \
-f target_url="$run_url" >/dev/null
gh pr comment "$PR_NUMBER" \
--repo "$GITHUB_REPOSITORY" \
--body "DSH \`${UPSTREAM_VERSION}\` with Environment \`${ENVIRONMENT_VERSION}\` **${label}** compatibility validation. [Open workflow run](${run_url})."
printf 'Candidate validation %s: %s\n' "$label" "$run_url" >> "$GITHUB_STEP_SUMMARY"
notify-discovered:
name: Notify upstream discovery
needs: discover
if: needs.discover.outputs.changed == 'true'
uses: yjrszcq/github-workflows/.github/workflows/gotify-notify.yml@c4c7fe9e17854cd4962913ac2792513eab0be988
with:
title: 发现 DSH 上游新版本 ${{ needs.discover.outputs.upstream }}
message: >-
当前支持版本：${{ needs.discover.outputs.current }}；上游候选版本：${{ needs.discover.outputs.upstream }}。
候选 Environment：${{ needs.discover.outputs.environment }}。
候选 PR：${{ needs.discover.outputs.pull_request_url }}
priority: 6
secrets:
gotify_url: ${{ secrets.GOTIFY_URL }}
gotify_token: ${{ secrets.GOTIFY_TOKEN }}
notify-result:
name: Notify upstream check result
needs: [discover, validate-candidate]
if: >-
always() &&
(needs.discover.result == 'failure' || needs.discover.outputs.changed == 'true')
uses: yjrszcq/github-workflows/.github/workflows/gotify-notify.yml@c4c7fe9e17854cd4962913ac2792513eab0be988
with:
title: >-
${{
needs.discover.result == 'failure' && 'DSH 上游检查失败' ||
(needs.validate-candidate.result == 'success' &&
format('DSH {0} 候选版本验证通过', needs.discover.outputs.upstream) ||
format('DSH {0} 候选版本验证失败', needs.discover.outputs.upstream))
}}
message: >-
${{
needs.discover.result == 'failure' &&
format('获取或生成 DSH 上游候选版本失败。工作流：{0}/{1}/actions/runs/{2}', github.server_url, github.repository, github.run_id) ||
(needs.validate-candidate.result == 'success' &&
format('DSH {0} 已通过 Environment {1} 兼容性验证。请审核并合并：{2}', needs.discover.outputs.upstream, needs.discover.outputs.environment, needs.discover.outputs.pull_request_url) ||
format('DSH {0} 未通过 Environment {1} 兼容性验证，需要适配 Environment。工作流：{2}/{3}/actions/runs/{4}', needs.discover.outputs.upstream, needs.discover.outputs.environment, github.server_url, github.repository, github.run_id))
}}
priority: ${{ (needs.discover.result == 'failure' || needs.validate-candidate.result != 'success') && 8 || 7 }}
secrets:
gotify_url: ${{ secrets.GOTIFY_URL }}
gotify_token: ${{ secrets.GOTIFY_TOKEN }}
upstream/.github/workflows/production-publish.yml
Vendored
+289
View File
@@ -0,0 +1,289 @@
name: Publish Supported Platform Target
on:
push:
branches:
- main
paths:
- container/environment/definition.json
- release/supported-target.json
- release/official-dsh-policy.json
workflow_dispatch:
concurrency:
group: production-supported-target
cancel-in-progress: false
permissions:
contents: read
jobs:
publish:
name: Sign and publish Supported Target
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 30
environment: production-release
permissions:
contents: write
steps:
- name: Checkout protected branch
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Set up Node.js
uses: actions/setup-node@v6
with:
node-version: 24
cache: npm
cache-dependency-path: |
container/control-plane/services/management/package-lock.json
container/environment/resources/plugins/platform-management/package/package-lock.json
- name: Install Management runtime dependencies
run: |
npm ci --omit=dev --ignore-scripts --no-audit --no-fund --prefix container/control-plane/services/management
npm ci --omit=dev --ignore-scripts --no-audit --no-fund --legacy-peer-deps --prefix container/environment/resources/plugins/platform-management/package
- name: Verify merged Supported Target
id: target
run: |
result="$(node container/platform/tools/supported-target.mjs validate \
release/supported-target.json container/environment/definition.json)"
dsh_version="$(jq -er '.target.latestSupportedDsh' <<<"$result")"
environment_version="$(jq -er '.target.environment' <<<"$result")"
[[ "$environment_version" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]
printf 'dsh_version=%s\n' "$dsh_version" >> "$GITHUB_OUTPUT"
printf 'environment_version=%s\n' "$environment_version" >> "$GITHUB_OUTPUT"
- name: Run release contract tests
run: |
npm test --prefix container/platform
npm test --prefix container/control-plane/services/gateway
node container/test/compose-config.mjs
- name: Install protected public trust bundle
env:
RECOVERY_ROOT_PUBLIC_KEY: ${{ secrets.DSH_RECOVERY_ROOT_PUBLIC_KEY }}
KEYRING_JSON_BASE64: ${{ secrets.DSH_KEYRING_JSON_BASE64 }}
KEYRING_SIGNATURE_BASE64: ${{ secrets.DSH_KEYRING_SIGNATURE_BASE64 }}
run: |
test -n "$RECOVERY_ROOT_PUBLIC_KEY"
test -n "$KEYRING_JSON_BASE64"
test -n "$KEYRING_SIGNATURE_BASE64"
mkdir -p "$RUNNER_TEMP/production-trust"
printf '%s\n' "$RECOVERY_ROOT_PUBLIC_KEY" > "$RUNNER_TEMP/production-trust/recovery-root.spki.base64"
printf '%s' "$KEYRING_JSON_BASE64" | base64 --decode > "$RUNNER_TEMP/production-trust/keyring.json"
printf '%s' "$KEYRING_SIGNATURE_BASE64" | base64 --decode > "$RUNNER_TEMP/production-trust/keyring.sig.json"
- name: Resolve signed channel and target sequence
id: sequence
run: |
channel="$RUNNER_TEMP/release-channel"
keyring_sha256="$(sha256sum "$RUNNER_TEMP/production-trust/keyring.json" | cut -d ' ' -f 1)"
previous=-
reuse=false
if git ls-remote --exit-code origin refs/heads/release-channel >/dev/null 2>&1; then
git fetch origin refs/heads/release-channel
channel_commit="$(git rev-parse FETCH_HEAD)"
git worktree add --detach "$channel" "$channel_commit"
current_sequence="$(jq -er .targetSequence "$channel/stable.json")"
previous="$channel/targets/$current_sequence"
test -d "$previous"
for name in \
stable.json \
stable.sig.json \
keyring.json \
keyring.sig.json \
bootstrap.manifest.json \
bootstrap.manifest.sig.json; do
test -f "$previous/$name"
done
for name in stable.json stable.sig.json keyring.json keyring.sig.json; do
cmp "$channel/$name" "$previous/$name"
done
if test -f "$channel/channel.json" \
&& test "$(jq -er .sourceCommit "$channel/channel.json")" = "$GITHUB_SHA" \
&& test "$(jq -er .keyringSha256 "$channel/channel.json")" = "$keyring_sha256"; then
sequence="$current_sequence"
reuse=true
else
test "$current_sequence" -lt 9007199254740991
sequence="$((current_sequence + 1))"
fi
else
git worktree add --detach "$channel" "$GITHUB_SHA"
git -C "$channel" checkout --orphan release-channel
git -C "$channel" rm -rf .
sequence=1
channel_commit=
fi
{
printf 'channel=%s\n' "$channel"
printf 'channel_commit=%s\n' "$channel_commit"
printf 'keyring_sha256=%s\n' "$keyring_sha256"
printf 'previous=%s\n' "$previous"
printf 'reuse=%s\n' "$reuse"
printf 'sequence=%s\n' "$sequence"
} >> "$GITHUB_OUTPUT"
printf 'Target sequence: %s (reuse: %s)\n' "$sequence" "$reuse" >> "$GITHUB_STEP_SUMMARY"
- name: Download reviewed DSH npm Artifact
if: steps.sequence.outputs.reuse != 'true'
env:
DSH_VERSION: ${{ steps.target.outputs.dsh_version }}
run: |
mkdir -p "$RUNNER_TEMP/platform-release-input"
npm pack "@deepseek-ai/dsh@${DSH_VERSION}" --pack-destination "$RUNNER_TEMP/platform-release-input" >/dev/null
tarball="$(find "$RUNNER_TEMP/platform-release-input" -maxdepth 1 -type f -name '*.tgz' -print -quit)"
test -n "$tarball"
printf 'DSH_TARBALL=%s\n' "$tarball" >> "$GITHUB_ENV"
- name: Prepare signed immutable target
if: steps.sequence.outputs.reuse != 'true'
env:
RELEASE_PRIVATE_KEY: ${{ secrets.DSH_RELEASE_PRIVATE_KEY }}
TARGET_SEQUENCE: ${{ steps.sequence.outputs.sequence }}
PREVIOUS_STABLE: ${{ steps.sequence.outputs.previous }}
run: |
test -n "$RELEASE_PRIVATE_KEY"
umask 077
private_key="$RUNNER_TEMP/current-release-private.pem"
trap 'rm -f "$private_key"' EXIT
printf '%s\n' "$RELEASE_PRIVATE_KEY" > "$private_key"
artifact_base_url="https://raw.githubusercontent.com/${GITHUB_REPOSITORY}/release-channel/targets/${TARGET_SEQUENCE}/"
SOURCE_DATE_EPOCH="$(git show -s --format=%ct "$GITHUB_SHA")"
export SOURCE_DATE_EPOCH
node container/platform/tools/prepare-release.mjs \
release/supported-target.json container/environment/definition.json release/official-dsh-policy.json \
"$RUNNER_TEMP/production-trust" "$private_key" "$DSH_TARBALL" "$PREVIOUS_STABLE" \
"$TARGET_SEQUENCE" "$artifact_base_url" "$RUNNER_TEMP/platform-release"
rm -f "$private_key"
trap - EXIT
- name: Reuse an already published target after retry
if: steps.sequence.outputs.reuse == 'true'
env:
TARGET_SEQUENCE: ${{ steps.sequence.outputs.sequence }}
CHANNEL: ${{ steps.sequence.outputs.channel }}
DSH_VERSION: ${{ steps.target.outputs.dsh_version }}
ENVIRONMENT_VERSION: ${{ steps.target.outputs.environment_version }}
run: |
cp -a "$CHANNEL/targets/$TARGET_SEQUENCE" "$RUNNER_TEMP/platform-release"
test "$(jq -er .desired.dsh.version "$RUNNER_TEMP/platform-release/stable.json")" = "$DSH_VERSION"
test "$(jq -er .desired.environment.version "$RUNNER_TEMP/platform-release/stable.json")" = "$ENVIRONMENT_VERSION"
- name: Derive Environment release identity
run: |
node container/platform/tools/environment-release.mjs \
"$RUNNER_TEMP/platform-release/bootstrap.manifest.json" \
"$RUNNER_TEMP/platform-release/environment.manifest.json" \
release/official-dsh-policy.json \
"$RUNNER_TEMP/environment-release.json"
- name: Validate Environment version ownership
id: environment-check
env:
GH_TOKEN: ${{ github.token }}
ENVIRONMENT_VERSION: ${{ steps.target.outputs.environment_version }}
run: |
tag="v${ENVIRONMENT_VERSION}"
publish=true
if gh release view "$tag" --json isDraft > "$RUNNER_TEMP/environment-release-state.json" 2>/dev/null; then
if test "$(jq -er .isDraft "$RUNNER_TEMP/environment-release-state.json")" = false; then
mkdir -p "$RUNNER_TEMP/published-environment"
gh release download "$tag" --pattern environment-release.json --dir "$RUNNER_TEMP/published-environment"
cmp "$RUNNER_TEMP/environment-release.json" "$RUNNER_TEMP/published-environment/environment-release.json"
publish=false
else
mkdir -p "$RUNNER_TEMP/draft-environment"
if gh release download "$tag" --pattern environment-release.json --dir "$RUNNER_TEMP/draft-environment" 2>/dev/null; then
cmp "$RUNNER_TEMP/environment-release.json" "$RUNNER_TEMP/draft-environment/environment-release.json"
fi
fi
elif git ls-remote --exit-code origin "refs/tags/$tag" >/dev/null 2>&1; then
echo "Git tag $tag exists without a matching Environment Release" >&2
exit 1
fi
printf 'publish=%s\n' "$publish" >> "$GITHUB_OUTPUT"
printf 'tag=%s\n' "$tag" >> "$GITHUB_OUTPUT"
- name: Publish signed channel target
env:
CHANNEL: ${{ steps.sequence.outputs.channel }}
EXISTING_CHANNEL_COMMIT: ${{ steps.sequence.outputs.channel_commit }}
KEYRING_SHA256: ${{ steps.sequence.outputs.keyring_sha256 }}
REUSE: ${{ steps.sequence.outputs.reuse }}
TARGET_SEQUENCE: ${{ steps.sequence.outputs.sequence }}
DSH_VERSION: ${{ steps.target.outputs.dsh_version }}
ENVIRONMENT_VERSION: ${{ steps.target.outputs.environment_version }}
run: |
if test "$REUSE" = true; then
commit="$EXISTING_CHANNEL_COMMIT"
else
target="$CHANNEL/targets/$TARGET_SEQUENCE"
test ! -e "$target"
mkdir -p "$target"
cp -a "$RUNNER_TEMP/platform-release/." "$target/"
for name in stable.json stable.sig.json keyring.json keyring.sig.json; do cp "$target/$name" "$CHANNEL/$name"; done
# shellcheck disable=SC2016
jq -n --arg sourceCommit "$GITHUB_SHA" --arg keyringSha256 "$KEYRING_SHA256" \
--arg dshVersion "$DSH_VERSION" --arg environmentVersion "$ENVIRONMENT_VERSION" \
--argjson targetSequence "$TARGET_SEQUENCE" \
'{schema:1,sourceCommit:$sourceCommit,keyringSha256:$keyringSha256,targetSequence:$targetSequence,dshVersion:$dshVersion,environmentVersion:$environmentVersion}' \
> "$CHANNEL/channel.json"
git -C "$CHANNEL" config user.name github-actions
git -C "$CHANNEL" config user.email github-actions@github.com
git -C "$CHANNEL" add channel.json keyring.json keyring.sig.json stable.json stable.sig.json "targets/$TARGET_SEQUENCE"
git -C "$CHANNEL" commit -m "release: publish target sequence $TARGET_SEQUENCE"
commit="$(git -C "$CHANNEL" rev-parse HEAD)"
git -C "$CHANNEL" push origin HEAD:refs/heads/release-channel
fi
# shellcheck disable=SC2016
printf 'Published signed channel sequence `%s` at `%s`.\n' "$TARGET_SEQUENCE" "$commit" >> "$GITHUB_STEP_SUMMARY"
- name: Publish Environment Release
env:
GH_TOKEN: ${{ github.token }}
PUBLISH_ENVIRONMENT: ${{ steps.environment-check.outputs.publish }}
ENVIRONMENT_TAG: ${{ steps.environment-check.outputs.tag }}
ENVIRONMENT_VERSION: ${{ steps.target.outputs.environment_version }}
DSH_VERSION: ${{ steps.target.outputs.dsh_version }}
TARGET_SEQUENCE: ${{ steps.sequence.outputs.sequence }}
run: |
prune_environment_assets() {
gh release view "$ENVIRONMENT_TAG" --json assets --jq '.assets[].name' | while IFS= read -r name; do
if test "$name" != environment-release.json; then
gh release delete-asset "$ENVIRONMENT_TAG" "$name" --yes
fi
done
}
if test "$PUBLISH_ENVIRONMENT" != true; then
prune_environment_assets
test "$(gh release view "$ENVIRONMENT_TAG" --json assets --jq '.assets[].name')" = environment-release.json
exit 0
fi
assets="$RUNNER_TEMP/environment-release-assets"
mkdir -p "$assets"
cp "$RUNNER_TEMP/environment-release.json" "$assets/"
notes="$RUNNER_TEMP/environment-release-notes.md"
# shellcheck disable=SC2016
{
printf 'Container Environment: `%s`\n\n' "$ENVIRONMENT_VERSION"
printf 'Supported DSH at publication: `%s`\n\n' "$DSH_VERSION"
printf 'Initial signed target sequence: `%s`\n' "$TARGET_SEQUENCE"
} > "$notes"
if ! gh release view "$ENVIRONMENT_TAG" >/dev/null 2>&1; then
gh release create "$ENVIRONMENT_TAG" --draft --target "$GITHUB_SHA" \
--title "$ENVIRONMENT_TAG" --notes-file "$notes"
fi
gh release edit "$ENVIRONMENT_TAG" --draft --target "$GITHUB_SHA" \
--title "$ENVIRONMENT_TAG" --notes-file "$notes"
gh release upload "$ENVIRONMENT_TAG" "$assets"/* --clobber
prune_environment_assets
find "$assets" -maxdepth 1 -type f -printf '%f\n' | sort > "$RUNNER_TEMP/local-environment-assets"
gh release view "$ENVIRONMENT_TAG" --json assets --jq '.assets[].name' | sort > "$RUNNER_TEMP/remote-environment-assets"
diff -u "$RUNNER_TEMP/local-environment-assets" "$RUNNER_TEMP/remote-environment-assets"
gh release edit "$ENVIRONMENT_TAG" --draft=false --latest \
--title "$ENVIRONMENT_TAG" --notes-file "$notes"
upstream/.github/workflows/workflow-notifications.yml
Vendored
+52
View File
@@ -0,0 +1,52 @@
name: Notify Workflow Results
on:
workflow_run:
workflows:
- Publish Supported Platform Target
- DSH Candidate Validation
types:
- completed
permissions:
contents: read
jobs:
notify-result:
name: Notify completed workflow result
if: >-
(github.event.workflow_run.name == 'Publish Supported Platform Target' &&
(github.event.workflow_run.conclusion == 'success' ||
github.event.workflow_run.conclusion == 'failure')) ||
(github.event.workflow_run.name == 'DSH Candidate Validation' &&
(github.event.workflow_run.event == 'pull_request_target' ||
github.event.workflow_run.event == 'workflow_dispatch') &&
github.event.workflow_run.head_branch != 'automation/dsh-upstream-candidate' &&
(github.event.workflow_run.conclusion == 'success' ||
github.event.workflow_run.conclusion == 'failure'))
uses: yjrszcq/github-workflows/.github/workflows/gotify-notify.yml@c4c7fe9e17854cd4962913ac2792513eab0be988
with:
title: >-
${{
github.event.workflow_run.name == 'Publish Supported Platform Target' &&
(github.event.workflow_run.conclusion == 'success' && 'DSH 正式目标已发布' || 'DSH 正式发布失败') ||
(github.event.workflow_run.conclusion == 'success' && 'DSH 候选版本验证通过' || 'DSH 候选版本验证失败')
}}
message: >-
${{
github.event.workflow_run.name == 'Publish Supported Platform Target' &&
(github.event.workflow_run.conclusion == 'success' &&
format('签名目标和必要的 Environment Release 已发布。独立镜像工作流已经启动；如果 production-image 启用了审批，镜像发布会等待批准。镜像工作流：{0}/{1}/actions/workflows/docker.yaml', github.server_url, github.repository) ||
format('签名目标或 Environment Release 发布失败。工作流：{0}', github.event.workflow_run.html_url)) ||
(github.event.workflow_run.conclusion == 'success' &&
(github.event.workflow_run.event == 'workflow_dispatch' &&
format('手动发起的 DSH 候选版本兼容性验证通过。候选提交：{0}。工作流：{1}', github.event.workflow_run.head_sha, github.event.workflow_run.html_url) ||
format('Pull Request 的 DSH 候选版本兼容性验证通过。候选提交：{0}。工作流：{1}', github.event.workflow_run.head_sha, github.event.workflow_run.html_url)) ||
(github.event.workflow_run.event == 'workflow_dispatch' &&
format('手动发起的 DSH 候选版本兼容性验证失败。候选提交：{0}。工作流：{1}', github.event.workflow_run.head_sha, github.event.workflow_run.html_url) ||
format('Pull Request 的 DSH 候选版本兼容性验证失败。候选提交：{0}。工作流：{1}', github.event.workflow_run.head_sha, github.event.workflow_run.html_url)))
}}
priority: ${{ github.event.workflow_run.conclusion == 'failure' && 8 || 7 }}
secrets:
gotify_url: ${{ secrets.GOTIFY_URL }}
gotify_token: ${{ secrets.GOTIFY_TOKEN }}
upstream/.gitignore
+11
View File
@@ -0,0 +1,11 @@
.DS_Store
.env
.env.*
!.env.example
temp/
node_modules/
/data/
/workspace/
.checkpoint-karpathy/private/
docker-compose.test.yaml
upstream/Dockerfile
+174
View File
@@ -0,0 +1,174 @@
ARG DSH_VERSION=latest
ARG ENVIRONMENT_VERSION=development
ARG TARGET_SEQUENCE=0
ARG SIGNED_IMAGE_INPUT=false
FROM node:24-bookworm-slim AS installer
ARG DSH_VERSION
ARG SIGNED_IMAGE_INPUT
COPY container/platform/image-input /opt/dsh-platform-image-input
COPY release/supported-target.json /opt/dsh-supported-target.json
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
g++ \
make \
python3 \
&& case "$SIGNED_IMAGE_INPUT" in \
true) \
test -f /opt/dsh-platform-image-input/dsh.tgz \
&& npm install --global /opt/dsh-platform-image-input/dsh.tgz \
;; \
false) \
requested_version="$DSH_VERSION"; \
if [ "$requested_version" = latest ]; then \
requested_version="$(node -p "JSON.parse(require('fs').readFileSync('/opt/dsh-supported-target.json', 'utf8')).latestSupportedDsh")"; \
fi; \
npm install --global "@deepseek-ai/dsh@${requested_version}" \
;; \
*) echo "SIGNED_IMAGE_INPUT must be true or false" >&2; exit 64 ;; \
esac \
&& rm -rf /var/lib/apt/lists/* /root/.npm
FROM node:24-bookworm-slim AS platform-seed
ARG PLATFORM_REVISION=development
ARG SIGNED_IMAGE_INPUT
COPY --from=installer /usr/local/lib/node_modules/@deepseek-ai/dsh /opt/installed-dsh
COPY container/control-plane/services/management/package.json container/control-plane/services/management/package-lock.json \
/opt/dsh-platform-source/control-plane/services/management/
COPY container/environment/resources/plugins/platform-management/package/package.json \
container/environment/resources/plugins/platform-management/package/package-lock.json \
/opt/dsh-platform-source/environment/resources/plugins/platform-management/package/
RUN npm ci --omit=dev --ignore-scripts --no-audit --no-fund \
--prefix /opt/dsh-platform-source/control-plane/services/management \
&& npm ci --omit=dev --ignore-scripts --no-audit --no-fund --legacy-peer-deps \
--prefix /opt/dsh-platform-source/environment/resources/plugins/platform-management/package
COPY container /opt/dsh-platform-source
RUN case "$SIGNED_IMAGE_INPUT" in \
true) image_input=/opt/dsh-platform-source/platform/image-input ;; \
false) image_input=- ;; \
*) echo "SIGNED_IMAGE_INPUT must be true or false" >&2; exit 64 ;; \
esac \
&& node /opt/dsh-platform-source/platform/tools/build-seed.mjs \
/opt/installed-dsh /opt/dsh-platform-seed "$image_input" "$PLATFORM_REVISION"
FROM node:24-bookworm-slim AS runtime
ARG PNPM_VERSION=11.7.0
ARG DSH_VERSION
ARG ENVIRONMENT_VERSION
ARG TARGET_SEQUENCE
LABEL org.opencontainers.image.title="DeepSeek Harness" \
org.opencontainers.image.description="Unofficial container image for DeepSeek Harness" \
org.opencontainers.image.source="https://github.com/yjrszcq/dsh-docker" \
org.opencontainers.image.licenses="MIT" \
org.opencontainers.image.version="$DSH_VERSION" \
io.dsh-docker.environment.version="$ENVIRONMENT_VERSION" \
io.dsh-docker.target-sequence="$TARGET_SEQUENCE"
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
bash \
ca-certificates \
curl \
g++ \
git \
jq \
make \
openssh-client \
p7zip-full \
procps \
python3 \
python3-venv \
ripgrep \
sudo \
tini \
unzip \
util-linux \
zip \
&& groupadd --system dsh-sudo-true \
&& groupadd --system dsh-sudo-false \
&& groupadd --system --gid 991 dsh-proxy \
&& useradd --system --uid 991 --gid 991 --home-dir /nonexistent --shell /usr/sbin/nologin dsh-proxy \
&& groupadd --system --gid 992 dsh-access \
&& useradd --system --uid 992 --gid 992 --home-dir /nonexistent --shell /usr/sbin/nologin dsh-access \
&& npm install --global "pnpm@${PNPM_VERSION}" \
&& rm -rf /var/lib/apt/lists/* /root/.npm
COPY docker-sudoers /etc/sudoers.d/dsh-sudo
RUN chmod 440 /etc/sudoers.d/dsh-sudo \
&& visudo -cf /etc/sudoers.d/dsh-sudo
COPY --from=platform-seed /opt/dsh-platform-seed /opt/dsh-platform/seed
COPY container/platform /opt/dsh-platform/runtime/platform
COPY container/control-plane /opt/dsh-platform/runtime/control-plane
COPY --from=platform-seed /opt/dsh-platform-source/control-plane/services/management/node_modules /opt/dsh-platform/runtime/control-plane/services/management/node_modules
COPY container/platform/tools/dsh-shim.sh /usr/local/bin/dsh
COPY container/platform/tools/dsh-platform-shim.sh /usr/local/bin/dsh-platform
RUN chmod 755 /usr/local/bin/dsh /usr/local/bin/dsh-platform
ENV DSH_PLATFORM_DATA=/data/platform \
DSH_PLATFORM_MANAGED=1 \
DSH_PLATFORM_RUN=/run/dsh-platform \
DSH_HOME=/data/dsh \
DSH_DEFAULT_WORKSPACE=/workspace \
DSH_PROXY_POLYFILL=true \
DSH_TELEMETRY_DISABLED=true \
DSH_UPDATE_METADATA_URL=https://raw.githubusercontent.com/yjrszcq/dsh-docker/release-channel/ \
DSH_LOG_MAX_BYTES=104857600 \
DSH_LOG_RETENTION_DAYS=14 \
DSH_ACTIVATION_TIMEOUT_SECONDS=60 \
DSH_EXPERIMENTAL_PROBATION_SECONDS=120
RUN mkdir -p /data/platform /data/dsh /workspace \
&& chown -R node:node /data/dsh /workspace
ARG INSTALL_DEVTOOLS=false
ARG UV_VERSION=0.11.32
RUN case "$INSTALL_DEVTOOLS" in \
true) apt-get update \
&& apt-get install -y --no-install-recommends \
bash-completion \
build-essential \
dnsutils \
file \
htop \
iproute2 \
iputils-ping \
less \
lsof \
nano \
netcat-openbsd \
openssl \
pkg-config \
rsync \
tmux \
tree \
unzip \
vim \
wget \
xz-utils \
zip \
&& curl --fail --location --silent --show-error \
"https://astral.sh/uv/${UV_VERSION}/install.sh" \
--output /tmp/uv-installer.sh \
&& UV_UNMANAGED_INSTALL=/usr/local/bin sh /tmp/uv-installer.sh \
&& mkdir -p /etc/uv \
&& printf '%s\n' 'python-downloads = "manual"' > /etc/uv/uv.toml \
&& rm /tmp/uv-installer.sh \
&& rm -rf /var/lib/apt/lists/* \
;; \
false) ;; \
*) echo "INSTALL_DEVTOOLS must be true or false" >&2; exit 64 ;; \
esac
WORKDIR /workspace
EXPOSE 3080
HEALTHCHECK --interval=30s --timeout=5s --start-period=60s --retries=3 \
CMD /usr/bin/curl --fail --silent --show-error --noproxy '*' http://127.0.0.1:3080/_dsh_gateway/health >/dev/null || exit 1
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["/usr/local/bin/node", "/opt/dsh-platform/runtime/platform/stage0/index.mjs"]
upstream/LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 yjrszcq
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
upstream/PROVENANCE.md
+11
View File
@@ -0,0 +1,11 @@
# Upstream provenance
`upstream/` — вендоренная копия <https://github.com/yjrszcq/dsh-docker> (MIT).
- commit: `2564866f7844f334379dc4913dc466a5a4138306`
- date: 2026-09-19 01:16:39 +0800
- причина вендоринга: самодостаточность репо (пересборка образа без внешних зависимостей);
лицензия MIT допускает копирование при сохранении уведомления (см. LICENSE).
Изменения относительно upstream: отсутствуют (чистая копия). Наши артефакты — compose,
скрипты и документация — лежат на верхнем уровне репозитория.
upstream/README.md
+220
View File
@@ -0,0 +1,220 @@
# DSH-Docker
English | [中文](README_CN.md)
An unofficial Docker image for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness). It installs the official `@deepseek-ai/dsh` package and adds persistent updates, rollback, gateway access control, and optional development tools.
> DeepSeek Harness is in Developer Preview and may introduce incompatible changes. This project is not affiliated with DeepSeek AI.
## Before You Start
- **Directory permissions:** Bind-mounted `data/dsh`, `data/platform`, and `workspace` must be writable by container UID/GID `1000:1000`. Keep both data directories when replacing or upgrading the container.
- **Port exposure:** The quick-start configuration binds `127.0.0.1:3080` and is reachable only from the Docker host. Changing it to `3080:3080` or `0.0.0.0:3080:3080` exposes the service on every host interface.
- **Remote access:** Before allowing LAN or Internet access, set `DSH_TRUSTED_HOSTS` to the exact IP addresses or domains used by browsers, initialize the local administrator account, and use HTTPS or another trusted network boundary. Do not expose privileged Docker or host resources to the container.
- **Root authority:** DSH/Agent passwordless sudo is disabled by default, including in the quick-start examples and repository Compose. Enable it only when unrestricted root access is intentional: set `DSH_SUDO_ENABLED=true` with repository Compose, add `group_add: [dsh-sudo-true]` to custom Compose, or add `--group-add dsh-sudo-true` to `docker run`. This setting does not restrict the standalone Management Console's root terminal or file manager, which must remain protected by authentication and a trusted network boundary.
- **DSH Management Console:** `/_dsh_platform/console/` remains available when DSH is stopped or fails to start. It provides updates and recovery, outbound proxy settings, live logs, bundled System Plugin and System Skill management, User Plugin recovery, User Skill management, container files, and a root terminal. The first browser visit creates the local administrator account. Management uses a separate session derived from the active DSH browser session; an optional Management password adds a second login layer.
| Variant | Rolling tag | Versioned tag | Contents |
| --- | --- | --- | --- |
| Standard | `latest` | `<version>` | DSH and normal runtime utilities |
| Devtools | `latest-devtools` | `<version>-devtools` | Standard image plus development tools |
Docker Hub tags follow the DSH version. `ghcr.io/yjrszcq/dsh-docker` mirrors only the Standard image with Environment version tags and `dsh-<version>` lookup tags; it does not publish Devtools images.
## Quick Start
Create local data directories:
```bash
mkdir -p data/dsh data/platform workspace
```
Use this minimal `docker-compose.yaml`:
```yaml
services:
deepseek-harness:
image: szcq/deepseek-harness:latest
container_name: deepseek-harness
restart: unless-stopped
ports:
- "127.0.0.1:3080:3080"
volumes:
- ./data/dsh:/data/dsh
- ./data/platform:/data/platform
- ./workspace:/workspace
```
Start it with:
```bash
docker compose up -d
```
Or use the equivalent `docker run` command:
```bash
docker run -d \
--name deepseek-harness \
--restart unless-stopped \
-p 127.0.0.1:3080:3080 \
-v "$(pwd)/data/platform:/data/platform" \
-v "$(pwd)/data/dsh:/data/dsh" \
-v "$(pwd)/workspace:/workspace" \
szcq/deepseek-harness:latest
```
Open <http://127.0.0.1:3080>.
## Storage
Keep both persistent locations:
| Path | Purpose |
| --- | --- |
| `/data/platform` | Platform state, managed rollback assets, snapshots, and logs |
| `/data/dsh` | DSH settings, sessions, credentials, and third-party plugins |
| `/workspace` | Default working directory |
Container workloads run as UID/GID `1000:1000`. If a bind mount is inaccessible:
```bash
sudo chown -R 1000:1000 data workspace
```
## Remote Access
For LAN or reverse-proxy access, set the browser-facing host:
```dotenv
DSH_TRUSTED_HOSTS=192.168.1.100,dsh.example.com
```
Use this minimal remote-access `docker-compose.yaml`, replacing the trusted hosts first:
```yaml
services:
deepseek-harness:
image: szcq/deepseek-harness:latest
container_name: deepseek-harness
restart: unless-stopped
ports:
- "3080:3080"
environment:
DSH_TRUSTED_HOSTS: "192.168.1.100,dsh.example.com"
volumes:
- ./data/dsh:/data/dsh
- ./data/platform:/data/platform
- ./workspace:/workspace
```
Or use the equivalent `docker run` command:
```bash
docker run -d \
--name deepseek-harness \
--restart unless-stopped \
-p 3080:3080 \
-e 'DSH_TRUSTED_HOSTS=192.168.1.100,dsh.example.com' \
-v "$(pwd)/data/platform:/data/platform" \
-v "$(pwd)/data/dsh:/data/dsh" \
-v "$(pwd)/workspace:/workspace" \
szcq/deepseek-harness:latest
```
A reverse proxy must preserve the original `Host` header. Terminate TLS outside this container. To publish only on the Docker host, use `127.0.0.1:3080:3080` instead of `3080:3080`.
After startup, open the browser-facing address and create the local administrator username and main password. The initialization page blocks DSH and Management until registration finishes. Use HTTPS or another trusted network boundary because credentials and authenticated sessions grant full DSH access.
Common settings:
| Variable | Default | Purpose |
| --- | --- | --- |
| `DSH_TRUSTED_HOSTS` | Loopback only | Allowed browser-facing hosts |
| `DSH_DEFAULT_WORKSPACE` | `/workspace` | Default workspace for directory pickers and standalone file management |
| `DSH_SUDO_ENABLED` | `false` | Compose-only passwordless sudo switch for DSH and its Agent |
See the [complete configuration reference](docs/en/guide.md#configuration) for all variables and validation rules.
## Online Updates
Open **Platform Management** in DSH settings or visit the standalone **DSH Management Console** at <http://127.0.0.1:3080/_dsh_platform/console/>. Checks do not download or activate anything until you confirm an update. Both interfaces can configure an existing HTTP or SOCKS5 proxy for selected managed outbound traffic; DSH Docker does not expose or create a public proxy service.
The standalone page remains available when DSH is down. It provides DSH lifecycle and recovery controls, live logs, bundled System Plugin and System Skill management, User Plugin and User Skill recovery, root file management, and a container terminal.
### Upgrade and authentication note
The Access Manager persists the administrator account. DSH and the Management Console use separate sessions, with optional Management-console password and two-factor authentication. The default Compose setting keeps `DSH_SUDO_ENABLED=false`. Preserve both `/data/dsh` and `/data/platform` (especially `/data/platform/state`) when replacing the image. Removing the platform volume resets the administrator account and authentication state and can route an existing deployment into migration or recovery.
Authentication forms are transport-only: the Gateway accepts the request and the Access Manager performs all authentication and credential decisions. A DSH login creates only a DSH Session; opening the Management Console exchanges that active session once for a separate Management Session. Direct Management access always completes the main DSH login first and then, when configured, asks for the Management console password. Logging out of DSH, or expiry of its session, also invalidates every linked Management Session. During a classified DSH outage, top-level browser navigation shows a generic recovery page before login so the standalone Management entry remains discoverable; API and WebSocket requests still fail closed. Authentication Settings lists signed-in devices with their browser, peer IP, and activity time; signing out one device revokes its DSH Session and linked Management Session together. Management can also move to a separately published public Origin or a local Origin using a user-selected host port (default `3081`).
Five consecutive password failures from the same browser start a 30-second retry wait; later failures double that wait up to 15 minutes. Each browser source is also limited to 12 failures per hour and 24 per 24 hours. Across all sources, the whole DSH instance has a wider flood limit of 20 failures per minute, 60 per hour, and 120 per 24 hours. The browser only displays the Access Manager decision and remaining time. A successful password check clears that browser's consecutive and rolling-source failures, but not the instance-wide history.
Lost credentials are recovered only from an interactive Root console. Passwords are read with input echo disabled and are never accepted as command arguments or piped input:
```bash
docker exec -it --user root deepseek-harness dsh-platform access status
docker exec -it --user root deepseek-harness dsh-platform access reset
docker exec -it --user root deepseek-harness dsh-platform access clear-retry
docker exec -it --user root deepseek-harness dsh-platform access clear-retry --global-only
docker exec -it --user root deepseek-harness dsh-platform access generate-key
```
`access clear-retry` asks for `y/[n]` confirmation and clears all browser retry waits, browser rolling windows, and instance-wide failed-attempt windows. Add `--global-only` to clear only the instance-wide windows while preserving every browser's retry wait and rolling limits. Neither form changes credentials or sessions.
A fresh empty volume opens the normal administrator registration page and needs no key. Only legacy migration or damaged authentication state uses `access generate-key` to issue a single-use authentication reset key valid for ten minutes; the recovery page can then recreate the administrator account.
Useful commands:
```bash
docker exec deepseek-harness dsh-platform status
docker exec deepseek-harness dsh-platform check
docker exec deepseek-harness dsh-platform update --wait
docker exec deepseek-harness dsh-platform stop --wait
docker exec deepseek-harness dsh-platform start --wait
docker exec deepseek-harness dsh-platform restart --wait
docker exec deepseek-harness dsh-platform rollback
```
Stable follows the supported DSH release. Experimental can install a newer verified upstream DSH and creates a data snapshot before activation. Returning to Stable may discard data written after that snapshot.
## System Plugins
The Container Environment includes these DSH-Docker integrations:
| Plugin | Purpose |
| --- | --- |
| `@dsh-docker/platform-management` | Adds **Platform Management** to DSH settings for updates, maintenance, logs, System Plugins, System Skills, and current-browser sign-out controls; this integration is platform-managed |
| `@dsh-docker/settings-navigation` | Keeps the desktop Settings directory independently scrollable and provides directory/detail navigation on narrow screens |
| `@dsh-docker/settings-document-editor` | Replaces desktop-only configuration-file opening with an optional browser editor for `settings.yaml` |
Other installed System Plugins can be enabled or disabled from **Platform Management** in DSH; it cannot modify itself. The standalone **DSH Management Console** can install, uninstall, enable, or disable bundled System Plugins, including restoring Platform Management when it is missing. Unsubmitted selections remain page-local; after Apply, the change is retained even when DSH is unavailable or the activation restart fails, and the next successful start uses it. Installation restores verified local Environment assets and does not download from GitHub or npm. Third-party User Plugins remain separate and are not treated as System Plugins.
Platform Management and the Settings Document Editor use the restricted DSH-side platform API through the authenticated DSH Session. They do not require a separate Management Session. Platform Management can revoke the current browser's DSH Session and its linked Management Sessions, but account names, passwords, and complete authentication settings remain available only in the standalone Management Console or Root CLI.
## User Plugins
The standalone Management Console inventories Bundle plugins in the DSH Web Profile and can enable, disable, or uninstall an exact User Plugin. Applying a change snapshots the complete Web Profile and restarts only DSH when it is available. A committed change remains stored if DSH is already down or the restart fails, so another faulty plugin can be corrected before retrying startup. Installation remains in DSH's normal plugin flow or the container terminal; the recovery page does not install arbitrary packages.
## System and User Skills
DSH Docker includes the signed `dsh-docker-operations` System Skill. It gives the Agent container-specific instructions for files, development tools, plugins, lifecycle, logs, updates, recovery, networking, and permissions, and can also be invoked explicitly as `/dsh-docker-operations`.
System Skills are available under **Platform Management** in DSH and **System Skills** in the standalone Management Console. Changes are applied immediately without restarting DSH. The standalone console can install, uninstall, enable, or disable a bundled skill; Platform Management can restore a missing skill and enable or disable it. Project and user skills with the same name retain DSH's native precedence and can override the bundled copy.
The standalone console also inventories native User Skills from the configured DSH and Agent Skill roots. It can enable, disable, or delete an exact entry without modifying project-level Skills.
## Security
An authenticated DSH Session has full DSH authority. A Management Session additionally grants a root container terminal and root file operations, so an admitted administrator can read credentials, execute commands, and modify container data. Use HTTPS and a trusted network boundary for remote access.
Repository Compose disables DSH/Agent passwordless sudo by default. Set `DSH_SUDO_ENABLED=true` only when DSH or its Agent intentionally needs root access. Do not combine sudo with privileged mode, the Docker socket, or sensitive host mounts without understanding the impact.
## Documentation
The [complete guide](docs/en/guide.md) covers:
- deployment prerequisites, both Compose layouts, and Docker Run;
- all configuration variables, storage, gateway, and remote-access behavior;
- platform architecture, online updates, System and User Plugins, System and User Skills, logs, trust, recovery, and rollback;
- standalone recovery tools, release automation, local builds, tests, and devtools.
upstream/README_CN.md
+220
View File
@@ -0,0 +1,220 @@
# DSH-Docker
[English](README.md) | 中文
[DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) 的非官方 Docker 镜像。镜像安装官方 `@deepseek-ai/dsh` 包，并提供持久化更新、回滚、Gateway 访问控制和可选开发工具。
> DeepSeek Harness 目前处于 Developer Preview，可能出现不兼容更新。本项目不隶属于 DeepSeek AI。
## 使用前须知
- **目录权限：** bind mount 的 `data/dsh`、`data/platform` 和 `workspace` 必须允许容器 UID/GID `1000:1000` 写入。替换或升级容器时必须保留两个数据目录。
- **端口暴露：** 快速开始默认绑定 `127.0.0.1:3080`，只能从 Docker 宿主机访问。改成 `3080:3080` 或 `0.0.0.0:3080:3080` 会向宿主机所有网络接口开放服务。
- **远程访问：** 允许局域网或互联网访问前，必须将 `DSH_TRUSTED_HOSTS` 设置为浏览器实际使用的 IP 地址或域名，初始化本地管理员账户，并通过 HTTPS 或其他可信网络边界提供服务；不要向容器暴露 Docker Socket、特权模式或敏感宿主机资源。
- **Root 权限：** DSH/Agent 的免密码 sudo 默认关闭，快速开始示例和仓库 Compose 均不会默认授予该权限。只有明确需要不受限制的 Root 权限时才开启：仓库 Compose 设置 `DSH_SUDO_ENABLED=true`，自定义 Compose 添加 `group_add: [dsh-sudo-true]`，或为 `docker run` 添加 `--group-add dsh-sudo-true`。该设置不会限制独立管理中心的 Root 终端和文件管理，因此管理中心仍须通过认证和可信网络边界保护。
- **DSH 管理中心：** `/_dsh_platform/console/` 在 DSH 停止或启动失败时仍可使用，提供更新与恢复、出站代理设置、实时日志、内置系统插件和系统技能管理、用户插件恢复、用户技能管理、容器文件和 Root 终端。首次浏览器访问会创建本地管理员账户。管理中心使用由当前 DSH 浏览器会话派生的独立会话；可选的管理中心密码构成第二层登录。
| 变体 | 滚动标签 | 固定版本标签 | 内容 |
| --- | --- | --- | --- |
| 标准版 | `latest` | `<version>` | DSH 和正常运行所需工具 |
| 开发工具版 | `latest-devtools` | `<version>-devtools` | 标准版加开发工具 |
Docker Hub 标签跟随 DSH 版本。`ghcr.io/yjrszcq/dsh-docker` 仅备份标准镜像，使用 Environment 版本标签和 `dsh-<version>` 定位标签，不发布 Devtools 镜像。
## 快速开始
创建本地数据目录：
```bash
mkdir -p data/dsh data/platform workspace
```
使用以下精简 `docker-compose.yaml`：
```yaml
services:
deepseek-harness:
image: szcq/deepseek-harness:latest
container_name: deepseek-harness
restart: unless-stopped
ports:
- "127.0.0.1:3080:3080"
volumes:
- ./data/dsh:/data/dsh
- ./data/platform:/data/platform
- ./workspace:/workspace
```
启动容器：
```bash
docker compose up -d
```
也可以使用等价的 `docker run` 命令：
```bash
docker run -d \
--name deepseek-harness \
--restart unless-stopped \
-p 127.0.0.1:3080:3080 \
-v "$(pwd)/data/platform:/data/platform" \
-v "$(pwd)/data/dsh:/data/dsh" \
-v "$(pwd)/workspace:/workspace" \
szcq/deepseek-harness:latest
```
打开 <http://127.0.0.1:3080>。
## 数据存储
以下两个持久化位置都必须保留：
| 路径 | 用途 |
| --- | --- |
| `/data/platform` | 平台状态、受管理的回滚资产、快照和日志 |
| `/data/dsh` | DSH 设置、会话、凭据和第三方插件 |
| `/workspace` | 默认工作目录 |
容器工作负载使用 UID/GID `1000:1000`。bind mount 无法访问时执行：
```bash
sudo chown -R 1000:1000 data workspace
```
## 远程访问
通过局域网或反向代理访问时，必须配置浏览器使用的主机：
```dotenv
DSH_TRUSTED_HOSTS=192.168.1.100,dsh.example.com
```
将可信主机替换为实际值后，可以使用以下支持远程访问的精简 `docker-compose.yaml`：
```yaml
services:
deepseek-harness:
image: szcq/deepseek-harness:latest
container_name: deepseek-harness
restart: unless-stopped
ports:
- "3080:3080"
environment:
DSH_TRUSTED_HOSTS: "192.168.1.100,dsh.example.com"
volumes:
- ./data/dsh:/data/dsh
- ./data/platform:/data/platform
- ./workspace:/workspace
```
也可以使用等价的 `docker run` 命令：
```bash
docker run -d \
--name deepseek-harness \
--restart unless-stopped \
-p 3080:3080 \
-e 'DSH_TRUSTED_HOSTS=192.168.1.100,dsh.example.com' \
-v "$(pwd)/data/platform:/data/platform" \
-v "$(pwd)/data/dsh:/data/dsh" \
-v "$(pwd)/workspace:/workspace" \
szcq/deepseek-harness:latest
```
反向代理必须保留原始 `Host` 请求头，TLS 在容器外终止。如需只监听 Docker 宿主机，请使用 `127.0.0.1:3080:3080`，不要使用 `3080:3080`。
启动后打开浏览器侧地址，创建本地管理员用户名和主密码。初始化完成前，注册页会阻断 DSH 和管理中心。凭据及认证会话拥有完整 DSH 权限，因此远程访问必须使用 HTTPS 或其他可信网络边界。
常用设置：
| 变量 | 默认值 | 用途 |
| --- | --- | --- |
| `DSH_TRUSTED_HOSTS` | 仅 loopback | 浏览器侧允许的主机 |
| `DSH_DEFAULT_WORKSPACE` | `/workspace` | 目录选择器和独立文件管理的默认工作目录 |
| `DSH_SUDO_ENABLED` | `false` | 仅供 Compose 使用的 DSH/Agent 免密码 sudo 开关 |
全部变量和校验规则见[完整配置参考](docs/cn/guide.md#配置参考)。
## 在线更新
在 DSH 设置中打开“平台管理”，或访问独立的 <http://127.0.0.1:3080/_dsh_platform/console/> DSH 管理中心。检查不会自动下载或激活，必须由用户确认更新。两个界面都可以为选定的受管出站流量配置现有 HTTP 或 SOCKS5 代理；DSH Docker 不会创建或对外开放代理服务。
独立页面在 DSH 无法启动时仍可使用，提供 DSH 生命周期与恢复、实时日志、内置系统插件和系统技能管理、用户插件和用户技能恢复、Root 文件管理及容器终端。
### 升级与认证提示
管理员账户由 Access Manager 持久化管理，DSH 与管理中心使用独立会话，并支持管理中心密码和两步验证。仓库 Compose 默认保持 `DSH_SUDO_ENABLED=false`。替换镜像时必须保留 `/data/dsh` 与 `/data/platform`（尤其是 `/data/platform/state`）。删除 platform volume 会重置管理员账户和认证状态，已有部署可能因此进入迁移或恢复流程。
认证表单只负责传输输入，所有凭据判断由 Access Manager 完成。登录 DSH 只创建 DSH Session；打开管理中心时会从这个有效会话通过一次性交接创建独立的 Management Session。直接访问管理中心也必须先完成 DSH 主密码登录；配置管理中心密码后，再进行第二层验证。DSH 注销或会话过期会同时使其关联的 Management Session 失效。DSH 发生已分类故障时，顶层页面导航会先显示不泄露故障细节的恢复页，使未登录用户仍能找到独立管理中心入口；API 和 WebSocket 继续关闭访问。“认证设置”按登录设备列出浏览器、对端 IP 和活动时间；注销一台设备会同时撤销它的 DSH Session 与关联 Management Session。管理中心也可切换到公开独立 Origin，或使用映射到容器独立入口的本机宿主机端口（默认 `3081`，可自定义）。
同一浏览器连续 5 次输入错误密码后，从等待 30 秒开始，后续失败会使等待时间翻倍，最高 15 分钟。每个浏览器来源还限制为每小时 12 次、每 24 小时 24 次失败；整个 DSH 实例跨所有来源使用更宽的防洪泛限制：每分钟 20 次、每小时 60 次、每 24 小时 120 次失败。浏览器只展示 Access Manager 的判定和剩余时间；密码验证成功会清除当前浏览器的连续失败和来源滚动窗口，不清除实例总量记录。
遗失凭据时只能从交互式 Root 控制台恢复。密码输入会关闭回显，且不接受命令参数或管道输入：
```bash
docker exec -it --user root deepseek-harness dsh-platform access status
docker exec -it --user root deepseek-harness dsh-platform access reset
docker exec -it --user root deepseek-harness dsh-platform access clear-retry
docker exec -it --user root deepseek-harness dsh-platform access clear-retry --global-only
docker exec -it --user root deepseek-harness dsh-platform access generate-key
```
`access clear-retry` 使用 `y/[n]` 确认，会清除全部浏览器的重试等待、来源滚动窗口和实例总量失败窗口。添加 `--global-only` 时只清除实例总量窗口，保留每个浏览器的重试等待与来源滚动限制。两种形式都不修改凭据或会话。
全新空卷直接显示普通管理员注册页，不需要密钥。只有旧部署迁移或认证状态损坏时，才使用 `access generate-key` 生成十分钟有效、单次使用的认证重置密钥；恢复页可据此重新创建管理员账户。
常用命令：
```bash
docker exec deepseek-harness dsh-platform status
docker exec deepseek-harness dsh-platform check
docker exec deepseek-harness dsh-platform update --wait
docker exec deepseek-harness dsh-platform stop --wait
docker exec deepseek-harness dsh-platform start --wait
docker exec deepseek-harness dsh-platform restart --wait
docker exec deepseek-harness dsh-platform rollback
```
Stable 跟随受支持的 DSH 版本。Experimental 可以安装经过验证的更新上游 DSH，并在激活前创建数据快照。返回 Stable 可能丢弃快照之后写入的数据。
## 系统插件
Container Environment 包含以下 DSH-Docker 集成：
| 插件 | 用途 |
| --- | --- |
| `@dsh-docker/platform-management` | 在 DSH 设置中增加“平台管理”，用于更新、运行维护、日志、系统资源和当前浏览器会话退出；该集成由平台托管 |
| `@dsh-docker/settings-navigation` | 让桌面端设置目录独立滚动，并在窄屏提供目录与详情分级导航 |
| `@dsh-docker/settings-document-editor` | 将只能在桌面打开配置文件的操作替换为可选的浏览器 `settings.yaml` 编辑器 |
除“平台管理”本身外，已安装的系统插件可在 DSH 的“平台管理”中启用或禁用；“平台管理”不能修改自身。独立的“DSH 管理中心”可以安装、卸载、启用或禁用 Environment 随附的系统插件，并能在“平台管理”缺失时将其恢复。未提交的选择只保留在当前页面；点击应用后，即使 DSH 已不可用或用于激活修改的重启失败，修改仍会保留，并在下次成功启动时生效。安装只使用经过验证的本地 Environment 资产，不会从 GitHub 或 npm 下载。第三方用户插件与系统插件分开管理，不会被视为系统插件。
“平台管理”和“设置文档编辑器”通过已认证的 DSH Session 使用 DSH 侧受限平台接口，不要求用户另行取得 Management Session。“平台管理”可以撤销当前浏览器的 DSH Session 及其关联的 Management Session，但用户名、密码和完整认证设置仍只能在独立管理中心或 Root CLI 中修改。
## 用户插件
独立管理中心会列出 DSH Web Profile 中的 Bundle 用户插件，并可精确启用、禁用或卸载指定插件。应用修改时会为完整 Web Profile 创建快照，并在 DSH 可用时只重启 DSH。若 DSH 已停止或重启失败，已提交的修改仍会保留，用户可继续处理其他故障插件后再尝试启动。插件安装仍使用 DSH 正常插件流程或容器终端；恢复页面不安装任意软件包。
## 系统技能与用户技能
DSH Docker 内置已签名的 `dsh-docker-operations` System Skill，为 Agent 提供文件、开发工具、插件、生命周期、日志、更新、恢复、网络和权限等容器环境操作指引，也可显式调用 `/dsh-docker-operations`。
系统技能可在 DSH 内的“平台管理”和独立管理中心的“系统技能”中管理，变更会立即生效，无需重启 DSH。独立管理中心支持安装、卸载、启用和禁用；“平台管理”可恢复缺失技能并启禁用。项目或用户的同名 Skill 仍按 DSH 原生优先级覆盖内置副本。
独立管理中心还会列出已配置 DSH 与 Agent Skill 根中的原生用户技能，可精确启用、禁用或删除指定条目，不会修改项目级 Skill。
## 安全提醒
已认证的 DSH Session 拥有完整 DSH 权限；Management Session 还可使用 root 容器终端和 root 文件操作，因此管理员可以读取凭据、执行命令并修改容器数据。远程访问必须使用 HTTPS 和可信网络边界。
仓库 Compose 默认关闭 DSH/Agent 的免密码 sudo。只有 DSH 或 Agent 明确需要 Root 权限时才设置 `DSH_SUDO_ENABLED=true`。不了解影响时，不要同时使用 sudo、特权模式、Docker Socket 或敏感宿主机挂载。
## 详细文档
[完整指南](docs/cn/guide.md)包括：
- 部署前提、两种 Compose 布局和 Docker Run；
- 全部配置变量、数据存储、Gateway 和远程访问行为；
- 平台架构、在线更新、系统与用户插件、系统与用户技能、日志、信任、恢复和回滚；
- 独立恢复工具、发布自动化、本地构建、测试和开发工具。
upstream/container/README.md
+15
View File
@@ -0,0 +1,15 @@
# Container source layout
The container source tree follows runtime ownership boundaries:
```text
container/
├── platform/ Stage-0, Bootstrap, shared contracts, and release tools
├── control-plane/ Bootstrap-owned services and platform managers
├── environment/ Reloadable definition, workloads, and packaged resources
└── test/ Container and Compose integration checks
```
`platform/` contains the trust boundary, Bootstrap, and shared protocol code. Persistent services and managers belong under `control-plane/`; the complete Container Environment source belongs under `environment/`. A Component Manifest lives with the implementation it starts. Environment inputs which are not processes live under `environment/resources/`: DSH overlays under `resources/plugins/<plugin>/`, exact source patches under `resources/patches/`, and Agent guidance under `resources/skills/<skill>/`. Environment-owned components, patches, System Plugins, and System Skills are identified by `id` and SHA-256 rather than independent release versions.
Bootstrap archives preserve the same `platform/` and `control-plane/` top-level layout. Image and Store References therefore resolve through the same `/run/dsh-platform/views/bootstrap` runtime view, without compatibility links or duplicate component copies.
upstream/container/control-plane/definition.json
+25
View File
@@ -0,0 +1,25 @@
{
"schema": 1,
"components": [
{
"id": "gateway",
"source": "services/gateway/component.json"
},
{
"id": "access-manager",
"source": "services/access-manager/component.json"
},
{
"id": "outbound-proxy",
"source": "services/outbound-proxy/component.json"
},
{
"id": "platform-recovery",
"source": "hooks/recovery/component.json"
},
{
"id": "platform-management",
"source": "services/management/component.json"
}
]
}
upstream/container/control-plane/hooks/dsh-web-ready.mjs
+201
View File
@@ -0,0 +1,201 @@
#!/usr/bin/env node
import { request } from 'node:http'
import { realpathSync } from 'node:fs'
import { fileURLToPath } from 'node:url'
const RUN_ROOT = process.env.DSH_PLATFORM_RUN ?? '/run/dsh-platform'
function fetchResponse(host, port, path, options = {}) {
return new Promise((resolve, reject) => {
const outgoing = request({ hostname: host, port, path, ...options }, response => {
const chunks = []
response.on('data', chunk => chunks.push(chunk))
response.once('end', () => {
resolve({
body: Buffer.concat(chunks),
headers: response.headers,
status: response.statusCode,
})
})
})
outgoing.setTimeout(2_000, () => outgoing.destroy(new Error('DSH Web request timed out')))
outgoing.once('error', reject)
outgoing.end(options.body)
})
}
async function fetch(host, port, path, options = {}) {
const response = await fetchResponse(host, port, path, options)
if (response.status !== 200) throw new Error(`${path} returned HTTP ${String(response.status)}`)
return response.body
}
function delay(milliseconds) {
return new Promise(resolve => setTimeout(resolve, milliseconds))
}
function lifecycleReadiness() {
const body = Buffer.from('{}')
return new Promise((resolve, reject) => {
const outgoing = request({
socketPath: `${RUN_ROOT}/dsh-lifecycle.sock`,
path: '/v1/runtime/readiness',
method: 'POST',
headers: {
'content-length': body.byteLength,
'content-type': 'application/json',
},
}, response => {
const chunks = []
response.on('data', chunk => chunks.push(chunk))
response.once('end', () => {
if (response.statusCode !== 200) {
reject(new Error(`DSH lifecycle readiness returned HTTP ${String(response.statusCode)}`))
return
}
try {
const value = JSON.parse(Buffer.concat(chunks).toString('utf8'))
resolve({
generation: typeof value?.generation === 'string' ? value.generation : null,
ready: value?.ready === true,
readyUrl: typeof value?.readyUrl === 'string' ? value.readyUrl : null,
})
} catch (error) {
reject(error)
}
})
})
outgoing.setTimeout(2_000, () => outgoing.destroy(new Error('DSH lifecycle readiness timed out')))
outgoing.once('error', reject)
outgoing.end(body)
})
}
async function waitForManagedReady(timeoutMs = 20_000) {
const deadline = Date.now() + timeoutMs
while (Date.now() < deadline) {
const readiness = await lifecycleReadiness()
if (readiness.ready) return readiness
await delay(100)
}
throw new Error('DSH managed Web boot did not complete')
}
async function authenticationHeaders(host, port, readyUrl) {
if (readyUrl === null) return {}
const url = new URL(readyUrl)
if (url.protocol !== 'http:' || url.hostname !== host || Number(url.port) !== port
|| url.pathname !== '/' || url.username !== '' || url.password !== '' || url.hash !== ''
|| url.searchParams.getAll('token').length !== 1) {
throw new Error('DSH Web readiness URL is invalid')
}
const exchange = await fetchResponse(host, port, `${url.pathname}${url.search}`, {
headers: { 'cache-control': 'no-store' },
})
const cookie = exchange.headers['set-cookie']?.[0]?.split(';', 1)[0]
// dsh 0.2.x redirects to './' instead of '/'
if (exchange.status !== 303 || !['/', './'].includes(exchange.headers.location) || typeof cookie !== 'string' || cookie === '') {
throw new Error('DSH Web authentication exchange failed')
}
return { cookie }
}
async function verifyBootManifest(host, port, headers) {
const html = (await fetch(host, port, '/', { headers })).toString('utf8')
const match = /(?:window\.__DSH_BOOT__|globalThis\["__DSH_BOOT__"\])\s*=\s*(\{.*?\})<\/script>/s.exec(html)
if (match === null) throw new Error('DSH boot manifest is unavailable')
const manifest = JSON.parse(match[1])
if (!Array.isArray(manifest.entries)) throw new Error('DSH boot manifest entries are invalid')
if (manifest.batches !== undefined && !Array.isArray(manifest.batches)) {
throw new Error('DSH boot manifest batches are invalid')
}
const resources = [
...manifest.entries.map(entry => entry?.url),
...(manifest.batches ?? []).map(batch => batch?.url),
]
await Promise.all([...new Set(resources)].map(async resource => {
// dsh 0.2.x emits plugin URLs relative to the root ('plugins/…'), 0.1.x absolute ('/plugins/…')
const url = typeof resource === 'string' ? `/${resource.replace(/^\/+/, '')}` : resource
if (typeof resource !== 'string' || !url.startsWith('/plugins/')) {
throw new Error('DSH boot manifest contains an invalid Plugin URL')
}
await fetch(host, port, url, { headers })
}))
return JSON.stringify({
batches: manifest.batches ?? [],
entries: manifest.entries,
rev: manifest.rev ?? null,
})
}
async function verifyPluginInventory(host, port, headers) {
const body = JSON.stringify({
type: 'client-request',
rpcId: 'dsh-platform-readiness',
method: 'pluginInventory/list',
payload: { args: {} },
})
const response = JSON.parse((await fetch(host, port, '/api/pluginInventory/list', {
method: 'POST',
headers: {
...headers,
'content-length': Buffer.byteLength(body),
'content-type': 'application/json',
},
body,
})).toString('utf8'))
if (response?.rpcId !== 'dsh-platform-readiness' || response?.result?.ok !== true) {
const message = response?.result?.error?.message
throw new Error(`DSH Plugin inventory is unavailable${typeof message === 'string' ? `: ${message}` : ''}`)
}
const entries = response.result.value?.entries
if (!Array.isArray(entries)) throw new Error('DSH Plugin inventory entries are invalid')
const unhealthy = entries.filter(entry => entry?.enabled === true && entry?.fiberPhase !== 'active')
if (unhealthy.length > 0) {
const detail = unhealthy.map(entry => `${String(entry?.moduleName ?? entry?.entryId ?? 'unknown')} (${String(entry?.fiberPhase)})`).join(', ')
throw new Error(`DSH Plugins are not active: ${detail}`)
}
return JSON.stringify(entries.map(entry => ({
enabled: entry?.enabled === true,
entryId: entry?.entryId ?? null,
fiberPhase: entry?.fiberPhase ?? null,
moduleName: entry?.moduleName ?? null,
})).sort((left, right) => JSON.stringify(left).localeCompare(JSON.stringify(right))))
}
function sameLifecycle(left, right) {
return left?.generation === right?.generation && left?.readyUrl === right?.readyUrl
}
export async function verifyDshWebReady({
host = '127.0.0.1',
port = 3079,
managedReady = waitForManagedReady,
} = {}) {
const firstReadiness = await managedReady()
const headers = await authenticationHeaders(host, port, firstReadiness?.readyUrl ?? null)
const firstManifest = await verifyBootManifest(host, port, headers)
const firstInventory = await verifyPluginInventory(host, port, headers)
const middleReadiness = await managedReady()
if (!sameLifecycle(firstReadiness, middleReadiness)) {
throw new Error('DSH lifecycle changed during Web readiness verification')
}
const secondManifest = await verifyBootManifest(host, port, headers)
const secondInventory = await verifyPluginInventory(host, port, headers)
const finalReadiness = await managedReady()
if (!sameLifecycle(firstReadiness, finalReadiness)) {
throw new Error('DSH lifecycle changed during Web readiness verification')
}
if (firstManifest !== secondManifest) {
throw new Error('DSH boot manifest changed during Web readiness verification')
}
if (firstInventory !== secondInventory) {
throw new Error('DSH Plugin inventory changed during Web readiness verification')
}
}
if (process.argv[1] !== undefined
&& realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url))) {
await verifyDshWebReady()
}
upstream/container/control-plane/hooks/recovery/component.json
+21
View File
@@ -0,0 +1,21 @@
{
"schema": 1,
"id": "platform-recovery",
"type": "oneshot",
"command": {
"executable": "/usr/local/bin/node",
"args": ["/run/dsh-platform/views/bootstrap/control-plane/hooks/recovery/index.mjs"],
"timeoutSeconds": 120
},
"environment": {},
"lifecycle": {
"prepare": null,
"preStart": null,
"postStart": null,
"preStop": null,
"stop": null,
"postStop": null
},
"health": null,
"logging": { "stdout": true, "stderr": true }
}
upstream/container/control-plane/hooks/recovery/index.mjs
+29
View File
@@ -0,0 +1,29 @@
#!/usr/bin/env node
import { realpathSync } from 'node:fs'
import { join } from 'node:path'
import { fileURLToPath } from 'node:url'
import { PlatformActivator } from '../../modules/updater/lib/activator.mjs'
import { UpdateJournal } from '../../modules/updater/lib/journal.mjs'
import { recoverInterruptedUpdate } from '../../modules/updater/lib/recovery.mjs'
import { PersistentStateSnapshots } from '../../modules/updater/lib/snapshots.mjs'
import { PlatformPaths } from '../../../platform/lib/paths.mjs'
export async function recoverPlatformUpdateBeforeDshStart({
dataRoot = process.env.DSH_PLATFORM_DATA ?? '/data/platform',
runRoot = process.env.DSH_PLATFORM_RUN ?? '/run/dsh-platform',
dshHome = process.env.DSH_HOME ?? '/data/dsh',
} = {}) {
const paths = new PlatformPaths(dataRoot, runRoot)
return recoverInterruptedUpdate({
journal: new UpdateJournal(join(paths.updaterStateRoot, 'transaction.json')),
snapshots: new PersistentStateSnapshots({ root: paths.snapshotsRoot, sourceRoot: dshHome }),
activator: new PlatformActivator({ dataRoot, runRoot }),
resume: false,
})
}
if (process.argv[1] !== undefined
&& realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url))) {
await recoverPlatformUpdateBeforeDshStart()
}
upstream/container/control-plane/modules/file-manager/archives.mjs
+110
View File
@@ -0,0 +1,110 @@
import { spawn } from 'node:child_process'
import { lstat, mkdir, readdir, readlink, rm } from 'node:fs/promises'
import { dirname, join, relative, resolve, sep } from 'node:path'
import { FileManagerError } from './index.mjs'
export const ARCHIVE_FORMATS = Object.freeze({
zip: { extension: '.zip', command: 'zip' },
'7z': { extension: '.7z', command: '7z' },
'tar.gz': { extension: '.tar.gz', command: 'tar' },
})
export function normalizeArchiveFormat(value) {
if (!Object.hasOwn(ARCHIVE_FORMATS, value)) throw new FileManagerError('archive format is invalid')
return value
}
function run(command, args, { cwd, signal, capture = false } = {}) {
return new Promise((resolvePromise, reject) => {
const child = spawn(command, args, { cwd, signal, stdio: ['ignore', capture ? 'pipe' : 'ignore', 'pipe'] })
let errorOutput = ''
let output = ''
if (capture) {
child.stdout.setEncoding('utf8')
child.stdout.on('data', chunk => { if (output.length < 8 * 1024 * 1024) output += chunk })
}
child.stderr.setEncoding('utf8')
child.stderr.on('data', chunk => { if (errorOutput.length < 16_384) errorOutput += chunk })
child.once('error', reject)
child.once('exit', (code, signalName) => {
if (code === 0) resolvePromise(output)
else reject(new FileManagerError(
`archive tool failed${errorOutput.trim() === '' ? '' : `: ${errorOutput.trim()}`}`,
400,
signalName === 'SIGTERM' ? 'FILE_TASK_CANCELLED' : 'ARCHIVE_TOOL_FAILED',
))
})
})
}
function validateEntryNames(names) {
for (const original of names) {
const name = original.replace(/^(\.\/)+/u, '')
if (name === '') continue
if (name.includes('\0') || name.startsWith('/') || name.split('/').includes('..')) {
throw new FileManagerError('archive contains an unsafe path', 400, 'ARCHIVE_UNSAFE')
}
}
}
function inside(root, path) {
const value = relative(root, path)
return value === '' || (!value.startsWith(`..${sep}`) && value !== '..')
}
export async function validateExtractedTree(root) {
const pending = [root]
let entries = 0
let bytes = 0
while (pending.length > 0) {
const path = pending.pop()
const details = await lstat(path)
if (details.isDirectory() && !details.isSymbolicLink()) {
for (const name of await readdir(path)) pending.push(join(path, name))
} else if (details.isSymbolicLink()) {
const target = await readlink(path)
if (target.startsWith('/') || !inside(root, resolve(dirname(path), target))) {
throw new FileManagerError('archive contains a symbolic link outside its destination', 400, 'ARCHIVE_UNSAFE')
}
} else if (!details.isFile()) {
throw new FileManagerError('archive contains an unsupported special file', 415, 'ARCHIVE_UNSAFE')
}
entries += 1
if (details.isFile()) bytes += details.size
}
return { entries: Math.max(0, entries - 1), bytes }
}
export async function createArchive({ format, sourceRoot, output, entries = ['.'], signal }) {
normalizeArchiveFormat(format)
if (!Array.isArray(entries) || entries.length === 0 || entries.some(entry => typeof entry !== 'string' || entry === '' || entry.includes('\0') || entry.includes('/'))) {
throw new FileManagerError('archive entry names are invalid')
}
if (format === 'zip') await run('zip', ['-q', '-r', output, '--', ...entries], { cwd: sourceRoot, signal })
else if (format === '7z') await run('7z', ['a', '-bd', '-y', '-t7z', output, '--', ...entries], { cwd: sourceRoot, signal })
else await run('tar', ['-czf', output, '-C', sourceRoot, '--', ...entries], { signal })
}
export async function extractArchive({ format, archive, output, signal }) {
normalizeArchiveFormat(format)
await mkdir(output, { recursive: true })
try {
if (format === 'zip') {
validateEntryNames((await run('unzip', ['-Z1', archive], { signal, capture: true })).split('\n'))
await run('unzip', ['-qq', archive, '-d', output], { signal })
} else if (format === '7z') {
const listed = await run('7z', ['l', '-slt', '--', archive], { signal, capture: true })
validateEntryNames(listed.split('\n').filter(line => line.startsWith('Path = ')).slice(1).map(line => line.slice(7)))
await run('7z', ['x', '-bd', '-y', `-o${output}`, '--', archive], { signal })
} else {
validateEntryNames((await run('tar', ['-tzf', archive], { signal, capture: true })).split('\n'))
await run('tar', ['--extract', '--gzip', '--file', archive, '--directory', output, '--no-same-owner', '--no-same-permissions'], { signal })
}
return await validateExtractedTree(output)
} catch (error) {
await rm(output, { recursive: true, force: true }).catch(() => {})
throw error
}
}
export const archiveInternals = Object.freeze({ inside, run, validateEntryNames })
upstream/container/control-plane/modules/file-manager/editor.mjs
+78
View File
@@ -0,0 +1,78 @@
import { link, open, rename, rm, stat } from 'node:fs/promises'
import { dirname, join } from 'node:path'
import { randomUUID } from 'node:crypto'
import {
FileManagerError, FileRevisionConflictError, MAX_TEXT_BYTES, fileError,
fileManagerInternals, isManagedPath, normalizeAbsolutePath,
} from './index.mjs'
async function syncDirectory(path) {
const handle = await open(path, 'r')
try { await handle.sync() } finally { await handle.close() }
}
function bytes(value) {
if (typeof value !== 'string') throw new FileManagerError('content must be a string')
const result = Buffer.from(value, 'utf8')
if (result.byteLength > MAX_TEXT_BYTES) throw new FileManagerError('file is too large to edit', 413, 'FILE_TOO_LARGE')
if (result.includes(0)) throw new FileManagerError('content contains NUL', 415, 'FILE_TYPE_UNSUPPORTED')
return result
}
export class AtomicFileEditor {
constructor({ isManaged = isManagedPath } = {}) {
this.isManaged = isManaged
}
async write(value, content, revision, { create = false } = {}) {
const path = normalizeAbsolutePath(value)
const body = bytes(content)
const parent = dirname(path)
let original
try {
original = await stat(path, { bigint: true })
if (!original.isFile()) throw new FileManagerError('path is not a regular file', 415, 'FILE_TYPE_UNSUPPORTED')
if (create) throw new FileManagerError('file already exists', 409, 'FILE_EXISTS')
if (typeof revision !== 'string' || revision !== fileManagerInternals.revisionFor(path, original)) {
throw new FileRevisionConflictError()
}
} catch (error) {
if (error?.code !== 'ENOENT') throw fileError(error)
if (!create) throw fileError(error)
if (revision !== null && revision !== undefined) throw new FileManagerError('new file revision must be null')
}
const temporary = join(parent, `.dsh-edit-${randomUUID()}.tmp`)
let handle
try {
handle = await open(temporary, 'wx', original === undefined ? 0o644 : Number(original.mode & 0o7777n))
await handle.writeFile(body)
await handle.sync()
if (original !== undefined) {
await handle.chmod(Number(original.mode & 0o7777n))
const current = await handle.stat({ bigint: true })
if (current.uid !== original.uid || current.gid !== original.gid) {
try { await handle.chown(Number(original.uid), Number(original.gid)) } catch {
throw new FileManagerError('cannot preserve file ownership; use the terminal', 403, 'OWNERSHIP_PRESERVE_FAILED')
}
}
}
await handle.close()
handle = undefined
if (original === undefined) {
await link(temporary, path)
await rm(temporary)
} else await rename(temporary, path)
await syncDirectory(parent)
const saved = await stat(path, { bigint: true })
return {
path, revision: fileManagerInternals.revisionFor(path, saved), size: body.byteLength,
mode: Number(saved.mode & 0o7777n), modifiedAt: new Date(Number(saved.mtimeNs / 1_000_000n)).toISOString(),
managed: this.isManaged(path),
}
} catch (error) {
await handle?.close().catch(() => {})
await rm(temporary, { force: true }).catch(() => {})
throw fileError(error)
}
}
}
upstream/container/control-plane/modules/file-manager/index.mjs
+518
View File
@@ -0,0 +1,518 @@
import { createHash, randomUUID } from 'node:crypto'
import { constants } from 'node:fs'
import { access, lstat, readFile, readdir, readlink, realpath, stat } from 'node:fs/promises'
import { basename, dirname, join, normalize, resolve } from 'node:path'
export const MAX_PATH_BYTES = 4096
export const MAX_TEXT_BYTES = 2 * 1024 * 1024
export const DEFAULT_LIST_LIMIT = 200
export const MAX_LIST_LIMIT = 1000
export const SEARCH_MAX_DEPTH = 16
export const SEARCH_MAX_SCANNED = 10_000
export const SEARCH_MAX_RESULTS = 1000
const SORTS = new Set(['name', 'size', 'type', 'modifiedAt'])
const ORDERS = new Set(['asc', 'desc'])
export class FileManagerError extends Error {
constructor(message, statusCode = 400, code = 'FILE_INVALID') {
super(message)
this.name = 'FileManagerError'
this.statusCode = statusCode
this.code = code
}
}
export class FileRevisionConflictError extends FileManagerError {
constructor(message = 'file revision changed') {
super(message, 409, 'REVISION_CONFLICT')
this.name = 'FileRevisionConflictError'
}
}
function statusCode(error) {
if (['EACCES', 'EPERM', 'EROFS'].includes(error?.code)) return 403
if (['ENOENT', 'ENOTDIR'].includes(error?.code)) return 404
if (error?.code === 'EFBIG') return 413
return 400
}
export function fileError(error, fallback = 'file operation failed') {
if (error instanceof FileManagerError) return error
const wrapped = new FileManagerError(error instanceof Error ? error.message : fallback, statusCode(error), error?.code ?? 'FILE_ERROR')
if (error instanceof Error) wrapped.cause = error
return wrapped
}
export function normalizeAbsolutePath(value) {
if (typeof value !== 'string' || value.length === 0) throw new FileManagerError('path must be a non-empty string')
if (!value.startsWith('/')) throw new FileManagerError('path must be absolute')
if (Buffer.byteLength(value, 'utf8') > MAX_PATH_BYTES) throw new FileManagerError('path is too long')
if(/[\u0000-\u001f\u007f]/u.test(value)) throw new FileManagerError('path contains control characters')
const path = normalize(value)
if (!path.startsWith('/')) throw new FileManagerError('path must be absolute')
return path
}
export function createManagedPathMatcher(platformData = '/data/platform') {
const data = normalizeAbsolutePath(platformData)
const roots = Object.freeze(['/opt/dsh-platform', '/run/dsh-platform', join(data, 'state'), join(data, 'store')])
return value => {
const path = normalizeAbsolutePath(value)
return roots.some(root => path === root || path.startsWith(`${root}/`))
}
}
export const isManagedPath = createManagedPathMatcher()
export function fileManagerLocations({
platformData = '/data/platform', dshHome = '/data/dsh', defaultWorkspace = '/workspace',
} = {}) {
const defaultPath = normalizeAbsolutePath(defaultWorkspace)
const shortcuts = [...new Set([defaultPath, normalizeAbsolutePath(dshHome), normalizeAbsolutePath(platformData), '/'])]
return Object.freeze({ defaultPath, shortcuts: Object.freeze(shortcuts) })
}
function parseIdentityFile(value, idIndex) {
const result = new Map()
for (const line of value.split('\n')) {
if (line === '' || line.startsWith('#')) continue
const fields = line.split(':')
const id = Number(fields[idIndex])
if (fields[0] !== '' && Number.isSafeInteger(id) && id >= 0 && !result.has(id)) result.set(id, fields[0])
}
return result
}
export class UnixIdentityResolver {
constructor({ passwdPath = '/etc/passwd', groupPath = '/etc/group' } = {}) {
this.passwdPath = passwdPath
this.groupPath = groupPath
this.loading = undefined
}
async #identities() {
this.loading ??= Promise.all([
readFile(this.passwdPath, 'utf8').then(value => parseIdentityFile(value, 2), () => new Map()),
readFile(this.groupPath, 'utf8').then(value => parseIdentityFile(value, 2), () => new Map()),
])
return this.loading
}
async resolve(uid, gid) {
const [users, groups] = await this.#identities()
return { user: users.get(uid) ?? String(uid), group: groups.get(gid) ?? String(gid) }
}
async userId(value) { return this.#identityId(value, 0, 'user') }
async groupId(value) { return this.#identityId(value, 1, 'group') }
async #identityId(value, index, label) {
if (typeof value !== 'string' || value === '' || value.length > 256 || /[:\u0000-\u001f\u007f]/u.test(value)) {
throw new FileManagerError(`${label} is invalid`, 400, 'FILE_IDENTITY_INVALID')
}
if (/^[0-9]+$/u.test(value)) {
const id = Number(value)
if (Number.isSafeInteger(id) && id >= 0 && id < 0xffff_ffff) return id
throw new FileManagerError(`${label} id is invalid`, 400, 'FILE_IDENTITY_INVALID')
}
const identities = (await this.#identities())[index]
for (const [id, name] of identities) if (name === value) return id
throw new FileManagerError(`${label} was not found`, 400, 'FILE_IDENTITY_INVALID')
}
}
function kind(stats) {
if (stats.isFile()) return 'file'
if (stats.isDirectory()) return 'directory'
if (stats.isSymbolicLink()) return 'symlink'
if (stats.isBlockDevice()) return 'block-device'
if (stats.isCharacterDevice()) return 'character-device'
if (stats.isFIFO()) return 'fifo'
if (stats.isSocket()) return 'socket'
return 'unknown'
}
function metadata(stats) {
return {
type: kind(stats),
size: Number(stats.size),
mode: Number(stats.mode & 0o7777n),
uid: Number(stats.uid),
gid: Number(stats.gid),
modifiedAt: new Date(Number(stats.mtimeNs / 1_000_000n)).toISOString(),
createdAt: new Date(Number(stats.birthtimeNs / 1_000_000n)).toISOString(),
}
}
function revisionFor(path, stats, extra = '') {
return `sha256:${createHash('sha256').update(JSON.stringify([
path, stats.dev.toString(), stats.ino.toString(), kind(stats), stats.size.toString(),
stats.mode.toString(), stats.uid.toString(), stats.gid.toString(), stats.mtimeNs.toString(), stats.ctimeNs.toString(), extra,
])).digest('hex')}`
}
async function permissions(path) {
const allowed = async mode => access(path, mode).then(() => true, () => false)
const [readable, writable] = await Promise.all([allowed(constants.R_OK), allowed(constants.W_OK)])
return { readable, writable }
}
async function symlinkDetails(path, stats) {
if (!stats.isSymbolicLink()) return {}
const linkTarget = await readlink(path)
try {
const target = await stat(path, { bigint: true })
return { linkTarget, targetExists: true, targetType: kind(target), realPath: await realpath(path) }
} catch (error) {
if (error?.code === 'ENOENT' || error?.code === 'ELOOP') return { linkTarget, targetExists: false, targetType: null, realPath: null }
throw error
}
}
async function describe(path, stats = undefined, managed = isManagedPath, identity = new UnixIdentityResolver()) {
const value = stats ?? await lstat(path, { bigint: true })
const link = await symlinkDetails(path, value)
const owner = await identity.resolve(Number(value.uid), Number(value.gid))
return {
name: basename(path) || '/',
path,
...metadata(value),
...owner,
...await permissions(path),
managed: managed(path),
revision: revisionFor(path, value, link.linkTarget ?? ''),
...link,
}
}
function compareEntries(left, right, sort, order) {
if (left.type === 'directory' && right.type !== 'directory') return -1
if (left.type !== 'directory' && right.type === 'directory') return 1
let result
if (sort === 'size') result = left.size - right.size
else if (sort === 'modifiedAt') result = left.modifiedAt.localeCompare(right.modifiedAt)
else if (sort === 'type') result = left.type.localeCompare(right.type)
else result = left.name.localeCompare(right.name, undefined, { numeric: true, sensitivity: 'base' })
if (result === 0) result = left.name.localeCompare(right.name)
return order === 'desc' ? -result : result
}
function encodeCursor(revision, offset) {
return Buffer.from(JSON.stringify({ revision, offset })).toString('base64url')
}
function decodeCursor(value, revision) {
if (value === undefined || value === null || value === '') return 0
try {
const parsed = JSON.parse(Buffer.from(value, 'base64url').toString('utf8'))
if (parsed.revision !== revision) throw new FileRevisionConflictError('directory changed while paging')
if (!Number.isSafeInteger(parsed.offset) || parsed.offset < 0) throw new Error('invalid offset')
return parsed.offset
} catch (error) {
if (error instanceof FileRevisionConflictError) throw error
throw new FileManagerError('cursor is invalid')
}
}
function listRevision(path, stats, entries) {
const digest = createHash('sha256')
for (const entry of entries) digest.update(`${entry.name}\0${entry.revision ?? entry.type}\0`)
return revisionFor(path, stats, digest.digest('hex'))
}
function directoryEntryType(entry) {
if (entry.isDirectory()) return 'directory'
if (entry.isFile()) return 'file'
if (entry.isSymbolicLink()) return 'symlink'
if (entry.isFIFO()) return 'fifo'
if (entry.isSocket()) return 'socket'
if (entry.isBlockDevice()) return 'block-device'
if (entry.isCharacterDevice()) return 'character-device'
return 'unknown'
}
function compareNames(left, right, order) {
if (left.type === 'directory' && right.type !== 'directory') return -1
if (left.type !== 'directory' && right.type === 'directory') return 1
const result = left.name.localeCompare(right.name, undefined, { numeric: true, sensitivity: 'base' })
|| left.name.localeCompare(right.name)
return order === 'desc' ? -result : result
}
function parseLimit(value, fallback = DEFAULT_LIST_LIMIT) {
if (value === undefined || value === null || value === '') return fallback
const limit = Number(value)
if (!Number.isSafeInteger(limit) || limit < 1 || limit > MAX_LIST_LIMIT) throw new FileManagerError(`limit must be between 1 and ${MAX_LIST_LIMIT}`)
return limit
}
function parseOffset(value) {
if (value === undefined || value === null || value === '') return undefined
const offset = Number(value)
if (!Number.isSafeInteger(offset) || offset < 0) throw new FileManagerError('offset must be a non-negative integer')
return offset
}
export class FileInventory {
constructor({ isManaged = isManagedPath, identity = new UnixIdentityResolver() } = {}) {
this.isManaged = isManaged
this.identity = identity
}
async stat(value) {
const path = normalizeAbsolutePath(value)
try {
const item = await describe(path, undefined, this.isManaged, this.identity)
return { ...item, realPath: item.realPath ?? await realpath(path).catch(() => path), parent: dirname(path) }
} catch (error) {
throw fileError(error)
}
}
async list(value, { cursor, offset: requestedOffset, limit, sort = 'name', order = 'asc' } = {}) {
const path = normalizeAbsolutePath(value)
if (!SORTS.has(sort)) throw new FileManagerError('sort is invalid')
if (!ORDERS.has(order)) throw new FileManagerError('order is invalid')
try {
const root = await lstat(path, { bigint: true })
if (!root.isDirectory()) throw new FileManagerError('path is not a directory', 415, 'FILE_TYPE_UNSUPPORTED')
const pageSize = parseLimit(limit)
const directOffset = parseOffset(requestedOffset)
if (directOffset !== undefined && cursor !== undefined && cursor !== null && cursor !== '') throw new FileManagerError('cursor and offset cannot be combined')
const names = await readdir(path, { withFileTypes: true })
let entries
let revision
let offset
if (sort === 'name') {
const summary = names.map(entry => ({ name: entry.name, type: directoryEntryType(entry) }))
summary.sort((left, right) => compareNames(left, right, order))
revision = listRevision(path, root, summary)
offset = directOffset ?? decodeCursor(cursor, revision)
entries = await Promise.all(summary.slice(offset, offset + pageSize).map(entry => (
describe(resolve(path, entry.name), undefined, this.isManaged, this.identity)
)))
} else {
entries = await Promise.all(names.map(entry => describe(resolve(path, entry.name), undefined, this.isManaged, this.identity)))
entries.sort((left, right) => compareEntries(left, right, sort, order))
revision = listRevision(path, root, entries)
offset = directOffset ?? decodeCursor(cursor, revision)
}
const page = sort === 'name' ? entries : entries.slice(offset, offset + pageSize)
const nextOffset = offset + page.length
return {
path,
realPath: await realpath(path),
revision,
entries: page,
nextCursor: nextOffset < names.length ? encodeCursor(revision, nextOffset) : null,
total: names.length,
managed: this.isManaged(path),
}
} catch (error) {
throw fileError(error)
}
}
async content(value) {
const path = normalizeAbsolutePath(value)
try {
const before = await stat(path, { bigint: true })
if (!before.isFile()) throw new FileManagerError('path is not a regular file', 415, 'FILE_TYPE_UNSUPPORTED')
if (before.size > BigInt(MAX_TEXT_BYTES)) throw new FileManagerError('file is too large to edit', 413, 'FILE_TOO_LARGE')
const bytes = await readFile(path)
const after = await stat(path, { bigint: true })
const beforeRevision = revisionFor(path, before)
if (beforeRevision !== revisionFor(path, after)) throw new FileRevisionConflictError('file changed while reading')
if (bytes.includes(0)) throw new FileManagerError('file is not UTF-8 text', 415, 'FILE_TYPE_UNSUPPORTED')
const content = new TextDecoder('utf-8', { fatal: true }).decode(bytes)
return {
path,
realPath: await realpath(path),
content,
encoding: 'utf-8',
newline: content.includes('\r\n') ? 'crlf' : 'lf',
revision: beforeRevision,
mode: Number(before.mode & 0o7777n),
size: bytes.byteLength,
modifiedAt: new Date(Number(before.mtimeNs / 1_000_000n)).toISOString(),
managed: this.isManaged(path),
}
} catch (error) {
throw fileError(error)
}
}
}
export class FileSearchManager {
constructor({ inventory = new FileInventory(), onState = () => {} } = {}) {
this.inventory = inventory
this.onState = onState
this.tasks = new Map()
}
start({ path: value, revision, query }) {
const path = normalizeAbsolutePath(value)
if (typeof query !== 'string' || query.trim().length === 0 || query.length > 256) throw new FileManagerError('search query is invalid')
const task = {
taskId: randomUUID(), operation: 'search', status: 'running', path, revision, query,
scanned: 0, matches: [], truncated: false, error: null, cancelRequested: false,
createdAt: new Date().toISOString(), updatedAt: new Date().toISOString(),
}
this.tasks.set(task.taskId, task)
this.#publish(task)
task.completion = this.#run(task).catch(error => {
task.status = task.cancelRequested ? 'cancelled' : 'failed'
task.error = error instanceof Error ? error.message : String(error)
this.#publish(task)
})
return this.public(task)
}
get(taskId, { cursor, limit } = {}) {
const task = this.tasks.get(taskId)
if (task === undefined) throw new FileManagerError('file task was not found', 404, 'FILE_TASK_NOT_FOUND')
const offset = cursor === undefined || cursor === '' ? 0 : Number(cursor)
if (!Number.isSafeInteger(offset) || offset < 0) throw new FileManagerError('task cursor is invalid')
const size = parseLimit(limit)
return { ...this.public(task), results: task.matches.slice(offset, offset + size), nextCursor: offset + size < task.matches.length ? String(offset + size) : null }
}
list() {
return [...this.tasks.values()].map(task => this.public(task)).reverse()
}
cancel(taskId) {
const task = this.tasks.get(taskId)
if (task === undefined) throw new FileManagerError('file task was not found', 404, 'FILE_TASK_NOT_FOUND')
if (task.status !== 'running') throw new FileManagerError('file task is already complete', 409, 'FILE_TASK_COMPLETE')
task.cancelRequested = true
this.#publish(task)
return this.public(task)
}
public(task) {
const { completion, matches, cancelRequested, ...value } = task
return { ...value, resultCount: matches.length, cancelRequested }
}
#publish(task) {
task.updatedAt = new Date().toISOString()
this.onState(this.public(task))
}
async #run(task) {
const root = await this.inventory.list(task.path, { limit: 1 })
if (task.revision !== undefined && task.revision !== null && task.revision !== root.revision) throw new FileRevisionConflictError('search root changed')
task.revision = root.revision
const needle = task.query.toLocaleLowerCase()
const pending = [{ path: task.path, depth: 0 }]
while (pending.length > 0 && task.scanned < SEARCH_MAX_SCANNED && task.matches.length < SEARCH_MAX_RESULTS) {
if (task.cancelRequested) {
task.status = 'cancelled'
this.#publish(task)
return
}
const current = pending.shift()
let names
try { names = await readdir(current.path, { withFileTypes: true }) } catch (error) {
if (['EACCES', 'EPERM', 'ENOENT'].includes(error?.code)) continue
throw error
}
for (const entry of names) {
task.scanned += 1
const child = resolve(current.path, entry.name)
const relative = child.slice(task.path === '/' ? 1 : task.path.length + 1)
if (relative.toLocaleLowerCase().includes(needle)) task.matches.push(await describe(child, undefined, this.inventory.isManaged, this.inventory.identity))
if (entry.isDirectory() && !entry.isSymbolicLink() && current.depth < SEARCH_MAX_DEPTH) pending.push({ path: child, depth: current.depth + 1 })
if (task.scanned >= SEARCH_MAX_SCANNED || task.matches.length >= SEARCH_MAX_RESULTS) break
}
this.#publish(task)
await new Promise(resolve => setImmediate(resolve))
}
task.truncated = pending.length > 0 || task.scanned >= SEARCH_MAX_SCANNED || task.matches.length >= SEARCH_MAX_RESULTS
task.status = 'success'
this.#publish(task)
}
}
export class FileSizeManager {
constructor({ inventory = new FileInventory(), onState = () => {} } = {}) {
this.inventory = inventory
this.onState = onState
this.tasks = new Map()
}
start({ path: value, revision } = {}) {
const path = normalizeAbsolutePath(value)
const task = {
taskId: randomUUID(), operation: 'size', status: 'running', path, revision,
bytes: 0, entries: 0, error: null, cancelRequested: false,
createdAt: new Date().toISOString(), updatedAt: new Date().toISOString(),
}
this.tasks.set(task.taskId, task)
this.#publish(task)
task.completion = this.#run(task).catch(error => {
task.status = task.cancelRequested ? 'cancelled' : 'failed'
task.error = error instanceof Error ? error.message : String(error)
this.#publish(task)
})
return this.public(task)
}
get(taskId) {
const task = this.tasks.get(taskId)
if (task === undefined) throw new FileManagerError('file task was not found', 404, 'FILE_TASK_NOT_FOUND')
return this.public(task)
}
list() { return [...this.tasks.values()].map(task => this.public(task)).reverse() }
cancel(taskId) {
const task = this.tasks.get(taskId)
if (task === undefined) throw new FileManagerError('file task was not found', 404, 'FILE_TASK_NOT_FOUND')
if (task.status !== 'running') throw new FileManagerError('file task is already complete', 409, 'FILE_TASK_COMPLETE')
task.cancelRequested = true
this.#publish(task)
return this.public(task)
}
public(task) {
const { completion, cancelRequested, ...value } = task
return { ...value, cancelRequested }
}
#publish(task) {
task.updatedAt = new Date().toISOString()
this.onState(this.public(task))
}
async #run(task) {
const root = await this.inventory.stat(task.path)
if (root.type !== 'directory') throw new FileManagerError('size target is not a directory', 415, 'FILE_TYPE_UNSUPPORTED')
if (typeof task.revision !== 'string' || task.revision !== root.revision) throw new FileRevisionConflictError('size target changed')
const pending = [task.path]
while (pending.length > 0) {
if (task.cancelRequested) {
task.status = 'cancelled'
this.#publish(task)
return
}
const directory = pending.pop()
for (const name of await readdir(directory)) {
const path = resolve(directory, name)
const details = await lstat(path, { bigint: true })
task.entries += 1
if (details.isDirectory() && !details.isSymbolicLink()) pending.push(path)
else if (details.isFile()) task.bytes += Number(details.size)
}
this.#publish(task)
await new Promise(resolveTask => setImmediate(resolveTask))
}
task.status = 'success'
this.#publish(task)
}
}
export const fileManagerInternals = Object.freeze({ describe, revisionFor, parseIdentityFile, parseLimit })
upstream/container/control-plane/modules/file-manager/tasks.mjs
+645
View File
@@ -0,0 +1,645 @@
import { randomUUID } from 'node:crypto'
import { chmod, cp, lchown, lstat, mkdir, open, readdir, readFile, readlink, rename, rm } from 'node:fs/promises'
import { basename, dirname, join } from 'node:path'
import { durableReplace } from '../../../platform/lib/atomic.mjs'
import { createArchive, extractArchive, normalizeArchiveFormat } from './archives.mjs'
import {
FileInventory, FileManagerError, FileRevisionConflictError, FileSearchManager, FileSizeManager, UnixIdentityResolver,
fileError, isManagedPath, normalizeAbsolutePath,
} from './index.mjs'
const OPERATIONS = new Set(['mkdir', 'touch', 'rename', 'copy', 'move', 'delete', 'attributes', 'archive', 'extract'])
const DEFAULT_PROTECTED_DELETE_ROOTS = ['/', '/data', '/data/dsh', '/data/platform', '/workspace', '/run/dsh-platform/deployment']
function now() { return new Date().toISOString() }
async function exists(path) {
return lstat(path).then(() => true, error => error?.code === 'ENOENT' ? false : Promise.reject(error))
}
async function syncDirectory(path) {
const handle = await open(path, 'r')
try { await handle.sync() } finally { await handle.close() }
}
function publicTask(task) {
const { completion, persisted, executor, resolveResult, rejectResult, cancelRequested, abortController, ...value } = task
return { ...value, cancelRequested: cancelRequested === true }
}
function cleanName(path) {
const name = basename(path)
if (name === '' || name === '.' || name === '..') throw new FileManagerError('destination name is invalid')
return name
}
function renamedPath(path, index) {
const dot = basename(path).lastIndexOf('.')
const directory = dirname(path)
const name = basename(path)
const stem = dot > 0 ? name.slice(0, dot) : name
const extension = dot > 0 ? name.slice(dot) : ''
return join(directory, `${stem} (${String(index)})${extension}`)
}
async function uniqueDestination(path, conflict) {
if (!await exists(path)) return path
if (conflict === 'reject') throw new FileManagerError('destination already exists', 409, 'FILE_EXISTS')
if (conflict === 'overwrite') return path
for (let index = 1; index < 10_000; index += 1) {
const candidate = renamedPath(path, index)
if (!await exists(candidate)) return candidate
}
throw new FileManagerError('could not choose a unique destination', 409, 'FILE_EXISTS')
}
async function copyEntry(source, staging) {
const sourceStats = await lstat(source)
if (sourceStats.isDirectory()) {
await cp(source, staging, { recursive: true, dereference: false, errorOnExist: true, force: false, preserveTimestamps: true })
} else if (sourceStats.isSymbolicLink()) {
const target = await readlink(source)
await import('node:fs/promises').then(fs => fs.symlink(target, staging))
} else if (sourceStats.isFile()) {
await cp(source, staging, { dereference: false, errorOnExist: true, force: false, preserveTimestamps: true })
} else throw new FileManagerError('special files cannot be copied', 415, 'FILE_TYPE_UNSUPPORTED')
}
async function treeMetrics(path) {
const value = await lstat(path)
if (!value.isDirectory() || value.isSymbolicLink()) return { entries: 1, bytes: value.isFile() ? value.size : 0 }
let entries = 1
let bytes = 0
for (const name of await readdir(path)) {
const child = await treeMetrics(join(path, name))
entries += child.entries
bytes += child.bytes
}
return { entries, bytes }
}
function normalizeAttributes(value) {
if (value === null || typeof value !== 'object' || Array.isArray(value)
|| Object.keys(value).some(key => !['user', 'group', 'mode', 'recursive'].includes(key))
|| typeof value.user !== 'string' || typeof value.group !== 'string'
|| typeof value.mode !== 'string' || !/^[0-7]{3,4}$/u.test(value.mode)
|| typeof value.recursive !== 'boolean') throw new FileManagerError('file attributes are invalid')
return { user: value.user, group: value.group, mode: value.mode, recursive: value.recursive }
}
function assertAttributesApplied(path, details, { uid, gid, mode }, symbolicLink) {
const ownershipMatches = details.uid === uid && details.gid === gid
const modeMatches = symbolicLink || (details.mode & 0o7777) === mode
if (ownershipMatches && modeMatches) return
throw new FileManagerError(
`filesystem did not apply Unix ownership or mode for ${path}; use a filesystem mounted with Unix metadata support`,
403,
'FILE_ATTRIBUTES_UNSUPPORTED',
)
}
export class FileTaskManager {
constructor({
root, inventory = new FileInventory(), isManaged = isManagedPath,
protectedRoots = DEFAULT_PROTECTED_DELETE_ROOTS,
onState = () => {}, platformBusy = () => false, report = async () => {}, identity,
} = {}) {
if (typeof root !== 'string') throw new Error('file task root is required')
this.root = root
this.inventory = inventory
this.isManaged = isManaged
this.protectedRoots = new Set(protectedRoots.map(normalizeAbsolutePath))
this.onState = onState
this.platformBusy = platformBusy
this.report = report
this.identity = identity ?? inventory.identity ?? new UnixIdentityResolver()
this.tasks = new Map()
this.activeMutation = undefined
this.queue = []
this.queueTail = Promise.resolve()
this.search = new FileSearchManager({ inventory, onState: state => onState(state) })
this.sizes = new FileSizeManager({ inventory, onState: state => onState(state) })
}
get hasManagedMutation() { return this.queue.some(task => ['queued', 'running'].includes(task.status) && task.managed === true) }
wouldManage(input) {
if (['search', 'size'].includes(input?.operation)) return false
const sourceManaged = Array.isArray(input?.sources) && input.sources.some(source => typeof source?.path === 'string' && this.isManaged(source.path))
return sourceManaged || (typeof input?.destination === 'string' && this.isManaged(input.destination))
}
async initialize() {
await mkdir(this.root, { recursive: true })
const files = (await readdir(this.root)).filter(name => name.endsWith('.json')).sort()
for (const name of files) {
try {
const task = JSON.parse(await readFile(join(this.root, name), 'utf8'))
if (task.schema !== 1 || typeof task.taskId !== 'string') continue
this.tasks.set(task.taskId, task)
if (task.status === 'running') await this.#recover(task)
else if (task.status === 'queued' && ['upload', 'download'].includes(task.operation)) {
task.status = 'running'
await this.#recover(task)
} else if (task.status === 'queued') this.#schedule(task)
} catch (error) {
this.onState({ operation: 'recovery', status: 'failed', error: error instanceof Error ? error.message : String(error) })
}
}
await this.#prune()
}
start(input) {
if (input?.operation === 'search') return this.search.start(input)
if (input?.operation === 'size') return this.sizes.start(input)
if (input === null || typeof input !== 'object' || !OPERATIONS.has(input.operation)) throw new FileManagerError('file task operation is invalid')
const sources = Array.isArray(input.sources) ? input.sources.map(source => ({ path: normalizeAbsolutePath(source.path), revision: source.revision })) : []
const destination = input.destination === undefined ? null : normalizeAbsolutePath(input.destination)
const conflict = input.conflict ?? 'reject'
if (!['reject', 'overwrite', 'rename'].includes(conflict)) throw new FileManagerError('file task conflict mode is invalid')
if (['copy', 'move', 'delete', 'rename', 'archive', 'extract'].includes(input.operation) && sources.length === 0) throw new FileManagerError('file task sources are required')
if (input.operation === 'extract' && sources.length !== 1) throw new FileManagerError('archive extraction requires one source')
if (input.operation === 'attributes' && sources.length !== 1) throw new FileManagerError('attribute changes require one source')
if (['mkdir', 'touch', 'copy', 'move', 'rename', 'archive', 'extract'].includes(input.operation) && destination === null) throw new FileManagerError('file task destination is required')
if (input.operation === 'delete' && sources.some(source => this.protectedRoots.has(source.path))) throw new FileManagerError('protected root cannot be deleted', 403, 'PROTECTED_ROOT')
const managed = sources.some(source => this.isManaged(source.path)) || (destination !== null && this.isManaged(destination))
if (managed && this.platformBusy()) throw new FileManagerError('a platform operation is already running', 409, 'FILE_TASK_CONFLICT')
const task = {
schema: 1, taskId: randomUUID(), operation: input.operation,
status: this.queue.length === 0 ? 'running' : 'queued', phase: this.queue.length === 0 ? 'validating' : 'queued',
sources, destination, destinationRevision: input.destinationRevision ?? null, conflict, managed,
attributes: input.operation === 'attributes' ? normalizeAttributes(input.attributes) : null,
archiveFormat: ['archive', 'extract'].includes(input.operation) ? normalizeArchiveFormat(input.archiveFormat) : null,
processedBytes: 0, totalBytes: 0, processedEntries: 0, totalEntries: 0, currentPath: null,
published: [], staging: [], hidden: [], currentSource: null, currentDestination: null,
error: null, errorCode: null, queuePosition: null, createdAt: now(), updatedAt: now(), cancelRequested: false,
}
this.tasks.set(task.taskId, task)
this.#schedule(task)
return publicTask(task)
}
startTransfer({ operation, path, totalBytes = null, managed = false }, executor) {
if (!['upload', 'download'].includes(operation) || typeof executor !== 'function') {
throw new FileManagerError('file transfer task is invalid')
}
const normalizedPath = normalizeAbsolutePath(path)
if (managed && this.platformBusy()) throw new FileManagerError('a platform operation is already running', 409, 'FILE_TASK_CONFLICT')
let resolveResult
let rejectResult
const result = new Promise((resolve, reject) => { resolveResult = resolve; rejectResult = reject })
const task = {
schema: 1, taskId: randomUUID(), operation,
status: this.queue.length === 0 ? 'running' : 'queued', phase: this.queue.length === 0 ? 'transferring' : 'queued',
path: normalizedPath, sources: [], destination: operation === 'upload' ? normalizedPath : null,
destinationRevision: null, conflict: null, managed, attributes: null, archiveFormat: null,
processedBytes: 0, totalBytes, processedEntries: 0, totalEntries: 1, currentPath: normalizedPath,
published: [], staging: [], hidden: [], currentSource: null, currentDestination: null,
error: null, errorCode: null, queuePosition: null, createdAt: now(), updatedAt: now(), cancelRequested: false,
executor, resolveResult, rejectResult,
}
this.tasks.set(task.taskId, task)
this.#schedule(task)
return { task: publicTask(task), result }
}
list() {
return [...this.search.list(), ...this.sizes.list(), ...[...this.tasks.values()].map(publicTask)].sort((a, b) => b.createdAt.localeCompare(a.createdAt)).slice(0, 100)
}
get(taskId, options = {}) {
if (this.tasks.has(taskId)) return publicTask(this.tasks.get(taskId))
try { return this.search.get(taskId, options) } catch (error) {
if (error?.statusCode !== 404) throw error
return this.sizes.get(taskId)
}
}
completion(taskId) {
const task = this.tasks.get(taskId) ?? this.search.tasks.get(taskId) ?? this.sizes.tasks.get(taskId)
if (task === undefined) throw new FileManagerError('file task was not found', 404, 'FILE_TASK_NOT_FOUND')
return Promise.resolve(task.completion).then(() => this.get(taskId))
}
cancel(taskId) {
const task = this.tasks.get(taskId)
if (task === undefined) {
try { return this.search.cancel(taskId) } catch (error) {
if (error?.statusCode !== 404) throw error
return this.sizes.cancel(taskId)
}
}
if (!['queued', 'running'].includes(task.status)) throw new FileManagerError('file task is already complete', 409, 'FILE_TASK_COMPLETE')
if (task.status === 'queued') {
task.cancelRequested = true
task.status = 'cancelled'
task.phase = 'cancelled'
task.updatedAt = now()
this.queue = this.queue.filter(value => value !== task)
void Promise.resolve(task.persisted).then(() => this.#terminal(task, 'cancelled', 'cancelled'))
task.rejectResult?.(Object.assign(new Error('file transfer cancelled'), { code: 'FILE_TASK_CANCELLED' }))
this.#publishQueue()
return publicTask(task)
}
if (['destination-committed', 'source-hidden', 'cleaning'].includes(task.phase)) throw new FileManagerError('file task passed its cancellation boundary', 409, 'FILE_TASK_COMMITTED')
task.cancelRequested = true
task.abortController?.abort()
this.#publish(task)
return publicTask(task)
}
#schedule(task) {
this.queue.push(task)
this.#publishQueue()
const previous = this.queueTail
task.persisted = this.#persist(task)
task.completion = Promise.all([previous.catch(() => {}), task.persisted]).then(async () => {
if (task.cancelRequested || task.status === 'cancelled') return
task.status = 'running'
task.phase = 'validating'
task.queuePosition = 0
this.activeMutation = task
await this.#phase(task, task.executor === undefined ? 'validating' : 'transferring')
if (task.executor === undefined) await this.#run(task).catch(() => {})
else await this.#runTransfer(task)
}).finally(() => {
if (this.activeMutation === task) this.activeMutation = undefined
this.queue = this.queue.filter(value => value !== task)
this.#publishQueue()
})
this.queueTail = task.completion
}
async #runTransfer(task) {
const controller = new AbortController()
task.abortController = controller
const controls = {
signal: controller.signal,
progress: ({ processedBytes, totalBytes, currentPath } = {}) => {
if (Number.isSafeInteger(processedBytes) && processedBytes >= 0) task.processedBytes = processedBytes
if (Number.isSafeInteger(totalBytes) && totalBytes >= 0) task.totalBytes = totalBytes
if (typeof currentPath === 'string') task.currentPath = currentPath
task.updatedAt = now()
this.#publish(task)
},
}
try {
const result = await task.executor(controls)
task.processedEntries = 1
if (task.totalBytes !== null) task.processedBytes = task.totalBytes
await this.#terminal(task, 'success', 'completed')
await this.report(`file-task.${task.operation}.completed`, publicTask(task))
task.resolveResult(result)
} catch (error) {
const status = task.cancelRequested ? 'cancelled' : 'failed'
task.error = error instanceof Error ? error.message : String(error)
task.errorCode = typeof error?.code === 'string' ? error.code : null
await this.#terminal(task, status, status)
await this.report(`file-task.${task.operation}.${status}`, { ...publicTask(task), error })
task.rejectResult(error)
} finally {
task.abortController = undefined
await this.#prune()
}
}
#publishQueue() {
let position = 0
for (const task of this.queue) {
if (task.status === 'queued') position += 1
task.queuePosition = task.status === 'queued' ? position : 0
this.#publish(task)
}
}
async #validate(task) {
for (const source of task.sources) {
const item = await this.inventory.stat(source.path)
if (source.revision !== item.revision) throw new FileRevisionConflictError(`source changed: ${source.path}`)
if (task.operation === 'attributes' && !['file', 'directory'].includes(item.type)) {
throw new FileManagerError('attributes can be changed only for files and directories', 415, 'FILE_TYPE_UNSUPPORTED')
}
const metrics = task.operation === 'attributes' && task.attributes.recursive === false
? { entries: 1, bytes: item.type === 'file' ? item.size : 0 }
: await treeMetrics(source.path)
task.totalEntries += metrics.entries
task.totalBytes += metrics.bytes
}
if (task.destinationRevision !== null) {
const destinationRoot = task.operation === 'rename' || task.operation === 'mkdir' || task.operation === 'touch'
? dirname(task.destination) : task.destination
const listed = await this.inventory.list(destinationRoot, { limit: 1 })
if (listed.revision !== task.destinationRevision) throw new FileRevisionConflictError('destination directory changed')
}
}
async #run(task) {
try {
await this.#validate(task)
await this.#phase(task, 'mutating')
if (task.operation === 'mkdir') await mkdir(task.destination)
else if (task.operation === 'touch') {
const handle = await open(task.destination, 'wx', 0o644)
await handle.sync(); await handle.close(); await syncDirectory(dirname(task.destination))
} else if (task.operation === 'rename') await this.#rename(task)
else if (task.operation === 'copy') await this.#copy(task, false)
else if (task.operation === 'move') await this.#copy(task, true)
else if (task.operation === 'delete') await this.#delete(task)
else if (task.operation === 'attributes') await this.#attributes(task)
else if (task.operation === 'archive') await this.#archive(task)
else if (task.operation === 'extract') await this.#extract(task)
await this.#terminal(task, 'success', 'completed')
await this.report(`file-task.${task.operation}.completed`, publicTask(task))
} catch (error) {
const status = task.cancelRequested ? 'cancelled' : 'failed'
task.error = error instanceof Error ? error.message : String(error)
task.errorCode = typeof error?.code === 'string' ? error.code : null
await this.#cleanupStaging(task)
await this.#terminal(task, status, status)
await this.report(`file-task.${task.operation}.${status}`, { ...publicTask(task), error })
throw error
} finally {
await this.#prune()
}
}
async #rename(task) {
if (task.sources.length !== 1) throw new FileManagerError('rename requires one source')
const source = task.sources[0].path
if (dirname(source) !== dirname(task.destination)) throw new FileManagerError('rename destination must use the same directory')
const destination = await uniqueDestination(task.destination, task.conflict)
task.currentSource = source
task.currentDestination = destination
await this.#phase(task, 'move-prepared')
if (task.conflict === 'overwrite' && await exists(destination)) await rm(destination, { recursive: true })
await rename(source, destination)
task.published.push(destination)
await this.#phase(task, 'destination-committed')
await syncDirectory(dirname(destination))
task.currentSource = null
task.currentDestination = null
await this.#phase(task, 'mutating')
}
async #copy(task, move) {
for (const source of task.sources) {
if (task.cancelRequested) throw new Error('file task cancelled')
const destination = await uniqueDestination(join(task.destination, cleanName(source.path)), task.conflict)
if (source.path === destination || destination.startsWith(`${source.path}/`)) throw new FileManagerError('directory cannot be copied into itself')
if (move) {
task.currentSource = source.path
task.currentDestination = destination
await this.#phase(task, 'move-prepared')
try {
if (task.conflict === 'overwrite' && await exists(destination)) await rm(destination, { recursive: true })
await rename(source.path, destination)
task.published.push(destination)
await this.#phase(task, 'destination-committed')
task.processedEntries += 1
task.currentSource = null
task.currentDestination = null
await this.#phase(task, 'mutating')
continue
} catch (error) {
if (error?.code !== 'EXDEV') throw error
}
}
const staging = join(dirname(destination), `.dsh-file-task-${task.taskId}-${randomUUID()}.tmp`)
task.staging.push(staging)
task.currentPath = source.path
task.currentSource = source.path
task.currentDestination = destination
await this.#persist(task)
await copyEntry(source.path, staging)
if (task.cancelRequested) throw new Error('file task cancelled')
if (task.conflict === 'overwrite' && await exists(destination)) await rm(destination, { recursive: true })
await rename(staging, destination)
task.staging = task.staging.filter(path => path !== staging)
task.published.push(destination)
await this.#phase(task, move ? 'destination-committed' : 'mutating')
if (move) await rm(source.path, { recursive: true })
task.processedEntries += 1
task.processedBytes = Math.min(task.totalBytes, task.processedBytes + (await treeMetrics(destination)).bytes)
task.currentSource = null
task.currentDestination = null
await this.#phase(task, 'mutating')
}
}
async #delete(task) {
for (const source of task.sources) {
if (task.cancelRequested) throw new Error('file task cancelled')
const hidden = join(dirname(source.path), `.dsh-delete-${task.taskId}-${randomUUID()}.tmp`)
await rename(source.path, hidden)
task.hidden.push(hidden)
await this.#phase(task, 'source-hidden')
const removed = await treeMetrics(hidden)
await rm(hidden, { recursive: true })
task.hidden = task.hidden.filter(path => path !== hidden)
task.processedEntries += removed.entries
task.processedBytes = Math.min(task.totalBytes, task.processedBytes + removed.bytes)
await this.#phase(task, 'cleaning')
}
}
async #attributes(task) {
const [uid, gid] = await Promise.all([
this.identity.userId(task.attributes.user),
this.identity.groupId(task.attributes.group),
])
const mode = Number.parseInt(task.attributes.mode, 8)
const pending = [task.sources[0].path]
task.resolvedAttributes = { uid, gid, mode }
while (pending.length > 0) {
if (task.cancelRequested) throw new Error('file task cancelled')
const path = pending.pop()
const details = await lstat(path)
if (task.attributes.recursive && details.isDirectory() && !details.isSymbolicLink()) {
for (const name of await readdir(path)) pending.push(join(path, name))
}
await lchown(path, uid, gid)
if (!details.isSymbolicLink()) await chmod(path, mode)
assertAttributesApplied(path, await lstat(path), { uid, gid, mode }, details.isSymbolicLink())
task.currentPath = path
task.processedEntries += 1
if (details.isFile()) task.processedBytes += details.size
if (task.processedEntries % 100 === 0) await this.#phase(task, 'mutating')
}
task.currentPath = null
}
async #archive(task) {
const output = await uniqueDestination(task.destination, task.conflict)
const sourceRoot = join(dirname(output), `.dsh-archive-input-${task.taskId}`)
const staging = join(dirname(output), `.dsh-archive-${task.taskId}.tmp`)
task.staging.push(sourceRoot, staging)
await mkdir(sourceRoot)
const names = new Set()
for (const source of task.sources) {
const name = cleanName(source.path)
if (names.has(name)) throw new FileManagerError('archive source names must be unique', 409, 'FILE_EXISTS')
names.add(name)
task.currentPath = source.path
await copyEntry(source.path, join(sourceRoot, name))
task.processedEntries += 1
await this.#phase(task, 'staging')
}
if (task.cancelRequested) throw new Error('file task cancelled')
const controller = new AbortController()
task.abortController = controller
try {
await createArchive({ format: task.archiveFormat, sourceRoot, output: staging, signal: controller.signal })
} finally {
task.abortController = undefined
}
if (task.cancelRequested) throw new Error('file task cancelled')
if (task.conflict === 'overwrite' && await exists(output)) await rm(output, { recursive: true })
await rename(staging, output)
task.staging = task.staging.filter(path => path !== staging)
task.published.push(output)
task.processedBytes = task.totalBytes
await rm(sourceRoot, { recursive: true, force: true })
task.staging = task.staging.filter(path => path !== sourceRoot)
task.currentPath = null
await this.#phase(task, 'destination-committed')
}
async #extract(task) {
const archive = task.sources[0].path
const staging = join(task.destination, `.dsh-extract-${task.taskId}.tmp`)
task.staging.push(staging)
task.currentPath = archive
await this.#persist(task)
const controller = new AbortController()
task.abortController = controller
let metrics
try {
metrics = await extractArchive({ format: task.archiveFormat, archive, output: staging, signal: controller.signal })
} finally {
task.abortController = undefined
}
task.totalEntries = metrics.entries
task.totalBytes = metrics.bytes
await this.#publishExtraction(task, staging)
}
async #publishExtraction(task, staging) {
const plan = []
for (const name of await readdir(staging)) {
plan.push({ source: join(staging, name), destination: await uniqueDestination(join(task.destination, name), task.conflict) })
}
for (const { source, destination } of plan) {
if (task.cancelRequested) throw new Error('file task cancelled')
if (task.conflict === 'overwrite' && await exists(destination)) await rm(destination, { recursive: true })
await rename(source, destination)
task.published.push(destination)
const item = await treeMetrics(destination)
task.processedEntries += item.entries
task.processedBytes += item.bytes
await this.#phase(task, 'publishing')
}
await rm(staging, { recursive: true, force: true })
task.staging = task.staging.filter(path => path !== staging)
task.currentPath = null
}
async #cleanupStaging(task) {
for (const path of task.staging) await rm(path, { recursive: true, force: true }).catch(() => {})
task.staging = []
}
async #recover(task) {
if (task.operation === 'attributes') {
try {
task.cancelRequested = false
task.processedEntries = 0
task.processedBytes = 0
await this.#phase(task, 'mutating')
await this.#attributes(task)
task.status = 'success'
await this.#phase(task, 'completed')
await this.report('file-task.attributes.recovered', publicTask(task))
} catch (error) {
task.status = 'failed'
task.error = error instanceof Error ? error.message : String(error)
task.errorCode = typeof error?.code === 'string' ? error.code : null
await this.#phase(task, 'failed')
await this.report('file-task.attributes.recovery-failed', { ...publicTask(task), error })
}
} else if (task.operation === 'archive' && task.phase === 'destination-committed') {
task.status = 'success'
await this.#phase(task, 'completed')
} else if (task.operation === 'extract' && task.phase === 'publishing' && task.staging?.length === 1 && await exists(task.staging[0])) {
try {
task.cancelRequested = false
await this.#publishExtraction(task, task.staging[0])
task.status = 'success'
await this.#phase(task, 'completed')
await this.report('file-task.extract.recovered', publicTask(task))
} catch (error) {
task.status = 'failed'
task.error = error instanceof Error ? error.message : String(error)
task.errorCode = typeof error?.code === 'string' ? error.code : null
await this.#phase(task, 'failed')
await this.report('file-task.extract.recovery-failed', { ...publicTask(task), error })
}
} else if (task.operation === 'delete' || task.phase === 'source-hidden' || task.phase === 'cleaning') {
for (const path of task.hidden ?? []) await rm(path, { recursive: true, force: true })
task.hidden = []
task.status = 'success'
await this.#phase(task, 'completed')
} else if (['move', 'rename'].includes(task.operation) && task.phase === 'destination-committed') {
if (task.currentSource !== null && await exists(task.currentSource)) await rm(task.currentSource, { recursive: true })
task.status = 'success'
await this.#phase(task, 'completed')
} else if (['move', 'rename'].includes(task.operation) && task.phase === 'move-prepared'
&& task.currentSource !== null && task.currentDestination !== null
&& !await exists(task.currentSource) && await exists(task.currentDestination)) {
task.published ??= []
if (!task.published.includes(task.currentDestination)) task.published.push(task.currentDestination)
task.status = 'success'
await this.#phase(task, 'completed')
} else {
await this.#cleanupStaging(task)
task.status = 'interrupted'
task.error = 'file operation was interrupted before a recoverable commit'
await this.#phase(task, 'interrupted')
}
}
async #phase(task, phase) {
task.phase = phase
task.updatedAt = now()
await this.#persist(task)
this.#publish(task)
}
async #terminal(task, status, phase) {
task.status = status
task.phase = phase
task.updatedAt = now()
await this.#persist(task)
if (this.activeMutation === task) this.activeMutation = undefined
this.queue = this.queue.filter(value => value !== task)
this.#publishQueue()
this.#publish(task)
}
async #persist(task) {
await mkdir(this.root, { recursive: true })
const value = publicTask(task)
await durableReplace(join(this.root, `${task.taskId}.json`), Buffer.from(`${JSON.stringify(value)}\n`))
}
#publish(task) { this.onState(publicTask(task)) }
async #prune() {
const completed = [...this.tasks.values()].filter(task => !['queued', 'running'].includes(task.status)).sort((a, b) => b.updatedAt.localeCompare(a.updatedAt))
for (const task of completed.slice(100)) {
this.tasks.delete(task.taskId)
await rm(join(this.root, `${task.taskId}.json`), { force: true })
}
}
}
export const fileTaskInternals = Object.freeze({ protectedDeleteRoots: DEFAULT_PROTECTED_DELETE_ROOTS, uniqueDestination, treeMetrics, normalizeAttributes, assertAttributesApplied })
upstream/container/control-plane/modules/file-manager/transfers.mjs
+205
View File
@@ -0,0 +1,205 @@
import { createReadStream } from 'node:fs'
import { link, lstat, mkdir, mkdtemp, open, readlink, rename, rm, stat } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { basename, dirname, extname, join } from 'node:path'
import { pipeline } from 'node:stream/promises'
import { Readable } from 'node:stream'
import { randomUUID } from 'node:crypto'
import {
FileManagerError, FileRevisionConflictError, fileError, fileManagerInternals,
isManagedPath, normalizeAbsolutePath,
} from './index.mjs'
import { createArchive } from './archives.mjs'
function parseRange(value, size) {
if (value === undefined) return { start: 0, end: Math.max(0, size - 1), partial: false }
const match = /^bytes=(\d*)-(\d*)$/.exec(value)
if (match === null || (match[1] === '' && match[2] === '')) throw new FileManagerError('range is invalid', 416, 'RANGE_INVALID')
let start
let end
if (match[1] === '') {
const suffix = Number(match[2])
if (!Number.isSafeInteger(suffix) || suffix < 1) throw new FileManagerError('range is invalid', 416, 'RANGE_INVALID')
start = Math.max(0, size - suffix)
end = size - 1
} else {
start = Number(match[1])
end = match[2] === '' ? size - 1 : Number(match[2])
}
if (!Number.isSafeInteger(start) || !Number.isSafeInteger(end) || start < 0 || start >= size || end < start) {
throw new FileManagerError('range is not satisfiable', 416, 'RANGE_INVALID')
}
return { start, end: Math.min(end, size - 1), partial: true }
}
function disposition(name) {
const fallback = name.replace(/[^\x20-\x7e]/g, '_').replace(/["\\]/g, '_') || 'download'
return `attachment; filename="${fallback}"; filename*=UTF-8''${encodeURIComponent(name)}`
}
async function syncDirectory(path) {
const handle = await open(path, 'r')
try { await handle.sync() } finally { await handle.close() }
}
async function existingMetadata(path, { requireFile = false } = {}) {
try {
const value = await stat(path, { bigint: true })
if (requireFile && !value.isFile()) throw new FileManagerError('upload target is not a regular file', 415, 'FILE_TYPE_UNSUPPORTED')
return value
} catch (error) {
if (error?.code === 'ENOENT') return undefined
throw error
}
}
function renamedPath(path, index) {
const extension = extname(path)
const stem = extension === '' ? path : path.slice(0, -extension.length)
return `${stem} (${String(index)})${extension}`
}
export class FileTransferManager {
constructor({ isManaged = isManagedPath, stagingRoot = tmpdir() } = {}) {
this.isManaged = isManaged
this.stagingRoot = stagingRoot
}
async openDownload(value, { revision, range, signal } = {}) {
const path = normalizeAbsolutePath(value)
let handle
let cleanup
try {
const pathStats = await lstat(path, { bigint: true })
if (pathStats.isDirectory()) {
if (range !== undefined) throw new FileManagerError('directory downloads do not support ranges', 416, 'RANGE_INVALID')
const actualRevision = fileManagerInternals.revisionFor(path, pathStats)
if (revision !== undefined && revision !== '' && revision !== actualRevision) throw new FileRevisionConflictError('download revision changed')
await mkdir(this.stagingRoot, { recursive: true })
const staging = await mkdtemp(join(this.stagingRoot, '.dsh-directory-download-'))
cleanup = () => rm(staging, { recursive: true, force: true })
const name = basename(path) || 'root'
const archive = join(staging, `${name}.zip`)
await createArchive({ format: 'zip', sourceRoot: path === '/' ? path : dirname(path), output: archive, entries: [path === '/' ? '.' : name], signal })
const download = await this.openDownload(archive, { signal })
download.path = path
download.revision = actualRevision
download.headers['content-disposition'] = disposition(`${name}.zip`)
download.cleanup = cleanup
cleanup = undefined
return download
}
handle = await open(path, 'r')
const before = await handle.stat({ bigint: true })
if (!before.isFile()) throw new FileManagerError('path is not a regular file', 415, 'FILE_TYPE_UNSUPPORTED')
const actualRevision = fileManagerInternals.revisionFor(path, before, await readlink(path).catch(() => ''))
if (revision !== undefined && revision !== '' && revision !== actualRevision) throw new FileRevisionConflictError('download revision changed')
const size = Number(before.size)
if (!Number.isSafeInteger(size)) throw new FileManagerError('file is too large', 413, 'FILE_TOO_LARGE')
if (size === 0 && range !== undefined) throw new FileManagerError('range is not satisfiable', 416, 'RANGE_INVALID')
const selected = size === 0 ? { start: 0, end: -1, partial: false } : parseRange(range, size)
const length = selected.end < selected.start ? 0 : selected.end - selected.start + 1
const stream = length === 0 ? Readable.from([]) : createReadStream(path, {
fd: handle.fd, autoClose: false, start: selected.start, end: selected.end,
})
return {
path, handle, stream, revision: actualRevision,
status: selected.partial ? 206 : 200,
headers: {
'accept-ranges': 'bytes',
'cache-control': 'no-store',
'content-disposition': disposition(basename(path)),
'content-length': String(length),
'content-type': 'application/octet-stream',
etag: `"${actualRevision}"`,
...(selected.partial ? { 'content-range': `bytes ${String(selected.start)}-${String(selected.end)}/${String(size)}` } : {}),
},
}
} catch (error) {
await handle?.close().catch(() => {})
await cleanup?.().catch(() => {})
throw fileError(error)
}
}
async sendDownload(response, download, { onProgress = () => {}, signal } = {}) {
response.writeHead(download.status, download.headers)
let sent = 0
download.stream.on('data', chunk => {
sent += chunk.byteLength
onProgress(sent, Number(download.headers['content-length']))
})
try {
await pipeline(download.stream, response, { signal })
} finally {
await download.handle.close().catch(() => {})
await download.cleanup?.().catch(() => {})
}
}
async upload(input, value, { conflict = 'reject', contentLength, onProgress = () => {}, signal } = {}) {
const requestedPath = normalizeAbsolutePath(value)
if (!['reject', 'overwrite', 'rename'].includes(conflict)) throw new FileManagerError('upload conflict mode is invalid')
const parent = dirname(requestedPath)
const parentStats = await stat(parent).catch(error => { throw fileError(error) })
if (!parentStats.isDirectory()) throw new FileManagerError('upload parent is not a directory', 415, 'FILE_TYPE_UNSUPPORTED')
const staging = join(parent, `.dsh-upload-${randomUUID()}.tmp`)
let handle
let received = 0
try {
const original = await existingMetadata(requestedPath, { requireFile: conflict === 'overwrite' })
if (original !== undefined && conflict === 'reject') throw new FileManagerError('upload target already exists', 409, 'FILE_EXISTS')
handle = await open(staging, 'wx', 0o600)
for await (const chunk of input) {
signal?.throwIfAborted()
const bytes = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk)
received += bytes.byteLength
await handle.write(bytes)
onProgress(received, contentLength)
}
if (contentLength !== undefined && received !== contentLength) throw new FileManagerError('upload size does not match Content-Length')
await handle.sync()
if (original === undefined) await handle.chmod(0o644)
else {
await handle.chmod(Number(original.mode & 0o7777n))
const current = await handle.stat({ bigint: true })
if (current.uid !== original.uid || current.gid !== original.gid) await handle.chown(Number(original.uid), Number(original.gid))
}
await handle.close()
handle = undefined
let path = requestedPath
if (conflict === 'overwrite') await rename(staging, path)
else if (conflict === 'reject') {
await link(staging, path)
await rm(staging)
} else {
let index = 0
for (;;) {
path = index === 0 ? requestedPath : renamedPath(requestedPath, index)
try {
await link(staging, path)
await rm(staging)
break
} catch (error) {
if (error?.code !== 'EEXIST') throw error
index += 1
}
}
}
await syncDirectory(parent)
const result = await stat(path, { bigint: true })
return {
path,
size: received,
revision: fileManagerInternals.revisionFor(path, result),
managed: this.isManaged(path),
}
} catch (error) {
await handle?.close().catch(() => {})
await rm(staging, { force: true }).catch(() => {})
throw fileError(error)
}
}
}
export const fileTransferInternals = Object.freeze({ parseRange, disposition, renamedPath })
upstream/container/control-plane/modules/log-manager/index.mjs
+473
View File
@@ -0,0 +1,473 @@
import { EventEmitter } from 'node:events'
import { randomUUID } from 'node:crypto'
import { createReadStream } from 'node:fs'
import { appendFile, chmod, chown, mkdir, readdir, readFile, rename, rm, stat } from 'node:fs/promises'
import { join } from 'node:path'
const SOURCE_PATTERN = /^[a-z0-9][a-z0-9._-]{0,127}$/
const MAX_ENTRY_BYTES = 64 * 1024
const CONSOLE_MARKER = 'dsh-platform-log-v1'
const LOG_LEVELS = new Set(['debug', 'info', 'warning', 'error'])
const MAX_DIAGNOSTIC_STRING = 16 * 1024
const MAX_ERROR_DEPTH = 3
const MAX_AGGREGATE_ERRORS = 8
const SUPPRESSION_KEY_PATTERN = /^[a-z0-9][a-z0-9._:-]{0,255}$/
function boundedString(value) {
const text = String(value)
return text.length <= MAX_DIAGNOSTIC_STRING ? text : `${text.slice(0, MAX_DIAGNOSTIC_STRING)}...[truncated]`
}
export function errorDetails(error, depth = 0) {
if (error === undefined || error === null) return {}
if (!(error instanceof Error)) return { error: boundedString(error), errorType: typeof error }
const details = {
error: boundedString(error.message || error.name),
errorName: boundedString(error.name),
}
if (typeof error.code === 'string' || typeof error.code === 'number') details.errorCode = error.code
if (typeof error.stack === 'string') details.errorStack = boundedString(error.stack)
if (depth >= MAX_ERROR_DEPTH) return details
if (error.cause !== undefined) details.errorCause = errorDetails(error.cause, depth + 1)
if (error instanceof AggregateError) {
details.errors = [...error.errors].slice(0, MAX_AGGREGATE_ERRORS).map(value => errorDetails(value, depth + 1))
if (error.errors.length > MAX_AGGREGATE_ERRORS) details.errorsTruncated = error.errors.length - MAX_AGGREGATE_ERRORS
}
return details
}
function validateSource(source) {
if (typeof source !== 'string' || !SOURCE_PATTERN.test(source)) throw new Error('log source is invalid')
return source
}
function validateFilter(value, label) {
if (value === undefined) return undefined
if (typeof value !== 'string' || value.length === 0 || value.length > 128 || /[\0\r\n]/u.test(value)) {
throw new Error(`${label} filter is invalid`)
}
return value
}
function matches(entry, filters) {
return (filters.taskId === undefined || entry.taskId === filters.taskId)
&& (filters.operation === undefined || entry.operation === filters.operation)
&& (filters.phase === undefined || entry.phase === filters.phase)
}
function parseLine(line) {
try { return JSON.parse(line) } catch { return undefined }
}
function parseForwardedLine(line) {
const entry = parseLine(line)
if (
entry?.platformLog !== CONSOLE_MARKER
|| typeof entry.timestamp !== 'string'
|| typeof entry.message !== 'string'
|| !['stdout', 'stderr', 'audit', 'platform'].includes(entry.stream)
) return undefined
try { validateSource(entry.source) } catch { return undefined }
return entry
}
export function capturedLogLevel(stream, message) {
if (stream !== 'stderr') return 'info'
const text = String(message).trim()
const lines = text.split('\n')
if (lines.some(line => /^(?:\s*at\s+|.*\b(?:error|fatal|failed|failure|exception|panic|unhandled)\b)|错误|失败|异常|致命/iu.test(line))) {
return 'error'
}
if (lines.some(line => /\b(?:warn|warning|deprecated|deprecation)\b|警告|已弃用/iu.test(line))) return 'warning'
return 'info'
}
const NODE_ERROR_LINE = /^\s*(?:AggregateError|Error|EvalError|RangeError|ReferenceError|SyntaxError|TypeError|URIError):/u
const NODE_LOCATION_LINE = /^(?:file:\/\/|\/).*:\d+(?::\d+)?$/u
function isNodeErrorContinuation(line) {
return line === ''
|| /^\s/u.test(line)
|| NODE_ERROR_LINE.test(line)
|| /^Node\.js v\d/u.test(line)
|| /^\s*(?:\[cause\]|\[errors\]|Caused by:)/u.test(line)
|| /^\s*[{}\[\]},]+$/u.test(line)
}
function capturedMessages(source, stream, lines) {
if (source !== 'dsh-runtime' || stream !== 'stderr') return lines
const messages = []
let cursor = 0
while (cursor < lines.length) {
const relativeError = lines.slice(cursor).findIndex(line => NODE_ERROR_LINE.test(line))
if (relativeError < 0) {
messages.push(...lines.slice(cursor))
break
}
const errorIndex = cursor + relativeError
let start = errorIndex
for (let index = errorIndex - 1; index >= cursor && index >= errorIndex - 6; index -= 1) {
if (NODE_LOCATION_LINE.test(lines[index])) {
start = index
break
}
}
messages.push(...lines.slice(cursor, start))
let end = errorIndex + 1
while (end < lines.length && isNodeErrorContinuation(lines[end])) {
const terminal = /^Node\.js v\d/u.test(lines[end])
end += 1
if (terminal) break
}
messages.push(boundedString(lines.slice(start, end).join('\n')))
cursor = end
}
return messages
}
export class JsonlLogManager extends EventEmitter {
constructor({
root,
maxBytes = 100 * 1024 * 1024,
retentionDays = 14,
rotateBytes = 10 * 1024 * 1024,
now = () => new Date(),
output,
fileMode = 0o660,
fileUid,
fileGid,
}) {
super()
if (!Number.isSafeInteger(maxBytes) || maxBytes < MAX_ENTRY_BYTES) throw new Error('log maxBytes is invalid')
if (!Number.isSafeInteger(retentionDays) || retentionDays < 1) throw new Error('log retentionDays is invalid')
if (!Number.isSafeInteger(rotateBytes) || rotateBytes < 1_024) throw new Error('log rotateBytes is invalid')
if (!Number.isSafeInteger(fileMode) || fileMode < 0 || fileMode > 0o777) throw new Error('log fileMode is invalid')
this.root = root
this.maxBytes = maxBytes
this.retentionMs = retentionDays * 86_400_000
this.rotateBytes = rotateBytes
this.now = now
this.output = output
this.fileMode = fileMode
this.fileUid = fileUid
this.fileGid = fileGid
this.queue = Promise.resolve()
this.suppressedDiagnostics = new Map()
}
prepare() {
return this.serialized(async () => {
await mkdir(this.root, { recursive: true })
const names = (await readdir(this.root)).filter(name => name.endsWith('.jsonl'))
for (const name of names) {
const path = join(this.root, name)
if (this.fileGid !== undefined) await chown(path, -1, this.fileGid)
await chmod(path, this.fileMode)
}
})
}
writeOutput(stream, line) {
const destination = stream === 'stderr' ? this.output?.stderr : this.output?.stdout
destination?.write(`${line}\n`)
}
mirror(entry) {
if (this.output === undefined) return
this.writeOutput(entry.stream, JSON.stringify({ ...entry, platformLog: CONSOLE_MARKER }))
}
serialized(operation) {
const result = this.queue.then(operation, operation)
this.queue = result.then(() => undefined, () => undefined)
return result
}
currentPath(source) {
return join(this.root, `${validateSource(source)}.jsonl`)
}
append(source, stream, message, fields = {}) {
return this.serialized(async () => {
validateSource(source)
if (!['stdout', 'stderr', 'audit', 'platform'].includes(stream)) throw new Error('log stream is invalid')
if (typeof message !== 'string') throw new Error('log message must be a string')
const level = fields.level ?? (stream === 'stderr' ? 'error' : 'info')
if (!LOG_LEVELS.has(level)) throw new Error('log level is invalid')
const entry = { ...fields, timestamp: this.now().toISOString(), source, stream, level, message }
const line = Buffer.from(`${JSON.stringify(entry)}\n`)
if (line.byteLength > MAX_ENTRY_BYTES) throw new Error('log entry exceeds 64 KiB')
await mkdir(this.root, { recursive: true })
const path = this.currentPath(source)
let exists = true
const size = await stat(path).then(value => value.size, error => {
if (error?.code !== 'ENOENT') throw error
exists = false
return 0
})
if (size > 0 && size + line.byteLength > this.rotateBytes) {
await rename(path, join(this.root, `${source}.${Date.now()}.${randomUUID()}.jsonl`))
exists = false
}
await appendFile(path, line, { mode: this.fileMode })
if (!exists) {
if (this.fileUid !== undefined || this.fileGid !== undefined) await chown(path, this.fileUid ?? -1, this.fileGid ?? -1)
await chmod(path, this.fileMode)
}
this.emit('entry', Object.freeze(entry))
this.mirror(entry)
await this.pruneUnlocked()
return Object.freeze(entry)
})
}
audit(action, fields = {}) {
return this.append('audit', 'audit', action, fields)
}
async diagnostic(source, message, { error, level = error === undefined || error === null ? 'info' : 'error', stream = 'platform', ...fields } = {}) {
const details = { ...fields, ...errorDetails(error), level }
try {
return await this.append(source, stream, message, details)
} catch (loggingError) {
try {
this.writeOutput('stderr', JSON.stringify({
timestamp: this.now().toISOString(),
source: 'log-manager',
stream: 'stderr',
level: 'error',
message: 'diagnostic.write.failed',
diagnostic: { source, stream, message, ...details },
loggingError: errorDetails(loggingError),
platformLog: CONSOLE_MARKER,
}))
} catch {}
return undefined
}
}
diagnosticRateLimited(key, source, message, fields = {}, { intervalMs = 30_000 } = {}) {
if (typeof key !== 'string' || !SUPPRESSION_KEY_PATTERN.test(key)) {
throw new Error('diagnostic suppression key is invalid')
}
if (!Number.isSafeInteger(intervalMs) || intervalMs < 1_000 || intervalMs > 86_400_000) {
throw new Error('diagnostic suppression interval is invalid')
}
const timestamp = this.now().getTime()
const previous = this.suppressedDiagnostics.get(key)
if (previous !== undefined && timestamp - previous.lastReportedAt < intervalMs) {
previous.suppressedCount += 1
return Promise.resolve(undefined)
}
const suppressedCount = previous?.suppressedCount ?? 0
this.suppressedDiagnostics.set(key, { lastReportedAt: timestamp, suppressedCount: 0 })
return this.diagnostic(source, message, {
...fields,
...(suppressedCount === 0 ? {} : { suppressedCount }),
})
}
async files() {
try {
const names = (await readdir(this.root)).filter(name => name.endsWith('.jsonl'))
return Promise.all(names.map(async name => ({ name, path: join(this.root, name), details: await stat(join(this.root, name)) })))
} catch (error) {
if (error?.code === 'ENOENT') return []
throw error
}
}
async pruneUnlocked() {
const cutoff = this.now().getTime() - this.retentionMs
let files = await this.files()
for (const file of files) {
if (file.details.mtimeMs < cutoff) await rm(file.path, { force: true })
}
files = (await this.files()).sort((left, right) => left.details.mtimeMs - right.details.mtimeMs)
let total = files.reduce((sum, file) => sum + file.details.size, 0)
for (const file of files) {
if (total <= this.maxBytes) break
await rm(file.path, { force: true })
total -= file.details.size
}
}
prune() {
return this.serialized(() => this.pruneUnlocked())
}
async query({ sources, since, taskId, operation, phase, limit = 200 } = {}) {
if (!Number.isSafeInteger(limit) || limit < 1 || limit > 5_000) throw new Error('log query limit is invalid')
const selected = sources === undefined ? undefined : new Set(sources.map(validateSource))
const threshold = since === undefined ? undefined : new Date(since).toISOString()
const filters = {
taskId: validateFilter(taskId, 'taskId'),
operation: validateFilter(operation, 'operation'),
phase: validateFilter(phase, 'phase'),
}
const entries = []
for (const file of await this.files()) {
const lines = (await readFile(file.path, 'utf8')).split('\n')
for (const line of lines) {
if (line === '') continue
const entry = parseLine(line)
if (
entry !== undefined
&& (selected === undefined || selected.has(entry.source))
&& (threshold === undefined || entry.timestamp >= threshold)
&& matches(entry, filters)
) entries.push(entry)
}
}
entries.sort((left, right) => left.timestamp.localeCompare(right.timestamp))
return Object.freeze(entries.slice(-limit).map(Object.freeze))
}
async sources() {
const result = new Set()
for (const file of await this.files()) {
for (const line of (await readFile(file.path, 'utf8')).split('\n')) {
const entry = parseLine(line)
if (entry?.source !== undefined) {
try { result.add(validateSource(entry.source)) } catch {}
}
}
}
return [...result].sort()
}
async follow({ sources, since, taskId, operation, phase } = {}, listener, { intervalMs = 250, onError = () => {} } = {}) {
if (typeof listener !== 'function') throw new Error('log follow listener is required')
if (!Number.isSafeInteger(intervalMs) || intervalMs < 50) throw new Error('log follow interval is invalid')
const selected = sources === undefined ? undefined : new Set(sources.map(validateSource))
const threshold = since === undefined ? undefined : new Date(since).toISOString()
const filters = {
taskId: validateFilter(taskId, 'taskId'),
operation: validateFilter(operation, 'operation'),
phase: validateFilter(phase, 'phase'),
}
await mkdir(this.root, { recursive: true })
let states = new Map()
for (const file of await this.files()) {
states.set(file.name, { identity: `${String(file.details.dev)}:${String(file.details.ino)}`, offset: file.details.size, pending: Buffer.alloc(0) })
}
let closed = false
let scanning = false
let scanAgain = false
const scan = async () => {
if (closed) return
if (scanning) {
scanAgain = true
return
}
scanning = true
try {
const files = await this.files()
const byIdentity = new Map([...states.values()].map(state => [state.identity, state]))
const nextStates = new Map()
const entries = []
for (const file of files) {
const identity = `${String(file.details.dev)}:${String(file.details.ino)}`
let state = states.get(file.name)
if (state?.identity !== identity) state = byIdentity.get(identity)
if (state === undefined || file.details.size < state.offset) state = { identity, offset: 0, pending: Buffer.alloc(0) }
if (file.details.size > state.offset) {
const chunks = []
const end = file.details.size - 1
for await (const chunk of createReadStream(file.path, { start: state.offset, end })) chunks.push(chunk)
const content = Buffer.concat([state.pending, ...chunks])
let lineStart = 0
for (let index = 0; index < content.length; index += 1) {
if (content[index] !== 0x0a) continue
const line = content.subarray(lineStart, index).toString('utf8')
lineStart = index + 1
if (line === '') continue
const entry = parseLine(line)
if (entry !== undefined
&& (selected === undefined || selected.has(entry.source))
&& (threshold === undefined || entry.timestamp >= threshold)
&& matches(entry, filters)) entries.push(entry)
}
state = { ...state, identity, offset: file.details.size, pending: content.subarray(lineStart) }
}
nextStates.set(file.name, state)
}
states = nextStates
entries.sort((left, right) => left.timestamp.localeCompare(right.timestamp))
for (const entry of entries) listener(Object.freeze(entry))
} catch (error) {
onError(error)
} finally {
scanning = false
if (scanAgain) {
scanAgain = false
void scan()
}
}
}
const timer = setInterval(() => { void scan() }, intervalMs)
timer.unref()
return Object.freeze({
close: () => {
closed = true
clearInterval(timer)
},
})
}
capture(child, source, declaration = { stdout: true, stderr: true }, { acceptForwarded = false } = {}) {
for (const streamName of ['stdout', 'stderr']) {
if (!declaration[streamName]) continue
const stream = child[streamName]
if (stream === undefined || stream === null) continue
let pending = ''
let completeLines = []
let flushScheduled = false
const appendLines = lines => {
if (lines.length === 0) return
const messages = capturedMessages(source, streamName, lines)
for (const message of messages) {
void this.append(source, streamName, message, { level: capturedLogLevel(streamName, message) })
.catch(error => this.emit('error', error))
}
}
const flush = () => {
flushScheduled = false
const lines = completeLines
completeLines = []
let unforwarded = []
const flushUnforwarded = () => {
appendLines(unforwarded)
unforwarded = []
}
for (const line of lines) {
const forwarded = acceptForwarded && this.output !== undefined ? parseForwardedLine(line) : undefined
if (forwarded === undefined) unforwarded.push(line)
else {
flushUnforwarded()
this.writeOutput(forwarded.stream, line)
}
}
flushUnforwarded()
}
const scheduleFlush = () => {
if (flushScheduled) return
flushScheduled = true
setImmediate(flush)
}
stream.setEncoding?.('utf8')
stream.on('data', chunk => {
pending += String(chunk)
const lines = pending.split('\n')
pending = lines.pop()
completeLines.push(...lines)
scheduleFlush()
})
stream.once('end', () => {
if (pending !== '') completeLines.push(pending)
pending = ''
flush()
})
}
}
}
upstream/container/control-plane/modules/patch-manager/index.mjs
+215
View File
@@ -0,0 +1,215 @@
import { createHash, randomUUID } from 'node:crypto'
import { spawn } from 'node:child_process'
import { cp, lstat, mkdir, readFile, readdir, readlink, rename, rm, symlink } from 'node:fs/promises'
import { basename, dirname, join, resolve } from 'node:path'
import { pathToFileURL } from 'node:url'
import { TrustError } from '../../../platform/lib/validation.mjs'
import { artifactForReference, parseEnvironmentManifest } from '../../../platform/lib/contracts.mjs'
async function patchModule(path) {
return import(`${pathToFileURL(resolve(path)).href}?runtime=${randomUUID()}`)
}
export async function verifyNpmIntegrity(path, integrity) {
if (typeof integrity !== 'string' || !integrity.startsWith('sha512-')) {
throw new TrustError('npm integrity must use SHA-512')
}
const expected = integrity.slice('sha512-'.length)
const bytes = await readFile(path)
const actual = createHash('sha512').update(bytes).digest('base64')
if (actual !== expected) throw new TrustError('npm tarball integrity mismatch', 'TRUST_ARTIFACT_MISMATCH')
return bytes.byteLength
}
export async function applyPatchSet(dshRoot, patchPaths) {
for (const path of patchPaths) {
const module = await patchModule(path)
if (typeof module.applyPatch !== 'function') {
throw new Error(`Patch ${path} does not export applyPatch(dshRoot)`)
}
await module.applyPatch(dshRoot)
}
}
export async function verifyPatchSet(dshRoot, patchPaths) {
for (const path of patchPaths) {
const module = await patchModule(path)
if (typeof module.verifyPatch !== 'function') {
throw new Error(`Patch ${path} does not export verifyPatch(dshRoot)`)
}
await module.verifyPatch(resolve(dshRoot))
}
}
export async function verifyRuntimePatches({ runtimeRoot, environmentRoot }) {
const patchSet = await loadEnvironmentPatchSet(environmentRoot)
await verifyPatchSet(join(resolve(runtimeRoot), 'package'), patchSet.paths)
return patchSet.ids
}
export async function loadEnvironmentPatchSet(environmentRoot) {
const environment = resolve(environmentRoot)
const manifest = parseEnvironmentManifest(await readFile(join(environment, 'environment.manifest.json')))
const paths = []
for (const reference of manifest.patches) {
const descriptor = artifactForReference(manifest, reference)
const path = join(environment, 'artifacts', descriptor.id)
const bytes = await readFile(path)
if (bytes.byteLength !== descriptor.size
|| createHash('sha256').update(bytes).digest('hex') !== descriptor.sha256) {
throw new TrustError(`Patch Artifact ${descriptor.id} differs from the Environment Manifest`)
}
paths.push(path)
}
return Object.freeze({
ids: Object.freeze(manifest.patches.map(reference => reference.id)),
paths: Object.freeze(paths),
})
}
async function measureTree(path) {
const details = await lstat(path)
if (details.isSymbolicLink()) return { bytes: 0, items: 1 }
if (!details.isDirectory()) return { bytes: details.size, items: 1 }
let bytes = 0
let items = 0
for (const entry of await readdir(path)) {
const measured = await measureTree(join(path, entry))
bytes += measured.bytes
items += measured.items
}
return { bytes, items }
}
async function copyTree(source, destination, onProgress = async () => {}) {
const details = await lstat(source)
if (details.isSymbolicLink()) {
await symlink(await readlink(source), destination)
await onProgress({ processedBytes: 0, processedItems: 1 })
return
}
if (!details.isDirectory()) {
await cp(source, destination, { preserveTimestamps: true, verbatimSymlinks: true })
await onProgress({ processedBytes: details.size, processedItems: 1 })
return
}
await mkdir(destination, { recursive: true })
for (const entry of await readdir(source)) {
await copyTree(join(source, entry), join(destination, entry), onProgress)
}
}
export async function buildRuntime({ pristineRoot, versionsRoot, runtimeId, patchPaths, onProgress = async () => {} }) {
if (typeof runtimeId !== 'string' || !/^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/.test(runtimeId)) {
throw new Error('Runtime ID is invalid')
}
const versions = resolve(versionsRoot)
const destination = join(versions, runtimeId)
try {
await lstat(destination)
throw new Error(`Runtime ${runtimeId} already exists`)
} catch (error) {
if (error?.code !== 'ENOENT') throw error
}
await mkdir(versions, { recursive: true })
const staging = join(versions, `.${runtimeId}.${randomUUID()}.tmp`)
try {
const packageRoot = join(staging, 'package')
await mkdir(staging)
const total = await measureTree(pristineRoot)
let processedBytes = 0
let processedItems = 0
await copyTree(pristineRoot, packageRoot, async update => {
processedBytes += update.processedBytes
processedItems += update.processedItems
await onProgress({ processedBytes, totalBytes: total.bytes, processedItems, totalItems: total.items })
})
await applyPatchSet(packageRoot, patchPaths)
const binTarget = join(packageRoot, 'lib', 'bin.js')
const details = await lstat(binTarget)
if (!details.isFile()) throw new Error('DSH Runtime has no lib/bin.js entrypoint')
await mkdir(join(staging, 'bin'))
await symlink('../package/lib/bin.js', join(staging, 'bin', 'dsh'))
await rename(staging, destination)
return destination
} catch (error) {
await rm(staging, { recursive: true, force: true })
throw error
}
}
async function optionalLink(path) {
try {
return basename(await import('node:fs/promises').then(({ readlink }) => readlink(path)))
} catch (error) {
if (error?.code === 'ENOENT') return undefined
throw error
}
}
async function replaceLink(root, name, runtimeId) {
const temporary = join(root, `.${name}.${randomUUID()}.tmp`)
await symlink(join('versions', runtimeId), temporary)
await rename(temporary, join(root, name))
}
export function cloneTree(source, destination) {
return new Promise((resolveClone, reject) => {
const child = spawn('cp', ['-a', '--reflink=auto', '--', resolve(source), resolve(destination)], {
stdio: ['ignore', 'ignore', 'pipe'],
})
const stderr = []
child.stderr.on('data', chunk => stderr.push(chunk))
child.once('error', reject)
child.once('exit', code => code === 0
? resolveClone(destination)
: reject(new Error(`Platform tree copy failed: ${Buffer.concat(stderr).toString('utf8').trim()}`)))
})
}
export class RuntimeSlots {
constructor(root) {
this.root = resolve(root)
}
async state() {
return Object.freeze({
current: await optionalLink(join(this.root, 'current')),
previous: await optionalLink(join(this.root, 'previous')),
})
}
async promote(runtimeId) {
await lstat(join(this.root, 'versions', runtimeId))
const state = await this.state()
if (state.current !== undefined && state.current !== runtimeId) await replaceLink(this.root, 'previous', state.current)
await replaceLink(this.root, 'current', runtimeId)
return this.state()
}
async rollback() {
const state = await this.state()
if (state.previous === undefined) throw new Error('no previous Runtime exists')
await replaceLink(this.root, 'current', state.previous)
if (state.current !== undefined) await replaceLink(this.root, 'previous', state.current)
return this.state()
}
async prune() {
const state = await this.state()
const retained = new Set([state.current, state.previous].filter(Boolean))
let versions
try { versions = await readdir(join(this.root, 'versions')) } catch (error) {
if (error?.code === 'ENOENT') return Object.freeze([])
throw error
}
const removed = []
for (const version of versions) {
if (!version.startsWith('.') && !retained.has(version)) {
await rm(join(this.root, 'versions', version), { recursive: true, force: true })
removed.push(version)
}
}
return Object.freeze(removed.sort())
}
}
upstream/container/control-plane/modules/plugin-manager/system.mjs
+441
View File
@@ -0,0 +1,441 @@
import { spawn } from 'node:child_process'
import { randomUUID } from 'node:crypto'
import { lstat, mkdir, readdir, readFile, readlink, rename, rm, symlink, writeFile } from 'node:fs/promises'
import { dirname, join, resolve } from 'node:path'
import { canonicalJson } from '../../../platform/lib/canonical-json.mjs'
import { artifactForReference, parseEnvironmentManifest } from '../../../platform/lib/contracts.mjs'
import { hashTree } from '../../../platform/lib/tree-hash.mjs'
const PLUGIN_ID_PATTERN = /^[a-z0-9][a-z0-9._-]{0,127}$/
const SELECTION_SCHEMA = 1
export const PROTECTED_SYSTEM_PLUGIN_IDS = Object.freeze(['platform-management'])
function assertPluginId(pluginId) {
if (typeof pluginId !== 'string' || !PLUGIN_ID_PATTERN.test(pluginId)) {
throw new Error('System Plugin ID is invalid')
}
return pluginId
}
function run(command, args) {
return new Promise((resolveRun, reject) => {
const child = spawn(command, args, { stdio: ['ignore', 'pipe', 'pipe'] })
const stdout = []
const stderr = []
child.stdout.on('data', chunk => stdout.push(chunk))
child.stderr.on('data', chunk => stderr.push(chunk))
child.once('error', reject)
child.once('exit', code => {
if (code === 0) resolveRun(Buffer.concat(stdout).toString('utf8'))
else reject(new Error(`${command} failed: ${Buffer.concat(stderr).toString('utf8').trim()}`))
})
})
}
async function extractPackage(archive, destination) {
const listing = (await run('tar', ['-tzf', archive])).split('\n').filter(Boolean)
if (listing.length === 0) throw new Error('System Plugin archive is empty')
for (const name of listing) {
if (!name.startsWith('package/') || name.startsWith('/') || name.split('/').includes('..')) {
throw new Error(`System Plugin archive path is unsafe: ${name}`)
}
}
await mkdir(destination, { recursive: true })
await run('tar', ['-xzf', archive, '--strip-components=1', '--no-same-owner', '--no-same-permissions', '-C', destination])
}
function validatePackage(value, pluginId) {
if (value === null || typeof value !== 'object' || Array.isArray(value)) {
throw new Error(`System Plugin ${pluginId} package.json must be an object`)
}
const expectedName = `@dsh-docker/${pluginId}`
if (value.name !== expectedName) throw new Error(`System Plugin ${pluginId} package name must be ${expectedName}`)
if (typeof value.main !== 'string' || value.main.length === 0
|| value.main.startsWith('/') || value.main.includes('\\') || value.main.split('/').includes('..')) {
throw new Error(`System Plugin ${pluginId} package main must be a safe relative path`)
}
return expectedName
}
function validatePatch(value, pluginId, packageName) {
if (!Array.isArray(value)) throw new Error(`System Plugin ${pluginId} cordis.patch.json must be an array`)
return value.map((entry, index) => {
if (entry === null || typeof entry !== 'object' || Array.isArray(entry)) {
throw new Error(`System Plugin ${pluginId} patch ${String(index)} must be an object`)
}
if (Object.keys(entry).length !== 1 || !Array.isArray(entry.insert) || entry.insert.length === 0) {
throw new Error(`System Plugin ${pluginId} patches must contain only a non-empty insert list`)
}
for (const row of entry.insert) {
if (row === null || typeof row !== 'object' || Array.isArray(row)) {
throw new Error(`System Plugin ${pluginId} inserted rows must be objects`)
}
if (typeof row.id !== 'string' || !row.id.startsWith(`dsh-docker.${pluginId}.`)) {
throw new Error(`System Plugin ${pluginId} patch IDs must use its dsh-docker namespace`)
}
if (typeof row.name !== 'string' || row.name.length === 0) {
throw new Error(`System Plugin ${pluginId} inserted rows must name a package`)
}
if (row.name !== packageName) {
throw new Error(`System Plugin ${pluginId} inserted rows must load its own package`)
}
}
return {
insert: entry.insert.map(row => ({ ...row })),
}
})
}
async function replaceLink(root, name, version) {
const temporary = join(root, `.${name}.${randomUUID()}.tmp`)
await symlink(join('versions', version), temporary)
await rename(temporary, join(root, name))
}
export async function reconcileSystemPlugins({ root, environmentVersion, plugins, artifactPath }) {
const managedRoot = resolve(root)
const versions = join(managedRoot, 'versions')
const destination = join(versions, environmentVersion)
const staging = join(versions, `.${environmentVersion}.${randomUUID()}.tmp`)
await mkdir(staging, { recursive: true })
const patches = []
try {
for (const plugin of plugins) {
const packageRoot = join(staging, 'packages', plugin.id)
await extractPackage(artifactPath(plugin), packageRoot)
const metadata = JSON.parse(await readFile(join(packageRoot, 'package.json'), 'utf8'))
const packageName = validatePackage(metadata, plugin.id)
const patch = JSON.parse(await readFile(join(packageRoot, 'cordis.patch.json'), 'utf8'))
patches.push(...validatePatch(patch, plugin.id, packageName))
}
await writeFile(join(staging, 'cordis.patch.yml'), canonicalJson(patches), { flag: 'wx' })
await rename(staging, destination)
const current = await import('node:fs/promises').then(({ readlink }) => readlink(join(managedRoot, 'current')).catch(error => error.code === 'ENOENT' ? undefined : Promise.reject(error)))
if (current !== undefined) await replaceLink(managedRoot, 'previous', current.split('/').at(-1))
await replaceLink(managedRoot, 'current', environmentVersion)
return destination
} catch (error) {
await rm(staging, { recursive: true, force: true })
throw error
}
}
async function readOverlay(root) {
const patch = JSON.parse(await readFile(join(root, 'cordis.patch.yml'), 'utf8'))
if (!Array.isArray(patch)) throw new Error('System Plugin overlay patch must be an array')
return patch
}
function defaultSelection(pluginIds, protectedIds) {
return Object.fromEntries(pluginIds.map(id => [id, {
installed: true,
enabled: true,
protected: protectedIds.has(id),
}]))
}
function validateSelectionDocument(value, pluginIds, protectedIds) {
if (value === null || typeof value !== 'object' || Array.isArray(value) || value.schema !== SELECTION_SCHEMA
|| value.plugins === null || typeof value.plugins !== 'object' || Array.isArray(value.plugins)) {
throw new Error('System Plugin selection state is invalid')
}
const selection = defaultSelection(pluginIds, protectedIds)
for (const id of pluginIds) {
const saved = value.plugins[id]
if (saved === undefined) continue
if (saved === null || typeof saved !== 'object' || Array.isArray(saved)
|| typeof saved.installed !== 'boolean' || typeof saved.enabled !== 'boolean') {
throw new Error(`System Plugin ${id} selection state is invalid`)
}
if (saved.enabled && !saved.installed) {
throw new Error(`System Plugin ${id} cannot be enabled while it is not installed`)
}
selection[id] = { installed: saved.installed, enabled: saved.enabled, protected: protectedIds.has(id) }
}
return selection
}
export class SystemPluginSelectionStore {
constructor(path, { protectedIds = PROTECTED_SYSTEM_PLUGIN_IDS } = {}) {
this.path = resolve(path)
this.protectedIds = new Set(protectedIds)
}
async read(pluginIds) {
const ids = pluginIds.map(assertPluginId)
let value
try {
value = JSON.parse(await readFile(this.path, 'utf8'))
} catch (error) {
if (error?.code === 'ENOENT') return defaultSelection(ids, this.protectedIds)
throw error
}
return validateSelectionDocument(value, ids, this.protectedIds)
}
async write(pluginIds, selection) {
const ids = pluginIds.map(assertPluginId)
const normalized = validateSelectionDocument({ schema: SELECTION_SCHEMA, plugins: selection }, ids, this.protectedIds)
const document = {
schema: SELECTION_SCHEMA,
plugins: Object.fromEntries(ids.map(id => [id, {
installed: normalized[id].installed,
enabled: normalized[id].enabled,
}])),
}
await mkdir(dirname(this.path), { recursive: true })
const temporary = `${this.path}.${randomUUID()}.tmp`
try {
await writeFile(temporary, `${canonicalJson(document)}\n`, { flag: 'wx', mode: 0o600 })
await rename(temporary, this.path)
} finally {
await rm(temporary, { force: true })
}
return normalized
}
async configure(pluginIds, pluginId, action) {
const ids = pluginIds.map(assertPluginId)
assertPluginId(pluginId)
if (!ids.includes(pluginId)) throw new Error(`System Plugin ${pluginId} is not provided by the current Environment`)
if (this.protectedIds.has(pluginId)) throw new Error(`System Plugin ${pluginId} is managed by the platform and cannot be changed`)
return this.configureSelection(ids, pluginId, action)
}
async recover(pluginIds, pluginId, action) {
const ids = pluginIds.map(assertPluginId)
assertPluginId(pluginId)
if (!ids.includes(pluginId)) throw new Error(`System Plugin ${pluginId} is not provided by the current Environment`)
if (!this.protectedIds.has(pluginId)) throw new Error(`System Plugin ${pluginId} is not a platform recovery target`)
return this.configureSelection(ids, pluginId, action)
}
async configureSelection(ids, pluginId, action) {
if (!['install', 'uninstall', 'enable', 'disable'].includes(action)) {
throw new Error('System Plugin action is invalid')
}
const selection = await this.read(ids)
const current = selection[pluginId]
if (action === 'install') selection[pluginId] = { ...current, installed: true, enabled: true }
else if (action === 'uninstall') selection[pluginId] = { ...current, installed: false, enabled: false }
else if (action === 'enable') {
if (!current.installed) throw new Error(`System Plugin ${pluginId} must be installed before it can be enabled`)
selection[pluginId] = { ...current, enabled: true }
} else selection[pluginId] = { ...current, enabled: false }
return this.write(ids, selection)
}
}
async function pluginCatalog(environmentRoot) {
const root = resolve(environmentRoot)
const manifest = parseEnvironmentManifest(await readFile(join(root, 'environment.manifest.json')))
return {
root,
plugins: manifest.systemPlugins.map(reference => ({
id: reference.id,
descriptor: artifactForReference(manifest, reference),
})),
}
}
async function readActiveSelection(root, plugins) {
if (root === undefined) return undefined
const overlay = await readOverlay(resolve(root))
const enabledPackages = new Set(overlay.flatMap(entry => (
Array.isArray(entry?.insert) ? entry.insert.map(row => row?.name).filter(Boolean) : []
)))
return Object.fromEntries(await Promise.all(plugins.map(async ({ id }) => {
const installed = await lstat(join(resolve(root), 'packages', id)).then(
() => true,
error => error?.code === 'ENOENT' ? false : Promise.reject(error),
)
return [id, { installed, enabled: installed && enabledPackages.has(`@dsh-docker/${id}`) }]
})))
}
export async function listManagedSystemPlugins({ environmentRoot, sourceRoot, selectionStore, activeRoot }) {
const catalog = await pluginCatalog(environmentRoot)
const selection = await selectionStore.read(catalog.plugins.map(plugin => plugin.id))
const active = await readActiveSelection(activeRoot, catalog.plugins)
return Object.freeze(await Promise.all(catalog.plugins.map(async ({ id, descriptor }) => {
const metadata = JSON.parse(await readFile(join(resolve(sourceRoot), 'packages', id, 'package.json'), 'utf8'))
validatePackage(metadata, id)
let description = null
const value = metadata.dshDocker?.description
if (value !== undefined) {
if (value === null || typeof value !== 'object' || Array.isArray(value)
|| typeof value.zh !== 'string' || typeof value.en !== 'string'
|| value.zh.length === 0 || value.en.length === 0 || value.zh.length > 240 || value.en.length > 240) {
throw new Error(`System Plugin ${id} description is invalid`)
}
description = Object.freeze({ zh: value.zh, en: value.en })
}
const activeInstalled = active?.[id].installed ?? selection[id].installed
const activeEnabled = active?.[id].enabled ?? selection[id].enabled
return Object.freeze({
id,
artifactId: descriptor.id,
sha256: descriptor.sha256,
description,
installed: selection[id].installed,
enabled: selection[id].enabled,
activeInstalled,
activeEnabled,
pendingRestart: activeInstalled !== selection[id].installed || activeEnabled !== selection[id].enabled,
protected: selection[id].protected,
reason: null,
})
})))
}
export async function discardSystemPluginSelection({ environmentRoot, sourceRoot, selectionStore, activeRoot }) {
const catalog = await pluginCatalog(environmentRoot)
const pluginIds = catalog.plugins.map(plugin => plugin.id)
const active = await readActiveSelection(activeRoot, catalog.plugins)
if (active === undefined) throw new Error('active System Plugin view is unavailable')
await selectionStore.write(pluginIds, active)
return listManagedSystemPlugins({ environmentRoot, sourceRoot, selectionStore, activeRoot })
}
export async function materializeSystemPluginSelection({
environmentRoot,
sourceRoot,
outputRoot,
selectionStore,
}) {
const catalog = await pluginCatalog(environmentRoot)
const pluginIds = catalog.plugins.map(plugin => plugin.id)
const selection = await selectionStore.read(pluginIds)
const root = resolve(outputRoot)
const staging = join(root, `.selection.${randomUUID()}.tmp`)
const destination = join(root, `selection-${randomUUID()}`)
const patches = []
await mkdir(join(staging, 'packages'), { recursive: true })
try {
for (const { id } of catalog.plugins) {
if (!selection[id].installed) continue
const packageRoot = join(resolve(sourceRoot), 'packages', id)
const metadata = JSON.parse(await readFile(join(packageRoot, 'package.json'), 'utf8'))
const packageName = validatePackage(metadata, id)
const patch = JSON.parse(await readFile(join(packageRoot, 'cordis.patch.json'), 'utf8'))
const validatedPatch = validatePatch(patch, id, packageName)
await symlink(packageRoot, join(staging, 'packages', id), 'dir')
if (selection[id].enabled) patches.push(...validatedPatch)
}
await writeFile(join(staging, 'cordis.patch.yml'), canonicalJson(patches), { flag: 'wx' })
await rename(staging, destination)
return Object.freeze({ path: destination, plugins: await listManagedSystemPlugins({ environmentRoot, sourceRoot, selectionStore }) })
} catch (error) {
await rm(staging, { recursive: true, force: true })
throw error
}
}
export async function pruneSystemPluginSelectionViews({ outputRoot, keepPath }) {
const root = resolve(outputRoot)
const keep = resolve(keepPath)
for (const entry of await readdir(root, { withFileTypes: true })) {
if (!entry.name.startsWith('selection-')) continue
const candidate = join(root, entry.name)
if (candidate !== keep) await rm(candidate, { recursive: true, force: true })
}
}
export async function listBundledSystemPlugins({ environmentRoot, viewRoot }) {
const manifest = parseEnvironmentManifest(await readFile(join(resolve(environmentRoot), 'environment.manifest.json')))
const overlay = await readOverlay(resolve(viewRoot)).catch(() => [])
const moduleNames = new Set(overlay.flatMap(entry => (
Array.isArray(entry?.insert) ? entry.insert.map(row => row?.name).filter(Boolean) : []
)))
const result = []
for (const reference of manifest.systemPlugins) {
const descriptor = artifactForReference(manifest, reference)
let installed = false
let reason = null
try {
const metadata = JSON.parse(await readFile(join(resolve(viewRoot), 'packages', reference.id, 'package.json'), 'utf8'))
const packageName = validatePackage(metadata, reference.id)
installed = moduleNames.has(packageName)
if (!installed) reason = 'plugin loader entry is absent'
} catch (error) {
reason = error instanceof Error ? error.message : 'plugin package is unavailable'
}
result.push(Object.freeze({
id: reference.id,
artifactId: descriptor.id,
sha256: descriptor.sha256,
installed,
reason,
}))
}
return Object.freeze(result)
}
export async function rebuildBundledSystemPluginView({
environmentRoot,
outputRoot,
expectedSha256,
requestedPluginId,
}) {
const source = resolve(environmentRoot)
const manifest = parseEnvironmentManifest(await readFile(join(source, 'environment.manifest.json')))
if (!manifest.systemPlugins.some(plugin => plugin.id === requestedPluginId)) {
throw new Error(`System Plugin ${requestedPluginId} is not bundled by the current Environment`)
}
const root = resolve(outputRoot)
const destination = join(root, expectedSha256)
try {
const details = await lstat(destination)
if (!details.isDirectory() || details.isSymbolicLink() || await hashTree(destination) !== expectedSha256) {
await rm(destination, { recursive: true, force: true })
} else {
return destination
}
} catch (error) {
if (error?.code !== 'ENOENT') throw error
}
const buildRoot = join(root, `.build.${randomUUID()}.tmp`)
await mkdir(root, { recursive: true })
try {
const built = await reconcileSystemPlugins({
root: buildRoot,
environmentVersion: expectedSha256,
plugins: manifest.systemPlugins,
artifactPath: reference => join(source, 'artifacts', artifactForReference(manifest, reference).id),
})
if (await hashTree(built) !== expectedSha256) {
throw new Error('rebuilt System Plugin view differs from the current Deployment Record')
}
await rename(built, destination)
return destination
} finally {
await rm(buildRoot, { recursive: true, force: true })
}
}
export async function linkSystemPluginScope({
dshHome,
viewRoot = '/run/dsh-platform/views/system-plugins',
}) {
const modulesRoot = join(resolve(dshHome), 'profiles', 'node_modules')
const scopeLink = join(modulesRoot, '@dsh-docker')
const target = join(resolve(viewRoot), 'packages')
await mkdir(modulesRoot, { recursive: true })
const existing = await lstat(scopeLink).catch(error => error?.code === 'ENOENT' ? undefined : Promise.reject(error))
if (existing !== undefined) {
if (!existing.isSymbolicLink()) {
throw new Error(`System Plugin scope ${scopeLink} exists and is not a symlink`)
}
if (await readlink(scopeLink) === target) return scopeLink
await rm(scopeLink)
}
const temporary = join(modulesRoot, `.@dsh-docker.${randomUUID()}.tmp`)
try {
await symlink(target, temporary, 'dir')
await rename(temporary, scopeLink)
} finally {
await rm(temporary, { force: true })
}
return scopeLink
}
upstream/container/control-plane/modules/plugin-manager/user-inventory.mjs
+203
View File
@@ -0,0 +1,203 @@
import { createHash } from 'node:crypto'
import { readFile } from 'node:fs/promises'
import { join, resolve } from 'node:path'
const PACKAGE_NAME_PATTERN = /^(?:@[a-z0-9][a-z0-9._-]{0,126}\/[a-z0-9][a-z0-9._-]{0,126}|[a-z0-9][a-z0-9._-]{0,213})$/i
const SELECTION_SCHEMA = 1
function packagePath(root, packageName) {
if (!PACKAGE_NAME_PATTERN.test(packageName)) return undefined
return join(root, 'node_modules', ...packageName.split('/'), 'package.json')
}
async function optionalBytes(path) {
return readFile(path).then(
bytes => ({ present: true, bytes }),
error => error?.code === 'ENOENT' ? { present: false, bytes: Buffer.alloc(0) } : Promise.reject(error),
)
}
function parseObject(bytes, label) {
let value
try {
value = JSON.parse(bytes.toString('utf8'))
} catch (error) {
throw new Error(`${label} is not valid JSON`, { cause: error })
}
if (value === null || typeof value !== 'object' || Array.isArray(value)) {
throw new Error(`${label} must contain a JSON object`)
}
return value
}
function profileFields(manifest) {
const dependencies = manifest.dependencies ?? {}
if (dependencies === null || typeof dependencies !== 'object' || Array.isArray(dependencies)) {
throw new Error('Web Profile dependencies must be an object')
}
for (const [name, spec] of Object.entries(dependencies)) {
if (typeof spec !== 'string' || spec.length === 0) {
throw new Error(`Web Profile dependency ${name} must have a non-empty string spec`)
}
}
const bundles = manifest.dsh?.profile?.bundles ?? []
if (!Array.isArray(bundles) || bundles.some(name => typeof name !== 'string')) {
throw new Error('Web Profile dsh.profile.bundles must be a string array')
}
return { dependencies, bundles }
}
function disabledPositions(value) {
if (value === undefined) return new Map()
if (value === null || typeof value !== 'object' || Array.isArray(value)
|| value.schema !== SELECTION_SCHEMA || !Array.isArray(value.disabled)) {
throw new Error('User Plugin selection state is invalid')
}
const positions = new Map()
for (const entry of value.disabled) {
if (entry === null || typeof entry !== 'object' || Array.isArray(entry)
|| !PACKAGE_NAME_PATTERN.test(entry.name) || !Number.isSafeInteger(entry.index) || entry.index < 0) {
throw new Error('User Plugin disabled-order entry is invalid')
}
if (positions.has(entry.name)) throw new Error(`User Plugin ${entry.name} has duplicate disabled-order entries`)
positions.set(entry.name, entry.index)
}
return positions
}
function sourceForSpec(spec) {
if (/^(?:file|link):/i.test(spec) || /^(?:\.{1,2}\/|\/)/.test(spec)) return 'file'
if (/^(?:git(?:\+[^:]+)?|github|gitlab|bitbucket):/i.test(spec) || /\.git(?:#|$)/i.test(spec)) return 'git'
if (/^https?:/i.test(spec)) return 'url'
if (/^npm:/i.test(spec) || /^(?:[~^<>=*]|\d|v\d)/i.test(spec) || spec === 'latest' || spec === 'next') return 'registry'
return 'other'
}
function updateRevision(hash, label, file) {
hash.update(`${label}:${file.present ? '1' : '0'}:${String(file.bytes.length)}:`)
hash.update(file.bytes)
hash.update('\0')
}
function metadataFailure(error) {
if (error?.code === 'ENOENT') return 'installed package metadata is missing'
if (error instanceof SyntaxError) return 'installed package metadata is not valid JSON'
return error instanceof Error ? error.message : String(error)
}
async function inspectDependency(profileRoot, name, spec, enabledBundles, disabled, reservedNames) {
const path = packagePath(profileRoot, name)
let metadata
let metadataError = null
if (path === undefined) {
metadataError = 'dependency name is invalid'
} else {
try {
metadata = JSON.parse(await readFile(path, 'utf8'))
if (metadata === null || typeof metadata !== 'object' || Array.isArray(metadata)) {
throw new Error('installed package metadata must be an object')
}
if (metadata.name !== name) throw new Error(`installed package name is ${String(metadata.name)}`)
} catch (error) {
metadataError = metadataFailure(error)
}
}
if (metadataError === null && metadata.version !== undefined
&& (typeof metadata.version !== 'string' || metadata.version.length === 0)) {
metadataError = 'installed package version is invalid'
}
const bundleDeclaration = metadata?.dsh?.bundle
const patch = bundleDeclaration?.patch
const isBundle = typeof patch === 'string' && patch.length > 0
const reservedNameConflict = reservedNames.has(name)
if (metadataError === null && !isBundle
&& (bundleDeclaration !== undefined || enabledBundles.has(name))) {
metadataError = 'installed package Bundle metadata is invalid'
}
if (!isBundle && metadataError === null && !reservedNameConflict) return undefined
return Object.freeze({
name,
spec,
source: sourceForSpec(spec),
version: typeof metadata?.version === 'string' ? metadata.version : null,
description: typeof metadata?.description === 'string' && metadata.description.trim() !== '' ? metadata.description.trim() : null,
enabled: !reservedNameConflict && enabledBundles.has(name),
previousIndex: disabled.get(name) ?? null,
damaged: metadataError !== null,
metadataError,
reservedNameConflict,
})
}
function loadedIdentity(plugin) {
return JSON.stringify([
plugin.name, plugin.spec, plugin.version, plugin.enabled, plugin.damaged,
plugin.metadataError, plugin.reservedNameConflict,
])
}
export function markUserPluginRestartState(current, loaded) {
if (loaded === undefined) return current
const loadedPlugins = new Map(loaded.plugins.map(plugin => [plugin.name, loadedIdentity(plugin)]))
const plugins = current.plugins.map(plugin => Object.freeze({
...plugin,
pendingRestart: loadedPlugins.get(plugin.name) !== loadedIdentity(plugin),
}))
const currentNames = new Set(current.plugins.map(plugin => plugin.name))
const removedLoadedPlugin = loaded.plugins.some(plugin => !currentNames.has(plugin.name))
return Object.freeze({
...current,
plugins: Object.freeze(plugins),
restartRequired: removedLoadedPlugin || plugins.some(plugin => plugin.pendingRestart),
})
}
export class UserPluginInventory {
constructor({ dshHome = '/data/dsh', profile = 'web', selectionPath, systemPluginNames = async () => [] } = {}) {
if (profile !== 'web') throw new Error('Only the web DSH Profile is supported')
this.profile = profile
this.profileRoot = join(resolve(dshHome), 'profiles', profile)
this.selectionPath = selectionPath === undefined
? join(resolve(dshHome), '..', 'platform', 'state', 'management', 'user-plugins.json')
: resolve(selectionPath)
this.systemPluginNames = systemPluginNames
}
async read() {
const manifestPath = join(this.profileRoot, 'package.json')
const lockfilePath = join(this.profileRoot, 'pnpm-lock.yaml')
const [manifestBytes, lockfile, selectionFile, names] = await Promise.all([
readFile(manifestPath),
optionalBytes(lockfilePath),
optionalBytes(this.selectionPath),
this.systemPluginNames(),
])
const manifest = parseObject(manifestBytes, 'Web Profile package.json')
const { dependencies, bundles } = profileFields(manifest)
const selection = !selectionFile.present
? undefined
: parseObject(selectionFile.bytes, 'User Plugin selection state')
const disabled = disabledPositions(selection)
const reservedNames = new Set(names)
if ([...reservedNames].some(name => typeof name !== 'string' || !PACKAGE_NAME_PATTERN.test(name))) {
throw new Error('Verified System Plugin names are invalid')
}
const enabledBundles = new Set(bundles)
const inspected = await Promise.all(Object.entries(dependencies).map(([name, spec]) => (
inspectDependency(this.profileRoot, name, spec, enabledBundles, disabled, reservedNames)
)))
const revisionHash = createHash('sha256')
updateRevision(revisionHash, 'manifest', { present: true, bytes: manifestBytes })
updateRevision(revisionHash, 'lockfile', lockfile)
updateRevision(revisionHash, 'selection', selectionFile)
const revision = revisionHash.digest('hex')
return Object.freeze({
schema: 1,
profile: this.profile,
revision: `sha256:${revision}`,
plugins: Object.freeze(inspected.filter(Boolean)),
})
}
}
export const userPluginInternals = Object.freeze({ PACKAGE_NAME_PATTERN, sourceForSpec, loadedIdentity })
upstream/container/control-plane/modules/plugin-manager/user-journal.mjs
+116
View File
@@ -0,0 +1,116 @@
import { readFile } from 'node:fs/promises'
import { durableReplace } from '../../../platform/lib/atomic.mjs'
import { userPluginInternals } from './user-inventory.mjs'
const PHASES = new Set([
'validated', 'paused', 'snapshotted', 'mutating', 'committed', 'restarting',
'restoring', 'completed', 'failed',
])
const TERMINAL = new Set(['completed', 'failed'])
const TRANSITIONS = new Map([
['validated', new Set(['paused', 'snapshotted', 'failed'])],
['paused', new Set(['snapshotted', 'restoring', 'failed'])],
['snapshotted', new Set(['mutating', 'restoring'])],
['mutating', new Set(['committed', 'restoring'])],
['committed', new Set(['restarting', 'completed', 'failed'])],
['restarting', new Set(['completed', 'failed'])],
['restoring', new Set(['failed'])],
['completed', new Set()],
['failed', new Set()],
])
function parseAction(value) {
if (value === null || typeof value !== 'object' || Array.isArray(value)
|| Object.keys(value).sort().join(',') !== 'action,name'
|| typeof value.name !== 'string' || !userPluginInternals.PACKAGE_NAME_PATTERN.test(value.name)
|| !['enable', 'disable', 'uninstall'].includes(value.action)) {
throw new Error('User Plugin journal action is invalid')
}
return Object.freeze({ name: value.name, action: value.action })
}
function parse(value) {
if (value === null || typeof value !== 'object' || Array.isArray(value)
|| Object.keys(value).sort().join(',') !== 'actions,error,phase,previousDisabled,recoveryResult,revision,schema,selectionPresent,snapshotId,taskId,updatedAt'
|| value.schema !== 1 || typeof value.taskId !== 'string' || !/^[A-Za-z0-9-]{1,128}$/.test(value.taskId)
|| !PHASES.has(value.phase) || typeof value.revision !== 'string' || !/^sha256:[a-f0-9]{64}$/.test(value.revision)
|| !Array.isArray(value.actions) || value.actions.length === 0
|| typeof value.selectionPresent !== 'boolean' || !Array.isArray(value.previousDisabled)
|| (value.snapshotId !== null && typeof value.snapshotId !== 'string')
|| (value.error !== null && typeof value.error !== 'string')
|| ![null, 'success', 'failed'].includes(value.recoveryResult)
|| typeof value.updatedAt !== 'string' || new Date(value.updatedAt).toISOString() !== value.updatedAt) {
throw new Error('User Plugin transaction journal is invalid')
}
const names = new Set()
const previousDisabled = value.previousDisabled.map(entry => {
if (entry === null || typeof entry !== 'object' || Array.isArray(entry)
|| Object.keys(entry).sort().join(',') !== 'index,name'
|| typeof entry.name !== 'string' || !userPluginInternals.PACKAGE_NAME_PATTERN.test(entry.name)
|| !Number.isSafeInteger(entry.index) || entry.index < 0
|| names.has(entry.name)) throw new Error('User Plugin journal disabled-order state is invalid')
names.add(entry.name)
return Object.freeze({ name: entry.name, index: entry.index })
})
return Object.freeze({
...value,
actions: Object.freeze(value.actions.map(parseAction)),
previousDisabled: Object.freeze(previousDisabled),
})
}
export class UserPluginJournal {
constructor(path, now = () => new Date()) {
this.path = path
this.now = now
this.queue = Promise.resolve()
}
exclusive(operation) {
const result = this.queue.then(operation, operation)
this.queue = result.then(() => undefined, () => undefined)
return result
}
async read() {
try { return parse(JSON.parse(await readFile(this.path, 'utf8'))) } catch (error) {
if (error?.code === 'ENOENT') return undefined
throw error
}
}
begin({ taskId, revision, actions, selection }) {
return this.exclusive(async () => {
const current = await this.read()
if (current !== undefined && !TERMINAL.has(current.phase)) {
throw new Error('a User Plugin transaction is already active')
}
const next = parse({
schema: 1, taskId, revision, actions,
selectionPresent: selection.present,
previousDisabled: selection.state.disabled,
phase: 'validated', snapshotId: null,
error: null, recoveryResult: null, updatedAt: this.now().toISOString(),
})
await durableReplace(this.path, `${JSON.stringify(next)}\n`)
return next
})
}
transition(phase, fields = {}) {
return this.exclusive(async () => {
const current = await this.read()
if (current === undefined || !TRANSITIONS.get(current.phase)?.has(phase)) {
throw new Error(`User Plugin journal cannot transition to ${phase}`)
}
const next = parse({ ...current, ...fields, phase, updatedAt: this.now().toISOString() })
if (next.taskId !== current.taskId || next.revision !== current.revision) {
throw new Error('User Plugin journal identity cannot change')
}
await durableReplace(this.path, `${JSON.stringify(next)}\n`)
return next
})
}
}
export const userPluginJournalInternals = Object.freeze({ parse, TERMINAL })
upstream/container/control-plane/modules/plugin-manager/user-snapshots.mjs
+143
View File
@@ -0,0 +1,143 @@
import { spawn } from 'node:child_process'
import { createHash, randomUUID } from 'node:crypto'
import { lstat, mkdir, open, readFile, readdir, rename, rm } from 'node:fs/promises'
import { join, resolve } from 'node:path'
import { canonicalJson } from '../../../platform/lib/canonical-json.mjs'
import { durableReplace } from '../../../platform/lib/atomic.mjs'
const ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/
function validId(id) {
if (typeof id !== 'string' || !ID_PATTERN.test(id)) throw new Error('User Plugin snapshot ID is invalid')
return id
}
function run(command, args) {
return new Promise((resolveRun, reject) => {
const child = spawn(command, args, { stdio: ['ignore', 'pipe', 'pipe'] })
const stdout = []
const stderr = []
child.stdout.on('data', chunk => stdout.push(chunk))
child.stderr.on('data', chunk => stderr.push(chunk))
child.once('error', reject)
child.once('exit', code => code === 0
? resolveRun(Buffer.concat(stdout).toString('utf8'))
: reject(new Error(`${command} failed: ${Buffer.concat(stderr).toString('utf8').trim()}`)))
})
}
async function measure(path) {
const handle = await open(path, 'r')
const hash = createHash('sha256')
let size = 0
try {
for await (const chunk of handle.createReadStream({ autoClose: false })) {
hash.update(chunk)
size += chunk.byteLength
}
} finally {
await handle.close()
}
return { sha256: hash.digest('hex'), size }
}
async function syncDirectory(path) {
const handle = await open(path, 'r')
try { await handle.sync() } finally { await handle.close() }
}
function parseManifest(value) {
if (value === null || typeof value !== 'object' || Array.isArray(value)
|| Object.keys(value).sort().join(',') !== 'archiveSha256,archiveSize,createdAt,id,schema,source'
|| value.schema !== 1 || !ID_PATTERN.test(value.id)
|| value.source !== '/data/dsh/profiles/web'
|| typeof value.createdAt !== 'string' || new Date(value.createdAt).toISOString() !== value.createdAt
|| typeof value.archiveSha256 !== 'string' || !/^[a-f0-9]{64}$/.test(value.archiveSha256)
|| !Number.isSafeInteger(value.archiveSize) || value.archiveSize < 0) {
throw new Error('User Plugin snapshot manifest is invalid')
}
return Object.freeze({ ...value })
}
export class UserPluginSnapshots {
constructor({ root, profileRoot = '/data/dsh/profiles/web', now = () => new Date(), runImpl = run }) {
this.root = resolve(root)
this.profileRoot = resolve(profileRoot)
this.now = now
this.run = runImpl
}
path(id) { return join(this.root, validId(id)) }
async create(id = randomUUID()) {
validId(id)
const details = await lstat(this.profileRoot)
if (!details.isDirectory() || details.isSymbolicLink()) throw new Error('Web Profile must be a real directory')
await mkdir(this.root, { recursive: true })
const destination = this.path(id)
await lstat(destination).then(
() => { throw new Error(`User Plugin snapshot ${id} already exists`) },
error => { if (error?.code !== 'ENOENT') throw error },
)
const staging = join(this.root, `.${id}.${randomUUID()}.tmp`)
await mkdir(staging)
const archive = join(staging, 'profile.tar')
try {
await this.run('tar', ['-cpf', archive, '-C', this.profileRoot, '.'])
const measured = await measure(archive)
const manifest = parseManifest({
schema: 1,
id,
source: '/data/dsh/profiles/web',
createdAt: this.now().toISOString(),
archiveSha256: measured.sha256,
archiveSize: measured.size,
})
await durableReplace(join(staging, 'manifest.json'), canonicalJson(manifest))
await rename(staging, destination)
await syncDirectory(this.root)
return Object.freeze({ ...manifest, path: destination })
} catch (error) {
await rm(staging, { recursive: true, force: true })
throw error
}
}
async inspect(id) {
const root = this.path(id)
const manifest = parseManifest(JSON.parse(await readFile(join(root, 'manifest.json'), 'utf8')))
if (manifest.id !== id) throw new Error('User Plugin snapshot ID does not match its directory')
const archive = await lstat(join(root, 'profile.tar')).then(
() => join(root, 'profile.tar'),
error => error?.code === 'ENOENT' ? join(root, 'profile.tar.gz') : Promise.reject(error),
)
const measured = await measure(archive)
if (measured.sha256 !== manifest.archiveSha256 || measured.size !== manifest.archiveSize) {
throw new Error('User Plugin snapshot archive does not match its manifest')
}
const entries = (await this.run('tar', [archive.endsWith('.gz') ? '-tzf' : '-tf', archive])).split('\n').filter(Boolean)
if (entries.length === 0 || entries.some(entry => entry.startsWith('/') || entry.split('/').includes('..'))) {
throw new Error('User Plugin snapshot archive contains unsafe paths')
}
return Object.freeze({ ...manifest, archive })
}
async restore(id) {
const snapshot = await this.inspect(id)
await mkdir(this.profileRoot, { recursive: true })
for (const entry of await readdir(this.profileRoot)) {
await rm(join(this.profileRoot, entry), { recursive: true, force: true })
}
const ownership = process.getuid?.() === 0 ? [] : ['--no-same-owner']
await this.run('tar', [snapshot.archive.endsWith('.gz') ? '-xzpf' : '-xpf', snapshot.archive, ...ownership, '-C', this.profileRoot])
await syncDirectory(this.profileRoot)
return snapshot
}
async remove(id) {
validId(id)
await rm(this.path(id), { recursive: true, force: true })
}
}
export const userPluginSnapshotInternals = Object.freeze({ parseManifest })
upstream/container/control-plane/modules/plugin-manager/user-state.mjs
+67
View File
@@ -0,0 +1,67 @@
import { open, readFile, rm } from 'node:fs/promises'
import { dirname } from 'node:path'
import { canonicalJson } from '../../../platform/lib/canonical-json.mjs'
import { durableReplace } from '../../../platform/lib/atomic.mjs'
import { userPluginInternals } from './user-inventory.mjs'
function parse(value) {
if (value === null || typeof value !== 'object' || Array.isArray(value)
|| Object.keys(value).sort().join(',') !== 'disabled,schema'
|| value.schema !== 1 || !Array.isArray(value.disabled)) {
throw new Error('User Plugin selection state is invalid')
}
const names = new Set()
const disabled = value.disabled.map(entry => {
if (entry === null || typeof entry !== 'object' || Array.isArray(entry)
|| Object.keys(entry).some(key => !['index', 'name'].includes(key))
|| !userPluginInternals.PACKAGE_NAME_PATTERN.test(entry.name)
|| !Number.isSafeInteger(entry.index) || entry.index < 0 || names.has(entry.name)) {
throw new Error('User Plugin disabled-order entry is invalid')
}
names.add(entry.name)
return Object.freeze({ name: entry.name, index: entry.index })
})
return Object.freeze({ schema: 1, disabled: Object.freeze(disabled) })
}
export class UserPluginSelectionStore {
constructor(path) {
this.path = path
}
async read() {
return (await this.snapshot()).state
}
async snapshot() {
try {
return Object.freeze({ present: true, state: parse(JSON.parse(await readFile(this.path, 'utf8'))) })
} catch (error) {
if (error?.code === 'ENOENT') {
return Object.freeze({ present: false, state: parse({ schema: 1, disabled: [] }) })
}
throw error
}
}
async write(value) {
const state = parse(value)
await durableReplace(this.path, `${canonicalJson(state).toString('utf8')}\n`)
return state
}
async restore(snapshot) {
if (snapshot === null || typeof snapshot !== 'object' || typeof snapshot.present !== 'boolean') {
throw new Error('User Plugin selection snapshot is invalid')
}
if (snapshot.present) return this.write(snapshot.state)
await rm(this.path, { force: true })
const directory = await open(dirname(this.path), 'r').catch(error => error?.code === 'ENOENT' ? undefined : Promise.reject(error))
if (directory !== undefined) {
try { await directory.sync() } finally { await directory.close() }
}
return parse({ schema: 1, disabled: [] })
}
}
export const userPluginStateInternals = Object.freeze({ parse })
upstream/container/control-plane/modules/plugin-manager/user-transaction.mjs
+414
View File
@@ -0,0 +1,414 @@
import { spawn } from 'node:child_process'
import { readFile } from 'node:fs/promises'
import { randomUUID } from 'node:crypto'
import { join } from 'node:path'
import { canonicalJson } from '../../../platform/lib/canonical-json.mjs'
import { durableReplace } from '../../../platform/lib/atomic.mjs'
import { userPluginInternals } from './user-inventory.mjs'
const PACKAGE_PATH_ENVIRONMENT = new Set([
'npm_config_cache',
'npm_config_prefix',
'npm_config_store_dir',
'npm_config_userconfig',
'pnpm_home',
])
function pluginEnvironment(dshHome, home = '/home/node') {
const environment = {
...process.env,
HOME: home,
XDG_CACHE_HOME: join(home, '.cache'),
XDG_CONFIG_HOME: join(home, '.config'),
XDG_DATA_HOME: join(home, '.local', 'share'),
XDG_STATE_HOME: join(home, '.local', 'state'),
DSH_HOME: dshHome,
}
for (const name of Object.keys(environment)) {
if (PACKAGE_PATH_ENVIRONMENT.has(name.toLowerCase())) delete environment[name]
}
return environment
}
function runDshPlugin({ dshHome, profile, name }, timeoutMs = 300_000) {
return new Promise((resolveRun, reject) => {
const child = spawn('/usr/local/bin/dsh', ['plugin', '--profile', profile, 'remove', name], {
env: pluginEnvironment(dshHome),
stdio: ['ignore', 'pipe', 'pipe'],
})
const stdout = []
const stderr = []
let size = 0
let settled = false
let timeoutError
let killTimer
const finish = (error) => {
if (settled) return
settled = true
clearTimeout(timer)
clearTimeout(killTimer)
if (error === undefined) resolveRun()
else reject(error)
}
const timer = setTimeout(() => {
timeoutError = new Error(`dsh plugin remove timed out for ${name}`)
child.kill('SIGTERM')
killTimer = setTimeout(() => child.kill('SIGKILL'), 5_000)
killTimer.unref()
}, timeoutMs)
timer.unref()
for (const stream of [child.stdout, child.stderr]) stream.on('data', chunk => {
size += chunk.byteLength
if (size <= 256 * 1024) (stream === child.stdout ? stdout : stderr).push(chunk)
})
child.once('error', finish)
child.once('exit', code => timeoutError !== undefined
? finish(timeoutError)
: code === 0
? finish()
: finish(new Error(`dsh plugin remove failed for ${name}: ${[
['stdout', Buffer.concat(stdout).toString('utf8').trim()],
['stderr', Buffer.concat(stderr).toString('utf8').trim()],
].filter(([, value]) => value !== '').map(([stream, value]) => `${stream}: ${value}`).join('\\n')}`)))
})
}
function message(error) {
return error instanceof Error ? error.message : String(error)
}
function validateActions(inventory, actions) {
if (!Array.isArray(actions) || actions.length === 0 || actions.length > 128) {
throw new Error('User Plugin actions must be a non-empty bounded array')
}
const plugins = new Map(inventory.plugins.map(plugin => [plugin.name, plugin]))
const names = new Set()
return Object.freeze(actions.map(value => {
if (value === null || typeof value !== 'object' || Array.isArray(value)
|| Object.keys(value).sort().join(',') !== 'action,name'
|| !userPluginInternals.PACKAGE_NAME_PATTERN.test(value.name)
|| !['enable', 'disable', 'uninstall'].includes(value.action)
|| names.has(value.name)) {
throw new Error('User Plugin action is invalid')
}
names.add(value.name)
const plugin = plugins.get(value.name)
if (plugin === undefined) throw new Error(`User Plugin ${value.name} is not managed by the Web Profile`)
if (value.action === 'enable' && (plugin.damaged || plugin.reservedNameConflict)) {
throw new Error(`User Plugin ${value.name} cannot be enabled`)
}
return Object.freeze({ name: value.name, action: value.action })
}))
}
async function mutateProfile({ inventory, selectionStore, actions, runPlugin }) {
const manifestPath = join(inventory.profileRoot, 'package.json')
const manifest = JSON.parse(await readFile(manifestPath, 'utf8'))
const bundles = [...(manifest.dsh?.profile?.bundles ?? [])]
const originalBundles = [...bundles]
const selection = await selectionStore.read()
const disabled = new Map(selection.disabled.map(entry => [entry.name, entry.index]))
for (const { name, action } of actions) {
if (action === 'disable') {
const index = bundles.indexOf(name)
if (index !== -1) {
disabled.set(name, originalBundles.indexOf(name))
for (let found = bundles.indexOf(name); found !== -1; found = bundles.indexOf(name)) bundles.splice(found, 1)
}
} else if (action === 'enable') {
if (!bundles.includes(name)) bundles.splice(Math.min(disabled.get(name) ?? bundles.length, bundles.length), 0, name)
disabled.delete(name)
} else if (action === 'uninstall') {
for (let found = bundles.indexOf(name); found !== -1; found = bundles.indexOf(name)) bundles.splice(found, 1)
}
}
manifest.dsh = { ...manifest.dsh, profile: { ...manifest.dsh?.profile, bundles } }
await durableReplace(manifestPath, `${canonicalJson(manifest).toString('utf8')}\n`)
for (const { name, action } of actions) {
if (action === 'uninstall') {
await runPlugin({ dshHome: join(inventory.profileRoot, '..', '..'), profile: inventory.profile, name })
disabled.delete(name)
}
}
if (actions.some(({ action }) => action === 'uninstall')) {
const installedManifest = JSON.parse(await readFile(manifestPath, 'utf8'))
installedManifest.dsh = {
...installedManifest.dsh,
profile: { ...installedManifest.dsh?.profile, bundles },
}
await durableReplace(manifestPath, `${canonicalJson(installedManifest).toString('utf8')}\n`)
}
await selectionStore.write({
schema: 1,
disabled: [...disabled].map(([name, index]) => ({ name, index })),
})
const resulting = await inventory.read()
const result = new Map(resulting.plugins.map(plugin => [plugin.name, plugin]))
for (const { name, action } of actions) {
if (action === 'uninstall' && result.has(name)) throw new Error(`User Plugin ${name} remains installed`)
if (action === 'enable' && result.get(name)?.enabled !== true) throw new Error(`User Plugin ${name} was not enabled`)
if (action === 'disable' && result.get(name)?.enabled !== false) throw new Error(`User Plugin ${name} was not disabled`)
}
return resulting
}
export class UserPluginTransactionManager {
constructor({ inventory, selectionStore, snapshots, journal, pauseDsh, restartDsh, runPlugin = runDshPlugin, report = async () => {} }) {
this.inventory = inventory
this.selectionStore = selectionStore
this.snapshots = snapshots
this.journal = journal
this.pauseDsh = pauseDsh
this.restartDsh = restartDsh
this.runPlugin = runPlugin
this.report = report
}
record(messageValue, fields = {}) {
return Promise.resolve().then(() => this.report(messageValue, fields)).catch(() => {})
}
async validate({ revision, actions }) {
const current = await this.inventory.read()
if (revision !== current.revision) {
const error = new Error('User Plugin inventory revision has changed')
error.code = 'REVISION_CONFLICT'
throw error
}
return Object.freeze({ revision, actions: validateActions(current, actions) })
}
async apply({ taskId = randomUUID(), revision, actions, strategy = 'restart', onProgress = () => {} }) {
if (!['restart', 'next-start'].includes(strategy)) throw new Error('User Plugin activation strategy is invalid')
const validated = await this.validate({ revision, actions })
const normalized = validated.actions
const publishProgress = state => { try { onProgress(state) } catch {} }
const selectionBefore = await this.selectionStore.snapshot()
let state = await this.journal.begin({ taskId, revision, actions: normalized, selection: selectionBefore })
publishProgress(state)
let paused = false
let snapshotCreated = false
await this.record('user-plugin.transaction.started', { taskId, actions: normalized })
try {
if (strategy === 'restart') {
await this.pauseDsh()
paused = true
state = await this.journal.transition('paused')
publishProgress(state)
}
await this.snapshots.create(taskId)
snapshotCreated = true
state = await this.journal.transition('snapshotted', { snapshotId: taskId })
publishProgress(state)
state = await this.journal.transition('mutating')
publishProgress(state)
const result = await mutateProfile({
inventory: this.inventory,
selectionStore: this.selectionStore,
actions: normalized,
runPlugin: this.runPlugin,
})
state = await this.journal.transition('committed')
publishProgress(state)
let completedStrategy = strategy
let activationError = null
if (strategy === 'restart') {
state = await this.journal.transition('restarting')
publishProgress(state)
try {
await this.restartDsh()
} catch (error) {
completedStrategy = 'next-start'
activationError = message(error)
await this.record('user-plugin.activation.deferred', { taskId, error })
}
}
state = await this.journal.transition('completed')
publishProgress(state)
await this.snapshots.remove(taskId).catch(error => this.record('user-plugin.snapshot.cleanup.failed', { taskId, error }))
await this.record('user-plugin.transaction.completed', { taskId })
return Object.freeze({
taskId, status: 'success', strategy: completedStrategy, activationError, inventory: result,
})
} catch (error) {
const committed = ['committed', 'restarting'].includes(state.phase)
if (committed) {
await this.journal.transition('failed', { error: message(error) })
await this.snapshots.remove(taskId).catch(cleanupError => this.record('user-plugin.snapshot.cleanup.failed', { taskId, error: cleanupError }))
} else if (snapshotCreated || state.snapshotId !== null) {
const snapshotId = state.snapshotId ?? taskId
try {
if (state.phase !== 'restoring') state = await this.journal.transition('restoring', {
error: message(error),
snapshotId,
})
await this.snapshots.restore(snapshotId)
await this.selectionStore.restore(selectionBefore)
if (strategy === 'restart') await this.restartDsh()
await this.journal.transition('failed', { recoveryResult: 'success' })
await this.snapshots.remove(snapshotId).catch(cleanupError => this.record('user-plugin.snapshot.cleanup.failed', {
taskId,
error: cleanupError,
}))
} catch (recoveryError) {
const aggregate = new AggregateError([error, recoveryError], 'User Plugin transaction and recovery failed')
aggregate.journal = await this.journal.transition('failed', {
error: `${message(error)}; recovery failed: ${message(recoveryError)}`,
recoveryResult: 'failed',
}).catch(() => {})
throw aggregate
}
} else if (paused) {
try {
await this.restartDsh()
await this.journal.transition('failed', { error: message(error), recoveryResult: 'success' })
} catch (recoveryError) {
const aggregate = new AggregateError([error, recoveryError], 'User Plugin transaction and recovery failed')
aggregate.journal = await this.journal.transition('failed', {
error: `${message(error)}; recovery failed: ${message(recoveryError)}`,
recoveryResult: 'failed',
}).catch(() => {})
throw aggregate
}
} else {
await this.journal.transition('failed', { error: message(error) }).catch(() => {})
}
await this.record('user-plugin.transaction.failed', { taskId, error, committed })
if (error !== null && typeof error === 'object') error.journal = await this.journal.read().catch(() => undefined)
throw error
}
}
async recover() {
const state = await this.journal.read()
if (state === undefined) return undefined
if (['completed', 'failed'].includes(state.phase)) {
if (state.snapshotId !== null) await this.snapshots.remove(state.snapshotId).catch(error => this.record('user-plugin.snapshot.cleanup.failed', {
taskId: state.taskId,
error,
}))
return state
}
if (['committed', 'restarting'].includes(state.phase)) {
try {
if (state.phase === 'committed') await this.journal.transition('restarting')
await this.restartDsh()
const completed = await this.journal.transition('completed')
if (state.snapshotId !== null) await this.snapshots.remove(state.snapshotId).catch(error => this.record('user-plugin.snapshot.cleanup.failed', {
taskId: state.taskId,
error,
}))
return completed
} catch (error) {
const failed = await this.journal.transition('failed', { error: message(error) })
if (state.snapshotId !== null) {
await this.snapshots.remove(state.snapshotId).catch(cleanupError => this.record('user-plugin.snapshot.cleanup.failed', {
taskId: state.taskId,
error: cleanupError,
}))
}
return failed
}
}
if (state.phase === 'validated') return this.journal.transition('failed', { error: 'transaction interrupted before DSH pause' })
await this.pauseDsh()
if (state.phase === 'paused') {
try {
await this.restartDsh()
return this.journal.transition('failed', { error: 'transaction interrupted before snapshot', recoveryResult: 'success' })
} catch (error) {
return this.journal.transition('failed', { error: message(error), recoveryResult: 'failed' })
}
}
try {
const restoring = state.phase === 'restoring' ? state : await this.journal.transition('restoring', {
error: state.error ?? 'transaction interrupted before commit',
})
await this.snapshots.restore(restoring.snapshotId)
await this.selectionStore.restore({
present: restoring.selectionPresent,
state: { schema: 1, disabled: restoring.previousDisabled },
})
await this.restartDsh()
const failed = await this.journal.transition('failed', { recoveryResult: 'success' })
await this.snapshots.remove(restoring.snapshotId).catch(error => this.record('user-plugin.snapshot.cleanup.failed', {
taskId: state.taskId,
error,
}))
return failed
} catch (error) {
return this.journal.transition('failed', { error: message(error), recoveryResult: 'failed' })
}
}
async recoverBeforeDshStart() {
return recoverUserPluginBeforeDshStart({
journal: this.journal,
report: (messageValue, fields) => this.record(messageValue, fields),
selectionStore: this.selectionStore,
snapshots: this.snapshots,
})
}
async completeDshStartup({ healthy, error = null }) {
const state = await this.journal.read()
if (state === undefined || state.phase !== 'restarting') return state
const next = healthy
? await this.journal.transition('completed')
: await this.journal.transition('failed', { error: error ?? 'DSH failed to start after the committed User Plugin change' })
if (state.snapshotId !== null) {
await this.snapshots.remove(state.snapshotId).catch(cleanupError => this.record('user-plugin.snapshot.cleanup.failed', {
taskId: state.taskId,
error: cleanupError,
}))
}
return next
}
}
export async function recoverUserPluginBeforeDshStart({ journal, selectionStore, snapshots, report = async () => {} }) {
const record = (messageValue, fields) => Promise.resolve()
.then(() => report(messageValue, fields))
.catch(() => {})
let state = await journal.read()
if (state === undefined) return undefined
if (['completed', 'failed'].includes(state.phase)) {
if (state.snapshotId !== null) await snapshots.remove(state.snapshotId).catch(error => record(
'user-plugin.snapshot.cleanup.failed', { taskId: state.taskId, error },
))
return state
}
if (state.phase === 'validated') {
return journal.transition('failed', {
error: 'transaction interrupted before DSH pause', recoveryResult: 'success',
})
}
if (state.phase === 'paused') {
return journal.transition('failed', {
error: 'transaction interrupted before snapshot', recoveryResult: 'success',
})
}
if (['committed', 'restarting'].includes(state.phase)) {
if (state.phase === 'committed') state = await journal.transition('restarting')
return state
}
try {
if (state.phase !== 'restoring') state = await journal.transition('restoring', {
error: state.error ?? 'transaction interrupted before commit',
})
await snapshots.restore(state.snapshotId)
await selectionStore.restore({
present: state.selectionPresent,
state: { schema: 1, disabled: state.previousDisabled },
})
const failed = await journal.transition('failed', { recoveryResult: 'success' })
await snapshots.remove(state.snapshotId).catch(error => record(
'user-plugin.snapshot.cleanup.failed', { taskId: state.taskId, error },
))
return failed
} catch (error) {
return journal.transition('failed', { error: message(error), recoveryResult: 'failed' })
}
}
export const userPluginTransactionInternals = Object.freeze({ mutateProfile, pluginEnvironment, runDshPlugin, validateActions })
upstream/container/control-plane/modules/skill-manager/index.mjs
+297
View File
@@ -0,0 +1,297 @@
import { createHash, randomUUID } from 'node:crypto'
import { spawn } from 'node:child_process'
import { cp, lstat, mkdir, readFile, readdir, readlink, rename, rm, symlink, writeFile } from 'node:fs/promises'
import { dirname, join, resolve } from 'node:path'
import { canonicalJson } from '../../../platform/lib/canonical-json.mjs'
import { parseEnvironmentManifest } from '../../../platform/lib/contracts.mjs'
import { hashTree } from '../../../platform/lib/tree-hash.mjs'
const ID_PATTERN = /^[a-z0-9]+(?:-[a-z0-9]+)*$/
const STATE_SCHEMA = 1
export const SYSTEM_SKILL_CATALOG_ARTIFACT_ID = 'system-skill-catalog'
export const SYSTEM_SKILL_CATALOG_MEDIA_TYPE = 'application/vnd.dsh-platform.system-skill-catalog.v1+tar+gzip'
function runTar(args, { capture = false } = {}) {
return new Promise((resolveTar, reject) => {
const child = spawn('tar', args, { stdio: ['ignore', capture ? 'pipe' : 'ignore', 'pipe'] })
const stdout = []
const stderr = []
child.stdout?.on('data', chunk => stdout.push(chunk))
child.stderr.on('data', chunk => stderr.push(chunk))
child.once('error', reject)
child.once('exit', code => code === 0
? resolveTar(Buffer.concat(stdout).toString('utf8'))
: reject(new Error(`System Skill archive failed: ${Buffer.concat(stderr).toString('utf8')}`)))
})
}
async function exists(path) {
return lstat(path).then(() => true, error => error?.code === 'ENOENT' ? false : Promise.reject(error))
}
export async function retainLegacySystemSkillCatalog({ sourceRoot, outputRoot }) {
const source = resolve(sourceRoot)
const destination = join(resolve(outputRoot), 'legacy-bootstrap')
if (await exists(join(destination, 'catalog.json'))) {
await readSystemSkillCatalog(destination)
return destination
}
await readSystemSkillCatalog(source)
const staging = `${destination}.${randomUUID()}.tmp`
try {
await cp(source, staging, { recursive: true, verbatimSymlinks: true })
try {
await rename(staging, destination)
} catch (error) {
if (error?.code !== 'EEXIST' && error?.code !== 'ENOTEMPTY') throw error
}
} finally {
await rm(staging, { recursive: true, force: true })
}
await readSystemSkillCatalog(destination)
return destination
}
export async function materializeSystemSkillCatalog({ environmentRoot, outputRoot }) {
const root = resolve(environmentRoot)
const manifest = parseEnvironmentManifest(await readFile(join(root, 'environment.manifest.json')))
const artifact = manifest.artifacts.find(entry => entry.id === SYSTEM_SKILL_CATALOG_ARTIFACT_ID)
if (artifact === undefined) return undefined
if (artifact.mediaType !== SYSTEM_SKILL_CATALOG_MEDIA_TYPE) {
throw new Error('System Skill catalog Artifact media type is invalid')
}
const archive = join(root, 'artifacts', artifact.id)
const bytes = await readFile(archive)
if (bytes.byteLength !== artifact.size || createHash('sha256').update(bytes).digest('hex') !== artifact.sha256) {
throw new Error('System Skill catalog Artifact differs from its Environment manifest')
}
const destination = join(resolve(outputRoot), artifact.sha256)
if (!await exists(destination)) {
const staging = `${destination}.${randomUUID()}.tmp`
await mkdir(staging, { recursive: true })
try {
const entries = (await runTar(['-tzf', archive], { capture: true })).split('\n').filter(Boolean)
if (entries.length === 0 || entries.some(entry => (
entry.startsWith('/') || !entry.startsWith('skills/') || entry.split('/').includes('..')
))) throw new Error('System Skill catalog archive contains an unsafe path')
await runTar(['-xzf', archive, '--no-same-owner', '--no-same-permissions', '--strip-components=1', '-C', staging])
await readSystemSkillCatalog(staging)
try {
await rename(staging, destination)
} catch (error) {
if (error?.code !== 'EEXIST' && error?.code !== 'ENOTEMPTY') throw error
}
} finally {
await rm(staging, { recursive: true, force: true })
}
}
await readSystemSkillCatalog(destination)
return destination
}
function exactKeys(value, keys, label) {
if (value === null || typeof value !== 'object' || Array.isArray(value)
|| Object.keys(value).sort().join(',') !== [...keys].sort().join(',')) {
throw new Error(`${label} is invalid`)
}
return value
}
function skillId(value) {
if (typeof value !== 'string' || !ID_PATTERN.test(value) || value.length > 64) {
throw new Error('System Skill ID is invalid')
}
return value
}
function description(value, label) {
exactKeys(value, ['en', 'zh'], label)
for (const language of ['en', 'zh']) {
if (typeof value[language] !== 'string' || value[language].length === 0 || value[language].length > 240) {
throw new Error(`${label}.${language} is invalid`)
}
}
return Object.freeze({ en: value.en, zh: value.zh })
}
export async function readSystemSkillCatalog(root) {
const sourceRoot = resolve(root)
const document = JSON.parse(await readFile(join(sourceRoot, 'catalog.json'), 'utf8'))
exactKeys(document, ['schema', 'skills'], 'System Skill catalog')
if (document.schema !== 1 || !Array.isArray(document.skills)) throw new Error('System Skill catalog is invalid')
const ids = new Set()
const skills = []
for (const entry of document.skills) {
exactKeys(entry, ['description', 'id', 'source'], 'System Skill catalog entry')
const id = skillId(entry.id)
if (ids.has(id)) throw new Error(`duplicate System Skill ID ${id}`)
ids.add(id)
if (typeof entry.source !== 'string' || entry.source !== id) throw new Error(`System Skill ${id} source is invalid`)
const source = resolve(sourceRoot, entry.source)
if (!source.startsWith(`${sourceRoot}/`)) throw new Error(`System Skill ${id} source escapes its catalog`)
const metadata = await lstat(join(source, 'SKILL.md'))
if (!metadata.isFile() || metadata.isSymbolicLink()) throw new Error(`System Skill ${id} SKILL.md is invalid`)
const bytes = await readFile(join(source, 'SKILL.md'), 'utf8')
if (!bytes.startsWith('---\n') || !bytes.includes(`\nname: ${id}\n`)) {
throw new Error(`System Skill ${id} frontmatter does not match its catalog`)
}
skills.push(Object.freeze({
id,
source,
sha256: await hashTree(source),
description: description(entry.description, `System Skill ${id} description`),
}))
}
return Object.freeze(skills)
}
function defaults(catalog) {
return Object.fromEntries(catalog.map(skill => [skill.id, { installed: true, enabled: true }]))
}
function normalizeState(value, catalog) {
const fallback = defaults(catalog)
if (value === undefined) return fallback
exactKeys(value, ['schema', 'skills'], 'System Skill selection state')
if (value.schema !== STATE_SCHEMA || value.skills === null || typeof value.skills !== 'object' || Array.isArray(value.skills)) {
throw new Error('System Skill selection state is invalid')
}
for (const skill of catalog) {
const saved = value.skills[skill.id]
if (saved === undefined) continue
exactKeys(saved, ['enabled', 'installed'], `System Skill ${skill.id} selection`)
if (typeof saved.installed !== 'boolean' || typeof saved.enabled !== 'boolean' || (saved.enabled && !saved.installed)) {
throw new Error(`System Skill ${skill.id} selection is invalid`)
}
fallback[skill.id] = { installed: saved.installed, enabled: saved.enabled }
}
return fallback
}
export class SystemSkillSelectionStore {
constructor(path) {
this.path = resolve(path)
}
async read(catalog) {
let value
try {
value = JSON.parse(await readFile(this.path, 'utf8'))
} catch (error) {
if (error?.code !== 'ENOENT') throw error
}
return normalizeState(value, catalog)
}
async write(catalog, state) {
const normalized = normalizeState({ schema: STATE_SCHEMA, skills: state }, catalog)
const document = {
schema: STATE_SCHEMA,
skills: Object.fromEntries(catalog.map(skill => [skill.id, normalized[skill.id]])),
}
await mkdir(dirname(this.path), { recursive: true })
const temporary = `${this.path}.${randomUUID()}.tmp`
try {
await writeFile(temporary, `${canonicalJson(document)}\n`, { flag: 'wx', mode: 0o600 })
await rename(temporary, this.path)
} finally {
await rm(temporary, { force: true })
}
return normalized
}
}
async function replaceSkillLink(viewRoot, skill) {
const path = join(viewRoot, skill.id)
const existing = await lstat(path).catch(error => error?.code === 'ENOENT' ? undefined : Promise.reject(error))
if (existing?.isSymbolicLink() && await readlink(path) === skill.source) return
const temporary = join(viewRoot, `.${skill.id}.${randomUUID()}.tmp`)
try {
await symlink(skill.source, temporary, 'dir')
await rename(temporary, path)
} finally {
await rm(temporary, { recursive: true, force: true })
}
}
export async function synchronizeSystemSkillView({ catalog, selection, viewRoot }) {
const root = resolve(viewRoot)
await mkdir(root, { recursive: true })
const expected = new Set(catalog.filter(skill => selection[skill.id].enabled).map(skill => skill.id))
for (const entry of await readdir(root, { withFileTypes: true })) {
if (!expected.has(entry.name)) await rm(join(root, entry.name), { recursive: true, force: true })
}
for (const skill of catalog) {
if (selection[skill.id].enabled) await replaceSkillLink(root, skill)
}
}
export class SystemSkillManager {
constructor({ sourceRoot, viewRoot, statePath }) {
this.sourceRoot = resolve(sourceRoot)
this.viewRoot = resolve(viewRoot)
this.store = new SystemSkillSelectionStore(statePath)
}
async initialize() {
const catalog = await readSystemSkillCatalog(this.sourceRoot)
const selection = await this.store.read(catalog)
await this.store.write(catalog, selection)
await synchronizeSystemSkillView({ catalog, selection, viewRoot: this.viewRoot })
return this.listFrom(catalog, selection)
}
async useSourceRoot(sourceRoot) {
const nextRoot = resolve(sourceRoot)
const catalog = await readSystemSkillCatalog(nextRoot)
const selection = await this.store.read(catalog)
await this.store.write(catalog, selection)
await synchronizeSystemSkillView({ catalog, selection, viewRoot: this.viewRoot })
this.sourceRoot = nextRoot
return this.listFrom(catalog, selection)
}
listFrom(catalog, selection) {
return Object.freeze(catalog.map(skill => Object.freeze({
id: skill.id,
sha256: skill.sha256,
description: skill.description,
installed: selection[skill.id].installed,
enabled: selection[skill.id].enabled,
})))
}
async list() {
const catalog = await readSystemSkillCatalog(this.sourceRoot)
return this.listFrom(catalog, await this.store.read(catalog))
}
async configure(idValue, action) {
const id = skillId(idValue)
if (!['install', 'uninstall', 'enable', 'disable'].includes(action)) throw new Error('System Skill action is invalid')
const catalog = await readSystemSkillCatalog(this.sourceRoot)
if (!catalog.some(skill => skill.id === id)) throw new Error(`System Skill ${id} is not provided by the current Environment`)
const before = await this.store.read(catalog)
const next = structuredClone(before)
const current = next[id]
if (action === 'install') next[id] = { installed: true, enabled: true }
else if (action === 'uninstall') next[id] = { installed: false, enabled: false }
else if (action === 'enable') {
if (!current.installed) throw new Error(`System Skill ${id} must be installed before it can be enabled`)
next[id] = { ...current, enabled: true }
} else next[id] = { ...current, enabled: false }
try {
await this.store.write(catalog, next)
await synchronizeSystemSkillView({ catalog, selection: next, viewRoot: this.viewRoot })
return this.listFrom(catalog, next)
} catch (error) {
try {
await this.store.write(catalog, before)
await synchronizeSystemSkillView({ catalog, selection: before, viewRoot: this.viewRoot })
} catch (rollbackError) {
throw new AggregateError([error, rollbackError], 'System Skill operation and rollback failed')
}
throw error
}
}
}
upstream/container/control-plane/modules/updater/lib/activator.mjs
+229
View File
@@ -0,0 +1,229 @@
import { spawn } from 'node:child_process'
import { cp, lstat, mkdir, readFile, rename, rm, writeFile } from 'node:fs/promises'
import { join } from 'node:path'
import { randomUUID } from 'node:crypto'
import { RuntimeSlots } from '../../patch-manager/index.mjs'
import { PlatformPaths } from '../../../../platform/lib/paths.mjs'
import { LocalApiClient } from './client.mjs'
import { ManagedDeploymentBuilder } from './managed-store.mjs'
import { deriveRecordId, parseDeploymentRecord } from '../../../../platform/lib/deployment-contracts.mjs'
function run(command, args) {
return new Promise((resolve, reject) => {
const child = spawn(command, args, { stdio: ['ignore', 'pipe', 'pipe'] })
const stdout = []
const stderr = []
child.stdout.on('data', chunk => stdout.push(chunk))
child.stderr.on('data', chunk => stderr.push(chunk))
child.once('error', reject)
child.once('exit', code => code === 0
? resolve(Buffer.concat(stdout))
: reject(new Error(Buffer.concat(stderr).toString('utf8'))))
})
}
async function exists(path) {
return lstat(path).then(() => true, error => error?.code === 'ENOENT' ? false : Promise.reject(error))
}
function matchesRecoveryJournal(record, deployment) {
if (record?.id === deployment.runtime) return true
if (record === null || record === undefined) return false
if (!Array.isArray(record.receiptTokens) || !Array.isArray(deployment.receiptTokens)) return false
const sameReceipts = [...record.receiptTokens].sort().join('\0')
=== [...deployment.receiptTokens].sort().join('\0')
return record.dshVersion === deployment.dsh
&& record.environmentVersion === deployment.environment
&& record.snapshotId === deployment.dataSnapshot
&& sameReceipts
}
export class PlatformActivator {
constructor({
dataRoot,
runRoot = process.env.DSH_PLATFORM_RUN ?? '/run/dsh-platform',
bootstrap,
stage0,
builder,
sleep = milliseconds => new Promise(resolve => setTimeout(resolve, milliseconds)),
}) {
this.dataRoot = dataRoot
this.paths = new PlatformPaths(dataRoot, runRoot)
this.bootstrap = bootstrap ?? new LocalApiClient(this.paths.bootstrapSocket)
this.stage0 = stage0 ?? new LocalApiClient(this.paths.trustSocket)
this.builder = builder ?? new ManagedDeploymentBuilder({ paths: this.paths })
this.sleep = sleep
this.runtimeSlots = new RuntimeSlots(this.paths.runtimesRoot)
this.environmentSlots = new RuntimeSlots(this.paths.environmentsRoot)
this.systemPluginSlots = new RuntimeSlots(this.paths.systemPluginsRoot)
this.experimentalCandidates = new Map()
}
prepareManaged(prepared, options) {
return this.builder.buildStable(prepared, options)
}
async pristine(version, receipt) {
const root = join(this.paths.pristineRoot, version)
if (!await exists(root)) {
const staging = `${root}.${randomUUID()}.tmp`
await mkdir(staging, { recursive: true })
try {
const entries = (await run('tar', ['-tzf', receipt.path])).toString('utf8').split('\n').filter(Boolean)
if (!entries.includes('package/package.json')) throw new Error('official DSH has no package metadata')
if (entries.some(entry => entry.startsWith('/') || !entry.startsWith('package/') || entry.split('/').includes('..'))) {
throw new Error('official DSH archive contains an unsafe path')
}
await run('tar', ['-xzf', receipt.path, '-C', staging])
await rename(staging, root)
} catch (error) {
await rm(staging, { recursive: true, force: true })
throw error
}
}
const packageMetadata = JSON.parse(await readFile(join(root, 'package', 'package.json'), 'utf8'))
if (packageMetadata.name !== '@deepseek-ai/dsh' || packageMetadata.version !== version) {
throw new Error('official DSH package metadata differs from its requested version')
}
return join(root, 'package')
}
async stageEnvironment(prepared) {
const version = prepared.environment.manifest.version
const versions = join(this.paths.environmentsRoot, 'versions')
const destination = join(versions, version)
if (await exists(destination)) return version
const staging = join(versions, `.${version}.${randomUUID()}.tmp`)
await mkdir(join(staging, 'artifacts'), { recursive: true })
try {
await cp(prepared.paths.get(prepared.stable.desired.environment.manifestArtifactId), join(staging, 'environment.manifest.json'))
for (const descriptor of prepared.environment.manifest.artifacts) {
await cp(prepared.paths.get(descriptor.id), join(staging, 'artifacts', descriptor.id))
}
await rename(staging, destination)
return version
} catch (error) {
await rm(staging, { recursive: true, force: true })
throw error
}
}
async activate(prepared, { onProgress = async () => {}, onSwitching = async () => {} } = {}) {
const desiredBootstrap = prepared.stable.desired.bootstrap.version
const artifacts = prepared.bootstrap.manifest.artifacts
if (artifacts.length !== 1) throw new Error('Bootstrap manifest must contain exactly one package')
const receipt = prepared.receipts.get(artifacts[0].id)
if (receipt === undefined) throw new Error('Bootstrap package receipt is required')
const staged = await this.stage0.stageBootstrap(receipt.token, desiredBootstrap)
// Older Stage-0 responses have only status=switching. Preserve that handoff
// contract while allowing content-identical targets to continue in place.
if (staged.restartRequired === true || staged.status === 'switching') await new Promise(() => {})
const managed = await this.prepareManaged(prepared, { onProgress })
await onSwitching()
await this.bootstrap.request('POST', '/v1/deployments/activate', { record: managed.record })
return managed.record
}
async rollback(recordId) {
return this.bootstrap.request('POST', '/v1/deployments/rollback', recordId === undefined ? {} : { recordId })
}
rollbackDeployments() {
return this.bootstrap.request('GET', '/v1/deployments/rollback-plan')
}
async currentDeployment({ required = true } = {}) {
let record
for (let attempt = 0; attempt < 100; attempt += 1) {
({ record } = await this.bootstrap.request('GET', '/v1/deployments/current'))
if (record !== null || !required) break
const status = await this.bootstrap.request('GET', '/v1/status')
if (status.startupComplete === true) break
await this.sleep(100)
}
if (record === null) {
if (required) throw new Error('current Deployment is required')
return null
}
return Object.freeze({
authority: record.authority,
targetSequence: record.targetSequence,
dsh: record.dshVersion,
runtime: record.id,
environment: record.environmentVersion,
dataSnapshot: record.snapshotId,
receiptTokens: Object.freeze([...record.receiptTokens]),
})
}
async prepareExperimental(prepared, { targetSequence }) {
const { record: current } = await this.bootstrap.request('GET', '/v1/deployments/current')
if (current === null) throw new Error('current Deployment is required')
const receiptTokens = await this.experimentalActivationTokens(prepared.receiptTokens)
const built = await this.builder.buildExperimental(prepared, current, receiptTokens, { targetSequence })
this.experimentalCandidates.set(built.record.id, built.record)
return Object.freeze({
runtimeId: built.record.id,
environmentVersion: built.record.environmentVersion,
dshVersion: built.record.dshVersion,
})
}
suspendDsh() { return this.bootstrap.request('POST', '/v1/components/dsh-runtime/suspend') }
resumeDsh() { return this.bootstrap.request('POST', '/v1/components/dsh-runtime/resume') }
health() { return this.bootstrap.request('GET', '/v1/health') }
async experimentalActivationTokens(experimentalTokens) {
const active = (await this.stage0.activeReceipts()).receipts
return Object.freeze([
...active.filter(receipt => receipt.authorityType === 'stable').map(receipt => receipt.token),
...experimentalTokens,
])
}
async switchExperimental(runtimeId) {
const record = this.experimentalCandidates.get(runtimeId)
if (record === undefined) throw new Error('Experimental Deployment candidate is not prepared')
return this.bootstrap.request('POST', '/v1/deployments/candidate', { record })
}
async bindExperimentalSnapshot(runtimeId, snapshotId) {
const record = this.experimentalCandidates.get(runtimeId)
if (record === undefined) throw new Error('Experimental Deployment candidate is not prepared')
const content = { ...record, snapshotId }
delete content.id
const bound = parseDeploymentRecord({ ...content, id: deriveRecordId('deployment-record', content) })
this.experimentalCandidates.delete(runtimeId)
this.experimentalCandidates.set(bound.id, bound)
return bound.id
}
async commitExperimental(runtimeId) {
const result = await this.bootstrap.request('POST', '/v1/deployments/candidate/commit', { recordId: runtimeId })
this.experimentalCandidates.delete(runtimeId)
return result
}
async restoreDeployment(deployment, { resume = true } = {}) {
await this.bootstrap.request('POST', '/v1/deployments/candidate/cancel')
const { record } = await this.bootstrap.request('GET', '/v1/deployments/current')
if (!matchesRecoveryJournal(record, deployment)) {
const { previous } = await this.rollbackDeployments()
if (previous === null) throw new Error('previous Deployment is unavailable')
if (!matchesRecoveryJournal(previous, deployment)) {
throw new Error('previous Deployment differs from the recovery journal')
}
await this.bootstrap.request('POST', '/v1/deployments/rollback', { recordId: previous.id })
}
if (resume) await this.resumeDsh()
}
async cleanup() {
const platform = await this.bootstrap.request('POST', '/v1/platform/gc')
return Object.freeze({
platform,
bootstrap: await this.stage0.collectBootstrap(),
trustedObjects: (await this.stage0.collectGarbage()).removed,
})
}
}
upstream/container/control-plane/modules/updater/lib/automatic-check.mjs
+170
View File
@@ -0,0 +1,170 @@
import { readFile } from 'node:fs/promises'
import { durableReplace } from '../../../../platform/lib/atomic.mjs'
import { compareDshVersions } from '../../../../platform/lib/supported-target.mjs'
export const AUTOMATIC_CHECK_INTERVALS = Object.freeze([3_600, 10_800, 21_600, 43_200, 86_400])
const DEFAULT_STATE = Object.freeze({
schema: 1,
automaticCheck: Object.freeze({
enabled: true,
intervalSeconds: 21_600,
notificationsEnabled: true,
}),
latestAutomatic: Object.freeze({ stable: null, upstream: null }),
})
function exactKeys(value, expected, label) {
if (value === null || typeof value !== 'object' || Array.isArray(value)) throw new Error(`${label} must be an object`)
const keys = Object.keys(value).sort()
const wanted = [...expected].sort()
if (keys.length !== wanted.length || keys.some((key, index) => key !== wanted[index])) {
throw new Error(`${label} fields are invalid`)
}
}
function parseConfig(value) {
exactKeys(value, ['enabled', 'intervalSeconds', 'notificationsEnabled'], 'automatic check config')
if (typeof value.enabled !== 'boolean') throw new Error('automatic check enabled must be boolean')
if (!AUTOMATIC_CHECK_INTERVALS.includes(value.intervalSeconds)) throw new Error('automatic check interval is invalid')
if (typeof value.notificationsEnabled !== 'boolean') throw new Error('update notifications enabled must be boolean')
return Object.freeze({ ...value })
}
function parseStable(value) {
if (value === null) return null
exactKeys(value, ['checkedAt', 'dsh', 'environment', 'targetSequence'], 'Stable notification')
if (!Number.isSafeInteger(value.targetSequence) || value.targetSequence < 1) throw new Error('Stable notification sequence is invalid')
compareDshVersions(value.dsh, value.dsh)
if (typeof value.environment !== 'string' || value.environment.length === 0) throw new Error('Stable notification Environment is invalid')
if (Number.isNaN(Date.parse(value.checkedAt))) throw new Error('Stable notification timestamp is invalid')
return Object.freeze({ ...value })
}
function parseUpstream(value) {
if (value === null) return null
exactKeys(value, ['checkedAt', 'version'], 'Upstream notification')
compareDshVersions(value.version, value.version)
if (Number.isNaN(Date.parse(value.checkedAt))) throw new Error('Upstream notification timestamp is invalid')
return Object.freeze({ ...value })
}
function parseState(value) {
exactKeys(value, ['automaticCheck', 'latestAutomatic', 'schema'], 'automatic check state')
if (value.schema !== 1) throw new Error('automatic check state schema must be 1')
exactKeys(value.latestAutomatic, ['stable', 'upstream'], 'automatic notification state')
return Object.freeze({
schema: 1,
automaticCheck: parseConfig(value.automaticCheck),
latestAutomatic: Object.freeze({
stable: parseStable(value.latestAutomatic.stable),
upstream: parseUpstream(value.latestAutomatic.upstream),
}),
})
}
function isHeld(version, environment, holds) {
return holds.some(hold => hold.dshVersion === version
&& (hold.type !== 'combination' || hold.environmentVersion === environment))
}
function stableSatisfied(candidate, current) {
return current.environment === candidate.environment && compareDshVersions(current.dsh, candidate.dsh) >= 0
}
export class AutomaticCheckStateStore {
constructor(path, now = () => new Date()) {
this.path = path
this.now = now
this.pending = Promise.resolve()
}
async read() {
try {
return parseState(JSON.parse(await readFile(this.path, 'utf8')))
} catch (error) {
if (error?.code === 'ENOENT') return DEFAULT_STATE
throw error
}
}
async replace(update) {
const operation = this.pending.then(async () => {
const next = parseState(await update(await this.read()))
await durableReplace(this.path, `${JSON.stringify(next)}\n`)
return next
})
this.pending = operation.catch(() => {})
return operation
}
configure(value) {
const config = parseConfig(value)
return this.replace(previous => ({
...previous,
automaticCheck: config,
latestAutomatic: !config.enabled || !config.notificationsEnabled
? { stable: null, upstream: null }
: previous.latestAutomatic,
}))
}
record({ channel, current, target, upstream, stableAvailable, holds = [] }) {
return this.replace(previous => {
if (!previous.automaticCheck.enabled || !previous.automaticCheck.notificationsEnabled) return previous
const checkedAt = this.now().toISOString()
let stable = previous.latestAutomatic.stable
let latestUpstream = previous.latestAutomatic.upstream
if (stable !== null && stableSatisfied(stable, current)) stable = null
if (
stableAvailable === true
&& (stable === null || target.targetSequence > stable.targetSequence)
) {
stable = {
targetSequence: target.targetSequence,
dsh: target.desired.dsh.version,
environment: target.desired.environment.version,
checkedAt,
}
}
if (channel !== 'experimental') {
latestUpstream = null
} else {
if (latestUpstream !== null && compareDshVersions(current.dsh, latestUpstream.version) >= 0) latestUpstream = null
if (
upstream !== null
&& compareDshVersions(upstream.version, current.dsh) > 0
&& !isHeld(upstream.version, target.desired.environment.version, holds)
&& (latestUpstream === null || compareDshVersions(upstream.version, latestUpstream.version) > 0)
) latestUpstream = { version: upstream.version, checkedAt }
}
return { ...previous, latestAutomatic: { stable, upstream: latestUpstream } }
})
}
clearSatisfied(current) {
return this.replace(previous => ({
...previous,
latestAutomatic: {
stable: previous.latestAutomatic.stable !== null
&& stableSatisfied(previous.latestAutomatic.stable, current)
? null
: previous.latestAutomatic.stable,
upstream: previous.latestAutomatic.upstream !== null
&& compareDshVersions(current.dsh, previous.latestAutomatic.upstream.version) >= 0
? null
: previous.latestAutomatic.upstream,
},
}))
}
clearUpstream() {
return this.replace(previous => ({
...previous,
latestAutomatic: { ...previous.latestAutomatic, upstream: null },
}))
}
}
upstream/container/control-plane/modules/updater/lib/channel-state.mjs
+169
View File
@@ -0,0 +1,169 @@
import { createHash, randomUUID } from 'node:crypto'
import { readFile } from 'node:fs/promises'
import { durableReplace } from '../../../../platform/lib/atomic.mjs'
import { compareDshVersions } from '../../../../platform/lib/supported-target.mjs'
import { isoTimestamp, plainObject, TrustError } from '../../../../platform/lib/validation.mjs'
const DEFAULT_STATE = Object.freeze({
schema: 1,
updateChannel: 'stable',
holds: Object.freeze([]),
experimentalBlocked: null,
})
function text(value, label) {
if (typeof value !== 'string' || value === '' || value.length > 512 || value.includes('\0')) {
throw new TrustError(`${label} is invalid`)
}
return value
}
function parseHold(value) {
const hold = plainObject(value, 'update hold')
const keys = Object.keys(hold).sort()
const expected = ['createdAt', 'dshVersion', 'environmentVersion', 'id', 'reason', 'type']
if (JSON.stringify(keys) !== JSON.stringify(expected)) throw new TrustError('update hold has unexpected fields')
if (!['version', 'combination'].includes(hold.type)) throw new TrustError('update hold type is invalid')
if (hold.type === 'version' && hold.environmentVersion !== null) throw new TrustError('version hold cannot name an Environment')
if (hold.type === 'combination' && typeof hold.environmentVersion !== 'string') {
throw new TrustError('combination hold requires an Environment')
}
isoTimestamp(hold.createdAt, 'update hold creation time')
return Object.freeze({
id: text(hold.id, 'update hold ID'),
type: hold.type,
dshVersion: text(hold.dshVersion, 'update hold DSH version'),
environmentVersion: hold.environmentVersion === null ? null : text(hold.environmentVersion, 'update hold Environment'),
reason: text(hold.reason, 'update hold reason'),
createdAt: hold.createdAt,
})
}
function parseState(value) {
const state = plainObject(value, 'channel state')
const keys = Object.keys(state).sort()
if (JSON.stringify(keys) !== JSON.stringify(['experimentalBlocked', 'holds', 'schema', 'updateChannel'])) {
throw new TrustError('channel state has unexpected fields')
}
if (state.schema !== 1) throw new TrustError('channel state schema must be 1')
if (!['stable', 'experimental'].includes(state.updateChannel)) throw new TrustError('update channel is invalid')
if (!Array.isArray(state.holds)) throw new TrustError('channel holds must be an array')
const holds = state.holds.map(parseHold)
if (new Set(holds.map(hold => hold.id)).size !== holds.length) throw new TrustError('update hold IDs must be unique')
const blocked = state.experimentalBlocked === null ? null : parseHold(state.experimentalBlocked)
if (blocked !== null && blocked.type !== 'combination') throw new TrustError('Experimental block must identify a combination')
return Object.freeze({ schema: 1, updateChannel: state.updateChannel, holds: Object.freeze(holds), experimentalBlocked: blocked })
}
export class ChannelStateStore {
constructor(path, now = () => new Date()) {
this.path = path
this.now = now
this.queue = Promise.resolve()
}
exclusive(operation) {
const result = this.queue.then(operation, operation)
this.queue = result.then(() => undefined, () => undefined)
return result
}
async read() {
try { return parseState(JSON.parse(await readFile(this.path, 'utf8'))) } catch (error) {
if (error?.code === 'ENOENT') return DEFAULT_STATE
throw error
}
}
write(value) { return durableReplace(this.path, `${JSON.stringify(parseState(value))}\n`) }
setChannel(updateChannel) {
return this.exclusive(async () => {
const current = await this.read()
const next = parseState({ ...current, updateChannel })
await this.write(next)
return next
})
}
addHold({ type, dshVersion, environmentVersion = null, reason }) {
return this.exclusive(async () => {
const current = await this.read()
const identity = `${type}\0${dshVersion}\0${environmentVersion ?? ''}`
const id = createHash('sha256').update(identity).digest('hex').slice(0, 24)
const hold = parseHold({
id, type, dshVersion, environmentVersion, reason,
createdAt: this.now().toISOString(),
})
const holds = [...current.holds.filter(item => item.id !== id), hold]
const next = parseState({ ...current, holds })
await this.write(next)
return hold
})
}
retry(id) {
return this.exclusive(async () => {
const current = await this.read()
const holds = current.holds.filter(hold => hold.id !== id)
const clearsBlock = current.experimentalBlocked?.id === id
if (holds.length === current.holds.length && !clearsBlock) throw new TrustError('update hold does not exist')
const next = parseState({ ...current, holds, experimentalBlocked: clearsBlock ? null : current.experimentalBlocked })
await this.write(next)
return next
})
}
block({ dshVersion, environmentVersion, reason }) {
return this.exclusive(async () => {
const current = await this.read()
const blocked = parseHold({
id: `blocked-${randomUUID()}`, type: 'combination', dshVersion, environmentVersion, reason,
createdAt: this.now().toISOString(),
})
const next = parseState({ ...current, experimentalBlocked: blocked })
await this.write(next)
return next
})
}
}
function held(local, dshVersion, environmentVersion) {
return local.holds.find(hold => (
hold.dshVersion === dshVersion
&& (hold.type === 'version' || hold.environmentVersion === environmentVersion)
))
}
export function planDesiredState({ local, current, supported, stableTargetSequence, upstream = null }) {
const aheadOfStable = compareDshVersions(current.dsh, supported.dsh) > 0
const stableIdentityMismatch = Number.isSafeInteger(stableTargetSequence) && (
current.targetSequence !== stableTargetSequence
|| (local.updateChannel === 'stable'
? current.authority !== 'stable'
: !['stable', 'experimental'].includes(current.authority))
)
const base = {
updateChannel: local.updateChannel,
current,
supported,
upstream,
aheadOfStable,
experimentalBlocked: local.experimentalBlocked,
holds: local.holds,
}
if (aheadOfStable) return Object.freeze({ ...base, action: 'frozen', reason: 'Stable has not caught up with the current DSH' })
if (compareDshVersions(current.dsh, supported.dsh) < 0 || current.environment !== supported.environment || stableIdentityMismatch) {
return Object.freeze({ ...base, action: 'stable', reason: 'converge the signed Stable target before Experimental DSH' })
}
if (local.updateChannel === 'stable' || upstream === null) {
return Object.freeze({ ...base, action: 'none', reason: null })
}
const matchingHold = held(local, upstream.version, supported.environment)
if (matchingHold !== undefined) return Object.freeze({ ...base, action: 'held', reason: matchingHold.reason, hold: matchingHold })
if (
local.experimentalBlocked?.dshVersion === upstream.version
&& local.experimentalBlocked.environmentVersion === supported.environment
) return Object.freeze({ ...base, action: 'blocked', reason: local.experimentalBlocked.reason })
return Object.freeze({ ...base, action: 'experimental', reason: null })
}
upstream/container/control-plane/modules/updater/lib/client.mjs
+66
View File
@@ -0,0 +1,66 @@
import { request } from 'node:http'
export class LocalApiClient {
constructor(socketPath) {
this.socketPath = socketPath
}
request(method, path, body, extraHeaders = {}) {
return new Promise((resolve, reject) => {
const bytes = body === undefined ? undefined : Buffer.from(JSON.stringify(body))
const req = request({
socketPath: this.socketPath,
method,
path,
headers: {
...extraHeaders,
...(bytes === undefined ? {} : {
'content-type': 'application/json',
'content-length': bytes.byteLength,
}),
},
}, response => {
const chunks = []
response.on('data', chunk => chunks.push(chunk))
response.on('end', () => {
let value
try { value = JSON.parse(Buffer.concat(chunks).toString('utf8')) } catch { value = {} }
if ((response.statusCode ?? 500) >= 400) {
const error = new Error(value.error ?? `local API returned ${String(response.statusCode)}`)
error.statusCode = response.statusCode
if (typeof value.code === 'string') error.code = value.code
if (Number.isInteger(value.retryAfterSeconds)) error.retryAfterSeconds = value.retryAfterSeconds
error.localApiPath = path
reject(error)
} else resolve(value)
})
})
req.once('error', reject)
req.end(bytes)
})
}
status() { return this.request('GET', '/v1/status') }
activeReceipts() { return this.request('GET', '/v1/receipts/active') }
acceptKeyring(document, signature) {
return this.request('POST', '/v1/keyring', { document: document.toString('base64'), signature })
}
acceptTarget(document, signature) {
return this.request('POST', '/v1/target', { document: document.toString('base64'), signature })
}
importArtifact(artifactId, sourcePath, parentReceipt = null) {
return this.request('POST', '/v1/artifacts/import', { artifactId, sourcePath, parentReceipt })
}
ensureOfficialDsh(version, metadataPath, tarballPath) {
return this.request('POST', '/v1/dsh/ensure', { version, metadataPath, tarballPath })
}
acceptManifest(receipt, signatureReceipt) {
return this.request('POST', '/v1/manifests/accept', { receipt, signatureReceipt })
}
activate(receipts) { return this.request('POST', '/v1/activate', { receipts }) }
collectGarbage() { return this.request('POST', '/v1/objects/collect', {}) }
stageBootstrap(receipt, version) {
return this.request('POST', '/v1/bootstrap/stage', { receipt, version })
}
collectBootstrap() { return this.request('POST', '/v1/bootstrap/collect', {}) }
}
upstream/container/control-plane/modules/updater/lib/coordinator.mjs
+894
View File
@@ -0,0 +1,894 @@
import { randomUUID } from 'node:crypto'
import { EventEmitter } from 'node:events'
import { recoverInterruptedUpdate } from './recovery.mjs'
import { planDesiredState } from './channel-state.mjs'
import { compareDshVersions } from '../../../../platform/lib/supported-target.mjs'
export class UpdateConflictError extends Error {}
function healthMetrics(health) {
const components = Array.isArray(health?.components) ? health.components : []
return {
readyServices: components.filter(component => component?.healthy === true).length,
totalServices: components.length,
}
}
function journalDeployment(deployment) {
return Object.freeze({
dsh: deployment.dsh,
environment: deployment.environment,
runtime: deployment.runtime,
dataSnapshot: deployment.dataSnapshot,
receiptTokens: deployment.receiptTokens,
})
}
function metricPercentage(processed, total) {
return typeof processed === 'number' && typeof total === 'number' && total > 0
? Math.floor(Math.max(0, Math.min(1, processed / total)) * 100)
: null
}
function holdReason(message) {
const lines = message.split(/\r?\n/u).map(line => line.trim()).filter(Boolean)
const npmNotarget = lines.find(line => /^npm error notarget\b/u.test(line))
const selected = npmNotarget?.replace(/^npm error notarget\s*/u, '') ?? lines[0] ?? 'Experimental update failed'
const normalized = selected.replace(/[\u0000-\u001f\u007f]/gu, ' ').replace(/\s+/gu, ' ').trim()
return (normalized || 'Experimental update failed').slice(0, 512)
}
export class UpdateCoordinator extends EventEmitter {
constructor({
metadata, preparer, activator, state, npm, journal, snapshots, channelState, completeRecovery, automaticChecks,
allowUnavailableMetadata = false,
probationSeconds = 120,
report = async () => {},
now = () => new Date(),
sleep = milliseconds => new Promise(resolve => setTimeout(resolve, milliseconds)),
}) {
super()
this.metadata = metadata
this.preparer = preparer
this.activator = activator
this.state = state
this.npm = npm
this.journal = journal
this.snapshots = snapshots
this.channelState = channelState
this.completeRecovery = completeRecovery
this.automaticChecks = automaticChecks
this.allowUnavailableMetadata = allowUnavailableMetadata
this.probationSeconds = probationSeconds
this.report = report
this.now = now
this.sleep = sleep
this.task = undefined
this.checkTask = undefined
this.reportTimes = new Map()
}
async transition(status, fields = {}) {
const previous = await this.state.read()
const phase = fields.phase ?? (
status === 'idle'
? null
: fields.rollbackPhase ?? (
['success', 'failed'].includes(status)
? previous.phase ?? previous.rollbackPhase ?? (['idle', 'success', 'failed'].includes(previous.status) ? null : previous.status)
: status
)
)
const contextChanged = phase !== (previous.phase ?? null)
|| (fields.operation !== undefined && fields.operation !== previous.operation)
const phaseMetrics = contextChanged ? {
processedBytes: null,
totalBytes: null,
processedItems: null,
totalItems: null,
readyServices: null,
totalServices: null,
detail: null,
} : {}
const value = await this.state.write(status, { ...phaseMetrics, ...fields, phase })
this.emit('state', value)
await this.record('update.phase.changed', {
status,
phase,
operation: value.operation ?? null,
rollbackPhase: value.rollbackPhase ?? null,
taskId: value.taskId ?? null,
progress: value.progress ?? null,
processedBytes: value.processedBytes ?? null,
totalBytes: value.totalBytes ?? null,
processedItems: value.processedItems ?? null,
totalItems: value.totalItems ?? null,
readyServices: value.readyServices ?? null,
totalServices: value.totalServices ?? null,
detail: value.detail ?? null,
targetSequence: value.targetSequence ?? value.available?.targetSequence ?? null,
outcome: value.outcome ?? null,
...(value.error === null || value.error === undefined ? {} : { error: value.error, level: 'error' }),
})
return value
}
async record(message, fields = {}) {
try {
const current = await this.state.read()
const context = {
...(current.taskId === null || current.taskId === undefined ? {} : { taskId: current.taskId }),
...(current.operation === null || current.operation === undefined ? {} : { operation: current.operation }),
...(current.phase === null || current.phase === undefined
? (current.status === null || current.status === undefined ? {} : { phase: current.status })
: { phase: current.phase }),
}
return await this.report(message, { ...context, ...fields })
} catch {}
}
async bestEffort(message, operation, fallback, fields = {}, cooldownMs = 0) {
try {
return await operation()
} catch (error) {
const timestamp = this.now().valueOf()
const previous = this.reportTimes.get(message)
if (cooldownMs === 0 || previous === undefined || timestamp - previous >= cooldownMs) {
this.reportTimes.set(message, timestamp)
await this.record(message, { ...fields, error, level: 'warning' })
}
return fallback
}
}
async reportHealth(status, fields) {
if (this.activator.health === undefined) return undefined
const health = await this.activator.health()
await this.transition(status, { ...fields, ...healthMetrics(health) })
return health
}
hasActiveTask() {
return this.task !== undefined
}
check(source = 'manual') {
if (this.task !== undefined && source === 'page-open') return Promise.resolve({ busy: true })
if (this.task !== undefined) return Promise.reject(new UpdateConflictError('an update task is already running'))
if (this.checkTask !== undefined) return this.checkTask
this.checkTask = this.runCheck(source).finally(() => { this.checkTask = undefined })
return this.checkTask
}
startTask(operation) {
if (this.task !== undefined) throw new UpdateConflictError('an update task is already running')
const taskId = randomUUID()
const precedingCheck = this.checkTask
this.task = (async () => {
if (precedingCheck !== undefined) {
try { await precedingCheck } catch {}
}
return operation(taskId)
})().finally(() => { this.task = undefined })
return { taskId, completion: this.task }
}
async runCheck(source) {
await this.transition('checking', {
checkSource: source, taskId: null, progress: 0, operation: 'check', rollbackPhase: null, error: null,
})
try {
if (this.allowUnavailableMetadata) return await this.recordUnavailableMetadata(source)
if (this.channelState !== undefined) {
const plan = await this.desiredState()
const rollbackPlan = await this.rollbackPlan()
const updateAvailable = ['stable', 'experimental'].includes(plan.action)
await this.transition('idle', {
available: {
targetSequence: plan.target.targetSequence,
bootstrap: plan.target.desired.bootstrap.version,
environment: plan.target.desired.environment.version,
dsh: plan.target.desired.dsh.version,
},
supported: plan.supported,
...(plan.upstream === null ? {} : { upstream: { version: plan.upstream.version } }),
current: { dsh: plan.current.dsh, environment: plan.current.environment, runtime: plan.current.runtime },
aheadOfStable: plan.aheadOfStable,
experimentalBlocked: plan.experimentalBlocked,
holds: plan.holds,
rollbackPlan,
updateAvailable,
checkedAt: this.now().toISOString(),
metadataUnavailable: false,
remoteCheckError: null,
remoteCheckFailedAt: null,
remoteCheckSource: null,
})
if (source === 'automatic' && this.automaticChecks !== undefined) {
await this.automaticChecks.record({
channel: plan.updateChannel,
current: plan.current,
target: plan.target,
upstream: plan.upstream,
stableAvailable: plan.action === 'stable',
holds: plan.holds,
})
this.emit('state', await this.state.read())
}
return { value: plan.target, updateAvailable }
}
const target = await this.metadata.check()
await this.transition('idle', {
available: {
targetSequence: target.value.targetSequence,
bootstrap: target.value.desired.bootstrap.version,
environment: target.value.desired.environment.version,
dsh: target.value.desired.dsh.version,
},
checkedAt: new Date().toISOString(),
metadataUnavailable: false,
remoteCheckError: null,
remoteCheckFailedAt: null,
remoteCheckSource: null,
})
return target
} catch (error) {
await this.record('update.check.failed', { error, checkSource: source })
await this.transition('idle', {
error: null,
remoteCheckError: error instanceof Error ? error.message : 'update check failed',
remoteCheckFailedAt: this.now().toISOString(),
remoteCheckSource: error?.remoteCheckSource === 'upstream' ? 'upstream' : 'stable',
})
throw error
}
}
async recordUnavailableMetadata(source) {
const local = await this.channelState?.read()
const upstream = local?.updateChannel === 'experimental' && this.npm !== undefined
? await this.bestEffort('update.upstream-discovery.failed', () => this.npm.discover(), null, { checkSource: source })
: null
await this.transition('idle', {
metadataUnavailable: true,
...(upstream === null ? {} : { upstream }),
updateAvailable: false,
checkedAt: this.now().toISOString(),
error: null,
})
return { unavailable: true, upstream }
}
start() {
return this.startTask(taskId => this.run(taskId))
}
startExperimental() {
if (this.npm === undefined || this.journal === undefined || this.snapshots === undefined) {
throw new Error('Experimental updates are not configured')
}
return this.startTask(taskId => this.runExperimental(taskId))
}
startReconcile() {
return this.startTask(taskId => this.runReconcile(taskId))
}
async setChannel(updateChannel) {
if (this.task !== undefined) throw new UpdateConflictError('an update task is already running')
if (this.channelState === undefined) throw new Error('update channels are not configured')
const result = await this.channelState.setChannel(updateChannel)
if (updateChannel === 'stable') await this.automaticChecks?.clearUpstream()
await this.record('update.channel.changed', { updateChannel })
return result
}
async retryHold(id) {
if (this.task !== undefined) throw new UpdateConflictError('an update task is already running')
if (this.channelState === undefined) throw new Error('update channels are not configured')
const result = await this.channelState.retry(id)
await this.record('update.hold.retried', { holdId: id })
return result
}
rollbackPlan() {
if (this.completeRecovery === undefined) return Promise.resolve(null)
return this.completeRecovery.plan()
}
startCompleteRollback(planId, options = {}) {
if (this.completeRecovery === undefined) throw new Error('complete rollback is not configured')
return this.startTask(taskId => this.runCompleteRollback(taskId, planId, options))
}
async runCompleteRollback(taskId, planId, options) {
const operation = options.requireConfirmation ? 'return-stable' : 'rollback'
try {
const rollbackPlan = await this.completeRecovery.plan()
let verifiedStable = null
await this.transition('restoring-data', {
taskId,
progress: 5,
operation,
rollbackPhase: 'preparing',
rollbackIncludesSnapshot: rollbackPlan?.snapshot !== null && rollbackPlan?.snapshot !== undefined,
error: null,
})
if (options.requireConfirmation) {
const [plan, target] = await Promise.all([this.completeRecovery.plan(), this.metadata.check()])
if (
plan === null
|| plan.planId !== planId
|| compareDshVersions(plan.previous.dsh, target.value.desired.dsh.version) > 0
) throw new Error('no verified pre-Experimental Stable recovery point is available')
verifiedStable = target.value
}
const result = await this.completeRecovery.restore(planId, {
...options,
onProgress: (() => {
let previousKey
return (rollbackPhase, progress, metrics = {}) => {
const key = [
rollbackPhase,
progress,
metricPercentage(metrics.processedBytes, metrics.totalBytes),
metricPercentage(metrics.processedItems, metrics.totalItems),
].join(':')
if (key === previousKey) return undefined
previousKey = key
return this.transition('restoring-data', {
taskId,
progress,
operation,
rollbackPhase,
processedBytes: metrics.processedBytes ?? null,
totalBytes: metrics.totalBytes ?? null,
processedItems: metrics.processedItems ?? null,
totalItems: metrics.totalItems ?? null,
})
}
})(),
})
await this.reportHealth('restoring-data', {
taskId,
progress: 95,
operation,
rollbackPhase: 'verifying',
})
const current = await this.activator.currentDeployment()
if (
rollbackPlan === null
|| current.dsh !== rollbackPlan.previous.dsh
|| current.environment !== rollbackPlan.previous.environment
|| current.runtime !== rollbackPlan.previous.runtime
) throw new Error('restored Deployment differs from the rollback plan')
const [stored, local] = await Promise.all([
this.state.read(),
this.channelState?.read() ?? Promise.resolve({ updateChannel: 'stable', holds: [], experimentalBlocked: null }),
])
const supported = verifiedStable === null
? stored.supported ?? null
: {
dsh: verifiedStable.desired.dsh.version,
environment: verifiedStable.desired.environment.version,
}
let aheadOfStable = false
let updateAvailable = false
if (supported !== null) {
const upstream = local.updateChannel === 'experimental'
&& typeof stored.upstream?.version === 'string'
&& compareDshVersions(stored.upstream.version, supported.dsh) > 0
? stored.upstream
: null
const plan = planDesiredState({
local,
current,
supported,
stableTargetSequence: verifiedStable?.targetSequence ?? stored.available?.targetSequence,
upstream,
})
aheadOfStable = plan.aheadOfStable
updateAvailable = ['stable', 'experimental'].includes(plan.action)
}
await this.bestEffort('update.notifications.cleanup.failed', () => this.clearSatisfiedNotifications(), undefined, { taskId })
await this.transition('success', {
taskId,
progress: 100,
current: { dsh: current.dsh, environment: current.environment, runtime: current.runtime },
aheadOfStable,
updateAvailable,
experimentalBlocked: local.experimentalBlocked,
holds: local.holds,
outcome: null,
probationUntil: null,
error: null,
})
return result
} catch (error) {
await this.record('update.rollback.failed', { error, taskId, operation })
await this.transition('failed', { taskId, error: error instanceof Error ? error.message : 'rollback failed' })
throw error
}
}
async desiredState() {
if (this.channelState === undefined) throw new Error('update channels are not configured')
const stable = (await this.metadata.check()).value
const [current, local] = await Promise.all([this.activator.currentDeployment(), this.channelState.read()])
const supported = { dsh: stable.desired.dsh.version, environment: stable.desired.environment.version }
let upstream = null
let upstreamCandidate = null
if (
local.updateChannel === 'experimental'
&& compareDshVersions(current.dsh, supported.dsh) <= 0
&& current.environment === supported.environment
) {
try {
upstream = await this.npm.discover(stable.officialDshPolicy)
} catch (error) {
const scoped = new Error(error instanceof Error ? error.message : 'official DSH discovery failed', { cause: error })
scoped.name = error instanceof Error ? error.name : 'Error'
scoped.code = error?.code
scoped.remoteCheckSource = 'upstream'
throw scoped
}
upstreamCandidate = compareDshVersions(upstream.version, supported.dsh) > 0 ? upstream : null
}
return Object.freeze({
...planDesiredState({ local, current, supported, stableTargetSequence: stable.targetSequence, upstream: upstreamCandidate }),
upstream,
target: stable,
})
}
async publicStatus() {
const [update, local, current, rollbackPlan, journal, automatic] = await Promise.all([
this.state.read(),
this.channelState?.read() ?? Promise.resolve({ updateChannel: 'stable', holds: [], experimentalBlocked: null }),
this.bestEffort(
'update.status.current.failed',
() => this.activator.currentDeployment({ required: false }),
null,
{},
60_000,
),
this.bestEffort('update.status.rollback-plan.failed', () => this.rollbackPlan(), null, {}, 60_000),
this.journal === undefined
? Promise.resolve(undefined)
: this.bestEffort('update.status.journal.failed', () => this.journal.read(), undefined, {}, 60_000),
this.automaticChecks?.read() ?? Promise.resolve(null),
])
const returnStableAvailable = rollbackPlan !== null && rollbackPlan.snapshot !== null && update.supported?.dsh !== undefined
&& compareDshVersions(rollbackPlan.previous.dsh, update.supported.dsh) <= 0
return Object.freeze({
update,
updateChannel: local.updateChannel,
current: current === null ? null : { dsh: current.dsh, environment: current.environment, runtime: current.runtime },
supported: update.supported ?? null,
upstream: update.upstream ?? null,
aheadOfStable: update.aheadOfStable ?? false,
experimentalBlocked: local.experimentalBlocked,
holds: local.holds,
probation: journal?.phase === 'probation' ? { until: journal.probationUntil } : null,
rollbackPlan: rollbackPlan === null ? null : { ...rollbackPlan, returnStableAvailable },
...(automatic === null ? {} : {
automaticCheck: automatic.automaticCheck,
latestAutomatic: automatic.latestAutomatic,
}),
})
}
async runReconcile(taskId) {
try {
const plan = await this.desiredState()
if (plan.action === 'stable') {
const blockExperimentalCombination = plan.current?.authority === 'experimental'
&& plan.current.environment !== plan.supported?.environment
await this.run(taskId, { complete: false })
const next = await this.desiredState()
if (next.action === 'experimental') {
return this.runExperimental(taskId, { blockCombination: blockExperimentalCombination })
}
return this.transition('success', {
taskId,
progress: 100,
current: { dsh: next.current.dsh, environment: next.current.environment, runtime: next.current.runtime },
aheadOfStable: next.aheadOfStable,
experimentalBlocked: next.experimentalBlocked,
holds: next.holds,
updateAvailable: ['stable', 'experimental'].includes(next.action),
error: null,
outcome: next.action,
})
}
if (plan.action === 'experimental') return this.runExperimental(taskId)
return this.transition('success', {
taskId, progress: 100, error: null,
outcome: plan.action,
})
} catch (error) {
await this.record('update.reconcile.failed', { error, taskId })
const current = await this.state.read()
if (current.status !== 'failed') {
await this.transition('failed', { taskId, error: error instanceof Error ? error.message : 'update planning failed' })
}
throw error
}
}
async run(taskId, { complete = true } = {}) {
try {
await this.transition('planning', {
taskId, progress: 0, operation: 'update', rollbackPhase: null, rollbackIncludesSnapshot: null,
processedBytes: null, totalBytes: null, processedItems: null, totalItems: null,
readyServices: null, totalServices: null, detail: null, error: null,
})
const target = await this.metadata.check()
const artifacts = Array.isArray(target.value.artifacts) ? target.value.artifacts : []
await this.transition('downloading', {
taskId,
progress: 10,
targetSequence: target.value.targetSequence,
processedBytes: 0,
totalBytes: artifacts.reduce((total, artifact) => total + artifact.size, 0),
processedItems: 0,
totalItems: artifacts.length,
detail: 'downloading',
})
let previousDownloadKey
let previousDownloadProgress = 10
const prepared = await this.preparer.prepare(target.value, {
onProgress: metrics => {
const byteRatio = Number.isFinite(metrics.totalBytes) && metrics.totalBytes > 0
? metrics.processedBytes / metrics.totalBytes
: null
const itemRatio = Number.isFinite(metrics.totalItems) && metrics.totalItems > 0
? metrics.processedItems / metrics.totalItems
: null
const ratio = byteRatio ?? itemRatio
const measuredProgress = ratio === null ? 10 : 10 + Math.round(Math.max(0, Math.min(1, ratio)) * 55)
const progress = Math.max(previousDownloadProgress, measuredProgress)
previousDownloadProgress = progress
const key = `${progress}:${metrics.processedItems}:${metrics.totalItems}`
if (key === previousDownloadKey) return undefined
previousDownloadKey = key
return this.transition('downloading', {
taskId,
progress,
processedBytes: metrics.processedBytes,
totalBytes: metrics.totalBytes,
processedItems: metrics.processedItems,
totalItems: metrics.totalItems,
detail: metrics.detail ?? null,
})
},
})
await this.transition('validating', { taskId, progress: 70 })
await this.transition('building-candidate', { taskId, progress: 75 })
let previousBuildProgress = 75
let buildComplete = false
await this.activator.activate(prepared, {
onProgress: progress => {
if (typeof progress === 'number') {
const stageProgress = Math.max(previousBuildProgress, progress)
if (stageProgress === previousBuildProgress) return undefined
previousBuildProgress = stageProgress
return this.transition('building-candidate', { taskId, progress: stageProgress })
}
const ratio = progress.totalBytes > 0 ? progress.processedBytes / progress.totalBytes : 0
const stageProgress = Math.max(
previousBuildProgress,
75 + Math.round(Math.max(0, Math.min(1, ratio)) * 14),
)
const complete = progress.processedBytes === progress.totalBytes
&& progress.processedItems === progress.totalItems
if (stageProgress === previousBuildProgress && (!complete || buildComplete)) return undefined
previousBuildProgress = stageProgress
buildComplete = complete
return this.transition('building-candidate', {
taskId,
progress: stageProgress,
processedBytes: progress.processedBytes,
totalBytes: progress.totalBytes,
processedItems: progress.processedItems,
totalItems: progress.totalItems,
})
},
onSwitching: () => this.transition('switching', { taskId, progress: 90 }),
})
const health = await this.reportHealth('switching', { taskId, progress: 95 })
const current = this.activator.currentDeployment === undefined
? null
: await this.activator.currentDeployment()
await this.bestEffort('update.notifications.cleanup.failed', () => this.clearSatisfiedNotifications(), undefined, { taskId })
return complete ? this.transition('success', {
taskId,
progress: 100,
...(current === null ? {} : {
current: { dsh: current.dsh, environment: current.environment, runtime: current.runtime },
updateAvailable: false,
}),
error: null,
}) : health
} catch (error) {
await this.record('update.stable.failed', { error, taskId })
await this.transition('failed', { taskId, error: error instanceof Error ? error.message : 'update failed' })
throw error
}
}
trustActivate(prepared) {
return this.preparer.trust.activate(prepared.receiptTokens)
}
async runExperimental(taskId, { blockCombination = false } = {}) {
let transaction
let candidate
let failureClass = 'check'
let obsoleteSnapshotId = null
try {
await this.transition('checking-upstream', {
taskId, progress: 0, operation: 'update', rollbackPhase: null, rollbackIncludesSnapshot: null, error: null,
})
const stable = (await this.metadata.check()).value
candidate = await this.npm.latest(stable)
if (candidate === null) return this.transition('success', { taskId, progress: 100, error: null })
const from = await this.activator.currentDeployment()
const bootstrap = await this.activator.bootstrap.status()
if (
from.environment !== stable.desired.environment.version
|| bootstrap.bootstrapVersion !== stable.desired.bootstrap.version
) throw new Error('Stable Environment and Bootstrap must converge before Experimental DSH')
await this.transition('downloading', { taskId, progress: 10 })
failureClass = 'candidate'
let previousDownloadProgress = 10
const prepared = await this.preparer.prepareExperimental(candidate, {
onProgress: metrics => {
const ratio = Number.isFinite(metrics.totalBytes) && metrics.totalBytes > 0
? metrics.processedBytes / metrics.totalBytes
: metrics.processedItems / metrics.totalItems
const progress = Math.max(previousDownloadProgress, 10 + Math.round(Math.max(0, Math.min(1, ratio)) * 20))
previousDownloadProgress = progress
return this.transition('downloading', {
taskId,
progress,
processedBytes: metrics.processedBytes,
totalBytes: metrics.totalBytes,
processedItems: metrics.processedItems,
totalItems: metrics.totalItems,
detail: metrics.detail ?? null,
})
},
})
await this.transition('building-candidate', { taskId, progress: 35 })
const built = await this.activator.prepareExperimental(prepared, { targetSequence: stable.targetSequence })
const previousJournal = await this.journal.read()
if (previousJournal !== undefined && ['committed', 'rolled-back', 'failed'].includes(previousJournal.phase)) {
obsoleteSnapshotId = previousJournal.snapshotId
}
transaction = await this.journal.begin({
transactionId: taskId,
mode: 'experimental',
from: journalDeployment(from),
to: { dsh: built.dshVersion, environment: built.environmentVersion, runtime: built.runtimeId },
})
transaction = await this.journal.transition('candidate-ready', { receiptTokens: prepared.receiptTokens })
await this.transition('snapshotting-data', { taskId, progress: 55 })
failureClass = 'snapshot'
await this.activator.suspendDsh()
transaction = await this.journal.transition('suspended')
let previousSnapshotProgress
const snapshot = await this.snapshots.create({
id: taskId,
runtimeId: from.runtime,
environmentVersion: from.environment,
dshVersion: from.dsh,
onProgress: metrics => {
const metricProgress = metricPercentage(metrics.processedItems, metrics.totalItems)
?? metricPercentage(metrics.processedBytes, metrics.totalBytes)
if (metricProgress === previousSnapshotProgress) return undefined
previousSnapshotProgress = metricProgress
return this.transition('snapshotting-data', {
taskId,
progress: 55,
processedBytes: metrics.processedBytes ?? null,
totalBytes: metrics.totalBytes ?? null,
processedItems: metrics.processedItems ?? null,
totalItems: metrics.totalItems ?? null,
})
},
})
const candidateRuntimeId = this.activator.bindExperimentalSnapshot === undefined
? built.runtimeId
: await this.activator.bindExperimentalSnapshot(built.runtimeId, snapshot.id)
transaction = await this.journal.transition('snapshot-created', {
snapshotId: snapshot.id,
to: { ...transaction.to, runtime: candidateRuntimeId },
})
await this.transition('switching', { taskId, progress: 70 })
await this.activator.switchExperimental(candidateRuntimeId)
failureClass = 'combination'
transaction = await this.journal.transition('switched')
const probationUntil = new Date(this.now().valueOf() + this.probationSeconds * 1000).toISOString()
transaction = await this.journal.transition('probation', { probationUntil })
await this.transition('probation', { taskId, progress: 85, probationUntil })
do {
const health = await this.activator.health()
const remainingMs = Math.max(0, new Date(probationUntil).valueOf() - this.now().valueOf())
const elapsedRatio = this.probationSeconds <= 0
? 1
: Math.max(0, Math.min(1, 1 - remainingMs / (this.probationSeconds * 1000)))
await this.transition('probation', {
taskId,
progress: 85 + Math.round(elapsedRatio * 14),
probationUntil,
detail: `probation:${String(Math.ceil(remainingMs / 1000))}`,
...healthMetrics(health),
})
if (!health.healthy) throw new Error('Experimental Runtime failed probation health checks')
if (this.now().valueOf() >= new Date(probationUntil).valueOf()) break
await this.sleep(Math.min(1_000, Math.max(0, new Date(probationUntil).valueOf() - this.now().valueOf())))
} while (true)
await this.activator.commitExperimental(candidateRuntimeId)
transaction = await this.journal.transition('committed')
await this.bestEffort('update.notifications.cleanup.failed', () => this.clearSatisfiedNotifications(), undefined, { taskId })
if (this.activator.cleanup !== undefined) {
await this.bestEffort('update.assets.cleanup.failed', () => this.activator.cleanup(), undefined, { taskId })
}
if (obsoleteSnapshotId !== null && obsoleteSnapshotId !== transaction.snapshotId) {
if (this.snapshots.remove !== undefined) {
await this.bestEffort('update.snapshot.cleanup.failed', () => this.snapshots.remove(obsoleteSnapshotId), undefined, {
snapshotId: obsoleteSnapshotId,
taskId,
})
}
}
return this.transition('success', {
taskId,
progress: 100,
current: { dsh: built.dshVersion, environment: built.environmentVersion, runtime: candidateRuntimeId },
aheadOfStable: true,
updateAvailable: false,
outcome: 'frozen',
probationUntil: null,
error: null,
})
} catch (error) {
let message = error instanceof Error ? error.message : 'Experimental update failed'
const reason = holdReason(message)
const failureState = await this.state.read()
const failurePhase = failureState.phase ?? failureState.status
let recoveryCompleted = false
await this.record('update.experimental.failed', { error, failureClass, taskId })
if (candidate !== undefined && this.channelState !== undefined) {
if (failureClass === 'candidate') {
await this.bestEffort('update.hold.persist.failed', () => this.channelState.addHold({
type: 'version', dshVersion: candidate.version, reason,
}), undefined, { dshVersion: candidate.version, holdType: 'version', taskId })
} else if (failureClass === 'combination') {
const environmentVersion = transaction?.to.environment
if (environmentVersion !== undefined) {
const persist = blockCombination
? () => this.channelState.block({ dshVersion: candidate.version, environmentVersion, reason })
: () => this.channelState.addHold({
type: 'combination', dshVersion: candidate.version, environmentVersion, reason,
})
await this.bestEffort(
blockCombination ? 'update.experimental-block.persist.failed' : 'update.hold.persist.failed',
persist,
undefined,
{
dshVersion: candidate.version,
environmentVersion,
...(blockCombination ? { blocked: true } : { holdType: 'combination' }),
taskId,
},
)
}
}
}
if (this.journal !== undefined) {
transaction = await this.bestEffort('update.journal.read.failed', () => this.journal.read(), transaction, { taskId })
}
if (transaction !== undefined && !['committed', 'rolled-back', 'failed'].includes(transaction.phase)) {
try {
if (['snapshot-created', 'switched', 'probation', 'restoring-data'].includes(transaction.phase)) {
if (transaction.phase !== 'restoring-data') transaction = await this.journal.transition('restoring-data', { error: message })
const recoveryProgress = offset => Math.min(99, Math.max(Number(failureState.progress) || 0, offset))
await this.transition('restoring-data', {
taskId,
operation: 'update',
progress: recoveryProgress(86),
detail: 'recovery:suspend',
rollbackIncludesSnapshot: transaction.snapshotId !== null,
error: null,
})
await this.bestEffort('update.rollback.suspend.failed', () => this.activator.suspendDsh(), undefined, { taskId })
await this.transition('restoring-data', {
taskId, operation: 'update', progress: recoveryProgress(88), detail: 'recovery:deployment',
})
await this.activator.restoreDeployment(transaction.from, { resume: false })
if (transaction.snapshotId !== null) {
await this.transition('restoring-data', {
taskId, operation: 'update', progress: recoveryProgress(90), detail: 'recovery:snapshot',
})
let previousRestoreProgress
await this.snapshots.restore(transaction.snapshotId, {
onProgress: metrics => {
const metricProgress = metricPercentage(metrics.processedItems, metrics.totalItems)
?? metricPercentage(metrics.processedBytes, metrics.totalBytes)
if (metricProgress === previousRestoreProgress) return undefined
previousRestoreProgress = metricProgress
return this.transition('restoring-data', {
taskId,
operation: 'update',
progress: recoveryProgress(90 + Math.round((metricProgress ?? 0) * 0.05)),
detail: 'recovery:snapshot',
processedBytes: metrics.processedBytes ?? null,
totalBytes: metrics.totalBytes ?? null,
processedItems: metrics.processedItems ?? null,
totalItems: metrics.totalItems ?? null,
})
},
})
}
await this.transition('restoring-data', {
taskId,
operation: 'update',
progress: recoveryProgress(96),
detail: 'recovery:runtime',
processedBytes: null,
totalBytes: null,
processedItems: null,
totalItems: null,
})
await this.activator.resumeDsh()
const health = await this.reportHealth('restoring-data', {
taskId, operation: 'update', progress: recoveryProgress(99), detail: 'recovery:health',
})
if (health?.healthy === false) throw new Error('restored Deployment failed health checks')
await this.journal.transition('rolled-back', { error: message })
recoveryCompleted = true
await this.record('update.experimental.rollback.completed', { taskId })
} else {
await this.bestEffort('update.rollback.resume.failed', () => this.activator.resumeDsh(), undefined, { taskId })
await this.journal.transition('failed', { error: message })
}
} catch (rollbackError) {
await this.record('update.experimental.rollback.failed', { error: rollbackError, taskId })
const rollbackMessage = rollbackError instanceof Error ? rollbackError.message : 'unknown error'
message = `${message}; rollback failed: ${rollbackMessage}`
}
}
await this.transition('failed', {
taskId,
error: message,
...(recoveryCompleted ? {
phase: failurePhase,
progress: failureState.progress,
detail: failureState.detail,
processedBytes: failureState.processedBytes,
totalBytes: failureState.totalBytes,
processedItems: failureState.processedItems,
totalItems: failureState.totalItems,
readyServices: failureState.readyServices,
totalServices: failureState.totalServices,
} : {}),
})
throw error
}
}
async recover() {
if (this.journal === undefined || this.snapshots === undefined) return undefined
return recoverInterruptedUpdate({ journal: this.journal, snapshots: this.snapshots, activator: this.activator })
}
async clearSatisfiedNotifications() {
if (this.automaticChecks === undefined) return
await this.automaticChecks.clearSatisfied(await this.activator.currentDeployment())
}
}
upstream/container/control-plane/modules/updater/lib/journal.mjs
+142
View File
@@ -0,0 +1,142 @@
import { open, readFile, rm } from 'node:fs/promises'
import { dirname } from 'node:path'
import { durableReplace } from '../../../../platform/lib/atomic.mjs'
import { exactKeys, isoTimestamp, plainObject, TrustError } from '../../../../platform/lib/validation.mjs'
const PHASES = new Set([
'planning', 'candidate-ready', 'suspended', 'snapshot-created', 'switched', 'probation',
'committed', 'restoring-data', 'rolled-back', 'failed',
])
const TRANSITIONS = new Map([
['planning', new Set(['candidate-ready', 'failed'])],
['candidate-ready', new Set(['suspended', 'failed'])],
['suspended', new Set(['snapshot-created', 'failed'])],
['snapshot-created', new Set(['switched', 'restoring-data', 'failed'])],
['switched', new Set(['probation', 'restoring-data'])],
['probation', new Set(['committed', 'restoring-data'])],
['restoring-data', new Set(['rolled-back', 'failed'])],
['committed', new Set(['restoring-data'])],
['rolled-back', new Set()],
['failed', new Set()],
])
function deployment(value, label, snapshot) {
const object = plainObject(value, label)
exactKeys(object, ['dsh', 'environment', 'runtime', ...(snapshot ? ['dataSnapshot', 'receiptTokens'] : [])], label)
for (const name of ['dsh', 'environment', 'runtime']) {
if (typeof object[name] !== 'string' || object[name] === '') throw new TrustError(`${label}.${name} is invalid`)
}
if (snapshot && object.dataSnapshot !== null && typeof object.dataSnapshot !== 'string') {
throw new TrustError(`${label}.dataSnapshot is invalid`)
}
if (snapshot && (!Array.isArray(object.receiptTokens) || object.receiptTokens.some(token => typeof token !== 'string'))) {
throw new TrustError(`${label}.receiptTokens is invalid`)
}
return Object.freeze({ ...object, ...(snapshot ? { receiptTokens: Object.freeze([...object.receiptTokens]) } : {}) })
}
function parseJournal(value) {
const object = plainObject(value, 'update journal')
exactKeys(object, [
'error', 'from', 'mode', 'phase', 'probationUntil', 'receiptTokens', 'schema',
'snapshotId', 'to', 'transactionId', 'updatedAt',
], 'update journal')
if (object.schema !== 1) throw new TrustError('update journal schema must be 1')
if (typeof object.transactionId !== 'string' || !/^[A-Za-z0-9-]{1,128}$/.test(object.transactionId)) {
throw new TrustError('update journal transaction ID is invalid')
}
if (!['stable', 'experimental'].includes(object.mode)) throw new TrustError('update journal mode is invalid')
if (!PHASES.has(object.phase)) throw new TrustError('update journal phase is invalid')
if (!Array.isArray(object.receiptTokens) || object.receiptTokens.some(token => typeof token !== 'string')) {
throw new TrustError('update journal receipt tokens are invalid')
}
for (const name of ['snapshotId', 'probationUntil', 'error']) {
if (object[name] !== null && typeof object[name] !== 'string') throw new TrustError(`update journal ${name} is invalid`)
}
if (object.probationUntil !== null) isoTimestamp(object.probationUntil, 'update journal probation deadline')
isoTimestamp(object.updatedAt, 'update journal update time')
return Object.freeze({
...object,
from: deployment(object.from, 'update journal from', true),
to: deployment(object.to, 'update journal to', false),
receiptTokens: Object.freeze([...object.receiptTokens]),
})
}
async function syncDirectory(path) {
const handle = await open(path, 'r')
try { await handle.sync() } finally { await handle.close() }
}
export class UpdateJournal {
constructor(path, now = () => new Date()) {
this.path = path
this.now = now
this.queue = Promise.resolve()
}
exclusive(operation) {
const result = this.queue.then(operation, operation)
this.queue = result.then(() => undefined, () => undefined)
return result
}
async read() {
try { return parseJournal(JSON.parse(await readFile(this.path, 'utf8'))) } catch (error) {
if (error?.code === 'ENOENT') return undefined
throw error
}
}
begin(value) {
return this.exclusive(async () => {
const previous = await this.read()
if (previous !== undefined && !['committed', 'rolled-back', 'failed'].includes(previous.phase)) {
throw new TrustError('an update transaction journal already exists')
}
const journal = parseJournal({
schema: 1,
transactionId: value.transactionId,
mode: value.mode,
phase: 'planning',
from: value.from,
to: value.to,
snapshotId: null,
receiptTokens: [],
probationUntil: null,
error: null,
updatedAt: this.now().toISOString(),
})
await durableReplace(this.path, `${JSON.stringify(journal)}\n`)
return journal
})
}
transition(phase, fields = {}) {
return this.exclusive(async () => {
const current = await this.read()
if (current === undefined) throw new TrustError('update transaction journal does not exist')
if (!TRANSITIONS.get(current.phase)?.has(phase)) {
throw new TrustError(`update journal cannot transition from ${current.phase} to ${phase}`)
}
const next = parseJournal({ ...current, ...fields, phase, updatedAt: this.now().toISOString() })
if (next.transactionId !== current.transactionId || next.mode !== current.mode) {
throw new TrustError('update journal identity cannot change')
}
await durableReplace(this.path, `${JSON.stringify(next)}\n`)
return next
})
}
clear() {
return this.exclusive(async () => {
const current = await this.read()
if (current !== undefined && !['committed', 'rolled-back', 'failed'].includes(current.phase)) {
throw new TrustError('cannot clear a nonterminal update journal')
}
await rm(this.path, { force: true })
await syncDirectory(dirname(this.path)).catch(error => { if (error?.code !== 'ENOENT') throw error })
})
}
}
upstream/container/control-plane/modules/updater/lib/managed-store.mjs
+275
View File
@@ -0,0 +1,275 @@
import { spawn } from 'node:child_process'
import { randomUUID } from 'node:crypto'
import { cp, lstat, mkdir, readFile, rename, rm, symlink } from 'node:fs/promises'
import { join } from 'node:path'
import { buildRuntime } from '../../patch-manager/index.mjs'
import { reconcileSystemPlugins } from '../../plugin-manager/system.mjs'
import { artifactForReference, parseEnvironmentManifest } from '../../../../platform/lib/contracts.mjs'
import { deriveRecordId, parseDeploymentRecord } from '../../../../platform/lib/deployment-contracts.mjs'
import { hashTree } from '../../../../platform/lib/tree-hash.mjs'
async function exists(path) {
return lstat(path).then(() => true, error => error?.code === 'ENOENT' ? false : Promise.reject(error))
}
function run(command, args) {
return new Promise((resolve, reject) => {
const child = spawn(command, args, { stdio: ['ignore', 'pipe', 'pipe'] })
const stdout = []
const stderr = []
child.stdout.on('data', chunk => stdout.push(chunk))
child.stderr.on('data', chunk => stderr.push(chunk))
child.once('error', reject)
child.once('exit', code => code === 0
? resolve(Buffer.concat(stdout).toString('utf8'))
: reject(new Error(Buffer.concat(stderr).toString('utf8'))))
})
}
const NPM_INSTALL_ATTEMPTS = 7
const NPM_INSTALL_RETRY_MS = 5_000
function retryableNpmInstall(error) {
return /(?:^|\n)npm error (?:code (?:ETARGET|E404|ENOVERSIONS)|notarget\b)/u.test(
error instanceof Error ? error.message : '',
)
}
async function verifyDirectory(path, expectedHash) {
const details = await lstat(path)
if (!details.isDirectory() || details.isSymbolicLink()) throw new Error(`Managed Store asset is not an immutable directory: ${path}`)
const sha256 = await hashTree(path)
if (expectedHash !== undefined && sha256 !== expectedHash) throw new Error(`Managed Store asset content conflicts at ${path}`)
return sha256
}
async function publish(staging, root, id) {
const sha256 = await hashTree(staging)
const destination = join(root, id)
try {
await rename(staging, destination)
} catch (error) {
if (error?.code !== 'EEXIST' && error?.code !== 'ENOTEMPTY') throw error
await verifyDirectory(destination, sha256)
}
return Object.freeze({ id, sha256, path: destination })
}
function storeReference(kind, asset) {
return Object.freeze({ storage: 'store', kind, id: asset.id, sha256: asset.sha256 })
}
export class ManagedDeploymentBuilder {
constructor({
paths,
npmInstall = (installRoot, archivePath, { preferOnline }) => run('npm', [
'install', '--global', '--prefix', installRoot, '--omit=dev', '--ignore-scripts',
'--no-audit', '--no-fund', preferOnline ? '--prefer-online' : '--prefer-offline',
'--cache', paths.npmCacheRoot, archivePath,
]),
sleep = milliseconds => new Promise(resolve => setTimeout(resolve, milliseconds)),
}) {
this.paths = paths
this.npmInstall = npmInstall
this.sleep = sleep
}
async pristine(version, receipt) {
const id = `pristine-npm-${receipt.objectSha256}`
const destination = join(this.paths.pristineRoot, id)
if (await exists(destination)) {
const sha256 = await verifyDirectory(destination)
return Object.freeze({ id, sha256, path: destination })
}
const installRoot = join(this.paths.pristineRoot, `.install.${randomUUID()}.tmp`)
const staging = join(this.paths.pristineRoot, `.${id}.${randomUUID()}.tmp`)
await mkdir(installRoot, { recursive: false })
try {
const entries = (await run('tar', ['-tzf', receipt.path])).split('\n').filter(Boolean)
if (!entries.includes('package/package.json') || entries.some(entry => (
entry.startsWith('/') || !entry.startsWith('package/') || entry.split('/').includes('..')
))) throw new Error('official DSH archive contains an unsafe path')
for (let attempt = 1; attempt <= NPM_INSTALL_ATTEMPTS; attempt += 1) {
const attemptRoot = join(installRoot, `attempt-${String(attempt)}`)
await mkdir(attemptRoot, { recursive: false })
try {
const archivePath = join(attemptRoot, 'official-dsh.tgz')
await symlink(receipt.path, archivePath, 'file')
await this.npmInstall(attemptRoot, archivePath, { preferOnline: attempt > 1 })
const installed = join(attemptRoot, 'lib', 'node_modules', '@deepseek-ai', 'dsh')
await rename(installed, staging)
break
} catch (error) {
if (attempt === NPM_INSTALL_ATTEMPTS || !retryableNpmInstall(error)) throw error
await this.sleep(Math.min(30_000, NPM_INSTALL_RETRY_MS * (2 ** (attempt - 1))))
} finally {
await rm(attemptRoot, { recursive: true, force: true })
}
}
const metadata = JSON.parse(await readFile(join(staging, 'package.json'), 'utf8'))
if (metadata.name !== '@deepseek-ai/dsh' || metadata.version !== version) {
throw new Error('official DSH package metadata differs from its requested version')
}
return await publish(staging, this.paths.pristineRoot, id)
} finally {
await rm(installRoot, { recursive: true, force: true })
await rm(staging, { recursive: true, force: true })
}
}
async environment(prepared) {
const manifest = prepared.environment.manifest
const identity = {
schema: 1,
version: manifest.version,
targetSequence: prepared.stable.targetSequence,
manifest: prepared.environment.manifestReceipt.objectSha256,
artifacts: manifest.artifacts.map(artifact => ({ id: artifact.id, sha256: artifact.sha256 })),
}
const id = deriveRecordId('environment', identity)
const destination = join(this.paths.environmentsRoot, id)
if (await exists(destination)) {
const sha256 = await verifyDirectory(destination)
return Object.freeze({ id, sha256, path: destination })
}
const staging = join(this.paths.environmentsRoot, `.${id}.${randomUUID()}.tmp`)
await mkdir(join(staging, 'artifacts'), { recursive: true })
try {
await cp(prepared.paths.get(prepared.stable.desired.environment.manifestArtifactId), join(staging, 'environment.manifest.json'))
for (const descriptor of manifest.artifacts) {
await cp(prepared.paths.get(descriptor.id), join(staging, 'artifacts', descriptor.id))
}
return await publish(staging, this.paths.environmentsRoot, id)
} finally {
await rm(staging, { recursive: true, force: true })
}
}
async runtime(prepared, pristine, environment, { onProgress = async () => {} } = {}) {
const patches = prepared.environment.manifest.patches.map(reference => {
const artifact = artifactForReference(prepared.environment.manifest, reference)
return { id: reference.id, artifactId: artifact.id, sha256: artifact.sha256 }
})
const id = deriveRecordId('runtime', {
schema: 1,
pristine: pristine.sha256,
environment: environment.sha256,
patches,
})
const destination = join(this.paths.runtimesRoot, id)
if (await exists(destination)) {
const sha256 = await verifyDirectory(destination)
return Object.freeze({ id, sha256, path: destination })
}
await buildRuntime({
pristineRoot: pristine.path,
versionsRoot: this.paths.runtimesRoot,
runtimeId: id,
patchPaths: patches.map(patch => prepared.paths.get(patch.artifactId)),
onProgress,
})
return Object.freeze({ id, sha256: await verifyDirectory(destination), path: destination })
}
async systemPlugins(prepared, environment) {
const plugins = prepared.environment.manifest.systemPlugins.map(reference => {
const artifact = artifactForReference(prepared.environment.manifest, reference)
return { id: reference.id, artifactId: artifact.id, sha256: artifact.sha256 }
})
const id = deriveRecordId('system-plugins', {
schema: 1,
environment: environment.sha256,
plugins,
})
const destination = join(this.paths.systemPluginsRoot, id)
if (await exists(destination)) {
const sha256 = await verifyDirectory(destination)
return Object.freeze({ id, sha256, path: destination })
}
const buildRoot = join(this.paths.systemPluginsRoot, `.build.${randomUUID()}.tmp`)
try {
const built = await reconcileSystemPlugins({
root: buildRoot,
environmentVersion: id,
plugins: prepared.environment.manifest.systemPlugins,
artifactPath: reference => prepared.paths.get(artifactForReference(prepared.environment.manifest, reference).id),
})
return await publish(built, this.paths.systemPluginsRoot, id)
} finally {
await rm(buildRoot, { recursive: true, force: true })
}
}
async buildStable(prepared, { onProgress = async () => {} } = {}) {
const pristine = await this.pristine(prepared.dsh.version, prepared.dsh.receipt)
await onProgress(80)
const environment = await this.environment(prepared)
await onProgress(82)
const runtime = await this.runtime(prepared, pristine, environment, {
onProgress: metrics => onProgress({ stage: 'runtime', ...metrics }),
})
await onProgress(87)
const systemPlugins = await this.systemPlugins(prepared, environment)
await onProgress(89)
const content = {
schema: 1,
authority: 'stable',
targetSequence: prepared.stable.targetSequence,
dshVersion: prepared.dsh.version,
environmentVersion: prepared.environment.manifest.version,
environment: storeReference('environment', environment),
pristine: storeReference('pristine', pristine),
runtime: storeReference('runtime', runtime),
systemPlugins: storeReference('system-plugins', systemPlugins),
receiptTokens: [...prepared.receiptTokens],
snapshotId: null,
}
return Object.freeze({
record: parseDeploymentRecord({ ...content, id: deriveRecordId('deployment-record', content) }),
assets: Object.freeze({ pristine, environment, runtime, systemPlugins }),
})
}
async buildExperimental(prepared, currentValue, receiptTokens, { targetSequence }) {
const current = parseDeploymentRecord(currentValue)
const pristine = await this.pristine(prepared.version, prepared.receipt)
const environmentRoot = join(this.paths.viewsRoot, 'environment')
const manifest = parseEnvironmentManifest(await readFile(join(environmentRoot, 'environment.manifest.json')))
if (manifest.version !== current.environmentVersion) throw new Error('current Environment view differs from its Deployment Record')
const patches = manifest.patches.map(reference => {
const artifact = artifactForReference(manifest, reference)
return { id: reference.id, artifactId: artifact.id, sha256: artifact.sha256 }
})
const runtimeId = deriveRecordId('runtime', {
schema: 1,
pristine: pristine.sha256,
environment: current.environment.sha256,
patches,
})
const runtimePath = join(this.paths.runtimesRoot, runtimeId)
if (!await exists(runtimePath)) {
await buildRuntime({
pristineRoot: pristine.path,
versionsRoot: this.paths.runtimesRoot,
runtimeId,
patchPaths: patches.map(patch => join(environmentRoot, 'artifacts', patch.artifactId)),
})
}
const runtime = Object.freeze({ id: runtimeId, sha256: await verifyDirectory(runtimePath), path: runtimePath })
const content = {
schema: 1,
authority: 'experimental',
targetSequence,
dshVersion: prepared.version,
environmentVersion: current.environmentVersion,
environment: current.environment,
pristine: storeReference('pristine', pristine),
runtime: storeReference('runtime', runtime),
systemPlugins: current.systemPlugins,
receiptTokens: [...receiptTokens],
snapshotId: null,
}
const record = parseDeploymentRecord({ ...content, id: deriveRecordId('deployment-record', content) })
return Object.freeze({ record, runtime })
}
}
upstream/container/control-plane/modules/updater/lib/metadata.mjs
+180
View File
@@ -0,0 +1,180 @@
import { setTimeout as delay } from 'node:timers/promises'
import { parseStable } from '../../../../platform/lib/contracts.mjs'
import { compareDshVersions } from '../../../../platform/lib/supported-target.mjs'
export class MetadataUnavailableError extends Error {
constructor() {
super('signed update metadata has not been published')
this.name = 'MetadataUnavailableError'
this.code = 'METADATA_UNAVAILABLE'
}
}
async function responseBytes(response, label, maxBytes = 10 * 1024 * 1024) {
if (!response.ok) throw new HttpResponseError(label, response.status)
const bytes = Buffer.from(await response.arrayBuffer())
if (bytes.byteLength > maxBytes) throw new Error(`${label} exceeds the download limit`)
return bytes
}
class HttpResponseError extends Error {
constructor(label, status) {
super(`${label} returned HTTP ${String(status)}`)
this.name = 'HttpResponseError'
this.status = status
}
}
function retryableRequest(error) {
return !(error instanceof HttpResponseError)
|| error.status === 408
|| error.status === 429
|| error.status >= 500
}
function exhaustedRequest(label, error, attempts) {
const timeout = error?.name === 'TimeoutError'
const detail = error instanceof Error && error.message !== '' ? `: ${error.message}` : ''
const wrapped = new Error(
timeout
? `${label} timed out after ${String(attempts)} attempts`
: `${label} failed after ${String(attempts)} attempts${detail}`,
{ cause: error },
)
wrapped.name = timeout ? 'TimeoutError' : 'Error'
if (error?.code !== undefined) wrapped.code = error.code
wrapped.requestExhausted = true
return wrapped
}
function timeoutForAttempt(initial, retry, attempt) {
return attempt === 1 ? initial : retry
}
export class MetadataClient {
constructor({
baseUrl,
trust,
fetchImpl = fetch,
attempts = 2,
retryMs = 250,
requestTimeoutMs = 5_000,
retryRequestTimeoutMs = requestTimeoutMs,
}) {
this.baseUrl = new URL(baseUrl.endsWith('/') ? baseUrl : `${baseUrl}/`)
if (this.baseUrl.protocol !== 'https:' && this.baseUrl.hostname !== '127.0.0.1' && this.baseUrl.hostname !== 'localhost') {
throw new Error('update metadata URL must use HTTPS')
}
this.trust = trust
this.fetchImpl = fetchImpl
this.attempts = attempts
this.retryMs = retryMs
this.requestTimeoutMs = requestTimeoutMs
this.retryRequestTimeoutMs = retryRequestTimeoutMs
}
async file(name) {
let lastError
for (let attempt = 1; attempt <= this.attempts; attempt += 1) {
try {
const response = await this.fetchImpl(new URL(name, this.baseUrl), {
signal: AbortSignal.timeout(timeoutForAttempt(this.requestTimeoutMs, this.retryRequestTimeoutMs, attempt)),
})
if (response.status === 404) throw new MetadataUnavailableError()
return await responseBytes(response, name)
} catch (error) {
if (error instanceof MetadataUnavailableError) throw error
if (!retryableRequest(error)) throw error
lastError = error
if (attempt < this.attempts) await delay(this.retryMs)
}
}
throw exhaustedRequest(name, lastError, this.attempts)
}
async verifiedPair(payloadName, signatureName, accept) {
let lastError
for (let attempt = 1; attempt <= this.attempts; attempt += 1) {
try {
const [payload, signatureBytes] = await Promise.all([
this.file(payloadName), this.file(signatureName),
])
await accept(payload, JSON.parse(signatureBytes.toString('utf8')))
return payload
} catch (error) {
if (error instanceof MetadataUnavailableError
|| error instanceof HttpResponseError
|| error?.requestExhausted === true) throw error
lastError = error
if (attempt < this.attempts) await delay(this.retryMs)
}
}
throw lastError
}
async check() {
await this.verifiedPair('keyring.json', 'keyring.sig.json', (bytes, signature) => (
this.trust.acceptKeyring(bytes, signature)
))
const stable = await this.verifiedPair('stable.json', 'stable.sig.json', (bytes, signature) => (
this.trust.acceptTarget(bytes, signature)
))
return Object.freeze({ bytes: stable, value: parseStable(stable) })
}
}
export class NpmRegistryClient {
constructor({
fetchImpl = fetch,
registry = 'https://registry.npmjs.org/',
packageName = '@deepseek-ai/dsh',
attempts = 2,
retryMs = 250,
requestTimeoutMs = 5_000,
retryRequestTimeoutMs = requestTimeoutMs,
}) {
this.fetchImpl = fetchImpl
this.registry = registry
this.packageName = packageName
this.attempts = attempts
this.retryMs = retryMs
this.requestTimeoutMs = requestTimeoutMs
this.retryRequestTimeoutMs = retryRequestTimeoutMs
}
async discover(policy = {}) {
const registry = policy.registry ?? this.registry
const packageName = policy.packageName ?? this.packageName
const packagePath = encodeURIComponent(packageName)
let bytes
let lastError
for (let attempt = 1; attempt <= this.attempts; attempt += 1) {
try {
const response = await this.fetchImpl(new URL(packagePath, registry), {
headers: { accept: 'application/vnd.npm.install-v1+json' },
redirect: 'error',
signal: AbortSignal.timeout(timeoutForAttempt(this.requestTimeoutMs, this.retryRequestTimeoutMs, attempt)),
})
bytes = await responseBytes(response, 'npm packument', 20 * 1024 * 1024)
break
} catch (error) {
if (!retryableRequest(error)) throw error
lastError = error
if (attempt < this.attempts) await delay(this.retryMs)
}
}
if (bytes === undefined) throw exhaustedRequest('npm packument', lastError, this.attempts)
let packument
try { packument = JSON.parse(bytes.toString('utf8')) } catch { throw new Error('npm packument is not valid JSON') }
const latest = packument?.['dist-tags']?.latest
const version = typeof latest === 'string' ? packument?.versions?.[latest]?.version : undefined
if (version !== latest) throw new Error('npm packument has no coherent latest DSH version')
return Object.freeze({ version })
}
async latest(stable) {
const found = await this.discover(stable.officialDshPolicy)
const version = found.version
return compareDshVersions(version, stable.desired.dsh.version) > 0 ? Object.freeze({ version }) : null
}
}
upstream/container/control-plane/modules/updater/lib/official-dsh-download.mjs
+192
View File
@@ -0,0 +1,192 @@
import { constants } from 'node:fs'
import { createWriteStream } from 'node:fs'
import { mkdir, open, readFile, rename, rm } from 'node:fs/promises'
import { randomUUID } from 'node:crypto'
import { join } from 'node:path'
import { Readable, Transform } from 'node:stream'
import { pipeline } from 'node:stream/promises'
import { setTimeout as delay } from 'node:timers/promises'
import {
OFFICIAL_DSH_METADATA_LIMIT,
OFFICIAL_DSH_PACKAGE,
OFFICIAL_DSH_TARBALL_LIMIT,
officialDshMetadataUrl,
officialDshTarballUrl,
} from '../../../../platform/lib/official-dsh-contracts.mjs'
import { compareDshVersions } from '../../../../platform/lib/supported-target.mjs'
function exactResponse(response, url, label) {
if (response.redirected || (response.url !== undefined && response.url !== '' && response.url !== url.href)) {
throw new Error(`${label} changed origin or redirected`)
}
if (!response.ok) {
const error = new Error(`${label} returned HTTP ${String(response.status)}`)
error.status = response.status
throw error
}
return response
}
function responseLength(response) {
const header = response.headers?.get?.('content-length')
if (header === null || header === undefined || header === '') return null
const value = Number(header)
return Number.isSafeInteger(value) && value >= 0 ? value : null
}
async function boundedBytes(response, label, maxBytes) {
const declared = responseLength(response)
if (declared !== null && declared > maxBytes) throw new Error(`${label} exceeds the download limit`)
const bytes = Buffer.from(await response.arrayBuffer())
if (bytes.byteLength > maxBytes) throw new Error(`${label} exceeds the download limit`)
return bytes
}
async function atomicBytes(path, bytes) {
const temporary = `${path}.${randomUUID()}.tmp`
try {
const handle = await open(temporary, constants.O_CREAT | constants.O_EXCL | constants.O_WRONLY, 0o600)
try {
await handle.writeFile(bytes)
await handle.sync()
} finally {
await handle.close()
}
await rename(temporary, path)
} catch (error) {
await rm(temporary, { force: true }).catch(() => {})
throw error
}
}
async function streamToFile(response, path, label, maxBytes, onProgress) {
if (response.body === null || response.body === undefined) throw new Error(`${label} has no response body`)
const declared = responseLength(response)
if (declared !== null && declared > maxBytes) throw new Error(`${label} exceeds the download limit`)
const temporary = `${path}.${randomUUID()}.tmp`
let processedBytes = 0
try {
const meter = new Transform({
transform(chunk, _encoding, callback) {
processedBytes += chunk.byteLength
if (processedBytes > maxBytes) {
callback(new Error(`${label} exceeds the download limit`))
return
}
Promise.resolve(onProgress({ processedBytes, totalBytes: Number.isFinite(declared) ? declared : null }))
.then(() => callback(null, chunk), callback)
},
})
await pipeline(Readable.fromWeb(response.body), meter, createWriteStream(temporary, { flags: 'wx', mode: 0o600 }))
const handle = await open(temporary, constants.O_RDWR)
try { await handle.sync() } finally { await handle.close() }
await rename(temporary, path)
return { processedBytes, totalBytes: declared }
} catch (error) {
await rm(temporary, { force: true }).catch(() => {})
throw error
}
}
export class OfficialDshDownloader {
constructor({ fetchImpl = fetch, attempts = 3, retryMs = 250 }) {
this.fetchImpl = fetchImpl
this.attempts = attempts
this.retryMs = retryMs
}
async retry(operation) {
let lastError
for (let attempt = 1; attempt <= this.attempts; attempt += 1) {
try {
return await operation(attempt)
} catch (error) {
lastError = error
const status = error?.status
if (error?.retryable === false
|| (Number.isInteger(status) && status < 500 && status !== 408 && status !== 429)
|| attempt === this.attempts) {
throw error
}
await delay(this.retryMs)
}
}
throw lastError
}
async download(version, root, { onProgress = async () => {} } = {}) {
compareDshVersions(version, version)
await mkdir(root, { recursive: true })
const metadataUrl = officialDshMetadataUrl()
const metadata = await this.retry(async () => {
const metadataResponse = exactResponse(await this.fetchImpl(metadataUrl, {
headers: { accept: 'application/vnd.npm.install-v1+json', 'accept-encoding': 'identity' },
redirect: 'error',
signal: AbortSignal.timeout(15_000),
}), metadataUrl, 'official DSH metadata')
return boundedBytes(metadataResponse, 'official DSH metadata', OFFICIAL_DSH_METADATA_LIMIT)
})
let packument
try { packument = JSON.parse(metadata.toString('utf8')) } catch { throw new Error('official DSH metadata is not valid JSON') }
const selected = packument?.versions?.[version]
if (selected?.name !== OFFICIAL_DSH_PACKAGE || selected?.version !== version) {
throw new Error('official DSH metadata has no coherent requested version')
}
const tarballUrl = officialDshTarballUrl(version)
if (selected?.dist?.tarball !== tarballUrl.href) throw new Error('official DSH tarball URL is not canonical')
const metadataPath = join(root, 'official-dsh-packument.json')
const tarballPath = join(root, 'official-dsh.tgz')
await atomicBytes(metadataPath, metadata)
try {
let reportedBytes = metadata.byteLength
const result = await this.retry(async () => {
const tarballResponse = exactResponse(await this.fetchImpl(tarballUrl, {
headers: { accept: 'application/octet-stream', 'accept-encoding': 'identity' },
redirect: 'error',
}), tarballUrl, 'official DSH tarball')
if (tarballResponse.headers?.get?.('content-encoding')) {
const error = new Error('official DSH tarball must not use content encoding')
error.retryable = false
throw error
}
const declaredTarball = responseLength(tarballResponse)
const knownTotal = declaredTarball === null ? null : metadata.byteLength + declaredTarball
await onProgress({
processedBytes: reportedBytes,
totalBytes: knownTotal,
processedItems: 1,
totalItems: 2,
detail: 'official-dsh-metadata',
})
return streamToFile(
tarballResponse,
tarballPath,
'official DSH tarball',
OFFICIAL_DSH_TARBALL_LIMIT,
metrics => {
reportedBytes = Math.max(reportedBytes, metadata.byteLength + metrics.processedBytes)
return onProgress({
processedBytes: reportedBytes,
totalBytes: knownTotal,
processedItems: 1,
totalItems: 2,
detail: 'official-dsh-tarball',
})
},
)
})
await onProgress({
processedBytes: Math.max(reportedBytes, metadata.byteLength + result.processedBytes),
totalBytes: result.totalBytes === null ? null : metadata.byteLength + result.totalBytes,
processedItems: 2,
totalItems: 2,
detail: 'official-dsh-tarball',
})
return Object.freeze({ metadataPath, tarballPath })
} catch (error) {
await rm(metadataPath, { force: true }).catch(() => {})
await rm(tarballPath, { force: true }).catch(() => {})
throw error
}
}
}
upstream/container/control-plane/modules/updater/lib/preparer.mjs
+206
View File
@@ -0,0 +1,206 @@
import { createHash, randomUUID } from 'node:crypto'
import { createWriteStream } from 'node:fs'
import { link, mkdir, open, readFile, rm } from 'node:fs/promises'
import { join } from 'node:path'
import { Readable, Transform } from 'node:stream'
import { pipeline } from 'node:stream/promises'
import { setTimeout as delay } from 'node:timers/promises'
import { parseBootstrapManifest, parseEnvironmentManifest } from '../../../../platform/lib/contracts.mjs'
import { OfficialDshDownloader } from './official-dsh-download.mjs'
async function verifyExisting(path, descriptor) {
const handle = await open(path, 'r')
const hash = createHash('sha256')
let size = 0
try {
for await (const chunk of handle.createReadStream({ autoClose: false })) {
size += chunk.byteLength
if (size > descriptor.size) return false
hash.update(chunk)
}
} finally {
await handle.close()
}
return size === descriptor.size && hash.digest('hex') === descriptor.sha256
}
async function download(descriptor, destination, fetchImpl, onChunk) {
let lastError
let reportedBytes = 0
for (let attempt = 1; attempt <= 3; attempt += 1) {
const temporary = `${destination}.${randomUUID()}.tmp`
try {
const response = await fetchImpl(descriptor.url, {
headers: { 'accept-encoding': 'identity' },
})
if (!response.ok) {
const error = new Error(`Artifact ${descriptor.id} returned HTTP ${String(response.status)}`)
error.status = response.status
throw error
}
if (response.body === null || response.body === undefined) throw new Error(`Artifact ${descriptor.id} has no response body`)
const contentEncoding = response.headers?.get?.('content-encoding')
if (contentEncoding !== null && contentEncoding !== undefined && contentEncoding !== '') {
const error = new Error(`Artifact ${descriptor.id} must not use content encoding`)
error.retryable = false
throw error
}
const declaredHeader = response.headers?.get?.('content-length')
if (declaredHeader !== null && declaredHeader !== undefined && declaredHeader !== '') {
const declared = Number(declaredHeader)
if (!Number.isSafeInteger(declared) || declared !== descriptor.size) {
throw new Error(`Artifact ${descriptor.id} does not match stable metadata`)
}
}
const hash = createHash('sha256')
let size = 0
const meter = new Transform({
transform(chunk, _encoding, callback) {
size += chunk.byteLength
if (size > descriptor.size) {
callback(new Error(`Artifact ${descriptor.id} does not match stable metadata`))
return
}
hash.update(chunk)
reportedBytes = Math.max(reportedBytes, size)
Promise.resolve(onChunk(reportedBytes)).then(() => callback(null, chunk), callback)
},
})
await pipeline(Readable.fromWeb(response.body), meter, createWriteStream(temporary, { flags: 'wx', mode: 0o600 }))
if (size !== descriptor.size || hash.digest('hex') !== descriptor.sha256) {
throw new Error(`Artifact ${descriptor.id} does not match stable metadata`)
}
const handle = await open(temporary, 'r+')
try { await handle.sync() } finally { await handle.close() }
try {
await link(temporary, destination)
await rm(temporary)
} catch (error) {
if (error?.code !== 'EEXIST') throw error
if (!await verifyExisting(destination, descriptor)) {
throw new Error(`existing Artifact ${descriptor.id} differs from the target`)
}
await rm(temporary, { force: true })
}
return destination
} catch (error) {
await rm(temporary, { force: true }).catch(() => {})
lastError = error
const status = error?.status
if (error?.retryable === false || (Number.isInteger(status) && status < 500 && status !== 408 && status !== 429) || attempt === 3) throw error
await delay(250)
}
}
throw lastError
}
export class TargetPreparer {
constructor({ untrustedRoot, trust, fetchImpl = fetch, officialDsh = new OfficialDshDownloader({ fetchImpl }) }) {
this.untrustedRoot = untrustedRoot
this.trust = trust
this.fetchImpl = fetchImpl
this.officialDsh = officialDsh
}
async prepare(stable, { onProgress = async () => {} } = {}) {
const taskRoot = join(this.untrustedRoot, `target-${String(stable.targetSequence)}`)
await mkdir(taskRoot, { recursive: true })
const downloads = new Map()
const paths = new Map()
const receipts = new Map()
const authorities = new Map()
let downloadedBytes = 0
let completedDownloads = 0
let totalBytes = stable.artifacts.reduce((total, descriptor) => total + descriptor.size, 0)
let totalItems = stable.artifacts.length
const importDescriptor = async (descriptor, parentReceipt = null) => {
const authority = parentReceipt ?? 'stable'
if (authorities.has(descriptor.id) && authorities.get(descriptor.id) !== authority) {
throw new Error(`Artifact ID ${descriptor.id} is reused across authorities`)
}
authorities.set(descriptor.id, authority)
if (!downloads.has(descriptor.id) && !stable.artifacts.some(root => root.id === descriptor.id)) {
totalBytes += descriptor.size
totalItems += 1
}
if (!downloads.has(descriptor.id)) {
const downloadBase = downloadedBytes
const downloaded = await download(
descriptor,
join(taskRoot, descriptor.id),
this.fetchImpl,
async processed => {
await onProgress({
processedBytes: downloadBase + processed,
totalBytes,
processedItems: completedDownloads,
totalItems,
})
},
)
downloadedBytes = downloadBase + descriptor.size
downloads.set(descriptor.id, downloaded)
completedDownloads += 1
await onProgress({ processedBytes: downloadedBytes, totalBytes, processedItems: completedDownloads, totalItems })
}
if (!receipts.has(descriptor.id)) {
const receipt = await this.trust.importArtifact(
descriptor.id,
downloads.get(descriptor.id),
parentReceipt,
)
receipts.set(descriptor.id, receipt)
paths.set(descriptor.id, receipt.path)
}
return receipts.get(descriptor.id)
}
for (const descriptor of stable.artifacts) await importDescriptor(descriptor)
const prepareManifest = async (reference, parser) => {
const manifestReceipt = receipts.get(reference.manifestArtifactId)
const signatureReceipt = receipts.get(reference.signatureArtifactId)
await this.trust.acceptManifest(manifestReceipt.token, signatureReceipt.token)
const manifest = parser(await readFile(paths.get(reference.manifestArtifactId)))
for (const descriptor of manifest.artifacts) await importDescriptor(descriptor, manifestReceipt.token)
return { manifest, manifestReceipt }
}
const bootstrap = await prepareManifest(stable.desired.bootstrap, parseBootstrapManifest)
const environment = await prepareManifest(stable.desired.environment, parseEnvironmentManifest)
const dshBaseBytes = downloadedBytes
const dshBaseItems = completedDownloads
const dshSources = await this.officialDsh.download(
stable.desired.dsh.version,
join(taskRoot, 'official-dsh'),
{
onProgress: metrics => onProgress({
processedBytes: dshBaseBytes + metrics.processedBytes,
totalBytes: metrics.totalBytes === null ? null : totalBytes + metrics.totalBytes,
processedItems: dshBaseItems + metrics.processedItems,
totalItems: totalItems + metrics.totalItems,
detail: metrics.detail,
}),
},
)
const dshReceipt = await this.trust.ensureOfficialDsh(
stable.desired.dsh.version,
dshSources.metadataPath,
dshSources.tarballPath,
)
return Object.freeze({
stable,
paths,
receipts,
bootstrap,
environment,
dsh: Object.freeze({ version: stable.desired.dsh.version, receipt: dshReceipt }),
receiptTokens: Object.freeze([...receipts.values(), dshReceipt].map(receipt => receipt.token)),
})
}
async prepareExperimental(candidate, { onProgress = async () => {} } = {}) {
const taskRoot = join(this.untrustedRoot, `experimental-${candidate.version}`)
const sources = await this.officialDsh.download(candidate.version, taskRoot, { onProgress })
const receipt = await this.trust.ensureOfficialDsh(candidate.version, sources.metadataPath, sources.tarballPath)
return Object.freeze({ version: candidate.version, receipt, receiptTokens: Object.freeze([receipt.token]) })
}
}
upstream/container/control-plane/modules/updater/lib/recovery.mjs
+82
View File
@@ -0,0 +1,82 @@
const TERMINAL_PHASES = new Set(['committed', 'rolled-back', 'failed'])
const PRE_SWITCH_PHASES = new Set(['planning', 'candidate-ready', 'suspended'])
export async function recoverInterruptedUpdate({ journal, snapshots, activator, resume = true }) {
const transaction = await journal.read()
if (transaction === undefined || TERMINAL_PHASES.has(transaction.phase)) return transaction
if (PRE_SWITCH_PHASES.has(transaction.phase)) {
if (resume) await activator.resumeDsh().catch(() => {})
return journal.transition('failed', { error: 'interrupted before Runtime switch' })
}
const restoring = transaction.phase === 'restoring-data'
? transaction
: await journal.transition('restoring-data', { error: 'recovering interrupted update' })
if (resume) await activator.suspendDsh().catch(() => {})
await activator.restoreDeployment(restoring.from, { resume: false })
if (restoring.snapshotId !== null) await snapshots.restore(restoring.snapshotId)
if (resume) await activator.resumeDsh()
return journal.transition('rolled-back', { error: 'recovered interrupted update' })
}
export async function reconcileRecoveredState({ journal, state }) {
const [transaction, persisted] = await Promise.all([journal.read(), state.read()])
const sameTask = transaction !== undefined && persisted.taskId === transaction.transactionId
// A metadata check has no journal and cannot be resumed after Management
// restarts. Leaving this state behind would keep every UI showing a spinner
// and could make startup mistake the check for an interrupted update.
if (persisted.status === 'checking' && !sameTask) {
return {
transaction,
persisted: await state.write('idle', {
taskId: null,
progress: 0,
error: null,
checkSource: null,
}),
}
}
if (sameTask && !['idle', 'success', 'failed'].includes(persisted.status)) {
if (transaction.phase === 'committed') {
return { transaction, persisted: await state.write('success', { progress: 100, error: null }) }
}
if (['rolled-back', 'failed'].includes(transaction.phase)) {
return {
transaction,
persisted: await state.write('failed', { error: transaction.error ?? 'interrupted update was not committed' }),
}
}
}
return { transaction, persisted }
}
export function resumeInterruptedReconcile({ coordinator, persisted, report, audit }) {
const task = coordinator.startReconcile()
const context = {
taskId: task.taskId,
interruptedTaskId: persisted.taskId ?? null,
}
const started = Promise.resolve()
.then(() => persisted.taskId === null || persisted.taskId === undefined
? undefined
: audit('update.failed', {
taskId: persisted.taskId,
outcome: 'interrupted',
resumedByTaskId: task.taskId,
level: 'warning',
}))
.then(() => report('update.resume.started', context))
.catch(() => {})
void task.completion.then(
async () => {
await started
await report('update.resume.completed', context)
},
async error => {
await started
await report('update.resume.failed', { ...context, error })
},
).catch(() => {})
return task
}
upstream/container/control-plane/modules/updater/lib/rollback.mjs
+143
View File
@@ -0,0 +1,143 @@
import { createHash } from 'node:crypto'
import { canonicalJson } from '../../../../platform/lib/canonical-json.mjs'
import { TrustError } from '../../../../platform/lib/validation.mjs'
function deploymentIdentity(deployment) {
return {
dsh: deployment.dsh,
environment: deployment.environment,
runtime: deployment.runtime,
receiptTokens: [...deployment.receiptTokens].sort(),
}
}
function recordIdentity(record) {
return deploymentIdentity({
dsh: record.dshVersion,
environment: record.environmentVersion,
runtime: record.id,
receiptTokens: record.receiptTokens,
})
}
function recordMatchesDeployment(record, deployment) {
return record.dshVersion === deployment.dsh
&& record.environmentVersion === deployment.environment
&& [...record.receiptTokens].sort().join('\0') === [...deployment.receiptTokens].sort().join('\0')
}
function sameRecordContent(left, right) {
return left.authority === right.authority
&& left.targetSequence === right.targetSequence
&& left.dshVersion === right.dshVersion
&& left.environmentVersion === right.environmentVersion
&& left.environment.sha256 === right.environment.sha256
&& left.pristine.sha256 === right.pristine.sha256
&& left.runtime.sha256 === right.runtime.sha256
&& left.systemPlugins.sha256 === right.systemPlugins.sha256
}
function bindPlan(value) {
return Object.freeze({
...value,
planId: createHash('sha256').update(canonicalJson(value)).digest('hex'),
})
}
export class CompleteStateRecovery {
constructor({ journal, snapshots, activator }) {
this.journal = journal
this.snapshots = snapshots
this.activator = activator
}
async plan() {
const transaction = await this.journal.read()
if (transaction?.phase === 'committed' && transaction.snapshotId !== null) {
const current = await this.activator.currentDeployment()
if (
current.dsh === transaction.to.dsh
&& current.environment === transaction.to.environment
&& current.runtime === transaction.to.runtime
) {
let previous = deploymentIdentity(transaction.from)
if (this.activator.rollbackDeployments !== undefined) {
const slots = await this.activator.rollbackDeployments()
if (
slots.current?.id !== current.runtime
|| slots.previous === null
|| !recordMatchesDeployment(slots.previous, transaction.from)
) return null
previous = recordIdentity(slots.previous)
}
const snapshot = await this.snapshots.inspect(transaction.snapshotId)
if (
snapshot.runtimeId !== transaction.from.runtime
|| snapshot.environmentVersion !== transaction.from.environment
|| snapshot.dshVersion !== transaction.from.dsh
) throw new TrustError('rollback snapshot does not describe the previous deployment')
return bindPlan({
transactionId: transaction.transactionId,
mode: transaction.mode,
current: deploymentIdentity(current),
previous,
snapshot: {
id: snapshot.id,
createdAt: snapshot.createdAt,
sha256: snapshot.archiveSha256,
size: snapshot.archiveSize,
},
})
}
}
if (this.activator.rollbackDeployments === undefined) return null
const { current, previous } = await this.activator.rollbackDeployments()
if (current === null || previous === null) return null
if (sameRecordContent(current, previous)) return null
return bindPlan({
transactionId: null,
mode: 'stable',
current: recordIdentity(current),
previous: recordIdentity(previous),
snapshot: null,
})
}
async restore(planId, {
confirmDataLoss = false,
requireConfirmation = false,
onProgress = async () => {},
} = {}) {
if (requireConfirmation && confirmDataLoss !== true) {
throw new TrustError('return to Stable requires explicit data-loss confirmation')
}
const plan = await this.plan()
if (plan === null || plan.planId !== planId) throw new TrustError('rollback plan is stale or unavailable')
if (requireConfirmation && plan.snapshot === null) {
throw new TrustError('return to Stable requires a verified data snapshot')
}
if (plan.snapshot === null) {
await onProgress('switching', 35, {})
await this.activator.rollback(plan.previous.runtime)
await onProgress('verifying', 90, {})
return Object.freeze({ status: 'rolled-back', transactionId: null })
}
let transaction = await this.journal.transition('restoring-data', { error: 'manual complete-state rollback' })
await onProgress('stopping', 15, {})
await this.activator.suspendDsh()
try {
await onProgress('switching', 35, {})
await this.activator.restoreDeployment({ ...transaction.from, runtime: plan.previous.runtime }, { resume: false })
await onProgress('restoring-data', 65, {})
await this.snapshots.restore(transaction.snapshotId, {
onProgress: metrics => onProgress('restoring-data', 65, metrics),
})
await onProgress('verifying', 85, {})
await this.activator.resumeDsh()
transaction = await this.journal.transition('rolled-back', { error: 'manual complete-state rollback completed' })
return Object.freeze({ status: 'rolled-back', transactionId: transaction.transactionId })
} catch (error) {
throw error
}
}
}
upstream/container/control-plane/modules/updater/lib/scheduler.mjs
+55
View File
@@ -0,0 +1,55 @@
export class UpdateScheduler {
constructor({
check,
intervalSeconds = 21_600,
onError = async () => {},
random = Math.random,
setTimer = setTimeout,
clearTimer = clearTimeout,
}) {
if (!Number.isSafeInteger(intervalSeconds) || intervalSeconds < 60) throw new Error('update interval must be at least 60 seconds')
this.check = check
this.onError = onError
this.intervalMs = intervalSeconds * 1_000
this.random = random
this.setTimer = setTimer
this.clearTimer = clearTimer
this.timer = undefined
this.enabled = false
}
schedule() {
const jitter = 0.9 + this.random() * 0.2
const timer = this.setTimer(async () => {
if (this.timer !== timer) return
this.timer = undefined
try {
await this.check()
} catch (error) {
await Promise.resolve().then(() => this.onError(error)).catch(() => {})
}
if (this.enabled && this.timer === undefined) this.schedule()
}, Math.round(this.intervalMs * jitter))
this.timer = timer
timer.unref?.()
}
start() {
this.enabled = true
if (this.timer === undefined) this.schedule()
}
configure({ enabled, intervalSeconds }) {
if (typeof enabled !== 'boolean') throw new Error('automatic check enabled must be boolean')
if (!Number.isSafeInteger(intervalSeconds) || intervalSeconds < 60) throw new Error('update interval must be at least 60 seconds')
this.stop()
this.intervalMs = intervalSeconds * 1_000
if (enabled) this.start()
}
stop() {
this.enabled = false
if (this.timer !== undefined) this.clearTimer(this.timer)
this.timer = undefined
}
}
upstream/container/control-plane/modules/updater/lib/snapshot-client.mjs
+29
View File
@@ -0,0 +1,29 @@
export class SnapshotClient {
constructor(api, scope) {
if (!['dsh', 'user-plugin'].includes(scope)) throw new Error('snapshot scope is invalid')
this.api = api
this.scope = scope
}
async call(operation, body, onProgress) {
const response = await this.api.request('POST', `/v1/${this.scope}/${operation}`, body)
if (response.progress !== null && onProgress !== undefined) await onProgress(response.progress)
return response.result
}
create(value = undefined) {
if (this.scope === 'dsh') {
const { onProgress, ...body } = value
return this.call('create', body, onProgress)
}
return this.call('create', { id: value })
}
inspect(id) { return this.call('inspect', { id }) }
restore(id, { onProgress } = {}) {
return this.call('restore', { id }, onProgress)
}
remove(id) { return this.call('remove', { id }) }
}
upstream/container/control-plane/modules/updater/lib/snapshots.mjs
+199
View File
@@ -0,0 +1,199 @@
import { spawn } from 'node:child_process'
import { createHash, randomUUID } from 'node:crypto'
import { lstat, mkdir, open, readFile, readdir, rename, rm } from 'node:fs/promises'
import { join, resolve } from 'node:path'
import { canonicalJson } from '../../../../platform/lib/canonical-json.mjs'
import { exactKeys, isoTimestamp, parseJsonDocument, plainObject, TrustError } from '../../../../platform/lib/validation.mjs'
function validId(value) {
if (typeof value !== 'string' || !/^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/.test(value)) {
throw new TrustError('snapshot ID is invalid')
}
return value
}
function version(value, label) {
if (typeof value !== 'string' || value === '' || value.length > 128 || /[\0/\\]/.test(value)) {
throw new TrustError(`${label} is invalid`)
}
return value
}
function parseManifest(bytes) {
const value = parseJsonDocument(bytes, 'snapshot manifest')
exactKeys(value, [
'archiveSha256', 'archiveSize', 'createdAt', 'dshVersion', 'environmentVersion',
'id', 'runtimeId', 'schema', 'source',
], 'snapshot manifest')
if (value.schema !== 1) throw new TrustError('snapshot manifest schema must be 1')
validId(value.id)
if (value.source !== '/data/dsh') throw new TrustError('snapshot source is invalid')
if (typeof value.archiveSha256 !== 'string' || !/^[a-f0-9]{64}$/.test(value.archiveSha256)) {
throw new TrustError('snapshot archive SHA-256 is invalid')
}
if (!Number.isSafeInteger(value.archiveSize) || value.archiveSize < 0) {
throw new TrustError('snapshot archive size is invalid')
}
isoTimestamp(value.createdAt, 'snapshot creation time')
version(value.runtimeId, 'snapshot runtime ID')
version(value.environmentVersion, 'snapshot Environment version')
version(value.dshVersion, 'snapshot DSH version')
return Object.freeze(value)
}
function run(command, args) {
return new Promise((resolveRun, reject) => {
const child = spawn(command, args, { stdio: ['ignore', 'pipe', 'pipe'] })
const stdout = []
const stderr = []
child.stdout.on('data', chunk => stdout.push(chunk))
child.stderr.on('data', chunk => stderr.push(chunk))
child.once('error', reject)
child.once('exit', code => code === 0
? resolveRun(Buffer.concat(stdout).toString('utf8'))
: reject(new Error(`${command} failed: ${Buffer.concat(stderr).toString('utf8').trim()}`)))
})
}
async function hashFile(path) {
const handle = await open(path, 'r')
const hash = createHash('sha256')
let size = 0
try {
for await (const chunk of handle.createReadStream({ autoClose: false })) {
hash.update(chunk)
size += chunk.byteLength
}
} finally {
await handle.close()
}
return { sha256: hash.digest('hex'), size }
}
async function syncDirectory(path) {
const handle = await open(path, 'r')
try { await handle.sync() } finally { await handle.close() }
}
async function countTree(path) {
const details = await lstat(path)
if (!details.isDirectory() || details.isSymbolicLink()) return 1
let count = 0
for (const entry of await readdir(path)) count += await countTree(join(path, entry))
return count
}
export class PersistentStateSnapshots {
constructor({ root, sourceRoot = '/data/dsh', now = () => new Date(), runImpl = run }) {
this.root = resolve(root)
this.sourceRoot = resolve(sourceRoot)
this.now = now
this.run = runImpl
}
path(id) {
return join(this.root, 'versions', validId(id))
}
async create({ id = randomUUID(), runtimeId, environmentVersion, dshVersion, onProgress = async () => {} }) {
validId(id)
const sourceDetails = await lstat(this.sourceRoot)
if (!sourceDetails.isDirectory() || sourceDetails.isSymbolicLink()) {
throw new TrustError('snapshot source must be a real directory')
}
const versions = join(this.root, 'versions')
const destination = this.path(id)
await lstat(destination).then(
() => { throw new TrustError(`snapshot ${id} already exists`) },
error => { if (error?.code !== 'ENOENT') throw error },
)
await mkdir(versions, { recursive: true })
const staging = join(versions, `.${id}.${randomUUID()}.tmp`)
await mkdir(staging)
const archive = join(staging, 'data.tar.gz')
try {
const totalItems = await countTree(this.sourceRoot)
await onProgress({ processedItems: 0, totalItems })
await this.run('tar', ['-czpf', archive, '-C', this.sourceRoot, '.'])
await onProgress({ processedItems: totalItems, totalItems })
const measured = await hashFile(archive)
const manifest = canonicalJson({
schema: 1,
id,
createdAt: this.now().toISOString(),
source: '/data/dsh',
runtimeId: version(runtimeId, 'snapshot runtime ID'),
environmentVersion: version(environmentVersion, 'snapshot Environment version'),
dshVersion: version(dshVersion, 'snapshot DSH version'),
archiveSha256: measured.sha256,
archiveSize: measured.size,
})
parseManifest(manifest)
const handle = await open(join(staging, 'manifest.json'), 'wx', 0o600)
try { await handle.writeFile(manifest); await handle.sync() } finally { await handle.close() }
await syncDirectory(staging)
await rename(staging, destination)
await syncDirectory(versions)
return Object.freeze({ ...parseManifest(manifest), path: destination })
} catch (error) {
await rm(staging, { recursive: true, force: true })
throw error
}
}
async inspect(id) {
const root = this.path(id)
const manifest = parseManifest(await readFile(join(root, 'manifest.json')))
if (manifest.id !== id) throw new TrustError('snapshot manifest ID does not match its directory')
const archive = join(root, 'data.tar.gz')
const measured = await hashFile(archive)
if (measured.sha256 !== manifest.archiveSha256 || measured.size !== manifest.archiveSize) {
throw new TrustError('snapshot archive does not match its manifest', 'TRUST_ARTIFACT_MISMATCH')
}
const entries = (await this.run('tar', ['-tzf', archive])).split('\n').filter(Boolean)
if (entries.length === 0) throw new TrustError('snapshot archive is empty')
for (const entry of entries) {
if (entry.startsWith('/') || entry.split('/').includes('..')) {
throw new TrustError(`snapshot archive path is unsafe: ${entry}`)
}
}
return Object.freeze({ ...manifest, archive })
}
async restore(id, { onProgress = async () => {} } = {}) {
const snapshot = await this.inspect(id)
await mkdir(this.sourceRoot, { recursive: true })
for (const entry of await readdir(this.sourceRoot)) {
await rm(join(this.sourceRoot, entry), { recursive: true, force: true })
}
const entries = (await this.run('tar', ['-tzf', snapshot.archive])).split('\n').filter(Boolean)
await onProgress({ processedItems: 0, totalItems: entries.length })
const ownership = process.getuid?.() === 0 ? [] : ['--no-same-owner']
await this.run('tar', ['-xzpf', snapshot.archive, ...ownership, '-C', this.sourceRoot])
await onProgress({ processedItems: entries.length, totalItems: entries.length })
await syncDirectory(this.sourceRoot)
return snapshot
}
async list() {
let names
try { names = await readdir(join(this.root, 'versions')) } catch (error) {
if (error?.code === 'ENOENT') return []
throw error
}
const snapshots = []
for (const name of names.filter(name => !name.startsWith('.')).sort()) snapshots.push(await this.inspect(name))
return Object.freeze(snapshots)
}
async remove(id) {
try {
await this.inspect(id)
} catch (error) {
if (error?.code === 'ENOENT') return false
throw error
}
await rm(this.path(id), { recursive: true, force: true })
return true
}
}
upstream/container/control-plane/modules/updater/lib/state.mjs
+32
View File
@@ -0,0 +1,32 @@
import { readFile } from 'node:fs/promises'
import { durableReplace } from '../../../../platform/lib/atomic.mjs'
export const TERMINAL_UPDATE_STATES = new Set(['idle', 'success', 'failed'])
export class UpdateStateStore {
constructor(path, now = () => new Date()) {
this.path = path
this.now = now
}
async read() {
try {
return JSON.parse(await readFile(this.path, 'utf8'))
} catch (error) {
if (error?.code === 'ENOENT') return { status: 'idle', taskId: null, progress: 0, error: null }
throw error
}
}
async write(status, fields = {}) {
const previous = await this.read()
const next = {
...previous,
...fields,
status,
updatedAt: this.now().toISOString(),
}
await durableReplace(this.path, `${JSON.stringify(next)}\n`)
return Object.freeze(next)
}
}
upstream/container/control-plane/services/access-manager/component.json
+30
View File
@@ -0,0 +1,30 @@
{
"schema": 1,
"id": "access-manager",
"type": "service",
"command": {
"executable": "/usr/local/bin/node",
"args": ["/run/dsh-platform/views/bootstrap/control-plane/services/access-manager/supervisor.mjs"],
"timeoutSeconds": 30
},
"environment": {},
"lifecycle": {
"prepare": null,
"preStart": null,
"postStart": null,
"preStop": null,
"stop": null,
"postStop": null
},
"health": {
"type": "exec",
"command": {
"executable": "/usr/local/bin/node",
"args": ["/run/dsh-platform/views/bootstrap/control-plane/services/access-manager/supervisor.mjs", "health"],
"timeoutSeconds": 5
},
"intervalSeconds": 1,
"timeoutSeconds": 30
},
"logging": { "stdout": true, "stderr": true }
}
upstream/container/control-plane/services/access-manager/index.mjs
+53
View File
@@ -0,0 +1,53 @@
#!/usr/bin/env node
import { chmod, mkdir, rm } from 'node:fs/promises'
import { dirname } from 'node:path'
import { PlatformPaths } from '../../../platform/lib/paths.mjs'
import { AccessStateStore } from './lib/store.mjs'
import { AccessService, createAccessHttpServer } from './lib/server.mjs'
const paths = new PlatformPaths(
process.env.DSH_PLATFORM_DATA ?? '/data/platform',
process.env.DSH_PLATFORM_RUN ?? '/run/dsh-platform',
)
async function listen(server, path) {
await mkdir(dirname(path), { recursive: true })
await rm(path, { force: true })
await new Promise((resolve, reject) => {
server.once('error', reject)
server.listen(path, resolve)
})
await chmod(path, 0o600)
}
const classificationToken = process.env.DSH_ACCESS_CLASSIFICATION_TOKEN
delete process.env.DSH_ACCESS_CLASSIFICATION_TOKEN
if (typeof classificationToken !== 'string' || classificationToken.length < 32) {
throw new Error('Access classification token is unavailable')
}
const store = new AccessStateStore({ root: paths.accessStateRoot })
await store.prepare()
const service = new AccessService({
store,
classificationToken,
report: (message, fields) => {
process.send?.({ type: 'diagnostic', message, fields })
},
})
await service.reconcileRuntimePolicy()
const access = createAccessHttpServer({ service })
const recovery = createAccessHttpServer({ service, surface: 'recovery' })
await listen(access, paths.accessSocket)
await listen(recovery, paths.accessRecoverySocket)
process.send?.({ type: 'ready', componentReady: true })
let resolveSignal
const signal = new Promise(resolve => { resolveSignal = resolve })
process.once('SIGINT', () => resolveSignal())
process.once('SIGTERM', () => resolveSignal())
await signal
await Promise.all([
new Promise(resolve => access.close(resolve)),
new Promise(resolve => recovery.close(resolve)),
])
upstream/container/control-plane/services/access-manager/lib/capabilities.mjs
+64
View File
@@ -0,0 +1,64 @@
import { createHash, randomBytes } from 'node:crypto'
function digest(value) { return createHash('sha256').update(value).digest('hex') }
function token(random) { return `dshcap_${random(32).toString('base64url')}` }
export function capabilityTarget(path) {
if (typeof path !== 'string' || path.length === 0 || path.length > 8_192 || !path.startsWith('/')) {
throw new TypeError('capability target is invalid')
}
const value = new URL(path, 'http://capability.internal')
if (value.origin !== 'http://capability.internal' || value.hash !== '') throw new TypeError('capability target is invalid')
return `${value.pathname}${value.search}`
}
export class CapabilityStore {
constructor({ now = Date.now, random = randomBytes, ttlMs = 5_000 } = {}) {
this.now = now
this.random = random
this.ttlMs = ttlMs
this.capabilities = new Map()
}
prune() {
const now = this.now()
for (const [key, value] of this.capabilities) if (value.expiresAt <= now) this.capabilities.delete(key)
}
issue(session, account, { audience, method, target }) {
if (!['management', 'maintenance', 'plugin'].includes(audience)) throw new TypeError('capability audience is invalid')
if (!/^(?:GET|HEAD|POST|PUT|DELETE)$/.test(method ?? '')) throw new TypeError('capability method is invalid')
this.prune()
const value = token(this.random)
this.capabilities.set(digest(value), {
audience,
method,
target: capabilityTarget(target),
sessionId: session.sessionId,
accountId: account.accountId,
mainCredentialVersion: account.mainCredential.version,
managementAdditionalCredentialVersion: account.managementAdditionalCredential.enabled
? account.managementAdditionalCredential.version : null,
managementAccessVersion: account.managementAccess.version,
expiresAt: this.now() + this.ttlMs,
})
return Object.freeze({ token: value, expiresAt: new Date(this.now() + this.ttlMs).toISOString() })
}
consume(value, account, { audience, method, target }) {
this.prune()
if (typeof value !== 'string' || value.length > 512) return undefined
const key = digest(value)
const capability = this.capabilities.get(key)
this.capabilities.delete(key)
const additionalVersion = account.managementAdditionalCredential.enabled
? account.managementAdditionalCredential.version : null
if (capability === undefined || capability.accountId !== account.accountId
|| capability.mainCredentialVersion !== account.mainCredential.version
|| capability.managementAdditionalCredentialVersion !== additionalVersion
|| capability.managementAccessVersion !== account.managementAccess.version
|| capability.audience !== audience || capability.method !== method
|| capability.target !== capabilityTarget(target)) return undefined
return Object.freeze({ sessionId: capability.sessionId, audience: capability.audience })
}
}
upstream/container/control-plane/services/access-manager/lib/credentials.mjs
+89
View File
@@ -0,0 +1,89 @@
import { randomBytes, scrypt as scryptCallback, timingSafeEqual } from 'node:crypto'
import { promisify } from 'node:util'
import { AccessError } from './errors.mjs'
const scrypt = promisify(scryptCallback)
const BIDI_CONTROLS = /[\u061c\u200e\u200f\u202a-\u202e\u2066-\u2069]/u
const CONTROL_CHARACTERS = /\p{Cc}/u
export const SCRYPT_POLICY = Object.freeze({ N: 32768, r: 8, p: 1, keyLength: 32, maxmem: 64 * 1024 * 1024 })
function codePoints(value) { return [...value].length }
export function normalizeUsername(input) {
if (typeof input !== 'string') throw new AccessError('USERNAME_INVALID', 'username is invalid')
const value = input.trim().normalize('NFC')
const bytes = Buffer.byteLength(value, 'utf8')
if (value.length === 0 || codePoints(value) > 64 || bytes > 256
|| CONTROL_CHARACTERS.test(value) || BIDI_CONTROLS.test(value)) {
throw new AccessError('USERNAME_INVALID', 'username is invalid')
}
return value
}
export function normalizePassword(input) {
if (typeof input !== 'string') throw new AccessError('PASSWORD_POLICY_VIOLATION', 'password does not meet policy')
const value = input.normalize('NFC')
const length = codePoints(value)
if (length < 8 || length > 1024 || Buffer.byteLength(value, 'utf8') > 1024
|| CONTROL_CHARACTERS.test(value) || BIDI_CONTROLS.test(value)) {
throw new AccessError('PASSWORD_POLICY_VIOLATION', 'password does not meet policy')
}
return value
}
function validParameters(value) {
return value !== null && typeof value === 'object'
&& Number.isInteger(value.N) && value.N >= 2
&& Number.isInteger(value.r) && value.r > 0
&& Number.isInteger(value.p) && value.p > 0
&& Number.isInteger(value.keyLength) && value.keyLength >= 16
&& Number.isInteger(value.maxmem) && value.maxmem >= 1024 * 1024
}
export async function createCredential(password, {
version = 1,
policy = SCRYPT_POLICY,
now = () => new Date(),
random = randomBytes,
derive = scrypt,
} = {}) {
const normalized = normalizePassword(password)
const salt = random(16)
const hash = await derive(normalized, salt, policy.keyLength, {
N: policy.N, r: policy.r, p: policy.p, maxmem: policy.maxmem,
})
return Object.freeze({
version,
algorithm: 'scrypt',
parameters: { ...policy },
salt: Buffer.from(salt).toString('base64url'),
hash: Buffer.from(hash).toString('base64url'),
changedAt: now().toISOString(),
})
}
export function validCredential(value) {
if (value === null || typeof value !== 'object' || value.algorithm !== 'scrypt'
|| !Number.isInteger(value.version) || value.version < 1 || !validParameters(value.parameters)
|| typeof value.salt !== 'string' || typeof value.hash !== 'string'
|| typeof value.changedAt !== 'string') return false
try {
return Buffer.from(value.salt, 'base64url').byteLength >= 16
&& Buffer.from(value.hash, 'base64url').byteLength === value.parameters.keyLength
&& Number.isFinite(Date.parse(value.changedAt))
} catch { return false }
}
export async function verifyCredential(password, credential, { derive = scrypt } = {}) {
if (!validCredential(credential)) return false
let normalized
try { normalized = normalizePassword(password) } catch { return false }
const expected = Buffer.from(credential.hash, 'base64url')
const salt = Buffer.from(credential.salt, 'base64url')
const parameters = credential.parameters
const actual = Buffer.from(await derive(normalized, salt, parameters.keyLength, {
N: parameters.N, r: parameters.r, p: parameters.p, maxmem: parameters.maxmem,
}))
return actual.byteLength === expected.byteLength && timingSafeEqual(actual, expected)
}
upstream/container/control-plane/services/access-manager/lib/errors.mjs
+19
View File
@@ -0,0 +1,19 @@
export class AccessError extends Error {
constructor(code, message, statusCode = 400, details = {}) {
super(message)
this.name = 'AccessError'
this.code = code
this.statusCode = statusCode
this.details = details
}
}
export function accessErrorBody(error) {
if (error instanceof AccessError) return {
error: error.message,
code: error.code,
...(Number.isInteger(error.details.retryAfterSeconds)
? { retryAfterSeconds: error.details.retryAfterSeconds } : {}),
}
return { error: 'access operation failed', code: 'ACCESS_INTERNAL_ERROR' }
}
upstream/container/control-plane/services/access-manager/lib/exchanges.mjs
+97
View File
@@ -0,0 +1,97 @@
import { createHash, randomBytes } from 'node:crypto'
function digest(value) { return createHash('sha256').update(value).digest('hex') }
function token(prefix, random) { return `${prefix}_${random(32).toString('base64url')}` }
export class ManagementExchangeStore {
constructor({ now = Date.now, random = randomBytes, handoffTtlMs = 30_000, pendingTtlMs = 5 * 60_000, pendingAttempts = 16 } = {}) {
this.now = now
this.random = random
this.handoffTtlMs = handoffTtlMs
this.pendingTtlMs = pendingTtlMs
this.pendingAttempts = pendingAttempts
this.handoffs = new Map()
this.pending = new Map()
}
prune() {
const now = this.now()
for (const [key, value] of this.handoffs) if (value.expiresAt <= now) this.handoffs.delete(key)
for (const [key, value] of this.pending) if (value.expiresAt <= now) this.pending.delete(key)
}
createHandoff(session, account, targetOrigin) {
this.prune()
const value = token('dshh', this.random)
this.handoffs.set(digest(value), {
accountId: account.accountId,
dshSessionId: session.sessionId,
mainCredentialVersion: account.mainCredential.version,
managementAccessVersion: account.managementAccess.version,
sourceDshOrigin: session.origin,
targetOrigin,
expiresAt: this.now() + this.handoffTtlMs,
})
return Object.freeze({ token: value, expiresAt: new Date(this.now() + this.handoffTtlMs).toISOString() })
}
consumeHandoff(value, account, targetOrigin) {
this.prune()
if (typeof value !== 'string' || value.length > 512) return undefined
const key = digest(value)
const handoff = this.handoffs.get(key)
this.handoffs.delete(key)
if (handoff === undefined || handoff.accountId !== account.accountId
|| handoff.mainCredentialVersion !== account.mainCredential.version
|| handoff.managementAccessVersion !== account.managementAccess.version
|| handoff.targetOrigin !== targetOrigin) return undefined
return handoff
}
createPending(account, { targetOrigin, sourceDshOrigin = null, sourceDshSessionId = null }) {
this.prune()
const value = token('dshmp', this.random)
this.pending.set(digest(value), {
accountId: account.accountId,
mainCredentialVersion: account.mainCredential.version,
managementAdditionalCredentialVersion: account.managementAdditionalCredential.version,
managementAccessVersion: account.managementAccess.version,
targetOrigin,
sourceDshOrigin,
sourceDshSessionId,
attempts: 0,
expiresAt: this.now() + this.pendingTtlMs,
})
return Object.freeze({ token: value, expiresAt: new Date(this.now() + this.pendingTtlMs).toISOString() })
}
peekPendingSource(value, account, targetOrigin) {
this.prune()
if (typeof value !== 'string' || value.length > 512) return undefined
const pending = this.pending.get(digest(value))
if (pending === undefined || pending.accountId !== account.accountId
|| pending.mainCredentialVersion !== account.mainCredential.version
|| pending.managementAccessVersion !== account.managementAccess.version
|| pending.targetOrigin !== targetOrigin) return undefined
return pending.sourceDshSessionId
}
inspectPending(value, account, targetOrigin) {
this.prune()
if (typeof value !== 'string' || value.length > 512) return undefined
const key = digest(value)
const pending = this.pending.get(key)
if (pending === undefined || pending.accountId !== account.accountId
|| pending.mainCredentialVersion !== account.mainCredential.version
|| pending.managementAccessVersion !== account.managementAccess.version
|| pending.targetOrigin !== targetOrigin) return undefined
pending.attempts += 1
if (pending.attempts > this.pendingAttempts) {
this.pending.delete(key)
return undefined
}
return { key, value: pending }
}
consumePending(key) { return this.pending.delete(key) }
}
upstream/container/control-plane/services/access-manager/lib/rate-limiter.mjs
+241
View File
@@ -0,0 +1,241 @@
import { AccessError } from './errors.mjs'
export class AuthenticationLimiter {
constructor({
globalLimit = 20,
windowMs = 60_000,
maxConcurrent = 2,
backoffThreshold = 5,
initialBackoffMs = 30_000,
maxBackoffMs = 15 * 60_000,
consecutiveResetMs = 24 * 60 * 60_000,
sourceWindows,
globalWindows,
clock = () => Date.now(),
} = {}) {
this.globalLimit = globalLimit
this.windowMs = windowMs
this.maxConcurrent = maxConcurrent
this.backoffThreshold = backoffThreshold
this.initialBackoffMs = initialBackoffMs
this.maxBackoffMs = maxBackoffMs
this.consecutiveResetMs = consecutiveResetMs
this.sourceWindows = sourceWindows ?? [
{ limit: 12, windowMs: 60 * 60_000 },
{ limit: 24, windowMs: 24 * 60 * 60_000 },
]
this.globalWindows = globalWindows ?? [
{ limit: globalLimit, windowMs },
{ limit: 60, windowMs: 60 * 60_000 },
{ limit: 120, windowMs: 24 * 60 * 60_000 },
]
this.maxSourceWindowMs = Math.max(...this.sourceWindows.map(value => value.windowMs))
this.maxGlobalWindowMs = Math.max(...this.globalWindows.map(value => value.windowMs))
this.clock = clock
this.global = []
this.sources = new Map()
this.consecutiveFailures = new Map()
this.active = 0
this.lastPruneAt = Number.NEGATIVE_INFINITY
}
prune(values, now, windowMs = this.windowMs) {
while (values.length > 0 && this.timestamp(values[0]) <= now - windowMs) values.shift()
}
timestamp(value) { return typeof value === 'number' ? value : value.at }
retryAfterSeconds(values, windows, now) {
let retryAfterMs = 0
for (const { limit, windowMs } of windows) {
const withinWindow = values.filter(value => this.timestamp(value) > now - windowMs)
if (withinWindow.length < limit) continue
retryAfterMs = Math.max(
retryAfterMs,
this.timestamp(withinWindow[withinWindow.length - limit]) + windowMs - now,
)
}
return Math.max(0, Math.ceil(retryAfterMs / 1_000))
}
retryKey(accountId, sourceId, credential = 'main') { return `${accountId}\u0000${credential}\u0000${sourceId}` }
retryPrefix(accountId, credential = 'all') {
return credential === 'all' ? `${accountId}\u0000` : `${accountId}\u0000${credential}\u0000`
}
pruneExpiredSources(now) {
if (now - this.lastPruneAt < this.windowMs) return
this.lastPruneAt = now
for (const [key, values] of this.sources) {
this.prune(values, now, this.maxSourceWindowMs)
if (values.length === 0) this.sources.delete(key)
}
for (const [key, retry] of this.consecutiveFailures) {
if (retry.lastFailureAt <= now - this.consecutiveResetMs) {
this.consecutiveFailures.delete(key)
}
}
}
currentRetry(accountId = 'unknown', sourceId = 'unknown', now = this.clock(), credential = 'main') {
const retryKey = this.retryKey(accountId, sourceId, credential)
const retry = this.consecutiveFailures.get(retryKey)
if (retry !== undefined && retry.lastFailureAt <= now - this.consecutiveResetMs) {
this.consecutiveFailures.delete(retryKey)
return undefined
}
return retry
}
checkRetry(accountId = 'unknown', sourceId = 'unknown', credential = 'main') {
const now = this.clock()
const retry = this.currentRetry(accountId, sourceId, now, credential)
if (retry !== undefined && retry.blockedUntil > now) {
const retryAfterSeconds = Math.max(1, Math.ceil((retry.blockedUntil - now) / 1_000))
throw new AccessError(
'AUTHENTICATION_RETRY_REQUIRED',
`authentication retry is available in ${retryAfterSeconds} seconds`,
429,
{ retryAfterSeconds },
)
}
}
enter(accountId = 'unknown', sourceId = 'unknown', credential = 'main') {
this.checkRetry(accountId, sourceId, credential)
const now = this.clock()
this.pruneExpiredSources(now)
const retryKey = this.retryKey(accountId, sourceId, credential)
const source = this.sources.get(retryKey) ?? []
this.prune(source, now, this.maxSourceWindowMs)
this.sources.set(retryKey, source)
this.prune(this.global, now, this.maxGlobalWindowMs)
if (this.active >= this.maxConcurrent) {
throw new AccessError(
'AUTHENTICATION_RATE_LIMITED',
'authentication is temporarily unavailable',
429,
{ retryAfterSeconds: 1 },
)
}
const globalRetryAfterSeconds = this.retryAfterSeconds(this.global, this.globalWindows, now)
const sourceRetryAfterSeconds = this.retryAfterSeconds(source, this.sourceWindows, now)
if (sourceRetryAfterSeconds > 0) {
throw new AccessError(
'AUTHENTICATION_RATE_LIMITED',
`browser authentication is available in ${sourceRetryAfterSeconds} seconds`,
429,
{ retryAfterSeconds: sourceRetryAfterSeconds },
)
}
if (globalRetryAfterSeconds > 0) {
throw new AccessError(
'AUTHENTICATION_RATE_LIMITED',
`administrator authentication is available in ${globalRetryAfterSeconds} seconds`,
429,
{ retryAfterSeconds: globalRetryAfterSeconds },
)
}
this.active += 1
let released = false
const release = success => {
if (released) return this.sourceStatus(accountId, sourceId, credential)
released = true
this.active -= 1
if (success !== true) {
const failedAt = this.clock()
this.global.push({ at: failedAt, credential })
source.push(failedAt)
const previous = this.consecutiveFailures.get(retryKey)
const failures = (previous !== undefined
&& previous.lastFailureAt > failedAt - this.consecutiveResetMs ? previous.failures : 0) + 1
const exponent = Math.max(0, failures - this.backoffThreshold)
const backoffMs = failures < this.backoffThreshold
? 0
: Math.min(this.maxBackoffMs, this.initialBackoffMs * (2 ** exponent))
this.consecutiveFailures.set(retryKey, {
failures,
blockedUntil: backoffMs === 0 ? 0 : failedAt + backoffMs,
lastFailureAt: failedAt,
})
} else {
this.consecutiveFailures.delete(this.retryKey(accountId, sourceId, credential))
this.sources.delete(this.retryKey(accountId, sourceId, credential))
}
return this.sourceStatus(accountId, sourceId, credential)
}
return { release }
}
sourceStatus(accountId = 'unknown', sourceId = 'unknown', credential = 'main') {
const retry = this.currentRetry(accountId, sourceId, this.clock(), credential)
if (retry === undefined) return { consecutiveFailures: 0, retryAfterSeconds: 0 }
return {
consecutiveFailures: retry.failures,
retryAfterSeconds: Math.max(0, Math.ceil((retry.blockedUntil - this.clock()) / 1_000)),
}
}
status(accountId = 'unknown', credential = 'all') {
const now = this.clock()
this.pruneExpiredSources(now)
this.prune(this.global, now, this.maxGlobalWindowMs)
const prefix = this.retryPrefix(accountId, credential)
for (const [key, retry] of this.consecutiveFailures) {
if (key.startsWith(prefix) && retry.lastFailureAt <= now - this.consecutiveResetMs) {
this.consecutiveFailures.delete(key)
}
}
const states = [...this.consecutiveFailures.entries()]
.filter(([key]) => key.startsWith(prefix))
.map(([, retry]) => ({
consecutiveFailures: retry.failures,
retryAfterSeconds: Math.max(0, Math.ceil((retry.blockedUntil - this.clock()) / 1_000)),
}))
const sourcePrefix = prefix
const sourceRetryAfterSeconds = Math.max(0, ...[...this.sources.entries()]
.filter(([key]) => key.startsWith(sourcePrefix))
.map(([, values]) => this.retryAfterSeconds(values, this.sourceWindows, now)))
const global = credential === 'all'
? this.global : this.global.filter(value => value.credential === credential)
return {
activeSources: states.length,
consecutiveFailures: Math.max(0, ...states.map(value => value.consecutiveFailures)),
retryAfterSeconds: Math.max(0, ...states.map(value => value.retryAfterSeconds)),
sourceRetryAfterSeconds,
globalFailures: global.length,
globalRetryAfterSeconds: this.retryAfterSeconds(global, this.globalWindows, now),
}
}
clear(accountId = 'unknown', credential = 'all') {
const previous = this.status(accountId, credential)
const prefix = this.retryPrefix(accountId, credential)
for (const key of this.consecutiveFailures.keys()) {
if (key.startsWith(prefix)) this.consecutiveFailures.delete(key)
}
for (const key of this.sources.keys()) {
if (key.startsWith(prefix)) this.sources.delete(key)
}
this.global = credential === 'all'
? [] : this.global.filter(value => value.credential !== credential)
return { cleared: previous.activeSources > 0 || previous.globalFailures > 0, ...previous }
}
clearGlobal(credential = 'all') {
const now = this.clock()
this.prune(this.global, now, this.maxGlobalWindowMs)
const matching = credential === 'all'
? this.global : this.global.filter(value => value.credential === credential)
const globalFailures = matching.length
const globalRetryAfterSeconds = this.retryAfterSeconds(matching, this.globalWindows, now)
this.global = credential === 'all'
? [] : this.global.filter(value => value.credential !== credential)
return {
cleared: globalFailures > 0,
globalFailures,
globalRetryAfterSeconds,
}
}
}
upstream/container/control-plane/services/access-manager/lib/runtime-capabilities.mjs
+62
View File
@@ -0,0 +1,62 @@
import { readFile } from 'node:fs/promises'
function groupId(groupFile, name) {
for (const line of groupFile.split('\n')) {
const fields = line.split(':')
if (fields[0] === name && /^\d+$/.test(fields[2] ?? '')) return Number(fields[2])
}
return null
}
function supplementaryGroups(status) {
const line = status.split('\n').find(value => value.startsWith('Groups:'))
if (line === undefined) return []
return line.slice('Groups:'.length).trim().split(/\s+/).filter(Boolean).map(Number).filter(Number.isInteger)
}
function mountOption(mounts, mountPoint, prefix) {
for (const line of mounts.split('\n')) {
const fields = line.split(' ')
if (fields[1] !== mountPoint) continue
return fields[3]?.split(',').find(value => value.startsWith(prefix)) ?? null
}
return null
}
async function optionalRead(path) {
try { return await readFile(path, 'utf8') } catch { return '' }
}
export async function detectRuntimeCapabilities({
groupPath = '/etc/group',
initStatusPath = '/proc/1/status',
mountsPath = '/proc/mounts',
ptraceScopePath = '/proc/sys/kernel/yama/ptrace_scope',
} = {}) {
const [groupsFile, initStatus, mounts, ptraceScopeValue] = await Promise.all([
optionalRead(groupPath),
optionalRead(initStatusPath),
optionalRead(mountsPath),
optionalRead(ptraceScopePath),
])
const enabledGroup = groupId(groupsFile, 'dsh-sudo-true')
const disabledGroup = groupId(groupsFile, 'dsh-sudo-false')
const initGroups = supplementaryGroups(initStatus)
const dshRootCapabilityEffective = enabledGroup !== null && initGroups.includes(enabledGroup)
const sudoSelection = dshRootCapabilityEffective
? 'enabled'
: (disabledGroup !== null && initGroups.includes(disabledGroup) ? 'disabled' : 'unknown')
const ptraceScope = /^\d+$/.test(ptraceScopeValue.trim()) ? Number(ptraceScopeValue.trim()) : null
const hidepid = mountOption(mounts, '/proc', 'hidepid=')
return {
dshRootCapabilityEffective,
agentIsolationEffective: false,
details: {
sudoSelection,
ptraceScope,
procHidepid: hidepid === null ? null : Number(hidepid.slice('hidepid='.length)),
agentIsolationReason: 'shared-node-process-identity',
},
}
}
upstream/container/control-plane/services/access-manager/lib/server.mjs
+1282
View File
File diff suppressed because it is too large. Load diff
upstream/container/control-plane/services/access-manager/lib/sessions.mjs
+301
View File
@@ -0,0 +1,301 @@
import { createHash, randomBytes, timingSafeEqual } from 'node:crypto'
function digest(value) {
return createHash('sha256').update(value).digest()
}
function identifier(random = randomBytes, bytes = 32) {
return random(bytes).toString('base64url')
}
function sameDigest(value, expected) {
const actual = digest(value ?? '')
return actual.byteLength === expected.byteLength && timingSafeEqual(actual, expected)
}
function sessionClient(value) {
if (value === null || typeof value !== 'object' || Array.isArray(value)) return Object.freeze({ ip: null, userAgent: null })
const bounded = (field, maximum) => typeof field === 'string' && field.length > 0 && field.length <= maximum
? field : null
return Object.freeze({
ip: bounded(value.ip, 128),
userAgent: bounded(value.userAgent, 512),
})
}
export class BrowserSessionStore {
constructor({
now = Date.now,
random = randomBytes,
dshAbsoluteMs = 30 * 24 * 60 * 60_000,
dshIdleMs = 7 * 24 * 60 * 60_000,
managementAbsoluteMs = 24 * 60 * 60_000,
managementIdleMs = 2 * 60 * 60_000,
activityReportMs = 5 * 60_000,
onEvent = () => {},
} = {}) {
this.now = now
this.random = random
this.policy = Object.freeze({
dsh: Object.freeze({ absoluteMs: dshAbsoluteMs, idleMs: dshIdleMs }),
management: Object.freeze({ absoluteMs: managementAbsoluteMs, idleMs: managementIdleMs }),
})
this.activityReportMs = activityReportMs
this.onEvent = onEvent
this.sessions = new Map()
}
setEventSink(onEvent) {
this.onEvent = typeof onEvent === 'function' ? onEvent : () => {}
}
emit(message, session, fields = {}) {
this.onEvent(message, {
accountId: session.accountId,
sessionId: session.sessionId,
kind: session.kind,
...fields,
})
}
prune() {
const now = this.now()
for (const [key, session] of this.sessions) {
const reason = session.expiresAt <= now
? 'absolute'
: session.lastSeenAt + session.idleMs <= now ? 'idle' : null
if (reason === null) continue
this.sessions.delete(key)
this.emit('access.session.expired', session, { reason })
}
for (const [key, session] of this.sessions) {
if (session.kind === 'management' && this.activeDshSession(session.sourceDshSessionId, session.accountId) === undefined) {
this.sessions.delete(key)
this.emit('access.session.invalidated', session, { reason: 'source-dsh-session' })
}
}
}
activeDshSession(sessionId, accountId) {
if (typeof sessionId !== 'string') return undefined
for (const session of this.sessions.values()) {
if (session.kind === 'dsh' && session.sessionId === sessionId && session.accountId === accountId) return session
}
return undefined
}
issue(kind, account, {
origin, sourceDshOrigin = null, sourceDshSessionId = null, client = null, authenticationSource = 'unknown',
} = {}) {
const policy = this.policy[kind]
if (policy === undefined) throw new TypeError('browser session kind is invalid')
if (typeof origin !== 'string' || origin.length === 0 || origin.length > 2_048) {
throw new TypeError('browser session origin is invalid')
}
this.prune()
const sourceDshSession = kind === 'management'
? this.activeDshSession(sourceDshSessionId, account.accountId)
: undefined
if (kind === 'management' && (sourceDshSession === undefined || sourceDshSession.origin !== sourceDshOrigin)) {
throw new TypeError('Management session requires an active DSH session')
}
const token = `${kind === 'dsh' ? 'dshs' : 'dshms'}_${identifier(this.random)}`
const csrfToken = `dshc_${identifier(this.random)}`
const sessionId = identifier(this.random, 16)
const createdAt = this.now()
const sourceClient = sourceDshSession === undefined ? sessionClient(client) : sourceDshSession.client
const expiresAt = Math.min(createdAt + policy.absoluteMs, sourceDshSession?.expiresAt ?? Number.POSITIVE_INFINITY)
this.sessions.set(digest(token).toString('hex'), {
sessionId,
kind,
accountId: account.accountId,
mainCredentialVersion: account.mainCredential.version,
managementAdditionalCredentialVersion: account.managementAdditionalCredential.enabled
? account.managementAdditionalCredential.version : null,
managementAccessVersion: account.managementAccess.version,
origin,
sourceDshOrigin,
sourceDshSessionId,
authenticationSource: sourceDshSession?.authenticationSource ?? authenticationSource,
client: sourceClient,
csrfDigest: digest(csrfToken),
createdAt,
lastSeenAt: createdAt,
activityReportedAt: createdAt,
expiresAt,
idleMs: policy.idleMs,
})
return Object.freeze({
token,
csrfToken,
sessionId,
expiresAt: new Date(expiresAt).toISOString(),
})
}
validate(token, kind, account, { origin, csrfToken, requireCsrf = false, touch = true } = {}) {
this.prune()
if (typeof token !== 'string' || token.length > 512 || this.policy[kind] === undefined) return undefined
const key = digest(token).toString('hex')
const session = this.sessions.get(key)
if (session === undefined || session.kind !== kind || session.accountId !== account.accountId
|| session.origin !== origin || (requireCsrf && !sameDigest(csrfToken, session.csrfDigest))) {
return undefined
}
let invalidationReason = null
if (session.mainCredentialVersion !== account.mainCredential.version) invalidationReason = 'main-credential-version'
else if (session.managementAccessVersion !== account.managementAccess.version) invalidationReason = 'management-access-version'
else if (kind === 'management'
&& session.managementAdditionalCredentialVersion !== (account.managementAdditionalCredential.enabled
? account.managementAdditionalCredential.version : null)) invalidationReason = 'management-credential-version'
if (invalidationReason !== null) {
this.sessions.delete(key)
this.emit('access.session.invalidated', session, { reason: invalidationReason })
return undefined
}
if (kind === 'management') {
const source = this.activeDshSession(session.sourceDshSessionId, account.accountId)
if (source === undefined || source.origin !== session.sourceDshOrigin
|| source.mainCredentialVersion !== account.mainCredential.version
|| source.managementAccessVersion !== account.managementAccess.version) {
this.sessions.delete(key)
this.emit('access.session.invalidated', session, { reason: 'source-dsh-session' })
return undefined
}
}
if (touch) {
const now = this.now()
session.lastSeenAt = now
if (now - session.activityReportedAt >= this.activityReportMs) {
session.activityReportedAt = now
this.emit('access.session.activity', session)
}
}
return Object.freeze({
sessionId: session.sessionId,
kind: session.kind,
accountId: session.accountId,
origin: session.origin,
sourceDshOrigin: session.sourceDshOrigin,
sourceDshSessionId: session.sourceDshSessionId,
createdAt: new Date(session.createdAt).toISOString(),
expiresAt: new Date(session.expiresAt).toISOString(),
})
}
revoke(token) {
if (typeof token !== 'string' || token.length > 512) return false
const key = digest(token).toString('hex')
const session = this.sessions.get(key)
if (!this.sessions.delete(key)) return false
if (session.kind === 'dsh') this.revokeManagementFromDsh(session.sessionId, 'source-dsh-session-revoked')
return true
}
revokeKind(kind) {
const dshSessionIds = []
let revoked = 0
for (const [key, session] of this.sessions) {
if (session.kind !== kind) continue
this.sessions.delete(key)
revoked += 1
if (kind === 'dsh') dshSessionIds.push(session.sessionId)
}
for (const sessionId of dshSessionIds) this.revokeManagementFromDsh(sessionId, 'source-dsh-session-revoked')
return revoked
}
countKind(kind) {
this.prune()
let count = 0
for (const session of this.sessions.values()) {
if (session.kind === kind) count += 1
}
return count
}
list(account, currentSessionId) {
this.prune()
const currentDshSessionId = this.sourceDshSessionId(currentSessionId) ?? currentSessionId
const managementSources = new Set([...this.sessions.values()]
.filter(session => session.kind === 'management')
.map(session => session.sourceDshSessionId))
return [...this.sessions.values()]
.filter(session => session.accountId === account.accountId
&& session.kind === 'dsh'
&& session.mainCredentialVersion === account.mainCredential.version
&& session.managementAccessVersion === account.managementAccess.version)
.map(session => Object.freeze({
sessionId: session.sessionId,
origin: session.origin,
current: session.sessionId === currentDshSessionId,
managementActive: managementSources.has(session.sessionId),
ip: session.client.ip,
userAgent: session.client.userAgent,
createdAt: new Date(session.createdAt).toISOString(),
lastSeenAt: new Date(session.lastSeenAt).toISOString(),
expiresAt: new Date(session.expiresAt).toISOString(),
}))
.sort((left, right) => right.lastSeenAt.localeCompare(left.lastSeenAt))
}
revokeDshSession(sessionId, accountId) {
if (typeof sessionId !== 'string' || typeof accountId !== 'string') return false
for (const [key, session] of this.sessions) {
if (session.kind !== 'dsh' || session.sessionId !== sessionId || session.accountId !== accountId) continue
this.sessions.delete(key)
this.revokeManagementFromDsh(session.sessionId, 'source-dsh-session-revoked')
return true
}
return false
}
revokeManagementFromDsh(sessionId, reason = 'source-dsh-session') {
let revoked = 0
for (const [key, session] of this.sessions) {
if (session.kind !== 'management' || session.sourceDshSessionId !== sessionId) continue
this.sessions.delete(key)
this.emit('access.session.invalidated', session, { reason })
revoked += 1
}
return revoked
}
sourceDshSessionId(sessionId) {
for (const session of this.sessions.values()) {
if (session.kind === 'management' && session.sessionId === sessionId) return session.sourceDshSessionId
}
return null
}
refreshDshSession(sessionId, account) {
const session = this.activeDshSession(sessionId, account.accountId)
if (session === undefined) return false
session.mainCredentialVersion = account.mainCredential.version
session.managementAccessVersion = account.managementAccess.version
this.emit('access.session.refreshed', session, { reason: 'management-access-change' })
return true
}
details(sessionId) {
this.prune()
for (const session of this.sessions.values()) {
if (session.sessionId !== sessionId) continue
return Object.freeze({
sessionId: session.sessionId,
kind: session.kind,
origin: session.origin,
sourceDshOrigin: session.sourceDshOrigin,
sourceDshSessionId: session.sourceDshSessionId,
authenticationSource: session.authenticationSource,
})
}
return undefined
}
revokeAll() {
const revoked = this.sessions.size
this.sessions.clear()
return revoked
}
}
upstream/container/control-plane/services/access-manager/lib/store.mjs
+249
View File
@@ -0,0 +1,249 @@
import { randomBytes, randomUUID } from 'node:crypto'
import { chmod, mkdir, open, readFile, rename, rm } from 'node:fs/promises'
import { dirname, join } from 'node:path'
import { AccessError } from './errors.mjs'
import { createCredential, normalizeUsername, validCredential } from './credentials.mjs'
import { validTotpSecret } from './totp.mjs'
const STATES = new Set(['never-initialized', 'migration-required', 'initialized', 'recovery-required'])
function identifier(random = randomBytes) { return random(32).toString('base64url') }
function timestamp(now) { return now().toISOString() }
async function syncDirectory(root) {
const handle = await open(root, 'r')
try { await handle.sync() } finally { await handle.close() }
}
async function atomicJson(path, value, mode = 0o600) {
const temporary = join(dirname(path), `.${path.split('/').at(-1)}.${randomUUID()}.tmp`)
const handle = await open(temporary, 'wx', mode)
try {
await handle.writeFile(`${JSON.stringify(value)}\n`, 'utf8')
await handle.sync()
} finally { await handle.close() }
await chmod(temporary, mode)
await rename(temporary, path)
await syncDirectory(dirname(path))
}
async function createExclusiveJson(path, value, mode = 0o600) {
const handle = await open(path, 'wx', mode)
try {
await handle.writeFile(`${JSON.stringify(value)}\n`, 'utf8')
await handle.sync()
} finally { await handle.close() }
await chmod(path, mode)
await syncDirectory(dirname(path))
}
async function readJson(path) {
try { return { exists: true, value: JSON.parse(await readFile(path, 'utf8')) } }
catch (error) {
if (error?.code === 'ENOENT') return { exists: false, value: undefined }
return { exists: true, error }
}
}
export function validInitialization(value) {
return value !== null && typeof value === 'object' && value.schema === 1
&& typeof value.instanceId === 'string' && Buffer.from(value.instanceId, 'base64url').byteLength === 32
&& STATES.has(value.state) && Number.isFinite(Date.parse(value.createdAt)) && Number.isFinite(Date.parse(value.updatedAt))
}
export function validAccount(value) {
const additional = value?.managementAdditionalCredential
const totp = value?.totp
const access = value?.managementAccess
const validOrigin = origin => {
try {
const parsed = new URL(origin)
return ['http:', 'https:'].includes(parsed.protocol) && parsed.origin !== 'null'
&& parsed.username === '' && parsed.password === '' && parsed.pathname === '/'
&& parsed.search === '' && parsed.hash === '' && origin === parsed.origin
} catch { return false }
}
const additionalValid = additional !== null && typeof additional === 'object'
&& typeof additional.enabled === 'boolean'
&& Number.isInteger(additional.version) && additional.version >= 1
&& Number.isFinite(Date.parse(additional.changedAt))
&& (additional.enabled ? validCredential(additional.verifier) : additional.verifier === null)
const totpValid = totp === undefined || (totp !== null && typeof totp === 'object'
&& typeof totp.enabled === 'boolean'
&& Number.isInteger(totp.version) && totp.version >= 1
&& Number.isFinite(Date.parse(totp.changedAt))
&& (totp.enabled ? validTotpSecret(totp.secret) : totp.secret === null))
const isolatedEntryValid = access?.mode === 'compat'
? access.isolatedEntry === null && access.dshPublicOrigin === null
: access?.mode === 'isolated' && access.isolatedEntry !== null && typeof access.isolatedEntry === 'object'
&& validOrigin(access.dshPublicOrigin)
&& (access.isolatedEntry.kind === 'local-only' && validOrigin(access.isolatedEntry.managementLocalOrigin)
|| (access.isolatedEntry.kind === 'public' && (() => {
try {
const parsed = new URL(access.isolatedEntry.managementPublicOrigin)
return ['http:', 'https:'].includes(parsed.protocol) && parsed.origin !== 'null'
&& parsed.username === '' && parsed.password === '' && parsed.pathname === '/'
&& parsed.search === '' && parsed.hash === ''
&& access.isolatedEntry.managementPublicOrigin === parsed.origin
} catch { return false }
})()))
return value !== null && typeof value === 'object' && value.schema === 1
&& typeof value.revision === 'string' && Buffer.from(value.revision, 'base64url').byteLength === 32
&& typeof value.accountId === 'string' && Buffer.from(value.accountId, 'base64url').byteLength === 32
&& (() => { try { return normalizeUsername(value.username) === value.username } catch { return false } })()
&& validCredential(value.mainCredential)
&& additionalValid && totpValid && isolatedEntryValid
&& Number.isInteger(access?.version) && access.version >= 1
&& Number.isFinite(Date.parse(access.changedAt))
&& Number.isFinite(Date.parse(value.createdAt)) && Number.isFinite(Date.parse(value.updatedAt))
}
function validEvidence(evidence) {
if (evidence === null || typeof evidence !== 'object' || Array.isArray(evidence)) return false
const entries = Object.entries(evidence)
return entries.length > 0 && entries.every(([key, value]) => (
/^[a-z][A-Za-z0-9]{0,63}$/.test(key) && typeof value === 'boolean'
))
}
export class AccessStateStore {
constructor({ root, now = () => new Date(), random = randomBytes, createVerifier = createCredential } = {}) {
if (typeof root !== 'string') throw new TypeError('Access state root is required')
this.root = root
this.initializationPath = join(root, 'initialization.json')
this.accountPath = join(root, 'account.json')
this.now = now
this.random = random
this.createVerifier = createVerifier
}
async prepare() {
await mkdir(this.root, { recursive: true, mode: 0o700 })
await chmod(this.root, 0o700)
}
async inspect() {
const [initialization, account] = await Promise.all([
readJson(this.initializationPath), readJson(this.accountPath),
])
return {
initialization: initialization.error === undefined && validInitialization(initialization.value)
? initialization.value : undefined,
initializationExists: initialization.exists,
initializationDamaged: initialization.error !== undefined || (initialization.exists && !validInitialization(initialization.value)),
account: account.error === undefined && validAccount(account.value) ? account.value : undefined,
accountExists: account.exists,
accountDamaged: account.error !== undefined || (account.exists && !validAccount(account.value)),
}
}
async transition(initialization, state) {
const next = { ...initialization, state, updatedAt: timestamp(this.now) }
await atomicJson(this.initializationPath, next)
return next
}
async classify(evidence) {
if (!validEvidence(evidence)) throw new AccessError('CLASSIFICATION_EVIDENCE_INVALID', 'classification evidence is invalid')
await this.prepare()
const inspected = await this.inspect()
if (inspected.initialization !== undefined) {
if (inspected.initialization.state === 'never-initialized' && inspected.account !== undefined) {
const initialization = await this.transition(inspected.initialization, 'initialized')
return { initialization, account: inspected.account }
}
if (inspected.initialization.state === 'initialized' && inspected.account === undefined) {
const initialization = await this.transition(inspected.initialization, 'recovery-required')
return { initialization, account: undefined }
}
if (inspected.accountDamaged) {
const initialization = inspected.initialization.state === 'recovery-required'
? inspected.initialization : await this.transition(inspected.initialization, 'recovery-required')
return { initialization, account: undefined }
}
return { initialization: inspected.initialization, account: inspected.account }
}
if (inspected.initializationDamaged || inspected.accountExists) {
const initialization = {
schema: 1, instanceId: identifier(this.random), state: 'recovery-required',
createdAt: timestamp(this.now), updatedAt: timestamp(this.now),
}
await atomicJson(this.initializationPath, initialization)
return { initialization, account: undefined }
}
const persistentEvidence = Object.entries(evidence)
.some(([key, value]) => key !== 'legacyAuthenticationConfigured' && value)
const initialization = {
schema: 1,
instanceId: identifier(this.random),
state: persistentEvidence ? 'migration-required' : 'never-initialized',
createdAt: timestamp(this.now),
updatedAt: timestamp(this.now),
}
try { await createExclusiveJson(this.initializationPath, initialization) }
catch (error) {
if (error?.code === 'EEXIST') return this.classify(evidence)
throw error
}
return { initialization, account: undefined }
}
async state() {
const inspected = await this.inspect()
if (inspected.initialization === undefined) {
return { initialization: undefined, account: undefined, state: 'classification-pending' }
}
if (inspected.initialization.state === 'initialized' && inspected.account === undefined) {
return { initialization: inspected.initialization, account: undefined, state: 'recovery-required' }
}
return { initialization: inspected.initialization, account: inspected.account, state: inspected.initialization.state }
}
async createAccount({ username, password }, allowedState) {
const current = await this.state()
if (current.state !== allowedState) {
throw new AccessError('ALREADY_INITIALIZED', 'administrator initialization is unavailable', 409)
}
const normalized = normalizeUsername(username)
const now = timestamp(this.now)
const verifier = await this.createVerifier(password, { now: this.now, random: this.random })
const account = {
schema: 1,
revision: identifier(this.random),
accountId: identifier(this.random),
username: normalized,
mainCredential: verifier,
totp: { enabled: false, version: 1, secret: null, changedAt: now },
managementAdditionalCredential: { enabled: false, version: 1, verifier: null, changedAt: now },
managementAccess: {
mode: 'compat', version: 1, isolatedEntry: null, dshPublicOrigin: null, changedAt: now,
},
createdAt: now,
updatedAt: now,
}
await atomicJson(this.accountPath, account)
await this.transition(current.initialization, 'initialized')
return account
}
initialize(value) { return this.createAccount(value, 'never-initialized') }
migrate(value) { return this.createAccount(value, 'migration-required') }
recover(value) { return this.createAccount(value, 'recovery-required') }
async replaceAccount(account, revision) {
const current = await this.state()
if (current.state !== 'initialized' || current.account === undefined) {
throw new AccessError('ACCESS_NOT_INITIALIZED', 'administrator access is not initialized', 409)
}
if (current.account.revision !== revision) throw new AccessError('REVISION_CONFLICT', 'account revision changed', 409)
if (!validAccount(account)) throw new AccessError('ACCOUNT_INVALID', 'account record is invalid')
await atomicJson(this.accountPath, account)
return account
}
async clear() {
await rm(this.accountPath, { force: true })
}
}
upstream/container/control-plane/services/access-manager/lib/totp.mjs
+293
View File
@@ -0,0 +1,293 @@
import { createHash, createHmac, randomBytes, timingSafeEqual } from 'node:crypto'
const BASE32_ALPHABET = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ234567'
const CODE_PATTERN = /^\d{6}$/
function digest(value) { return createHash('sha256').update(value).digest('hex') }
function token(prefix, random) { return `${prefix}_${random(32).toString('base64url')}` }
function base32Encode(value) {
let bits = 0
let buffer = 0
let output = ''
for (const byte of value) {
buffer = (buffer << 8) | byte
bits += 8
while (bits >= 5) {
output += BASE32_ALPHABET[(buffer >>> (bits - 5)) & 31]
bits -= 5
}
}
if (bits > 0) output += BASE32_ALPHABET[(buffer << (5 - bits)) & 31]
return output
}
function base32Decode(value) {
if (typeof value !== 'string' || value.length === 0 || /[^A-Z2-7]/u.test(value)) return undefined
let bits = 0
let buffer = 0
const output = []
for (const character of value) {
buffer = (buffer << 5) | BASE32_ALPHABET.indexOf(character)
bits += 5
if (bits >= 8) {
output.push((buffer >>> (bits - 8)) & 255)
bits -= 8
}
}
return Buffer.from(output)
}
function counterBuffer(counter) {
const value = Buffer.alloc(8)
value.writeBigUInt64BE(BigInt(counter))
return value
}
export function generateTotpSecret(random = randomBytes) {
return base32Encode(random(20))
}
export function validTotpSecret(secret) {
return base32Decode(secret)?.byteLength === 20
}
export function totpCode(secret, { now = Date.now(), offset = 0 } = {}) {
const key = base32Decode(secret)
if (key?.byteLength !== 20) throw new TypeError('TOTP secret is invalid')
const counter = Math.floor(now / 30_000) + offset
const hmac = createHmac('sha1', key).update(counterBuffer(counter)).digest()
const position = hmac.at(-1) & 15
const value = (hmac.readUInt32BE(position) & 0x7fffffff) % 1_000_000
return String(value).padStart(6, '0')
}
export function verifyTotpCode(code, secret, { now = Date.now(), window = 1 } = {}) {
if (typeof code !== 'string' || !CODE_PATTERN.test(code)) return false
const actual = Buffer.from(code)
for (let offset = -window; offset <= window; offset += 1) {
const expected = Buffer.from(totpCode(secret, { now, offset }))
if (timingSafeEqual(actual, expected)) return true
}
return false
}
export function totpUri({ secret, username, issuer = 'DSH Docker' }) {
if (!validTotpSecret(secret)) throw new TypeError('TOTP secret is invalid')
const label = `${issuer}:${username}`
const query = new URLSearchParams({ secret, issuer, algorithm: 'SHA1', digits: '6', period: '30' })
return `otpauth://totp/${encodeURIComponent(label)}?${query}`
}
export class TotpFlowStore {
constructor({ now = Date.now, random = randomBytes, enrollmentTtlMs = 5 * 60_000, loginTtlMs = 5 * 60_000 } = {}) {
this.now = now
this.random = random
this.enrollmentTtlMs = enrollmentTtlMs
this.loginTtlMs = loginTtlMs
this.enrollments = new Map()
this.disableConfirmations = new Map()
this.logins = new Map()
}
prune() {
const current = this.now()
for (const [key, value] of this.enrollments) if (value.expiresAt <= current) this.enrollments.delete(key)
for (const [key, value] of this.disableConfirmations) if (value.expiresAt <= current) this.disableConfirmations.delete(key)
for (const [key, value] of this.logins) if (value.expiresAt <= current) this.logins.delete(key)
}
createEnrollment(account, sessionId) {
this.prune()
for (const [key, value] of this.enrollments) {
if (value.accountId === account.accountId && value.sessionId === sessionId) this.enrollments.delete(key)
}
const value = token('dshte', this.random)
const secret = generateTotpSecret(this.random)
const expiresAt = this.now() + this.enrollmentTtlMs
this.enrollments.set(digest(value), {
accountId: account.accountId,
accountRevision: account.revision,
sessionId,
secret,
confirmed: false,
expiresAt,
})
return Object.freeze({ token: value, secret, expiresAt: new Date(expiresAt).toISOString() })
}
enrollment(value, account, sessionId) {
this.prune()
if (typeof value !== 'string' || value.length > 512) return undefined
const key = digest(value)
const enrollment = this.enrollments.get(key)
if (enrollment === undefined || enrollment.accountId !== account.accountId
|| enrollment.accountRevision !== account.revision || enrollment.sessionId !== sessionId) return undefined
return { key, value: enrollment }
}
cancelEnrollment(value, account, sessionId) {
const enrollment = this.enrollment(value, account, sessionId)
return enrollment === undefined ? false : this.enrollments.delete(enrollment.key)
}
confirmEnrollment(key) {
const enrollment = this.enrollments.get(key)
if (enrollment === undefined) return false
enrollment.confirmed = true
return true
}
consumeEnrollment(key) { return this.enrollments.delete(key) }
createDisableConfirmation(account, sessionId) {
this.prune()
for (const [key, value] of this.disableConfirmations) {
if (value.accountId === account.accountId && value.sessionId === sessionId) this.disableConfirmations.delete(key)
}
const value = token('dshtd', this.random)
const expiresAt = this.now() + this.enrollmentTtlMs
this.disableConfirmations.set(digest(value), {
accountId: account.accountId,
accountRevision: account.revision,
sessionId,
confirmed: false,
expiresAt,
})
return Object.freeze({ token: value, expiresAt: new Date(expiresAt).toISOString() })
}
disableConfirmation(value, account, sessionId) {
this.prune()
if (typeof value !== 'string' || value.length > 512) return undefined
const key = digest(value)
const confirmation = this.disableConfirmations.get(key)
if (confirmation === undefined || confirmation.accountId !== account.accountId
|| confirmation.accountRevision !== account.revision || confirmation.sessionId !== sessionId) return undefined
return { key, value: confirmation }
}
cancelDisableConfirmation(value, account, sessionId) {
const confirmation = this.disableConfirmation(value, account, sessionId)
return confirmation === undefined ? false : this.disableConfirmations.delete(confirmation.key)
}
confirmDisable(key) {
const confirmation = this.disableConfirmations.get(key)
if (confirmation === undefined) return false
confirmation.confirmed = true
return true
}
consumeDisableConfirmation(key) { return this.disableConfirmations.delete(key) }
createLogin(account, details) {
this.prune()
const value = token('dshtl', this.random)
const expiresAt = this.now() + this.loginTtlMs
this.logins.set(digest(value), {
accountId: account.accountId,
accountRevision: account.revision,
...details,
expiresAt,
})
return Object.freeze({ token: value, expiresAt: new Date(expiresAt).toISOString() })
}
login(value, account) {
this.prune()
if (typeof value !== 'string' || value.length > 512) return undefined
const key = digest(value)
const login = this.logins.get(key)
if (login === undefined || login.accountId !== account.accountId
|| login.accountRevision !== account.revision) return undefined
return { key, value: login }
}
consumeLogin(key) { return this.logins.delete(key) }
clear() {
this.enrollments.clear()
this.disableConfirmations.clear()
this.logins.clear()
}
}
export class TotpRetryLimiter {
constructor({
now = Date.now,
threshold = 3,
retryMs = 10_000,
dailyWindowMs = 24 * 60 * 60_000,
sourceDailyLimit = 25,
globalDailyLimit = 50,
} = {}) {
this.now = now
this.threshold = threshold
this.retryMs = retryMs
this.dailyWindowMs = dailyWindowMs
this.sourceDailyLimit = sourceDailyLimit
this.globalDailyLimit = globalDailyLimit
this.entries = new Map()
this.globalFailures = new Map()
}
key(accountId, source) { return `${accountId}\u0000${source}` }
prune(values) {
const cutoff = this.now() - this.dailyWindowMs
while (values.length > 0 && values[0] <= cutoff) values.shift()
}
retry(accountId, source) {
const entry = this.entries.get(this.key(accountId, source))
const global = this.globalFailures.get(accountId) ?? []
if (entry !== undefined) this.prune(entry.failuresInWindow)
this.prune(global)
const until = values => Math.max(1, Math.ceil((values[0] + this.dailyWindowMs - this.now()) / 1000))
if (global.length >= this.globalDailyLimit) return { kind: 'rate', retryAfterSeconds: until(global) }
if (entry?.failuresInWindow.length >= this.sourceDailyLimit) {
return { kind: 'rate', retryAfterSeconds: until(entry.failuresInWindow) }
}
if (entry !== undefined && entry.blockedUntil > this.now()) {
return { kind: 'retry', retryAfterSeconds: Math.max(1, Math.ceil((entry.blockedUntil - this.now()) / 1000)) }
}
return { kind: null, retryAfterSeconds: 0 }
}
fail(accountId, source) {
const key = this.key(accountId, source)
const entry = this.entries.get(key) ?? { consecutiveFailures: 0, blockedUntil: 0, failuresInWindow: [] }
const global = this.globalFailures.get(accountId) ?? []
this.prune(entry.failuresInWindow)
this.prune(global)
entry.consecutiveFailures += 1
entry.failuresInWindow.push(this.now())
global.push(this.now())
if (entry.consecutiveFailures >= this.threshold) entry.blockedUntil = this.now() + this.retryMs
this.entries.set(key, entry)
this.globalFailures.set(accountId, global)
return this.retry(accountId, source)
}
succeed(accountId, source) {
const entry = this.entries.get(this.key(accountId, source))
if (entry === undefined) return
entry.consecutiveFailures = 0
entry.blockedUntil = 0
}
clearDailyLimits(accountId, { globalOnly = false } = {}) {
let cleared = this.globalFailures.delete(accountId) ? 1 : 0
if (globalOnly) return { cleared }
for (const [key, entry] of this.entries) {
if (!key.startsWith(`${accountId}\u0000`)) continue
if (entry.failuresInWindow.length > 0) {
entry.failuresInWindow = []
cleared += 1
}
}
return { cleared }
}
}
upstream/container/control-plane/services/access-manager/lib/transitions.mjs
+120
View File
@@ -0,0 +1,120 @@
import { createHash, randomBytes, timingSafeEqual } from 'node:crypto'
function digest(value) { return createHash('sha256').update(value).digest() }
function token(prefix, random, bytes = 32) { return `${prefix}_${random(bytes).toString('base64url')}` }
function sameDigest(value, expected) {
if (typeof value !== 'string' || expected === null) return false
const actual = digest(value)
return actual.byteLength === expected.byteLength && timingSafeEqual(actual, expected)
}
export class ManagementTransitionStore {
constructor({ now = Date.now, random = randomBytes, ttlMs = 30_000 } = {}) {
this.now = now
this.random = random
this.ttlMs = ttlMs
this.transitions = new Map()
this.continuations = new Map()
}
prune() {
const now = this.now()
for (const [id, value] of this.transitions) if (value.expiresAt <= now) this.transitions.delete(id)
for (const [key, value] of this.continuations) if (value.expiresAt <= now) this.continuations.delete(key)
}
create({ account, instanceId, sessionId, sourceOrigin, sourceDshOrigin, sourceDshSessionId, mode, isolatedEntry, candidateOrigin }) {
this.prune()
const transitionId = token('dshmt', this.random, 16)
const nonce = candidateOrigin === null ? null : token('dshmn', this.random)
const expiresAt = this.now() + this.ttlMs
this.transitions.set(transitionId, {
accountId: account.accountId,
accountRevision: account.revision,
managementAccessVersion: account.managementAccess.version,
instanceId,
sessionId,
sourceOrigin,
sourceDshOrigin,
sourceDshSessionId,
mode,
isolatedEntry,
candidateOrigin,
nonceDigest: nonce === null ? null : digest(nonce),
proofDigest: null,
expiresAt,
})
return Object.freeze({
transitionId,
nonce,
candidateOrigin,
expiresAt: new Date(expiresAt).toISOString(),
})
}
probe({ transitionId, nonce, candidateOrigin, sourceOrigin, instanceId }) {
this.prune()
const transition = this.transitions.get(transitionId)
if (transition === undefined || transition.candidateOrigin !== candidateOrigin
|| transition.sourceOrigin !== sourceOrigin || transition.instanceId !== instanceId
|| !sameDigest(nonce, transition.nonceDigest)) return undefined
transition.nonceDigest = null
const proof = token('dshmpf', this.random)
transition.proofDigest = digest(proof)
return Object.freeze({ proof, expiresAt: new Date(transition.expiresAt).toISOString() })
}
inspect({ transitionId, proof, account, sessionId }) {
this.prune()
const transition = this.transitions.get(transitionId)
if (transition === undefined || transition.accountId !== account.accountId
|| transition.accountRevision !== account.revision
|| transition.managementAccessVersion !== account.managementAccess.version
|| transition.sessionId !== sessionId
|| (transition.candidateOrigin !== null && !sameDigest(proof, transition.proofDigest))) return undefined
return transition
}
consume(value) {
const transition = this.inspect(value)
if (transition === undefined) return undefined
const { transitionId } = value
this.transitions.delete(transitionId)
return transition
}
createContinuation({ account, targetOrigin, sourceDshOrigin, sourceDshSessionId }) {
if (targetOrigin === null) return null
this.prune()
const continuation = token('dshmc', this.random)
const expiresAt = this.now() + this.ttlMs
this.continuations.set(digest(continuation).toString('hex'), {
accountId: account.accountId,
accountRevision: account.revision,
managementAccessVersion: account.managementAccess.version,
targetOrigin,
sourceDshOrigin,
sourceDshSessionId,
expiresAt,
})
return Object.freeze({ token: continuation, expiresAt: new Date(expiresAt).toISOString() })
}
consumeContinuation({ token: value, account, targetOrigin }) {
this.prune()
if (typeof value !== 'string' || value.length > 512) return undefined
const key = digest(value).toString('hex')
const continuation = this.continuations.get(key)
this.continuations.delete(key)
if (continuation === undefined || continuation.accountId !== account.accountId
|| continuation.accountRevision !== account.revision
|| continuation.managementAccessVersion !== account.managementAccess.version
|| continuation.targetOrigin !== targetOrigin) return undefined
return continuation
}
clear() {
this.transitions.clear()
this.continuations.clear()
}
}
upstream/container/control-plane/services/access-manager/supervisor.mjs
+49
View File
@@ -0,0 +1,49 @@
#!/usr/bin/env node
import { LocalApiClient } from '../../modules/updater/lib/client.mjs'
import { PlatformPaths } from '../../../platform/lib/paths.mjs'
const paths = new PlatformPaths(
process.env.DSH_PLATFORM_DATA ?? '/data/platform',
process.env.DSH_PLATFORM_RUN ?? '/run/dsh-platform',
)
const launch = new LocalApiClient(paths.accessLaunchSocket)
if (process.argv[2] === 'health') {
const status = await launch.request('GET', '/v1/status')
process.exit(status.componentReady === true ? 0 : 1)
}
const token = process.env.DSH_ACCESS_LAUNCH_TOKEN
delete process.env.DSH_ACCESS_LAUNCH_TOKEN
if (typeof token !== 'string' || token.length < 32) throw new Error('Access Manager launch token is unavailable')
await launch.request('POST', '/v1/start', { token })
let stopping = false
let resolveStop
const stopped = new Promise(resolve => { resolveStop = resolve })
const stop = () => {
if (stopping) return
stopping = true
void launch.request('POST', '/v1/stop', { token }).catch(() => {}).finally(resolveStop)
}
process.once('SIGINT', stop)
process.once('SIGTERM', stop)
const poll = setInterval(() => {
void launch.request('GET', '/v1/status').then(status => {
if (!stopping && status.running !== true) {
clearInterval(poll)
process.exitCode = 1
resolveStop()
}
}, () => {
if (!stopping) {
clearInterval(poll)
process.exitCode = 1
resolveStop()
}
})
}, 500)
await stopped
clearInterval(poll)
upstream/container/control-plane/services/gateway/component.json
+31
View File
@@ -0,0 +1,31 @@
{
"schema": 1,
"id": "gateway",
"type": "service",
"command": {
"executable": "/usr/local/bin/node",
"args": ["/run/dsh-platform/views/bootstrap/control-plane/services/gateway/index.mjs"],
"timeoutSeconds": 30
},
"environment": {},
"lifecycle": {
"prepare": null,
"preStart": null,
"postStart": null,
"preStop": null,
"stop": null,
"postStop": null
},
"health": {
"type": "http",
"host": "127.0.0.1",
"port": 3080,
"path": "/_dsh_gateway/health",
"intervalSeconds": 1,
"timeoutSeconds": 30
},
"logging": {
"stdout": true,
"stderr": true
}
}
upstream/container/control-plane/services/gateway/index.mjs
+38
View File
@@ -0,0 +1,38 @@
#!/usr/bin/env node
import { loadConfig } from './lib/config.mjs'
import { UsageError } from './lib/errors.mjs'
import { runGateway } from './lib/lifecycle.mjs'
import { JsonlLogManager } from '../../modules/log-manager/index.mjs'
import { PlatformPaths } from '../../../platform/lib/paths.mjs'
const paths = new PlatformPaths(
process.env.DSH_PLATFORM_DATA ?? '/data/platform',
process.env.DSH_PLATFORM_RUN ?? '/run/dsh-platform',
)
const logs = new JsonlLogManager({
root: paths.logsRoot,
maxBytes: Number(process.env.DSH_LOG_MAX_BYTES ?? 104857600),
retentionDays: Number(process.env.DSH_LOG_RETENTION_DAYS ?? 14),
output: { stdout: process.stdout, stderr: process.stderr },
})
logs.on('error', error => { void logs.diagnostic('log-manager', 'capture.failed', { error }) })
const report = (message, fields = {}) => message === 'gateway.access.login-failed'
? logs.diagnosticRateLimited(
`gateway.access.login-failed:${typeof fields.code === 'string' ? fields.code.toLowerCase() : 'unknown'}`,
'gateway', message, fields,
)
: logs.diagnostic('gateway', message, fields)
try {
const config = await loadConfig()
process.exitCode = await runGateway(config, { report, managementPort: config.managementPort })
} catch (error) {
if (error instanceof UsageError) {
await logs.diagnostic('gateway', 'gateway.configuration.failed', { error, level: 'warning' })
process.exitCode = error.exitCode
} else {
await logs.diagnostic('gateway', 'gateway.process.failed', { error })
process.exitCode = 1
}
}
upstream/container/control-plane/services/gateway/lib/availability.mjs
+151
View File
@@ -0,0 +1,151 @@
import { request as httpRequest } from 'node:http'
// A Bootstrap handoff stops DSH while the updater that initiated it is still
// recorded as building the candidate. This state is consulted only after the
// DSH upstream becomes unavailable, so ordinary candidate builds stay visible.
const SWITCHING_STATES = new Set(['building-candidate', 'snapshotting-data', 'switching', 'probation'])
const RECOVERING_STATES = new Set(['restoring-data'])
const COPY = Object.freeze({
en: Object.freeze({
starting: 'DeepSeek Harness is starting',
stopping: 'DeepSeek Harness is stopping',
stopped: 'DeepSeek Harness is stopped',
restarting: 'DeepSeek Harness is restarting',
switching: 'Switching the DeepSeek Harness runtime',
'runtime-recovering': 'Restoring the DeepSeek Harness runtime',
recovering: 'DeepSeek Harness stopped unexpectedly and recovery is in progress',
failed: 'DeepSeek Harness could not start',
'plugin-failed': 'DeepSeek Harness plugins repeatedly failed to load',
unavailable: 'DeepSeek Harness is temporarily unavailable and recovery is in progress',
}),
zh: Object.freeze({
starting: 'DeepSeek Harness 正在启动',
stopping: 'DeepSeek Harness 正在停止',
stopped: 'DeepSeek Harness 已停止',
restarting: 'DeepSeek Harness 正在重新启动',
switching: '正在切换 DeepSeek Harness 运行版本',
'runtime-recovering': '正在恢复 DeepSeek Harness 运行版本',
recovering: 'DeepSeek Harness 意外停止，正在尝试恢复',
failed: 'DeepSeek Harness 启动失败',
'plugin-failed': 'DeepSeek Harness 插件持续加载失败',
unavailable: 'DeepSeek Harness 暂时不可用，正在尝试恢复',
}),
})
const MANAGEMENT_LINK = Object.freeze({
en: 'Open DSH Management Console for diagnostics and recovery',
zh: '打开 DSH 管理中心进行检查和恢复',
})
const REFRESH_LINK = Object.freeze({ en: 'Reload page', zh: '刷新页面' })
export class DshAvailability {
constructor({ now = () => Date.now(), failures = 3, failureWindowMs = 1_500 } = {}) {
this.now = now
this.failures = failures
this.failureWindowMs = failureWindowMs
this.everReady = false
this.consecutiveFailures = 0
this.firstFailureAt = null
}
observe(ready) {
if (ready) {
this.everReady = true
this.consecutiveFailures = 0
this.firstFailureAt = null
return
}
this.consecutiveFailures += 1
this.firstFailureAt ??= this.now()
}
classify(platform = {}) {
if (platform.operation === 'restart-failed') return 'failed'
if (platform.operation === 'restarting') return 'restarting'
if (platform.operation === 'recovering') return 'runtime-recovering'
if (platform.operation === 'switching') return 'switching'
const updateState = platform.update?.status
if (RECOVERING_STATES.has(updateState)) return 'runtime-recovering'
if (SWITCHING_STATES.has(updateState)) return 'switching'
const lifecycle = platform.dshLifecycle ?? {}
if (['starting', 'stopping', 'stopped', 'restarting', 'recovering', 'failed'].includes(lifecycle.state)) {
return lifecycle.state
}
if (platform.recoveryMode !== null && platform.recoveryMode !== undefined) return 'failed'
if (!this.everReady) return 'starting'
if (
this.consecutiveFailures >= this.failures
&& this.firstFailureAt !== null
&& this.now() - this.firstFailureAt >= this.failureWindowMs
) return 'unavailable'
return 'unknown'
}
}
export function probeDsh({ host, port, timeoutMs = 750 }) {
return new Promise(resolve => {
const request = httpRequest({ hostname: host, port, path: '/', method: 'GET' }, response => {
response.resume()
resolve((response.statusCode ?? 500) < 500)
})
request.setTimeout(timeoutMs, () => request.destroy())
request.once('error', () => resolve(false))
request.end()
})
}
function cookieLocale(cookie = '') {
for (const part of cookie.split(';')) {
const [name, value] = part.trim().split('=', 2)
if (name === 'dsh_locale' && (value === 'zh' || value === 'en')) return value
}
return undefined
}
export function language(headers) {
const saved = cookieLocale(headers.cookie)
if (saved !== undefined) return saved
for (const item of (headers['accept-language'] ?? '').split(',')) {
const primary = item.trim().split(';', 1)[0].split('-', 1)[0].toLowerCase()
if (primary === 'zh' || primary === 'en') return primary
}
return 'en'
}
export function stateMessage(state, headers = {}, lifecycle = {}) {
const locale = language(headers)
const message = COPY[locale][state] ?? COPY.en.unavailable
if (state !== 'recovering' || !(lifecycle.attempt > 0) || !(lifecycle.maxAttempts > 0)) return message
return locale === 'zh'
? `${message}（第 ${String(lifecycle.attempt)} / ${String(lifecycle.maxAttempts)} 次）`
: `${message} (attempt ${String(lifecycle.attempt)} of ${String(lifecycle.maxAttempts)})`
}
export function availabilityPage(state, headers = {}, { lifecycle = {}, returnPath = null, poll = true, allowRefresh = false, managementHref = '/_dsh_platform/console/' } = {}) {
const locale = language(headers)
const message = stateMessage(state, headers, lifecycle)
const managementLink = MANAGEMENT_LINK[locale]
const refreshLink = REFRESH_LINK[locale]
const target = JSON.stringify(returnPath).replaceAll('<', '\\u003c')
return `<!doctype html>
<html lang="${locale === 'zh' ? 'zh-CN' : 'en'}">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>DeepSeek Harness</title>
<style>
html{height:100%}body{min-height:100%;margin:0;display:grid;place-items:center;background:#151517;color:#f3f3f4;font-family:Inter,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif}.boot{display:flex;flex-direction:column;align-items:center;gap:16px;padding:24px;text-align:center}.wordmark{font-size:16px;line-height:24px;font-weight:600;letter-spacing:0}.status{max-width:520px;font-size:13px;line-height:20px;color:#a7a7ad}.actions{display:flex;flex-wrap:wrap;justify-content:center;gap:10px}.management{padding:7px 12px;border:1px solid #46464b;border-radius:6px;color:#d7d7db;font-size:13px;line-height:20px;text-decoration:none}.management:hover{background:#232326}.management:focus-visible{outline:2px solid #8ca8ff;outline-offset:2px}@media(prefers-color-scheme:light){body{background:#f9fafb;color:#0f1115}.status{color:#6d7178}.management{border-color:#d6d8dc;color:#34373d}.management:hover{background:#f0f1f3}}
</style>
</head>
<body>
<main class="boot"><div class="wordmark">HARNESS</div><div class="status" role="status">${message}</div><div class="actions"><a class="management" href="${String(managementHref).replaceAll('"', '&quot;')}">${managementLink}</a>${allowRefresh ? `<a class="management refresh" href="">${refreshLink}</a>` : ''}</div></main>
${poll ? `<script>
const returnPath=${target};const status=document.querySelector('.status');const management=document.querySelector('.management:not(.refresh)');
async function check(){try{const response=await fetch('/_dsh_gateway/readiness',{cache:'no-store'});const value=await response.json();if(typeof value.managementHref==='string')management.href=value.managementHref;if(value.ready){if(returnPath===null)location.reload();else location.replace(returnPath);return}if(typeof value.message==='string')status.textContent=value.message}catch{}setTimeout(check,1000)}
setTimeout(check,600);
</script>` : ''}
</body>
</html>`
}
upstream/container/control-plane/services/gateway/lib/browser-auth.mjs
+1088
View File
File diff suppressed because it is too large. Load diff
upstream/container/control-plane/services/gateway/lib/config.mjs
+75
View File
@@ -0,0 +1,75 @@
import { UsageError } from './errors.mjs'
export function parseBoolean(name, value, fallback) {
const resolved = value ?? String(fallback)
if (resolved === 'true') return true
if (resolved === 'false') return false
throw new UsageError(`${name} must be true or false`)
}
function parseUrlAuthority(value) {
try {
return new URL(`http://${value}`)
} catch {
return undefined
}
}
function canonicalAuthority(value, parsed) {
const port = parsed.port !== '' ? parsed.port : new URL(`https://${value}`).port
return port === '' ? parsed.hostname : `${parsed.hostname}:${port}`
}
export function parseTrustedAuthority(value) {
const parsed = parseUrlAuthority(value)
if (
parsed === undefined
|| parsed.hostname.includes('*')
|| canonicalAuthority(value, parsed) !== value.toLowerCase()
) {
throw new UsageError(
`DSH_TRUSTED_HOSTS entry ${JSON.stringify(value)} must be a canonical host or host:port authority`,
)
}
const authority = canonicalAuthority(value, parsed)
return Object.freeze({
hostname: parsed.hostname,
authority,
anyPort: authority === parsed.hostname,
matchAuthority: parsed.host,
})
}
export function parseTrustedHosts(environment = process.env) {
const raw = environment.DSH_TRUSTED_HOSTS?.trim() ?? ''
if (raw === '') return Object.freeze({ wildcard: false, authorities: Object.freeze([]) })
const entries = raw.split(',').map(entry => entry.trim())
if (entries.some(entry => entry === '')) {
throw new UsageError('DSH_TRUSTED_HOSTS cannot contain an empty entry')
}
if (entries.includes('*')) {
if (entries.length !== 1) throw new UsageError('DSH_TRUSTED_HOSTS wildcard must be used alone')
return Object.freeze({ wildcard: true, authorities: Object.freeze([]) })
}
const unique = new Map()
for (const entry of entries) {
const parsed = parseTrustedAuthority(entry)
unique.set(`${parsed.authority}|${String(parsed.anyPort)}`, parsed)
}
return Object.freeze({ wildcard: false, authorities: Object.freeze([...unique.values()]) })
}
export async function loadConfig(environment = process.env) {
const managementPort = environment.DSH_MANAGEMENT_PORT === undefined || environment.DSH_MANAGEMENT_PORT.trim() === ''
? 3081 : Number(environment.DSH_MANAGEMENT_PORT)
if (managementPort !== null && (!Number.isInteger(managementPort) || managementPort < 1 || managementPort > 65535)) {
throw new UsageError('DSH_MANAGEMENT_PORT must be an integer between 1 and 65535')
}
return Object.freeze({
trustedHosts: parseTrustedHosts(environment),
polyfill: parseBoolean('DSH_PROXY_POLYFILL', environment.DSH_PROXY_POLYFILL, true),
managementPort,
})
}
upstream/container/control-plane/services/gateway/lib/dsh-upstream-auth.mjs
+119
View File
@@ -0,0 +1,119 @@
import { request } from 'node:http'
function lifecycleReadiness(socketPath) {
const body = Buffer.from('{}')
return new Promise((resolve, reject) => {
const outgoing = request({
socketPath,
path: '/v1/runtime/readiness',
method: 'POST',
headers: {
'content-length': body.byteLength,
'content-type': 'application/json',
},
}, response => {
const chunks = []
response.on('data', chunk => chunks.push(chunk))
response.once('end', () => {
if (response.statusCode !== 200) {
reject(new Error(`DSH lifecycle readiness returned HTTP ${String(response.statusCode)}`))
return
}
try {
resolve(JSON.parse(Buffer.concat(chunks).toString('utf8')))
} catch (error) {
reject(error)
}
})
})
outgoing.setTimeout(750, () => outgoing.destroy(new Error('DSH lifecycle readiness timed out')))
outgoing.once('error', reject)
outgoing.end(body)
})
}
function exchange(host, port, readyUrl) {
const url = new URL(readyUrl)
if (url.protocol !== 'http:' || url.hostname !== host || Number(url.port) !== port
|| url.pathname !== '/' || url.username !== '' || url.password !== '' || url.hash !== ''
|| url.searchParams.getAll('token').length !== 1) {
throw new Error('DSH Web readiness URL is invalid')
}
return new Promise((resolve, reject) => {
const outgoing = request({
hostname: host,
port,
path: `${url.pathname}${url.search}`,
method: 'GET',
headers: { 'cache-control': 'no-store' },
}, response => {
response.resume()
response.once('end', () => {
const cookie = response.headers['set-cookie']?.[0]?.split(';', 1)[0]
if (response.statusCode !== 303 || response.headers.location !== '/'
|| typeof cookie !== 'string' || cookie === '') {
reject(new Error('DSH Web authentication exchange failed'))
} else resolve(cookie)
})
})
outgoing.setTimeout(750, () => outgoing.destroy(new Error('DSH Web authentication exchange timed out')))
outgoing.once('error', reject)
outgoing.end()
})
}
export class DshUpstreamAuthentication {
constructor({
socketPath = '/run/dsh-platform/dsh-lifecycle.sock',
host = '127.0.0.1',
port = 3079,
now = () => Date.now(),
legacyProbeIntervalMs = 250,
} = {}) {
this.socketPath = socketPath
this.host = host
this.port = port
this.now = now
this.legacyProbeIntervalMs = legacyProbeIntervalMs
this.cachedCookie = undefined
this.cachedGeneration = null
this.nextProbeAt = 0
this.pending = null
}
async publicReady() {
const readiness = await lifecycleReadiness(this.socketPath)
const ready = readiness?.publicReady === true
|| (readiness?.publicReady === undefined && readiness?.ready === true)
if (ready) this.nextProbeAt = 0
return ready
}
async cookie() {
if (this.pending !== null) return this.pending
if (this.now() < this.nextProbeAt) return null
this.pending = this.refresh().finally(() => { this.pending = null })
return this.pending
}
async refresh() {
const readiness = await lifecycleReadiness(this.socketPath)
const publicReady = readiness?.publicReady === true
|| (readiness?.publicReady === undefined && readiness?.ready === true)
if (!publicReady || readiness?.ready !== true || typeof readiness.readyUrl !== 'string') {
if (!publicReady) {
this.cachedCookie = undefined
this.cachedGeneration = null
}
this.nextProbeAt = this.now() + this.legacyProbeIntervalMs
return null
}
if (this.cachedCookie !== undefined && this.cachedGeneration === (readiness.generation ?? null)) {
return this.cachedCookie
}
const cookie = await exchange(this.host, this.port, readiness.readyUrl)
this.cachedCookie = cookie
this.cachedGeneration = readiness.generation ?? null
return cookie
}
}
upstream/container/control-plane/services/gateway/lib/errors.mjs
+7
View File
@@ -0,0 +1,7 @@
export class UsageError extends Error {
constructor(message) {
super(message)
this.name = 'UsageError'
this.exitCode = 64
}
}
upstream/container/control-plane/services/gateway/lib/lifecycle.mjs
+157
View File
@@ -0,0 +1,157 @@
import { createGatewayServer, closeGatewayServer } from './proxy.mjs'
import { createBrowserAuthentication } from './browser-auth.mjs'
import { LocalApiClient } from '../../../modules/updater/lib/client.mjs'
import { safeReturnPath } from './proxy.mjs'
import { DshUpstreamAuthentication } from './dsh-upstream-auth.mjs'
export const EXTERNAL_HOST = '0.0.0.0'
export const EXTERNAL_PORT = 3080
function listen(server, port, host) {
return new Promise((resolve, reject) => {
const onError = (error) => {
server.off('listening', onListening)
reject(error)
}
const onListening = () => {
server.off('error', onError)
resolve()
}
server.once('error', onError)
server.once('listening', onListening)
server.listen(port, host)
})
}
export async function runGateway(config, {
signalSource = process,
gatewayFactory = createGatewayServer,
externalHost = EXTERNAL_HOST,
externalPort = EXTERNAL_PORT,
managementPort = null,
managementHost = externalHost,
managementSocketPath = process.env.DSH_PLATFORM_MANAGEMENT_SOCKET ?? '/run/dsh-platform/management.sock',
managementCliSocketPath = process.env.DSH_PLATFORM_MANAGEMENT_CLI_SOCKET ?? '/run/dsh-platform/management-cli.sock',
maintenanceSocketPath = process.env.DSH_PLATFORM_MAINTENANCE_SOCKET ?? '/run/dsh-platform/maintenance.sock',
lifecycleSocketPath = process.env.DSH_PLATFORM_LIFECYCLE_SOCKET ?? '/run/dsh-platform/dsh-lifecycle.sock',
accessSocketPath = process.env.DSH_PLATFORM_ACCESS_SOCKET ?? '/run/dsh-platform/access/access.sock',
accessClient,
managementClient,
platformStatusPollMs = 250,
platformStatusCacheMs = 10_000,
now = () => Date.now(),
report = async () => {},
} = {}) {
const record = (message, fields = {}) => Promise.resolve().then(() => report(message, fields)).catch(() => {})
const management = managementClient ?? new LocalApiClient(managementCliSocketPath)
const access = accessClient ?? new LocalApiClient(accessSocketPath)
const accessRequest = access.request.bind(access)
const resilientAccess = accessClient === undefined
? { request: async (method, path, body, headers) => {
for (let attempt = 0; ; attempt++) {
try {
return await accessRequest(method, path, body, headers)
} catch (error) {
const retryable = ['ENOENT', 'ECONNREFUSED', 'ECONNRESET', 'EPIPE'].includes(error?.code)
if (!retryable || attempt >= 20) throw error
await new Promise(resolve => setTimeout(resolve, Math.min(250, 25 * (attempt + 1))))
}
}
} } : access
const browserAuthentication = createBrowserAuthentication({ access: resilientAccess, safeReturnPath, report: record })
const dshUpstreamAuthentication = new DshUpstreamAuthentication({ socketPath: lifecycleSocketPath })
const isolatedBrowserAuthentication = createBrowserAuthentication({
access: resilientAccess,
safeReturnPath,
report: record,
paths: { authPrefix: '/auth/', accessPrefix: '/access/', transitionPrefix: '/transition/', consolePath: '/' },
})
let lastPlatformStatus
let lastPlatformStatusAt = 0
const refreshPlatformStatus = async () => {
const status = await management.request('GET', '/_dsh_platform/api/v1/status')
lastPlatformStatus = status
lastPlatformStatusAt = now()
return status
}
const platformStatus = async () => {
try {
return await refreshPlatformStatus()
} catch (error) {
if (lastPlatformStatus !== undefined && now() - lastPlatformStatusAt <= platformStatusCacheMs) {
return lastPlatformStatus
}
throw error
}
}
const server = gatewayFactory({
polyfill: config.polyfill,
trustedHosts: config.trustedHosts,
managementSocketPath,
maintenanceSocketPath,
browserAuthentication,
dshUpstreamAuthentication,
platformStatus,
report: record,
surface: 'compat',
})
const managementServer = managementPort === null || managementPort === undefined
? null
: gatewayFactory({
polyfill: false,
trustedHosts: config.trustedHosts,
managementSocketPath,
maintenanceSocketPath,
browserAuthentication: isolatedBrowserAuthentication,
dshUpstreamAuthentication,
platformStatus,
report: record,
surface: 'management',
})
const platformStatusTimer = setInterval(() => {
void refreshPlatformStatus().catch(() => {})
}, platformStatusPollMs)
platformStatusTimer.unref?.()
void refreshPlatformStatus().catch(() => {})
let resolveSignal
const receivedSignal = new Promise(resolve => { resolveSignal = resolve })
const onSignal = signal => resolveSignal(signal)
const onSigint = () => onSignal('SIGINT')
const onSigterm = () => onSignal('SIGTERM')
signalSource.once('SIGINT', onSigint)
signalSource.once('SIGTERM', onSigterm)
try {
await record('gateway.starting', {
externalHost, externalPort,
...(managementServer === null ? {} : { managementHost, managementPort }),
})
await listen(server, externalPort, externalHost)
if (managementServer !== null) await listen(managementServer, managementPort, managementHost)
await record('gateway.ready', {
externalHost, externalPort,
...(managementServer === null ? {} : { managementHost, managementPort }),
})
const serverFailed = new Promise((_, reject) => server.once('error', reject))
const signal = await Promise.race([receivedSignal, serverFailed])
await record('gateway.stopping', { signal })
await closeGatewayServer(server)
if (managementServer !== null) await closeGatewayServer(managementServer)
await record('gateway.stopped')
return 0
} catch (error) {
await record('gateway.fatal', { error })
if (server.listening || managementServer?.listening) {
try {
if (server.listening) await closeGatewayServer(server)
if (managementServer?.listening) await closeGatewayServer(managementServer)
} catch (closeError) {
await record('gateway.stop.failed', { error: closeError, cause: error instanceof Error ? error.message : String(error) })
}
}
throw error
} finally {
clearInterval(platformStatusTimer)
signalSource.off('SIGINT', onSigint)
signalSource.off('SIGTERM', onSigterm)
}
}
upstream/container/control-plane/services/gateway/lib/polyfill.mjs
+60
View File
@@ -0,0 +1,60 @@
export const RANDOM_UUID_POLYFILL = '<script>(function(){try{var c=globalThis.crypto;if(c&&typeof c.randomUUID!=="function"&&typeof c.getRandomValues==="function"){c.randomUUID=function(){var b=c.getRandomValues(new Uint8Array(16));b[6]=(b[6]&15)|64;b[8]=(b[8]&63)|128;var h="";for(var i=0;i<16;i++){h+=b[i].toString(16).padStart(2,"0")}return h.slice(0,8)+"-"+h.slice(8,12)+"-"+h.slice(12,16)+"-"+h.slice(16,20)+"-"+h.slice(20)}}}catch(e){}})();</script>'
export const LIFECYCLE_TRANSITION_GUARD = `<script>(function(){
try{
var api="/_dsh_platform/plugin-api/v1",wait="/_dsh_gateway/wait",navigating=false;
function transition(value){return ["restarting","switching","recovering","restart-failed"].includes(value.operation)||["snapshotting-data","switching","probation","restoring-data"].includes(value.update&&value.update.status)||["starting","stopping","stopped","restarting","recovering","failed"].includes(value.dshLifecycle&&value.dshLifecycle.state)}
function navigate(){if(navigating||location.pathname===wait)return;navigating=true;var back=location.pathname+location.search+location.hash;location.replace(wait+"?return="+encodeURIComponent(back))}
async function inspect(){try{var response=await fetch(api+"/status",{cache:"no-store"}),value=await response.json();if(transition(value))navigate()}catch(e){}}
var events=new EventSource(api+"/events");events.addEventListener("state",inspect);inspect();
}catch(e){}
})();</script>`
export const SESSION_VALIDITY_GUARD = `<script>(function(){
try{
var checking=false;
async function inspect(){if(checking)return;checking=true;try{var response=await fetch("/_dsh_platform/auth/session-context",{cache:"no-store",credentials:"same-origin"});if(response.status===401)location.reload()}catch(e){}finally{checking=false}}
setInterval(inspect,5000);addEventListener("focus",inspect);document.addEventListener("visibilitychange",function(){if(document.visibilityState==="visible")inspect()});
}catch(e){}
})();</script>`
export const PLUGIN_RECOVERY_GUARD = `<script>(function(){
try{
var nativeFetch=globalThis.fetch.bind(globalThis),markerKey="dsh-platform:plugin-load-recovery",waitPath="/_dsh_gateway/wait",failurePath="/_dsh_gateway/plugin-failure",navigating=false;
function plugin(url){var match=url.pathname.match(/^\\/plugins\\/(.+)\\/client\\.js$/);if(!match)return null;try{return decodeURIComponent(match[1])}catch(e){return match[1]}}
function target(input){try{return new URL(typeof input==="string"?input:input.url,location.href)}catch(e){return null}}
function marker(){try{var value=JSON.parse(sessionStorage.getItem(markerKey));return value&&Date.now()-value.createdAt<120000?value:null}catch(e){return null}}
function save(value){try{sessionStorage.setItem(markerKey,JSON.stringify(value))}catch(e){}}
function report(event,fields){nativeFetch("/_dsh_gateway/client-event",{method:"POST",headers:{"content-type":"application/json"},body:JSON.stringify(Object.assign({event:event,pathname:location.pathname},fields)),keepalive:true}).catch(function(){})}
async function json(path){try{var response=await nativeFetch(path,{cache:"no-store"});return await response.json()}catch(e){return {}}}
function transition(value){return ["restarting","switching","recovering","restart-failed"].includes(value.operation)||["snapshotting-data","switching","probation","restoring-data"].includes(value.update&&value.update.status)||["starting","stopping","stopped","restarting","recovering","failed"].includes(value.dshLifecycle&&value.dshLifecycle.state)}
async function recover(url,pluginId,reason){
var values=await Promise.all([json("/_dsh_gateway/readiness"),json("/_dsh_platform/plugin-api/v1/status")]),readiness=values[0],status=values[1],lifecycle=status.dshLifecycle||{},updated=Date.parse(lifecycle.updatedAt),recent=typeof lifecycle.taskId==="string"&&Number.isFinite(updated)&&Date.now()-updated<30000,eligible=transition(status)||recent||readiness.pluginRecoveryEligible===true;
var identity=typeof lifecycle.taskId==="string"?lifecycle.taskId:(lifecycle.updatedAt||readiness.state||"unknown"),previous=marker(),attempt=previous&&previous.identity===identity?previous.attempt+1:1,fields={pluginId:pluginId,revision:url.searchParams.get("rev"),lifecycleState:lifecycle.state||readiness.state||null,lifecycleTaskId:lifecycle.taskId||null,recoveryAttempt:attempt,reason:reason};
if(attempt>=3){report("browser.plugin-load.recovery.failed",Object.assign({level:"error"},fields));save({identity:identity,taskId:lifecycle.taskId||null,url:url.pathname+url.search,pluginId:pluginId,attempt:attempt,createdAt:Date.now(),completed:false});if(!navigating){navigating=true;location.replace(failurePath)}return true}
if(!eligible){report("browser.plugin-load.failed",Object.assign({level:"error"},fields));return false}
report("browser.plugin-load.failed",Object.assign({level:"warning"},fields));
save({identity:identity,taskId:lifecycle.taskId||null,url:url.pathname+url.search,pluginId:pluginId,attempt:attempt,createdAt:Date.now(),completed:false});
report("browser.plugin-load.recovery.started",Object.assign({level:"warning"},fields));
if(!navigating){navigating=true;var back=location.pathname+location.search+location.hash;location.replace(waitPath+"?return="+encodeURIComponent(back))}return true
}
function completed(url,pluginId){var value=marker();if(!value||value.completed||value.url!==url.pathname+url.search)return;value.completed=true;save(value);report("browser.plugin-load.recovery.completed",{level:"info",pluginId:pluginId,revision:url.searchParams.get("rev"),lifecycleTaskId:value.taskId||null,recoveryAttempt:value.attempt})}
globalThis.fetch=function(input,init){var url=target(input),pluginId=url&&plugin(url),pending=nativeFetch(input,init);if(!pluginId)return pending;return pending.then(function(response){if(response.ok){completed(url,pluginId);return response}if(response.status!==502&&response.status!==503)return response;return recover(url,pluginId,"HTTP "+response.status).then(function(active){return active?new Promise(function(){}):response})},function(error){return recover(url,pluginId,error&&error.name||"network error").then(function(active){if(active)return new Promise(function(){});throw error})})}
function errorUrl(value){
var candidates=[value&&value.filename,value&&value.error&&value.error.stack,value&&value.error&&value.error.message,value&&value.reason&&value.reason.stack,value&&value.reason&&value.reason.message];
for(var i=0;i<candidates.length;i++){if(typeof candidates[i]!=="string")continue;var match=candidates[i].match(/(?:https?:\\/\\/[^\\s)]+)?(\\/plugins\\/(?:@[^/\\s]+\\/)?[^/\\s]+\\/client\\.js(?:\\?[^\\s)]+)?)/);if(!match)continue;var url=target(match[1]);if(url)return url}
return null
}
function handleError(value,reason){var url=errorUrl(value);if(!url)return;var pluginId=plugin(url);if(!pluginId)return;recover(url,pluginId,reason).catch(function(){})}
addEventListener("error",function(event){handleError(event,"plugin module error")},true);
addEventListener("unhandledrejection",function(event){handleError({reason:event.reason},"plugin module rejection")},true);
}catch(e){}
})();</script>`
export function injectRandomUuidPolyfill(html) {
const match = /<head(?:\s[^>]*)?>/i.exec(html)
const injected = `${RANDOM_UUID_POLYFILL}${LIFECYCLE_TRANSITION_GUARD}${SESSION_VALIDITY_GUARD}${PLUGIN_RECOVERY_GUARD}`
if (match === null) return `${injected}${html}`
const end = match.index + match[0].length
return `${html.slice(0, end)}${injected}${html.slice(end)}`
}
upstream/container/control-plane/services/gateway/lib/proxy.mjs
+1310
View File
File diff suppressed because it is too large. Load diff
upstream/container/control-plane/services/gateway/lib/trust.mjs
+56
View File
@@ -0,0 +1,56 @@
function parseRequestAuthority(value) {
if (
value === ''
|| /[\s/@?#]/.test(value)
|| value.endsWith(':')
) return undefined
try {
const parsed = new URL(`http://${value}`)
return parsed.hostname === '' ? undefined : parsed
} catch {
return undefined
}
}
export function isLoopbackHostname(hostname) {
if (hostname === 'localhost' || hostname === '[::1]') return true
const parts = hostname.split('.')
return parts.length === 4
&& parts[0] === '127'
&& parts.every(part => /^\d{1,3}$/.test(part) && Number(part) <= 255)
}
function matchesTrustedAuthority(requestUrl, trustedHosts) {
if (trustedHosts.wildcard) return true
return trustedHosts.authorities.some((entry) => (
entry.anyPort ? entry.hostname === requestUrl.hostname : entry.matchAuthority === requestUrl.host
))
}
export function inspectExternalRequest(headers, trustedHosts, {
allowCrossOrigin = false,
allowUntrustedAuthority = false,
} = {}) {
const host = headers.host
if (typeof host !== 'string') return Object.freeze({ accepted: false, reason: 'missing-host' })
const hostUrl = parseRequestAuthority(host)
if (hostUrl === undefined) return Object.freeze({ accepted: false, reason: 'invalid-host' })
if (!allowUntrustedAuthority && !isLoopbackHostname(hostUrl.hostname) && !matchesTrustedAuthority(hostUrl, trustedHosts)) {
return Object.freeze({ accepted: false, reason: 'untrusted-host' })
}
if (!allowCrossOrigin && headers['sec-fetch-site'] === 'cross-site') {
return Object.freeze({ accepted: false, reason: 'cross-site' })
}
const origin = headers.origin
if (!allowCrossOrigin && origin !== undefined) {
if (typeof origin !== 'string') return Object.freeze({ accepted: false, reason: 'invalid-origin' })
try {
if (new URL(origin).host !== hostUrl.host) {
return Object.freeze({ accepted: false, reason: 'origin-mismatch' })
}
} catch {
return Object.freeze({ accepted: false, reason: 'invalid-origin' })
}
}
return Object.freeze({ accepted: true, externalAuthority: hostUrl.host })
}
upstream/container/control-plane/services/gateway/package.json
+11
View File
@@ -0,0 +1,11 @@
{
"name": "dsh-docker-gateway",
"private": true,
"type": "module",
"scripts": {
"test": "node --test"
},
"engines": {
"node": ">=24"
}
}
upstream/container/control-plane/services/gateway/test/auth.test.mjs
+149
View File
@@ -0,0 +1,149 @@
import assert from 'node:assert/strict'
import { createServer, request as httpRequest } from 'node:http'
import { connect as netConnect } from 'node:net'
import { mkdtemp } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import test from 'node:test'
import { parseTrustedHosts } from '../lib/config.mjs'
import { closeGatewayServer, createGatewayServer, HEALTH_PATH } from '../lib/proxy.mjs'
async function listen(server) {
await new Promise((resolve, reject) => {
server.once('error', reject)
server.listen(0, '127.0.0.1', resolve)
})
return server.address().port
}
async function close(server) {
server.closeAllConnections?.()
await new Promise(resolve => server.close(resolve))
}
function request(port, { path = '/', method = 'GET', headers = {}, body = '' } = {}) {
return new Promise((resolve, reject) => {
const outgoing = httpRequest({ hostname: '127.0.0.1', port, path, method, headers }, response => {
const chunks = []
response.on('data', chunk => chunks.push(chunk))
response.on('end', () => resolve({
status: response.statusCode,
headers: response.headers,
body: Buffer.concat(chunks).toString('utf8'),
}))
})
outgoing.once('error', reject)
outgoing.end(body)
})
}
const initialized = Object.freeze({ status: async () => ({ state: 'initialized' }), handle: async () => false })
test('Gateway fails closed without an authenticated browser session while health remains available', async () => {
let upstreamRequests = 0
const upstream = createServer((_incoming, response) => { upstreamRequests += 1; response.end('upstream') })
const upstreamPort = await listen(upstream)
const gateway = createGatewayServer({ trustedHosts: parseTrustedHosts({}), upstreamPort })
const port = await listen(gateway)
try {
assert.equal((await request(port, { path: HEALTH_PATH, headers: { host: '127.0.0.1' } })).status, 200)
const navigation = await request(port, { headers: { accept: 'text/html', host: '127.0.0.1' } })
assert.equal(navigation.status, 303)
assert.match(navigation.headers.location, /^\/_dsh_platform\/auth\//)
assert.equal((await request(port, { path: '/api/sessions', headers: { host: '127.0.0.1' } })).status, 401)
assert.equal(upstreamRequests, 0)
} finally {
await Promise.all([closeGatewayServer(gateway), close(upstream)])
}
})
test('Gateway reports an unavailable authentication backend as a service failure', async () => {
const unavailable = new Error('access socket unavailable')
unavailable.browserAuthenticationBackend = true
const browserAuthentication = {
status: async () => { throw unavailable },
handle: async () => false,
}
const gateway = createGatewayServer({
trustedHosts: parseTrustedHosts({}), browserAuthentication, upstreamPort: 1,
})
const port = await listen(gateway)
try {
assert.equal((await request(port, {
path: HEALTH_PATH, headers: { host: '127.0.0.1' },
})).status, 200)
const unavailable = await request(port, { headers: { host: '127.0.0.1', accept: 'text/html' } })
assert.equal(unavailable.status, 503)
assert.match(unavailable.body, /authentication service unavailable/)
} finally { await closeGatewayServer(gateway) }
})
test('DSH and Management sessions authorize only their own route surfaces', async () => {
const root = await mkdtemp(join(tmpdir(), 'dsh-gateway-session-domains-'))
const socketPath = join(root, 'management.sock')
const management = createServer((incoming, response) => response.end(`management:${incoming.url}`))
await new Promise((resolve, reject) => {
management.once('error', reject)
management.listen(socketPath, resolve)
})
const upstream = createServer((_incoming, response) => response.end('dsh'))
const upstreamPort = await listen(upstream)
const browserAuthentication = {
...initialized,
validateDsh: async request => ({ authenticated: request.headers.cookie === 'dsh=yes' }),
authorizePlugin: async request => request.headers.cookie === 'dsh=yes'
? { authorized: true, capability: { token: 'plugin-capability' } }
: { authorized: false },
validateManagement: async request => ({ authenticated: request.headers.cookie === 'management=yes' }),
enterManagement: async (_request, response) => { response.writeHead(303, { location: '/_dsh_platform/auth/management' }); response.end() },
}
const gateway = createGatewayServer({
trustedHosts: parseTrustedHosts({}), browserAuthentication,
managementSocketPath: socketPath, upstreamPort,
})
const port = await listen(gateway)
try {
assert.equal((await request(port, { headers: { host: '127.0.0.1', cookie: 'dsh=yes' } })).body, 'dsh')
assert.equal((await request(port, {
path: '/_dsh_platform/plugin-api/v1/status', headers: { host: '127.0.0.1', cookie: 'dsh=yes' },
})).body, 'management:/_dsh_platform/api/v1/status')
assert.equal((await request(port, {
path: '/_dsh_platform/api/v1/status', headers: { host: '127.0.0.1', cookie: 'dsh=yes' },
})).status, 401)
assert.equal((await request(port, {
path: '/_dsh_platform/api/v1/status', headers: { host: '127.0.0.1', cookie: 'management=yes' },
})).body, 'management:/_dsh_platform/api/v1/status')
assert.equal((await request(port, {
path: '/', headers: { host: '127.0.0.1', cookie: 'management=yes' },
})).status, 401)
} finally {
await Promise.all([closeGatewayServer(gateway), close(management), close(upstream)])
}
})
test('terminal WebSocket requires Management rather than DSH authentication', async () => {
const browserAuthentication = {
...initialized,
validateDsh: async () => ({ authenticated: true }),
validateManagement: async () => ({ authenticated: false }),
enterManagement: async () => {},
}
const gateway = createGatewayServer({ trustedHosts: parseTrustedHosts({}), browserAuthentication, upstreamPort: 1 })
const port = await listen(gateway)
const client = netConnect(port, '127.0.0.1')
try {
await new Promise((resolve, reject) => { client.once('connect', resolve); client.once('error', reject) })
client.write([
'GET /_dsh_platform/api/v1/terminal/sessions/123e4567-e89b-42d3-a456-426614174000/stream HTTP/1.1',
'Host: 127.0.0.1', 'Connection: Upgrade', 'Upgrade: websocket', '', '',
].join('\r\n'))
const response = await new Promise((resolve, reject) => {
client.once('data', data => resolve(data.toString('utf8')))
client.once('error', reject)
})
assert.match(response, /^HTTP\/1\.1 401 Unauthorized/)
} finally {
client.destroy()
await closeGatewayServer(gateway)
}
})
upstream/container/control-plane/services/gateway/test/availability.test.mjs
+486
View File
@@ -0,0 +1,486 @@
import assert from 'node:assert/strict'
import { createServer, request as httpRequest } from 'node:http'
import { connect as netConnect } from 'node:net'
import test from 'node:test'
import { DshAvailability, availabilityPage, language } from '../lib/availability.mjs'
import { closeGatewayServer, createGatewayServer as createGatewayServerBase, PLUGIN_FAILURE_PATH, READINESS_PATH, WAIT_PATH, safeReturnPath } from '../lib/proxy.mjs'
import { parseTrustedHosts } from '../lib/config.mjs'
const AUTHENTICATED_BROWSER = Object.freeze({
status: async () => ({ state: 'initialized' }),
validateDsh: async () => ({ authenticated: true }),
validateManagement: async () => ({ authenticated: true }),
enterManagement: async () => { throw new Error('unexpected Management login') },
handle: async () => false,
})
const UNAUTHENTICATED_BROWSER = Object.freeze({
status: async () => ({ state: 'initialized' }),
validateDsh: async () => ({ authenticated: false }),
validateManagement: async () => ({ authenticated: false }),
enterManagement: async () => { throw new Error('unexpected Management login') },
handle: async () => false,
})
function createGatewayServer(options) {
return createGatewayServerBase({
...options,
browserAuthentication: options.browserAuthentication ?? AUTHENTICATED_BROWSER,
})
}
const ISOLATED_UNAUTHENTICATED_BROWSER = Object.freeze({
...UNAUTHENTICATED_BROWSER,
status: async () => ({ state: 'initialized', account: {
managementAccess: {
mode: 'isolated',
isolatedEntry: { kind: 'public', managementPublicOrigin: 'https://management.example' },
},
} }),
})
async function listen(server) {
await new Promise((resolve, reject) => {
server.once('error', reject)
server.listen(0, '127.0.0.1', resolve)
})
return server.address().port
}
function request(port, path = '/', { method = 'GET', accept = 'text/html', headers = {} } = {}) {
return new Promise((resolve, reject) => {
const outgoing = httpRequest({
hostname: '127.0.0.1', port, path, method,
headers: { host: 'dsh.example', accept, ...headers },
}, response => {
const chunks = []
response.on('data', chunk => chunks.push(chunk))
response.on('end', () => resolve({
status: response.statusCode,
headers: response.headers,
body: Buffer.concat(chunks).toString('utf8'),
}))
})
outgoing.once('error', reject)
outgoing.end()
})
}
async function unavailableGateway({ platform = {}, availability = new DshAvailability(), probe = async () => false, report, browserAuthentication } = {}) {
const placeholder = createServer()
const unavailablePort = await listen(placeholder)
await new Promise(resolve => placeholder.close(resolve))
const gateway = createGatewayServer({
trustedHosts: parseTrustedHosts({ DSH_TRUSTED_HOSTS: 'dsh.example' }),
upstreamPort: unavailablePort,
platformStatus: async () => platform,
availability,
probe,
browserAuthentication,
report,
probeIntervalMs: 60_000,
})
return { gateway, port: await listen(gateway) }
}
test('authenticated DSH traffic waits for authoritative Bootstrap publication', async t => {
let upstreamRequests = 0
const upstream = createServer((_request, response) => {
upstreamRequests += 1
response.writeHead(200, { 'content-type': 'text/plain' })
response.end('published')
})
const upstreamPort = await listen(upstream)
t.after(() => upstream.close())
let published = false
const gateway = createGatewayServer({
trustedHosts: parseTrustedHosts({ DSH_TRUSTED_HOSTS: 'dsh.example' }),
upstreamPort,
platformStatus: async () => ({ dshLifecycle: { state: 'running' } }),
probe: async () => true,
dshUpstreamAuthentication: {
cookie: async () => null,
publicReady: async () => published,
},
probeIntervalMs: 60_000,
})
const port = await listen(gateway)
try {
const socket = netConnect(port, '127.0.0.1')
await new Promise((resolve, reject) => {
socket.once('connect', resolve)
socket.once('error', reject)
})
socket.write('GET /events HTTP/1.1\r\nHost: dsh.example\r\nConnection: Upgrade\r\nUpgrade: websocket\r\n\r\n')
const upgrade = await new Promise((resolve, reject) => {
socket.once('data', data => resolve(data.toString('utf8')))
socket.once('error', reject)
})
assert.match(upgrade, /^HTTP\/1\.1 503 Service Unavailable/)
socket.destroy()
const waiting = await request(port)
assert.equal(waiting.status, 200)
assert.match(waiting.body, /DeepSeek Harness is starting/)
assert.equal(upstreamRequests, 0)
assert.deepEqual(JSON.parse((await request(port, READINESS_PATH, { accept: 'application/json' })).body).ready, false)
const dedicatedWaiting = await request(port, `${WAIT_PATH}?return=%2Fsessions%2Fcurrent`)
assert.equal(dedicatedWaiting.status, 200)
assert.match(dedicatedWaiting.body, /DeepSeek Harness is starting/)
assert.match(dedicatedWaiting.body, /\/sessions\/current/)
published = true
const ready = await request(port)
assert.equal(ready.status, 200)
assert.equal(ready.body, 'published')
assert.equal(upstreamRequests, 1)
assert.equal(JSON.parse((await request(port, READINESS_PATH, { accept: 'application/json' })).body).ready, true)
} finally {
await closeGatewayServer(gateway)
}
})
test('an intentional stopped state does not report an upstream proxy failure', async () => {
const reports = []
const context = await unavailableGateway({
platform: { dshLifecycle: { state: 'stopped' } },
report: (message, fields) => { reports.push({ message, fields }) },
})
try {
const response = await request(context.port)
assert.equal(response.status, 200)
assert.match(response.body, /DeepSeek Harness is stopped/)
await new Promise(resolve => setImmediate(resolve))
assert.equal(reports.some(entry => entry.message === 'gateway.upstream.failed'), false)
} finally {
await closeGatewayServer(context.gateway)
}
})
test('an unauthenticated top-level navigation sees classified DSH recovery before the login page', async () => {
const gateway = createGatewayServer({
trustedHosts: parseTrustedHosts({ DSH_TRUSTED_HOSTS: 'dsh.example' }),
upstreamPort: 1,
platformStatus: async () => ({ dshLifecycle: { state: 'failed', error: 'private runtime detail' } }),
probe: async () => false,
browserAuthentication: UNAUTHENTICATED_BROWSER,
probeIntervalMs: 60_000,
})
const port = await listen(gateway)
try {
const page = await request(port)
assert.equal(page.status, 200)
assert.match(page.body, /DeepSeek Harness could not start/)
assert.match(page.body, /href="\/_dsh_platform\/console\/"/)
assert.doesNotMatch(page.body, /private runtime detail|autocomplete="current-password"/)
assert.equal((await request(port, '/api/sessions', { accept: 'application/json' })).status, 503)
} finally {
await closeGatewayServer(gateway)
}
})
test('official-style holding page is self-contained and replaces the spinner with platform status', () => {
const page = availabilityPage('switching', { 'accept-language': 'zh-CN' })
assert.match(page, /HARNESS/)
assert.match(page, /正在切换 DeepSeek Harness 运行版本/)
assert.match(page, /href="\/_dsh_platform\/console\/"/)
assert.match(page, /打开 DSH 管理中心进行检查和恢复/)
assert.match(page, new RegExp(READINESS_PATH))
assert.match(page, /prefers-color-scheme:light/)
assert.match(page, /name="viewport"/)
assert.match(page, /html\{height:100%\}body\{min-height:100%;margin:0;display:grid;place-items:center;/)
assert.doesNotMatch(page, /100(?:d|s|l)?vh/)
assert.match(page, /max-width:520px/)
assert.match(page, /management\.href=value\.managementHref/)
assert.doesNotMatch(page, /spinner|Loading plugins/)
assert.doesNotMatch(page, /letter-spacing:-/)
})
test('persistent plugin failure page is terminal, localized, and links to Management', async () => {
const page = availabilityPage('plugin-failed', { 'accept-language': 'zh-CN' }, { poll: false, allowRefresh: true })
assert.match(page, /DeepSeek Harness 插件持续加载失败/)
assert.match(page, /打开 DSH 管理中心进行检查和恢复/)
assert.match(page, /刷新页面/)
assert.doesNotMatch(page, /setTimeout\(check/)
assert.doesNotMatch(availabilityPage('failed', {}, { poll: false }), /Reload page/)
const context = await unavailableGateway()
try {
const response = await request(context.port, PLUGIN_FAILURE_PATH, {
headers: { 'accept-language': 'zh-CN' },
})
assert.equal(response.status, 200)
assert.match(response.body, /DeepSeek Harness 插件持续加载失败/)
assert.equal((await request(context.port, PLUGIN_FAILURE_PATH, { method: 'POST' })).status, 405)
} finally {
await closeGatewayServer(context.gateway)
}
})
test('Gateway temporary DSH pages link to the configured isolated Management entry', async () => {
const context = await unavailableGateway({ browserAuthentication: ISOLATED_UNAUTHENTICATED_BROWSER })
try {
const response = await request(context.port, PLUGIN_FAILURE_PATH)
assert.equal(response.status, 200)
assert.match(response.body, /href="https:\/\/management\.example\/"/)
assert.doesNotMatch(response.body, /href="\/_dsh_platform\/console\/"/)
} finally {
await closeGatewayServer(context.gateway)
}
})
test('Gateway readiness publishes the current Management entry for an already-open temporary page', async () => {
let access = { state: 'initialized', account: { managementAccess: { mode: 'compat' } } }
const browserAuthentication = {
...UNAUTHENTICATED_BROWSER,
status: async () => access,
}
const context = await unavailableGateway({ browserAuthentication })
try {
const compatible = JSON.parse((await request(context.port, READINESS_PATH)).body)
assert.equal(compatible.managementHref, '/_dsh_platform/console/')
access = await ISOLATED_UNAUTHENTICATED_BROWSER.status()
const isolated = JSON.parse((await request(context.port, READINESS_PATH)).body)
assert.equal(isolated.managementHref, 'https://management.example/')
} finally {
await closeGatewayServer(context.gateway)
}
})
test('holding pages prefer the last DSH locale cookie over browser language', () => {
assert.equal(language({ cookie: 'theme=dark; dsh_locale=en', 'accept-language': 'zh-CN' }), 'en')
assert.equal(language({ cookie: 'dsh_locale=zh', 'accept-language': 'en-US' }), 'zh')
assert.equal(language({ cookie: 'dsh_locale=fr', 'accept-language': 'en-US' }), 'en')
assert.equal(language({ 'accept-language': 'fr-FR, zh-CN;q=0.8, en;q=0.5' }), 'zh')
assert.match(availabilityPage('recovering', { cookie: 'dsh_locale=zh', 'accept-language': 'en-US' }), /意外停止/)
assert.match(availabilityPage('recovering', { cookie: 'dsh_locale=en', 'accept-language': 'zh-CN' }), /Open DSH Management Console for diagnostics and recovery/)
})
test('every classified outage page links to the standalone Platform Management console', () => {
for (const state of ['starting', 'stopping', 'stopped', 'restarting', 'switching', 'runtime-recovering', 'recovering', 'failed', 'unavailable']) {
const page = availabilityPage(state, { 'accept-language': 'en-US' })
assert.match(page, /href="\/_dsh_platform\/console\/"/)
}
})
test('intentional DSH restarts have explicit holding and failure states', () => {
const availability = new DshAvailability()
assert.equal(availability.classify({ operation: 'restarting' }), 'restarting')
assert.equal(availability.classify({ operation: 'restart-failed' }), 'failed')
assert.match(availabilityPage('restarting', { 'accept-language': 'zh-CN' }), /正在重新启动/)
assert.equal(availability.classify({ dshLifecycle: { state: 'stopping' } }), 'stopping')
assert.equal(availability.classify({ dshLifecycle: { state: 'stopped' } }), 'stopped')
assert.equal(availability.classify({ dshLifecycle: { state: 'recovering', attempt: 2 } }), 'recovering')
assert.match(availabilityPage('recovering', { 'accept-language': 'zh-CN' }, {
lifecycle: { attempt: 2, maxAttempts: 3 },
}), /第 2 \/ 3 次/)
})
test('holding-page return paths stay same-origin and reject ambiguous inputs', () => {
assert.equal(safeReturnPath('/sessions/current?view=chat#latest'), '/sessions/current?view=chat#latest')
for (const value of [null, '', '//example.com/path', '/\\example.com', 'https://example.com/', '/bad\npath']) {
assert.equal(safeReturnPath(value), '/')
}
const page = availabilityPage('restarting', {}, { returnPath: '/</script><script>alert(1)</script>' })
assert.doesNotMatch(page, /const returnPath="\/<\/script>/)
assert.match(page, /\\u003c\/script>/)
})
test('cold start serves a holding page while API and WebSocket-shaped HTTP requests receive 503', async () => {
const { gateway, port } = await unavailableGateway()
try {
const page = await request(port)
assert.equal(page.status, 200)
assert.match(page.body, /DeepSeek Harness is starting/)
assert.equal(page.headers['cache-control'], 'no-store')
assert.equal((await request(port, '/api/sessions', { accept: 'application/json' })).status, 503)
assert.equal((await request(port, '/', { method: 'POST', accept: 'text/html' })).status, 503)
} finally {
await closeGatewayServer(gateway)
}
})
test('an unresponsive management service does not block the cold-start holding page', async () => {
const placeholder = createServer()
const unavailablePort = await listen(placeholder)
await new Promise(resolve => placeholder.close(resolve))
const gateway = createGatewayServer({
trustedHosts: parseTrustedHosts({ DSH_TRUSTED_HOSTS: 'dsh.example' }),
upstreamPort: unavailablePort,
platformStatus: () => new Promise(() => {}),
probe: async () => false,
probeIntervalMs: 60_000,
})
const port = await listen(gateway)
try {
const startedAt = Date.now()
const page = await request(port)
assert.equal(page.status, 200)
assert.match(page.body, /DeepSeek Harness is starting/)
assert.ok(Date.now() - startedAt < 1_000)
} finally {
await closeGatewayServer(gateway)
}
})
test('cold-start management status failures remain diagnostic warnings', async () => {
const reported = []
const gateway = createGatewayServer({
trustedHosts: parseTrustedHosts({ DSH_TRUSTED_HOSTS: 'dsh.example' }),
upstreamPort: 1,
platformStatus: async () => { throw Object.assign(new Error('socket is not ready'), { code: 'ENOENT' }) },
probe: async () => false,
probeIntervalMs: 60_000,
report: async (message, fields) => reported.push({ message, fields }),
})
const port = await listen(gateway)
try {
assert.equal((await request(port)).status, 200)
await new Promise(resolve => setImmediate(resolve))
assert.equal(reported.length, 1)
assert.equal(reported[0].message, 'gateway.platform-status.failed')
assert.equal(reported[0].fields.level, 'warning')
assert.equal(reported[0].fields.error.code, 'ENOENT')
} finally {
await closeGatewayServer(gateway)
}
})
test('platform switching, recovery, and startup failure select distinct page messages', async () => {
for (const [platform, expected] of [
[{ operation: 'switching' }, /Switching the DeepSeek Harness runtime/],
[{ operation: 'recovering' }, /Restoring the DeepSeek Harness runtime/],
[{ recoveryMode: 'candidate failed' }, /DeepSeek Harness could not start/],
]) {
const { gateway, port } = await unavailableGateway({ platform })
try {
const page = await request(port)
assert.equal(page.status, 200)
assert.match(page.body, expected)
assert.equal((await request(port, '/api/status', { accept: 'application/json' })).status, 503)
} finally {
await closeGatewayServer(gateway)
}
}
})
test('dedicated holding route preserves a safe return path and redirects when ready', async () => {
let ready = false
const platform = { dshLifecycle: { state: 'stopping' } }
const context = await unavailableGateway({
platform,
probe: async () => ready,
})
try {
const target = '/sessions/current?view=chat#latest'
const waiting = await request(context.port, `${WAIT_PATH}?return=${encodeURIComponent(target)}`)
assert.equal(waiting.status, 200)
assert.match(waiting.body, /DeepSeek Harness is stopping/)
assert.match(waiting.body, /location\.replace\(returnPath\)/)
assert.match(waiting.body, /\/sessions\/current\?view=chat#latest/)
const invalid = await request(context.port, `${WAIT_PATH}?return=${encodeURIComponent('//example.com/')}`)
assert.equal(invalid.status, 200)
assert.match(invalid.body, /const returnPath="\/"/)
ready = true
const premature = await request(context.port, `${WAIT_PATH}?return=${encodeURIComponent(target)}`)
assert.equal(premature.status, 200)
assert.match(premature.body, /DeepSeek Harness is stopping/)
const prematureReadiness = await request(context.port, READINESS_PATH)
assert.equal(prematureReadiness.status, 503)
assert.equal(JSON.parse(prematureReadiness.body).state, 'stopping')
platform.dshLifecycle.state = 'running'
const redirected = await request(context.port, `${WAIT_PATH}?return=${encodeURIComponent(target)}`)
assert.equal(redirected.status, 302)
assert.equal(redirected.headers.location, target)
assert.deepEqual(JSON.parse((await request(context.port, READINESS_PATH)).body), {
ready: true, state: 'ready', pluginRecoveryEligible: false,
managementHref: '/_dsh_platform/console/',
})
assert.equal((await request(context.port, WAIT_PATH, { method: 'POST' })).status, 405)
} finally {
await closeGatewayServer(context.gateway)
}
})
test('WebSocket receives 503 during a classified DSH outage', async () => {
const reports = []
const { gateway, port } = await unavailableGateway({
platform: { operation: 'switching' },
report: (message, fields) => reports.push({ message, fields }),
})
const socket = netConnect(port, '127.0.0.1')
try {
await new Promise((resolve, reject) => {
socket.once('connect', resolve)
socket.once('error', reject)
})
socket.write('GET /events HTTP/1.1\r\nHost: dsh.example\r\nConnection: Upgrade\r\nUpgrade: websocket\r\n\r\n')
const response = await new Promise((resolve, reject) => {
socket.once('data', data => resolve(data.toString('utf8')))
socket.once('error', reject)
})
assert.match(response, /^HTTP\/1\.1 503 Service Unavailable/)
await new Promise(resolve => setImmediate(resolve))
const failure = reports.find(entry => entry.message === 'gateway.websocket.failed')
assert.equal(failure?.fields.level, 'warning')
assert.equal(failure?.fields.upstream, 'dsh')
} finally {
socket.destroy()
await closeGatewayServer(gateway)
}
})
test('one unexpected local failure remains 502 until repeated probes confirm unavailability', async () => {
let now = 0
const availability = new DshAvailability({ now: () => now, failures: 3, failureWindowMs: 1_500 })
availability.observe(true)
const context = await unavailableGateway({ availability })
try {
assert.equal((await request(context.port)).status, 502)
availability.observe(false)
now = 1_600
availability.observe(false)
const recovered = await request(context.port)
assert.equal(recovered.status, 200)
assert.match(recovered.body, /temporarily unavailable/)
} finally {
await closeGatewayServer(context.gateway)
}
})
test('a Bootstrap candidate handoff serves the switching page instead of a bare 502', async () => {
const context = await unavailableGateway({ platform: { update: { status: 'building-candidate' } } })
try {
const response = await request(context.port)
assert.equal(response.status, 200)
assert.match(response.body, /Switching the DeepSeek Harness runtime/)
} finally {
await closeGatewayServer(context.gateway)
}
})
test('readiness immediately reports ready and clears confirmed failure state', async () => {
let ready = false
let now = 0
const availability = new DshAvailability({ now: () => now, failures: 1, failureWindowMs: 0 })
availability.observe(true)
availability.observe(false)
const context = await unavailableGateway({ availability, probe: async () => ready })
try {
const unavailable = await request(context.port, READINESS_PATH, { accept: 'application/json' })
assert.equal(unavailable.status, 503)
assert.equal(JSON.parse(unavailable.body).state, 'unavailable')
ready = true
const healthy = await request(context.port, READINESS_PATH, { accept: 'application/json' })
assert.equal(healthy.status, 200)
assert.deepEqual(JSON.parse(healthy.body), {
ready: true, state: 'ready', pluginRecoveryEligible: false,
managementHref: '/_dsh_platform/console/',
})
assert.equal(availability.classify({}), 'unknown')
} finally {
await closeGatewayServer(context.gateway)
}
})
upstream/container/control-plane/services/gateway/test/browser-auth.test.mjs
+1400
View File
File diff suppressed because it is too large. Load diff
upstream/container/control-plane/services/gateway/test/browser-loopback-patch.test.mjs
+40
View File
@@ -0,0 +1,40 @@
import assert from 'node:assert/strict'
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import test from 'node:test'
import { patchBrowserLoopback } from '../../../../environment/resources/patches/browser-loopback.mjs'
const BEFORE = 'isLoopback: pageLocation === void 0 || isLoopbackHostname(pageLocation.hostname),'
const BEFORE_WITH_TRANSPORT = 'isLoopback: transport?.ownsHost === true || pageLocation === void 0 || isLoopbackHostname(pageLocation.hostname),'
test('browser loopback patch changes exactly one connection classification', async () => {
const directory = await mkdtemp(join(tmpdir(), 'dsh-browser-loopback-patch-'))
const target = join(directory, 'client.js')
try {
await writeFile(target, `before;${BEFORE}after;`)
patchBrowserLoopback(target)
assert.equal(await readFile(target, 'utf8'), 'before;isLoopback: true,after;')
assert.throws(() => patchBrowserLoopback(target), /found 0/)
await writeFile(target, `${BEFORE}${BEFORE}`)
assert.throws(() => patchBrowserLoopback(target), /found 2/)
} finally {
await rm(directory, { recursive: true })
}
})
test('browser loopback patch accepts the transport-owned Host classification', async () => {
const directory = await mkdtemp(join(tmpdir(), 'dsh-browser-loopback-patch-'))
const target = join(directory, 'client.js')
try {
await writeFile(target, `before;${BEFORE_WITH_TRANSPORT}after;`)
patchBrowserLoopback(target)
assert.equal(await readFile(target, 'utf8'), 'before;isLoopback: true,after;')
await writeFile(target, `${BEFORE}${BEFORE_WITH_TRANSPORT}`)
assert.throws(() => patchBrowserLoopback(target), /found 2/)
} finally {
await rm(directory, { recursive: true })
}
})
upstream/container/control-plane/services/gateway/test/config.test.mjs
+85
View File
@@ -0,0 +1,85 @@
import assert from 'node:assert/strict'
import test from 'node:test'
import {
loadConfig,
parseBoolean,
parseTrustedAuthority,
parseTrustedHosts,
} from '../lib/config.mjs'
import { UsageError } from '../lib/errors.mjs'
test('parseBoolean accepts only explicit Compose booleans', () => {
assert.equal(parseBoolean('FLAG', undefined, true), true)
assert.equal(parseBoolean('FLAG', 'false', true), false)
assert.throws(() => parseBoolean('FLAG', '1', true), UsageError)
})
test('parseTrustedAuthority accepts canonical host authorities', () => {
assert.deepEqual(parseTrustedAuthority('Example.COM'), {
hostname: 'example.com',
authority: 'example.com',
anyPort: true,
matchAuthority: 'example.com',
})
assert.deepEqual(parseTrustedAuthority('example.com:8443'), {
hostname: 'example.com',
authority: 'example.com:8443',
anyPort: false,
matchAuthority: 'example.com:8443',
})
assert.deepEqual(parseTrustedAuthority('[fd00::1]:3080'), {
hostname: '[fd00::1]',
authority: '[fd00::1]:3080',
anyPort: false,
matchAuthority: '[fd00::1]:3080',
})
})
test('parseTrustedAuthority rejects URL-like and noncanonical entries', () => {
for (const value of [
'https://example.com',
'example.com/path',
'user@example.com',
'example.com:',
'*.example.com',
' example.com',
]) {
assert.throws(() => parseTrustedAuthority(value), UsageError, value)
}
})
test('parseTrustedHosts supports empty, lists, deduplication, and wildcard', () => {
assert.deepEqual(parseTrustedHosts({}), { wildcard: false, authorities: [] })
const list = parseTrustedHosts({ DSH_TRUSTED_HOSTS: 'a.example, b.example:8443,a.example' })
assert.equal(list.wildcard, false)
assert.equal(list.authorities.length, 2)
assert.deepEqual(parseTrustedHosts({ DSH_TRUSTED_HOSTS: '*' }), {
wildcard: true,
authorities: [],
})
})
test('parseTrustedHosts rejects malformed lists', () => {
assert.throws(() => parseTrustedHosts({ DSH_TRUSTED_HOSTS: 'a.example,,b.example' }), UsageError)
assert.throws(() => parseTrustedHosts({ DSH_TRUSTED_HOSTS: '*,a.example' }), UsageError)
})
test('loadConfig owns only Gateway settings', async () => {
const config = await loadConfig({
DSH_DEFAULT_WORKSPACE: 'not-a-gateway-setting',
DSH_TELEMETRY_DISABLED: 'not-a-gateway-setting',
DSH_PROXY_USERNAME: 'unused',
DSH_PROXY_PASSWORD: '',
})
assert.equal(Object.hasOwn(config, 'password'), false)
assert.equal(Object.hasOwn(config, 'username'), false)
assert.equal(Object.hasOwn(config, 'platformPassword'), false)
assert.equal(Object.hasOwn(config, 'workspace'), false)
assert.equal(Object.hasOwn(config, 'telemetryDisabled'), false)
assert.equal(config.managementPort, 3081)
assert.doesNotReject(() => loadConfig({
DSH_PROXY_USERNAME: 'invalid:name',
DSH_PROXY_PASSWORD: 'secret',
DSH_PLATFORM_PASSWORD: 'legacy-secret',
}))
})
upstream/container/control-plane/services/gateway/test/directory-picker-patch.test.mjs
+25
View File
@@ -0,0 +1,25 @@
import assert from 'node:assert/strict'
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import test from 'node:test'
import { patchDirectoryPicker } from '../../../../environment/resources/patches/directory-picker.mjs'
test('directory picker patch changes exactly one default without touching explicit paths', async () => {
const directory = await mkdtemp(join(tmpdir(), 'dsh-directory-patch-'))
const target = join(directory, 'index.js')
try {
await writeFile(target, 'before;const target = resolve(path ?? home);after;')
patchDirectoryPicker(target)
assert.equal(
await readFile(target, 'utf8'),
'before;const target = resolve(path ?? process.env.DSH_DEFAULT_WORKSPACE ?? home);after;',
)
assert.throws(() => patchDirectoryPicker(target), /found 0/)
await writeFile(target, 'const target = resolve(path ?? home);const target = resolve(path ?? home);')
assert.throws(() => patchDirectoryPicker(target), /found 2/)
} finally {
await rm(directory, { recursive: true })
}
})
upstream/container/control-plane/services/gateway/test/dsh-upstream-auth.test.mjs
+84
View File
@@ -0,0 +1,84 @@
import assert from 'node:assert/strict'
import { chmod, mkdtemp } from 'node:fs/promises'
import { createServer } from 'node:http'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import test from 'node:test'
import { DshUpstreamAuthentication } from '../lib/dsh-upstream-auth.mjs'
async function listen(server, target, host) {
await new Promise((resolve, reject) => {
server.once('error', reject)
server.listen(target, host, resolve)
})
}
test('exchanges and caches the private DSH launch URL', async t => {
const root = await mkdtemp(join(tmpdir(), 'dsh-upstream-auth-'))
const socketPath = join(root, 'lifecycle.sock')
let exchanges = 0
const dsh = createServer((request, response) => {
exchanges += 1
assert.equal(request.url, '/?token=fixture-token')
response.writeHead(303, {
location: '/',
'set-cookie': 'dsh-auth-fixture=signed; Path=/; HttpOnly; SameSite=Strict',
})
response.end()
})
await listen(dsh, 0, '127.0.0.1')
const port = dsh.address().port
let generation = 'generation-one'
let publicReady = false
const lifecycle = createServer((_request, response) => {
response.writeHead(200, { 'content-type': 'application/json' })
response.end(JSON.stringify({
generation,
publicReady,
ready: true,
readyUrl: `http://127.0.0.1:${String(port)}/?token=fixture-token`,
}))
})
await listen(lifecycle, socketPath)
await chmod(socketPath, 0o600)
t.after(() => { dsh.close(); lifecycle.close() })
const authentication = new DshUpstreamAuthentication({ socketPath, port })
assert.equal(await authentication.publicReady(), false)
assert.equal(await authentication.cookie(), null)
assert.equal(exchanges, 0)
publicReady = true
assert.equal(await authentication.publicReady(), true)
assert.equal(await authentication.cookie(), 'dsh-auth-fixture=signed')
assert.equal(await authentication.cookie(), 'dsh-auth-fixture=signed')
assert.equal(exchanges, 1)
generation = 'generation-two'
assert.equal(await authentication.cookie(), 'dsh-auth-fixture=signed')
assert.equal(exchanges, 2)
publicReady = false
assert.equal(await authentication.cookie(), null)
publicReady = true
assert.equal(await authentication.publicReady(), true)
assert.equal(await authentication.cookie(), 'dsh-auth-fixture=signed')
assert.equal(exchanges, 3)
})
test('keeps probing legacy DSH readiness without inventing a cookie', async t => {
const root = await mkdtemp(join(tmpdir(), 'dsh-upstream-auth-legacy-'))
const socketPath = join(root, 'lifecycle.sock')
let now = 0
let probes = 0
const lifecycle = createServer((_request, response) => {
probes += 1
response.writeHead(200, { 'content-type': 'application/json' })
response.end(JSON.stringify({ ready: true, readyUrl: null }))
})
await listen(lifecycle, socketPath)
t.after(() => lifecycle.close())
const authentication = new DshUpstreamAuthentication({ socketPath, now: () => now })
assert.equal(await authentication.cookie(), null)
assert.equal(await authentication.cookie(), null)
assert.equal(probes, 1)
now = 250
assert.equal(await authentication.cookie(), null)
assert.equal(probes, 2)
})
upstream/container/control-plane/services/gateway/test/lifecycle.test.mjs
+127
View File
@@ -0,0 +1,127 @@
import assert from 'node:assert/strict'
import { EventEmitter } from 'node:events'
import test from 'node:test'
import { runGateway } from '../lib/lifecycle.mjs'
class FakeServer extends EventEmitter {
listening = false
listen(port, host) {
this.listening = true
this.bound = { port, host }
queueMicrotask(() => this.emit('listening'))
}
close(callback) {
this.listening = false
queueMicrotask(() => callback())
}
closeAllConnections() {}
}
const config = Object.freeze({
polyfill: true,
trustedHosts: Object.freeze({ wildcard: false, authorities: Object.freeze([]) }),
})
test('gateway owns only its listener and binds browser authentication to Access Manager', async () => {
const signalSource = new EventEmitter()
const server = new FakeServer()
let options
const reports = []
const running = runGateway(config, {
externalHost: '127.0.0.1',
externalPort: 8080,
managementSocketPath: '/run/platform.sock',
maintenanceSocketPath: '/run/maintenance.sock',
accessClient: { request: async () => ({ state: 'recovery-required' }) },
signalSource,
report: (message, fields) => { reports.push({ message, fields }) },
gatewayFactory: value => { options = value; return server },
})
await new Promise(resolve => setImmediate(resolve))
assert.deepEqual(server.bound, { host: '127.0.0.1', port: 8080 })
assert.equal(options.managementSocketPath, '/run/platform.sock')
assert.equal(options.maintenanceSocketPath, '/run/maintenance.sock')
assert.equal(typeof options.browserAuthentication.handle, 'function')
assert.equal(typeof options.platformStatus, 'function')
signalSource.emit('SIGTERM')
assert.equal(await running, 0)
assert.equal(server.listening, false)
assert.deepEqual(reports.map(entry => entry.message), [
'gateway.starting', 'gateway.ready', 'gateway.stopping', 'gateway.stopped',
])
assert.equal(reports[2].fields.signal, 'SIGTERM')
})
test('gateway propagates listener failures without creating a DSH process', async () => {
const server = new FakeServer()
const reports = []
server.listen = () => queueMicrotask(() => server.emit('error', new Error('bind failed')))
await assert.rejects(runGateway(config, {
gatewayFactory: () => server,
accessClient: { request: async () => ({ state: 'recovery-required' }) },
signalSource: new EventEmitter(),
report: (message, fields) => { reports.push({ message, fields }) },
}), /bind failed/)
assert.equal(reports.some(entry => entry.message === 'gateway.fatal'
&& entry.fields.error.message === 'bind failed'), true)
})
test('gateway can expose an isolated Management listener from the same process', async () => {
const signalSource = new EventEmitter()
const servers = []
const running = runGateway(config, {
externalHost: '127.0.0.1',
externalPort: 8080,
managementPort: 8081,
managementHost: '127.0.0.1',
gatewayFactory: options => {
const server = new FakeServer()
server.options = options
servers.push(server)
return server
},
accessClient: { request: async () => ({ state: 'recovery-required' }) },
signalSource,
})
await new Promise(resolve => setImmediate(resolve))
assert.equal(servers.length, 2)
assert.deepEqual(servers[0].bound, { host: '127.0.0.1', port: 8080 })
assert.deepEqual(servers[1].bound, { host: '127.0.0.1', port: 8081 })
assert.equal(servers[0].options.surface, 'compat')
assert.equal(servers[1].options.surface, 'management')
signalSource.emit('SIGTERM')
assert.equal(await running, 0)
assert.equal(servers.every(server => server.listening === false), true)
})
test('gateway retains a recent platform status while Management is handed off', async () => {
const signalSource = new EventEmitter()
const server = new FakeServer()
const expected = { update: { status: 'building-candidate' } }
let requests = 0
let options
const running = runGateway(config, {
gatewayFactory: value => { options = value; return server },
accessClient: { request: async () => ({ state: 'recovery-required' }) },
signalSource,
managementClient: {
request: async () => {
requests += 1
if (requests === 1) return expected
throw Object.assign(new Error('Management is switching'), { code: 'ECONNREFUSED' })
},
},
platformStatusPollMs: 5,
})
try {
await new Promise(resolve => setTimeout(resolve, 15))
assert.deepEqual(await options.platformStatus(), expected)
assert.equal(requests >= 2, true)
} finally {
signalSource.emit('SIGTERM')
assert.equal(await running, 0)
}
})
upstream/container/control-plane/services/gateway/test/polyfill.test.mjs
+147
View File
@@ -0,0 +1,147 @@
import assert from 'node:assert/strict'
import test from 'node:test'
import vm from 'node:vm'
import { injectRandomUuidPolyfill, LIFECYCLE_TRANSITION_GUARD, PLUGIN_RECOVERY_GUARD, RANDOM_UUID_POLYFILL, SESSION_VALIDITY_GUARD } from '../lib/polyfill.mjs'
async function simulatePluginError(bundlePath, { eventType = 'error', storage = new Map() } = {}) {
let replacement
const events = []
const listeners = new Map()
const context = {
URL,
crypto: globalThis.crypto,
location: {
href: 'http://gateway.local/sessions/current',
pathname: '/sessions/current',
search: '',
hash: '',
replace: value => { replacement = value },
},
sessionStorage: {
getItem: key => storage.get(key) ?? null,
setItem: (key, value) => storage.set(key, value),
},
addEventListener: (name, callback) => listeners.set(name, callback),
fetch: async (input, init) => {
if (input === '/_dsh_gateway/readiness') return { json: async () => ({ state: 'ready', pluginRecoveryEligible: true }) }
if (input === '/_dsh_platform/plugin-api/v1/status') return {
json: async () => ({
dshLifecycle: {
state: 'running',
taskId: 'lifecycle-task',
updatedAt: new Date().toISOString(),
},
}),
}
if (input === '/_dsh_gateway/client-event') {
events.push(JSON.parse(init.body))
return { json: async () => ({}) }
}
return { ok: true, json: async () => ({}) }
},
}
vm.runInNewContext(PLUGIN_RECOVERY_GUARD.slice('<script>'.length, -'</script>'.length), context)
if (eventType === 'error') listeners.get('error')?.({ filename: `http://gateway.local${bundlePath}` })
else if (eventType === 'unhandledrejection') listeners.get('unhandledrejection')?.({
reason: { stack: `TypeError: failed to import http://gateway.local${bundlePath}` },
})
await new Promise(resolve => setImmediate(resolve))
return { events, replacement, storage }
}
test('polyfill is inserted immediately after the head tag', () => {
assert.equal(
injectRandomUuidPolyfill('<html><head data-x="1"><title>x</title></head></html>'),
`<html><head data-x="1">${RANDOM_UUID_POLYFILL}${LIFECYCLE_TRANSITION_GUARD}${SESSION_VALIDITY_GUARD}${PLUGIN_RECOVERY_GUARD}<title>x</title></head></html>`,
)
})
test('polyfill falls back to the start when no head exists', () => {
assert.equal(injectRandomUuidPolyfill('<main>x</main>'), `${RANDOM_UUID_POLYFILL}${LIFECYCLE_TRANSITION_GUARD}${SESSION_VALIDITY_GUARD}${PLUGIN_RECOVERY_GUARD}<main>x</main>`)
})
test('gateway lifecycle guard moves an already-open DSH page into the holding flow', async () => {
let inspect
let replacement
const context = {
EventSource: class {
addEventListener(name, callback) { if (name === 'state') inspect = callback }
},
URL,
fetch: async () => ({ json: async () => ({ dshLifecycle: { state: 'restarting' } }) }),
location: {
pathname: '/sessions/current', search: '?view=chat', hash: '#latest',
replace: value => { replacement = value },
},
}
vm.runInNewContext(LIFECYCLE_TRANSITION_GUARD.slice('<script>'.length, -'</script>'.length), context)
await new Promise(resolve => setImmediate(resolve))
assert.equal(replacement, '/_dsh_gateway/wait?return=%2Fsessions%2Fcurrent%3Fview%3Dchat%23latest')
replacement = undefined
await inspect()
assert.equal(replacement, undefined)
})
test('plugin recovery guard runs before DSH modules and bounds lifecycle recovery', () => {
assert.doesNotThrow(() => new Function(PLUGIN_RECOVERY_GUARD.slice('<script>'.length, -'</script>'.length)))
assert.match(PLUGIN_RECOVERY_GUARD, /globalThis\.fetch=function/)
assert.match(PLUGIN_RECOVERY_GUARD, /\/_dsh_gateway\/client-event/)
assert.match(PLUGIN_RECOVERY_GUARD, /\/_dsh_gateway\/wait/)
assert.match(PLUGIN_RECOVERY_GUARD, /attempt>=3/)
assert.match(PLUGIN_RECOVERY_GUARD, /\/_dsh_gateway\/plugin-failure/)
assert.match(PLUGIN_RECOVERY_GUARD, /browser\.plugin-load\.recovery\.completed/)
assert.match(PLUGIN_RECOVERY_GUARD, /addEventListener\("error"/)
assert.match(PLUGIN_RECOVERY_GUARD, /unhandledrejection/)
assert.doesNotMatch(PLUGIN_RECOVERY_GUARD, /Failed to load plugins/)
assert.doesNotMatch(PLUGIN_RECOVERY_GUARD, /localStorage/)
})
test('Gateway session guard reloads DSH pages only after Access Manager rejects the session', () => {
assert.doesNotThrow(() => new Function(SESSION_VALIDITY_GUARD.slice('<script>'.length, -'</script>'.length)))
assert.match(SESSION_VALIDITY_GUARD, /auth\/session-context/)
assert.match(SESSION_VALIDITY_GUARD, /response\.status===401/)
assert.doesNotMatch(SESSION_VALIDITY_GUARD, /innerText|textContent/)
})
test('plugin recovery guard catches official and third-party dynamic import failures', async () => {
for (const [path, pluginId] of [
['/plugins/@deepseek-ai/dsh-client-ui-input-trigger/client.js?rev=abc123', '@deepseek-ai/dsh-client-ui-input-trigger'],
['/plugins/community-plugin/client.js?rev=def456', 'community-plugin'],
]) {
const result = await simulatePluginError(path)
assert.match(result.replacement, /^\/_dsh_gateway\/wait\?return=/)
assert.deepEqual(result.events.map(value => [value.event, value.pluginId]), [
['browser.plugin-load.failed', pluginId],
['browser.plugin-load.recovery.started', pluginId],
])
}
})
test('plugin recovery guard retries twice and opens the Gateway failure page after the third failure', async () => {
const path = '/plugins/@deepseek-ai/dsh-client-ui-input-trigger/client.js?rev=abc123'
const storage = new Map()
const first = await simulatePluginError(path, { storage })
const second = await simulatePluginError(path, { storage })
const third = await simulatePluginError(path, { storage })
assert.match(first.replacement, /^\/_dsh_gateway\/wait\?return=/)
assert.match(second.replacement, /^\/_dsh_gateway\/wait\?return=/)
assert.equal(third.replacement, '/_dsh_gateway/plugin-failure')
assert.equal(first.events.at(-1).recoveryAttempt, 1)
assert.equal(second.events.at(-1).recoveryAttempt, 2)
assert.deepEqual(third.events.map(value => [value.event, value.recoveryAttempt]), [
['browser.plugin-load.recovery.failed', 3],
])
})
test('plugin recovery guard does not inspect page text', async () => {
const result = await simulatePluginError('/plugins/example/client.js?rev=abc123', { eventType: 'none' })
assert.equal(result.replacement, undefined)
assert.deepEqual(result.events, [])
})
test('polyfill is feature guarded and uses no weak random source', () => {
assert.match(RANDOM_UUID_POLYFILL, /typeof c\.randomUUID!=="function"/)
assert.match(RANDOM_UUID_POLYFILL, /c\.getRandomValues/)
assert.doesNotMatch(RANDOM_UUID_POLYFILL, /Math\.random/)
})
upstream/container/control-plane/services/gateway/test/proxy.test.mjs
+1142
View File
File diff suppressed because it is too large. Load diff
upstream/container/control-plane/services/gateway/test/trust.test.mjs
+86
View File
@@ -0,0 +1,86 @@
import assert from 'node:assert/strict'
import test from 'node:test'
import { parseTrustedHosts } from '../lib/config.mjs'
import { inspectExternalRequest, isLoopbackHostname } from '../lib/trust.mjs'
test('isLoopbackHostname follows DSH loopback semantics', () => {
for (const hostname of ['localhost', '127.0.0.1', '127.255.255.255', '[::1]']) {
assert.equal(isLoopbackHostname(hostname), true, hostname)
}
for (const hostname of ['example.com', '128.0.0.1', '127.0.0.999', '[::2]']) {
assert.equal(isLoopbackHostname(hostname), false, hostname)
}
})
test('empty trusted Hosts accepts loopback only', () => {
const trusted = parseTrustedHosts({})
assert.equal(inspectExternalRequest({ host: '127.0.0.1:3080' }, trusted).accepted, true)
assert.equal(inspectExternalRequest({ host: 'localhost:3080' }, trusted).accepted, true)
assert.deepEqual(inspectExternalRequest({ host: '192.168.1.10:3080' }, trusted), {
accepted: false,
reason: 'untrusted-host',
})
})
test('trusted Hosts match optional ports and wildcard', () => {
const trusted = parseTrustedHosts({
DSH_TRUSTED_HOSTS: 'lan.example,exact.example:8443,default.example:80',
})
assert.equal(inspectExternalRequest({ host: 'lan.example:9999' }, trusted).accepted, true)
assert.equal(inspectExternalRequest({ host: 'exact.example:8443' }, trusted).accepted, true)
assert.equal(inspectExternalRequest({ host: 'exact.example:3080' }, trusted).accepted, false)
assert.equal(inspectExternalRequest({ host: 'default.example' }, trusted).accepted, true)
assert.equal(inspectExternalRequest({ host: 'default.example:80' }, trusted).accepted, true)
const wildcard = parseTrustedHosts({ DSH_TRUSTED_HOSTS: '*' })
assert.equal(inspectExternalRequest({ host: 'anything.example' }, wildcard).accepted, true)
})
test('request inspection rejects cross-site and mismatched origins', () => {
const trusted = parseTrustedHosts({ DSH_TRUSTED_HOSTS: 'dsh.example' })
assert.deepEqual(inspectExternalRequest({
host: 'dsh.example',
'sec-fetch-site': 'cross-site',
}, trusted), { accepted: false, reason: 'cross-site' })
assert.deepEqual(inspectExternalRequest({
host: 'dsh.example',
origin: 'https://evil.example',
}, trusted), { accepted: false, reason: 'origin-mismatch' })
assert.equal(inspectExternalRequest({
host: 'dsh.example',
origin: 'https://dsh.example',
'sec-fetch-site': 'same-origin',
}, trusted).accepted, true)
})
test('explicit cross-origin inspection still requires a trusted Host', () => {
const trusted = parseTrustedHosts({ DSH_TRUSTED_HOSTS: 'management.example' })
assert.equal(inspectExternalRequest({
host: 'management.example',
origin: 'https://dsh.example',
'sec-fetch-site': 'cross-site',
}, trusted, { allowCrossOrigin: true }).accepted, true)
assert.equal(inspectExternalRequest({
host: 'untrusted.example',
origin: 'https://dsh.example',
'sec-fetch-site': 'cross-site',
}, trusted, { allowCrossOrigin: true }).accepted, false)
})
test('request inspection rejects missing and malformed browser authorities', () => {
const trusted = parseTrustedHosts({ DSH_TRUSTED_HOSTS: '*' })
assert.deepEqual(inspectExternalRequest({}, trusted), { accepted: false, reason: 'missing-host' })
assert.deepEqual(inspectExternalRequest({ host: 'bad host' }, trusted), {
accepted: false,
reason: 'invalid-host',
})
for (const host of ['ok.example/path', 'user@ok.example', 'ok.example?', 'ok.example#x', 'ok.example:']) {
assert.deepEqual(inspectExternalRequest({ host }, trusted), {
accepted: false,
reason: 'invalid-host',
}, host)
}
assert.deepEqual(inspectExternalRequest({ host: 'ok.example', origin: 'null' }, trusted), {
accepted: false,
reason: 'invalid-origin',
})
})
upstream/container/control-plane/services/management/cli.mjs
+442
View File
@@ -0,0 +1,442 @@
import { access, rm } from 'node:fs/promises'
import { createInterface } from 'node:readline/promises'
import { stdin, stdout } from 'node:process'
import { LocalApiClient } from '../../modules/updater/lib/client.mjs'
import { PlatformPaths } from '../../../platform/lib/paths.mjs'
const API_PREFIX = '/_dsh_platform/api/v1'
function usage() {
return `usage: dsh-platform <command> [options]
commands:
status
check
update [--wait]
start|stop|restart [--wait]
channel [stable|experimental]
retry|rollback|return-stable
recover
logs [--source NAME] [--since ISO] [--limit N]
access status|reset|set-username|reset-password
access reset-management-password|disable-management-password|disable-two-factor|clear-sessions [--management-only]|generate-key
access clear-retry [--global-only] [--main-password|--management-password|--two-factor]
trust status|reset`
}
export function parseCli(argv) {
const [command, ...rest] = argv
if (['status', 'check', 'retry', 'rollback', 'return-stable', 'recover'].includes(command)) {
if (rest.length !== 0) throw new Error(usage())
return { command }
}
if (command === 'channel' && (rest.length === 0 || (rest.length === 1 && ['stable', 'experimental'].includes(rest[0])))) {
return { command, channel: rest[0] }
}
if (['update', 'start', 'stop', 'restart'].includes(command)) {
if (rest.some(value => value !== '--wait') || rest.filter(value => value === '--wait').length > 1) throw new Error(usage())
return { command, wait: rest.includes('--wait') }
}
if (command === 'logs') {
const options = { command, sources: [] }
for (let index = 0; index < rest.length; index += 2) {
const name = rest[index]
const value = rest[index + 1]
if (value === undefined) throw new Error(usage())
if (name === '--source') options.sources.push(value)
else if (name === '--since') options.since = value
else if (name === '--limit') options.limit = Number(value)
else throw new Error(usage())
}
return options
}
if (command === 'trust' && rest.length === 1 && ['status', 'reset'].includes(rest[0])) {
return { command, operation: rest[0] }
}
if (command === 'access' && rest.length === 1 && ['status', 'reset', 'set-username', 'reset-password', 'reset-management-password', 'disable-management-password', 'disable-two-factor', 'clear-sessions', 'generate-key', 'image-baseline', 'clear-retry'].includes(rest[0])) {
return { command, operation: rest[0] }
}
if (command === 'access' && rest[0] === 'clear-retry') {
const options = rest.slice(1)
const credentialOptions = options.filter(value => ['--main-password', '--management-password', '--two-factor'].includes(value))
if (options.length > 0 && options.length <= 2 && new Set(options).size === options.length
&& credentialOptions.length <= 1
&& options.every(value => ['--global-only', '--main-password', '--management-password', '--two-factor'].includes(value))) {
return {
command,
operation: 'clear-retry',
...(options.includes('--global-only') ? { globalOnly: true } : {}),
...(options.includes('--main-password') ? { credential: 'main' } : {}),
...(options.includes('--management-password') ? { credential: 'management' } : {}),
...(options.includes('--two-factor') ? { credential: 'totp' } : {}),
}
}
}
if (command === 'access' && rest[0] === 'clear-sessions') {
const options = rest.slice(1)
if (options.length <= 1 && (options.length === 0 || options[0] === '--management-only')) {
return { command, operation: 'clear-sessions', ...(options.length === 1 ? { managementOnly: true } : {}) }
}
}
throw new Error(usage())
}
export async function recoverImageBaseline({
recovery,
input = stdin,
output = stdout,
getuid = () => process.getuid?.(),
ask,
} = {}) {
if (getuid() !== 0) throw new Error('image baseline recovery must run as root from the container console')
if (!input.isTTY || !output.isTTY) throw new Error('image baseline recovery requires an interactive container console')
const status = await recovery.request('GET', '/v1/status')
const imageBuildId = status.imageBaseline?.imageBuildId
if (typeof imageBuildId !== 'string') throw new Error('image baseline is unavailable')
const expected = `RECOVER IMAGE BASELINE ${imageBuildId}`
let answer
if (ask !== undefined) answer = await ask(expected, status)
else {
const prompt = createInterface({ input, output })
answer = await prompt.question(
`Current Deployment may be incompatible with image ${status.imageBaseline.dsh}. Type ${expected}: `,
)
prompt.close()
}
if (answer !== expected) throw new Error('image baseline recovery cancelled')
return recovery.request('POST', '/v1/recover-image-baseline', { confirm: imageBuildId })
}
export async function resetTrust({
dataRoot = '/data/platform',
runRoot = '/run/dsh-platform',
input = stdin,
output = stdout,
getuid = () => process.getuid?.(),
} = {}) {
if (getuid() !== 0) throw new Error('trust reset must run as root from the container console')
if (!input.isTTY || !output.isTTY) throw new Error('trust reset requires an interactive container console')
const paths = new PlatformPaths(dataRoot, runRoot)
const socket = paths.trustSocket
if (await access(socket).then(() => true, () => false)) throw new Error('stop Stage-0 before resetting trust')
const prompt = createInterface({ input, output })
const answer = await prompt.question('Type RESET DSH TRUST to clear accepted trust state: ')
prompt.close()
if (answer !== 'RESET DSH TRUST') throw new Error('trust reset cancelled')
await rm(paths.trustStateRoot, { recursive: true, force: true })
return { status: 'reset', recoveryRoot: 'image' }
}
function queryString(options) {
const query = new URLSearchParams()
for (const source of options.sources ?? []) query.append('source', source)
if (options.since !== undefined) query.set('since', options.since)
if (options.limit !== undefined) query.set('limit', String(options.limit))
const value = query.toString()
return value === '' ? '' : `?${value}`
}
function json(value) {
return JSON.stringify(value)
}
function managementPasswordAction(value) {
const normalized = value.trim().toLowerCase()
if (['', 'n'].includes(normalized)) return 'preserve'
if (normalized === 'd') return 'disable'
if (normalized === 'y') return 'reset'
throw new Error('management password choice must be y, n, or d')
}
function affirmative(value, label) {
const normalized = value.trim().toLowerCase()
if (normalized === 'y') return true
if (['', 'n'].includes(normalized)) return false
throw new Error(`${label} choice must be y or n`)
}
export async function readSecret(input, output, prompt) {
output.write(prompt)
if (typeof input.setRawMode !== 'function') throw new Error('password input must be an interactive TTY')
input.setRawMode(true)
input.resume()
return new Promise((resolve, reject) => {
let value = ''
const cleanup = () => {
input.off('data', onData)
input.off('end', onEnd)
input.off('error', onError)
input.setRawMode(false)
input.pause()
}
const finish = (error, result) => {
cleanup()
if (error === undefined) resolve(result)
else reject(error)
}
const onData = chunk => {
for (const character of String(chunk)) {
if (character === '\r' || character === '\n') {
output.write('\n')
finish(undefined, value)
return
}
if (character === '\u0003') {
finish(new Error('access recovery cancelled'))
return
}
if (character === '\u007f') value = value.slice(0, -1)
else if (character >= ' ') value += character
}
}
const onEnd = () => finish(new Error('password input ended before confirmation'))
const onError = error => finish(error)
input.on('data', onData)
input.once('end', onEnd)
input.once('error', onError)
})
}
const RETRYABLE_CONTROL_PLANE_ERRORS = new Set(['ENOENT', 'ECONNREFUSED', 'ECONNRESET', 'EPIPE'])
function isRetryableControlPlaneError(error) {
return RETRYABLE_CONTROL_PLANE_ERRORS.has(error?.code)
}
export async function runCli({
argv = process.argv.slice(2),
management = new LocalApiClient(process.env.DSH_PLATFORM_MANAGEMENT_CLI_SOCKET ?? '/run/dsh-platform/management-cli.sock'),
trust = new LocalApiClient(process.env.DSH_PLATFORM_TRUST_SOCKET ?? '/run/dsh-platform/stage0-trust.sock'),
reset = resetTrust,
recovery = new LocalApiClient(process.env.DSH_PLATFORM_RECOVERY_SOCKET ?? '/run/dsh-platform/recovery.sock'),
access = new LocalApiClient(process.env.DSH_PLATFORM_ACCESS_RECOVERY_SOCKET ?? '/run/dsh-platform/access/recovery.sock'),
recover = recoverImageBaseline,
write = value => process.stdout.write(`${value}\n`),
delay = milliseconds => new Promise(resolve => setTimeout(resolve, milliseconds)),
input = stdin,
output = stdout,
getuid = () => process.getuid?.(),
readPassword = readSecret,
ask = async question => {
const prompt = createInterface({ input, output })
try { return await prompt.question(question) } finally { prompt.close() }
},
} = {}) {
const parsed = parseCli(argv)
if (parsed.command === 'recover') {
if (getuid() !== 0) throw new Error('image baseline recovery must run as root from the container console')
if (!input.isTTY || !output.isTTY) throw new Error('image baseline recovery requires an interactive container console')
write(json(await recover({ recovery, input, output })))
return 0
}
if (parsed.command === 'access') {
if (getuid() !== 0) throw new Error('access recovery requires root')
if (!input.isTTY || !output.isTTY) throw new Error('access recovery requires an interactive container console')
const current = await access.request('GET', '/v1/recovery/status')
if (parsed.operation === 'status') {
write(json(current))
return 0
}
const confirm = async question => affirmative(await ask(question), 'confirmation')
const cancel = () => {
write(json({ status: 'cancelled' }))
return 0
}
if (parsed.operation === 'generate-key') {
if (!await confirm('Generate an administrator authentication reset key? y/[n]: ')) return cancel()
write(json(await access.request('POST', '/v1/recovery/generate-key')))
return 0
}
if (parsed.operation === 'clear-sessions') {
const question = parsed.managementOnly
? 'Clear all Management console sessions? y/[n]: '
: 'Clear all DSH and Management console sessions? y/[n]: '
if (!await confirm(question)) return cancel()
write(json(await access.request('POST', '/v1/recovery/clear-sessions', {
managementOnly: parsed.managementOnly === true,
})))
return 0
}
if (parsed.operation === 'clear-retry') {
const credentialLabel = parsed.credential === 'main'
? 'main password' : parsed.credential === 'management' ? 'Management console password'
: parsed.credential === 'totp' ? 'two-factor' : 'all administrator'
const question = parsed.credential === 'totp'
? parsed.globalOnly
? 'Clear the instance-wide two-factor daily limit? y/[n]: '
: 'Clear two-factor daily limits? The fixed 10-second retry remains active. y/[n]: '
: parsed.globalOnly
? 'Clear instance-wide administrator login rate limits? y/[n]: '
: `Clear ${credentialLabel} login retry limits? y/[n]: `
if (!await confirm(question)) return cancel()
const authenticationRetry = await access.request('POST', '/v1/recovery/clear-retry', {
scope: parsed.globalOnly ? 'global' : 'all',
...(parsed.credential === undefined ? {} : { credential: parsed.credential }),
})
write(json(authenticationRetry))
return 0
}
if (current.account === null) throw new Error('administrator account is unavailable')
const revision = current.account.revision
if (parsed.operation === 'set-username') {
if (!await confirm('Change administrator username? y/[n]: ')) return cancel()
const username = await ask('New administrator username: ')
write(json(await access.request('POST', '/v1/recovery/set-username', { revision, username })))
return 0
}
if (parsed.operation === 'disable-management-password') {
if (!await confirm('Disable management password? y/[n]: ')) return cancel()
write(json(await access.request('POST', '/v1/recovery/disable-management-password', { revision })))
return 0
}
if (parsed.operation === 'disable-two-factor') {
if (!await confirm('Disable two-factor authentication? y/[n]: ')) return cancel()
write(json(await access.request('POST', '/v1/recovery/reset-access', {
revision,
managementPasswordAction: 'preserve',
totpAction: 'disable',
})))
return 0
}
if (parsed.operation === 'reset') {
let username
let password
if (affirmative(await ask('Change administrator username? y/[n]: '), 'username')) {
username = await ask('New administrator username: ')
}
if (affirmative(await ask('Change administrator password? y/[n]: '), 'password')) {
password = await readPassword(input, output, 'New administrator password: ')
}
let action = 'preserve'
let managementPassword
if (current.account.managementAdditionalCredential.enabled) {
action = managementPasswordAction(await ask(
'Change Management console password? (y)es/[n]o/(d)isable: ',
))
if (action === 'reset') {
managementPassword = await readPassword(input, output, 'New management password: ')
}
}
let totpAction = 'preserve'
if (current.account.totp?.enabled === true) {
if (affirmative(await ask('Disable two-factor authentication? y/[n]: '), 'two-factor')) {
totpAction = 'disable'
}
}
if (username === undefined && password === undefined && action === 'preserve' && totpAction === 'preserve') return cancel()
write(json(await access.request('POST', '/v1/recovery/reset-access', {
revision,
...(username === undefined ? {} : { username }),
...(password === undefined ? {} : { password }),
managementPasswordAction: action,
...(totpAction === 'preserve' ? {} : { totpAction }),
...(managementPassword === undefined ? {} : { managementPassword }),
})))
return 0
}
const confirmed = parsed.operation === 'reset-management-password'
? await confirm('Set or change management password? y/[n]: ')
: await confirm('Change administrator password? y/[n]: ')
if (!confirmed) return cancel()
const password = await readPassword(input, output, parsed.operation === 'reset-management-password'
? 'New management password: ' : 'New administrator password: ')
const route = parsed.operation === 'reset-management-password'
? '/v1/recovery/reset-management-password' : '/v1/recovery/reset-password'
const value = await access.request('POST', route, { revision, password })
write(json(value))
return 0
}
if (parsed.command === 'trust') {
const value = parsed.operation === 'status' ? await trust.status() : await reset()
write(json(value))
return 0
}
if (parsed.command === 'logs') {
const value = await management.request('GET', `${API_PREFIX}/logs${queryString(parsed)}`)
for (const entry of value.entries) write(JSON.stringify(entry))
return 0
}
if (parsed.command === 'update') {
const started = await management.request('POST', `${API_PREFIX}/update`)
write(json(started))
if (!parsed.wait) return 0
let controlPlaneRestarted = false
for (;;) {
let value
try {
value = await management.request('GET', `${API_PREFIX}/status`)
} catch (error) {
if (!isRetryableControlPlaneError(error)) throw error
controlPlaneRestarted = true
await delay(1_000)
continue
}
const terminal = ['success', 'failed'].includes(value.update.status)
const sameTask = value.update.taskId === started.taskId
const resumedTask = controlPlaneRestarted
&& value.update.taskId !== null
&& value.update.operation === 'update'
if (terminal && (sameTask || resumedTask)) {
write(json(value.update))
return value.update.status === 'success' ? 0 : 1
}
await delay(1_000)
}
}
if (['start', 'stop', 'restart'].includes(parsed.command)) {
const started = await management.request('POST', `${API_PREFIX}/${parsed.command}-dsh`)
write(json(started))
if (!parsed.wait) return 0
for (;;) {
const value = await management.request('GET', `${API_PREFIX}/status`)
const lifecycle = value.dshLifecycle
const successState = parsed.command === 'stop' ? 'stopped' : 'running'
if (lifecycle.taskId === started.taskId && [successState, 'failed'].includes(lifecycle.state)) {
write(json(lifecycle))
return lifecycle.state === successState ? 0 : 1
}
await delay(1_000)
}
}
if (parsed.command === 'channel') {
if (parsed.channel === undefined) {
const value = await management.request('GET', `${API_PREFIX}/status`)
write(value.updateChannel)
} else {
const value = await management.request('PUT', `${API_PREFIX}/channel`, { channel: parsed.channel })
write(value.updateChannel)
}
return 0
}
if (parsed.command === 'retry') {
const status = await management.request('GET', `${API_PREFIX}/status`)
const choices = [...status.holds, ...(status.experimentalBlocked === null ? [] : [status.experimentalBlocked])]
const unique = [...new Map(choices.map(value => [value.id, value])).values()]
if (unique.length !== 1) throw new Error('retry requires exactly one active Hold or Blocked combination')
const value = await management.request('POST', `${API_PREFIX}/holds/retry`, { id: unique[0].id })
write(json(value))
return 0
}
if (parsed.command === 'rollback' || parsed.command === 'return-stable') {
const { plan } = await management.request('GET', `${API_PREFIX}/rollback-plan`)
if (plan === null) throw new Error('no complete rollback plan is available')
let confirmDataLoss = false
if (parsed.command === 'return-stable') {
if (!input.isTTY || !output.isTTY) throw new Error('return-stable requires an interactive container console')
const prompt = createInterface({ input, output })
const answer = await prompt.question(`Restore data snapshot from ${plan.snapshot.createdAt}? Type RETURN STABLE AND LOSE NEWER DATA: `)
prompt.close()
if (answer !== 'RETURN STABLE AND LOSE NEWER DATA') throw new Error('return-stable cancelled')
confirmDataLoss = true
}
const value = await management.request('POST', `${API_PREFIX}/${parsed.command}`, {
planId: plan.planId,
...(parsed.command === 'return-stable' ? { confirmDataLoss } : {}),
})
write(json(value))
return 0
}
const method = parsed.command === 'status' ? 'GET' : 'POST'
const value = await management.request(method, `${API_PREFIX}/${parsed.command}`)
write(json(value))
return 0
}
upstream/container/control-plane/services/management/component.json
+30
View File
@@ -0,0 +1,30 @@
{
"schema": 1,
"id": "platform-management",
"type": "service",
"command": {
"executable": "/usr/local/bin/node",
"args": ["/run/dsh-platform/views/bootstrap/control-plane/services/management/index.mjs"],
"timeoutSeconds": 30
},
"environment": {},
"lifecycle": {
"prepare": null,
"preStart": null,
"postStart": null,
"preStop": null,
"stop": null,
"postStop": null
},
"health": {
"type": "exec",
"command": {
"executable": "/usr/local/bin/node",
"args": ["/run/dsh-platform/views/bootstrap/control-plane/services/management/healthcheck.mjs"],
"timeoutSeconds": 5
},
"intervalSeconds": 1,
"timeoutSeconds": 30
},
"logging": { "stdout": true, "stderr": true }
}
upstream/container/control-plane/services/management/dsh-platform.mjs
+10
View File
@@ -0,0 +1,10 @@
#!/usr/bin/env node
import { runCli } from './cli.mjs'
try {
process.exitCode = await runCli()
} catch (error) {
console.error(error instanceof Error ? error.message : 'dsh-platform failed')
process.exitCode = 1
}
upstream/container/control-plane/services/management/healthcheck.mjs
+20
View File
@@ -0,0 +1,20 @@
import { request } from 'node:http'
const socketPath = process.env.DSH_PLATFORM_MANAGEMENT_SOCKET ?? '/run/dsh-platform/management.sock'
const status = await new Promise((resolve, reject) => {
const probe = request({
socketPath,
path: '/_dsh_platform/internal/health',
method: 'GET',
timeout: 2_000,
}, response => {
response.resume()
resolve(response.statusCode)
})
probe.once('timeout', () => probe.destroy(new Error('management health check timed out')))
probe.once('error', reject)
probe.end()
})
if (status !== 200) throw new Error(`management health check returned ${String(status)}`)
upstream/container/control-plane/services/management/index.mjs
+437
View File
@@ -0,0 +1,437 @@
#!/usr/bin/env node
import { readFile } from 'node:fs/promises'
import { randomBytes } from 'node:crypto'
import { existsSync } from 'node:fs'
import { join } from 'node:path'
import { JsonlLogManager } from '../../modules/log-manager/index.mjs'
import { createManagementServer, listenManagement } from './server.mjs'
import { LocalApiClient } from '../../modules/updater/lib/client.mjs'
import { consumeInternalCapability } from '../../../platform/lib/access-capability.mjs'
import { MetadataClient, NpmRegistryClient } from '../../modules/updater/lib/metadata.mjs'
import { TargetPreparer } from '../../modules/updater/lib/preparer.mjs'
import { UpdateCoordinator } from '../../modules/updater/lib/coordinator.mjs'
import { UpdateScheduler } from '../../modules/updater/lib/scheduler.mjs'
import { UpdateStateStore } from '../../modules/updater/lib/state.mjs'
import { AutomaticCheckStateStore } from '../../modules/updater/lib/automatic-check.mjs'
import { PlatformActivator } from '../../modules/updater/lib/activator.mjs'
import { UpdateJournal } from '../../modules/updater/lib/journal.mjs'
import { SnapshotClient } from '../../modules/updater/lib/snapshot-client.mjs'
import { reconcileRecoveredState, resumeInterruptedReconcile } from '../../modules/updater/lib/recovery.mjs'
import { ChannelStateStore } from '../../modules/updater/lib/channel-state.mjs'
import { CompleteStateRecovery } from '../../modules/updater/lib/rollback.mjs'
import { PlatformPaths } from '../../../platform/lib/paths.mjs'
import { readDeploymentStatus } from '../../../platform/lib/deployment-status.mjs'
import { parseImageInventory } from '../../../platform/lib/deployment-contracts.mjs'
import { SettingsDocumentStore } from './settings-document.mjs'
import { markUserPluginRestartState, UserPluginInventory } from '../../modules/plugin-manager/user-inventory.mjs'
import { UserPluginJournal } from '../../modules/plugin-manager/user-journal.mjs'
import { UserPluginSelectionStore } from '../../modules/plugin-manager/user-state.mjs'
import { UserPluginTransactionManager } from '../../modules/plugin-manager/user-transaction.mjs'
import { UserSkillManager } from './user-skills.mjs'
import { createScopedFetch } from './scoped-fetch.mjs'
import { OutboundProxyControlClient } from './outbound-proxy-client.mjs'
import { ProviderInventory } from './provider-inventory.mjs'
import { DshUpstreamAuthentication } from '../gateway/lib/dsh-upstream-auth.mjs'
import { PROXY_SCOPE_CATALOG } from '../outbound-proxy/lib/scope-catalog.mjs'
import { defaultProxyConfiguration } from '../outbound-proxy/lib/contracts.mjs'
import { requestOrigin } from '../gateway/lib/browser-auth.mjs'
import { createRestrictedCliServer, restrictedCliRoute } from './restricted-cli.mjs'
import qrcode from 'qrcode-generator'
const dataRoot = process.env.DSH_PLATFORM_DATA ?? '/data/platform'
const runRoot = process.env.DSH_PLATFORM_RUN ?? '/run/dsh-platform'
const paths = new PlatformPaths(dataRoot, runRoot)
const seedRoot = process.env.DSH_PLATFORM_SEED ?? '/opt/dsh-platform/seed'
const imageInventory = parseImageInventory(await readFile(join(seedRoot, 'inventory.json')))
const trust = new LocalApiClient(paths.trustSocket)
const bootstrap = new LocalApiClient(paths.bootstrapSocket)
const access = new LocalApiClient(paths.accessSocket)
const snapshotApi = new LocalApiClient(paths.snapshotSocket)
const userSkillApi = new LocalApiClient(paths.userSkillSocket)
const outboundProxy = new OutboundProxyControlClient(paths.proxyControlSocket)
const dshHome = process.env.DSH_HOME ?? '/data/dsh'
const logs = new JsonlLogManager({
root: paths.logsRoot,
maxBytes: Number(process.env.DSH_LOG_MAX_BYTES ?? 104857600),
retentionDays: Number(process.env.DSH_LOG_RETENTION_DAYS ?? 14),
output: { stdout: process.stdout, stderr: process.stderr },
})
const updateFetch = createScopedFetch('updates')
const dshUpstreamAuthentication = new DshUpstreamAuthentication({ socketPath: paths.dshLifecycleSocket })
const providerInventory = new ProviderInventory({
cachePath: paths.proxyProviderInventoryPath,
authentication: dshUpstreamAuthentication,
})
function qrCodeDataUri(value) {
const code = qrcode(0, 'M')
code.addData(value, 'Byte')
code.make()
const svg = code.createSvgTag({ cellSize: 5, margin: 10, scalable: true })
return `data:image/svg+xml;base64,${Buffer.from(svg).toString('base64')}`
}
let lastProxyConfiguration
function unavailableProxyConfiguration() {
const fallback = lastProxyConfiguration ?? {
...defaultProxyConfiguration(),
revision: null,
routeHealth: {},
lastTest: null,
}
return Object.freeze({
...fallback,
componentReady: false,
configurationAvailable: lastProxyConfiguration !== undefined,
scopeCatalog: PROXY_SCOPE_CATALOG,
})
}
function proxySnapshot(view) {
return Object.freeze({
revision: view.revision,
configuration: Object.freeze({
schema: view.schema,
enabled: view.enabled,
proxy: view.proxy,
scopes: view.scopes,
environment: view.environment,
modelApi: view.modelApi,
noProxy: view.noProxy,
bypass: view.bypass,
}),
})
}
const proxyConfiguration = async () => {
try {
lastProxyConfiguration = await outboundProxy.configuration()
return Object.freeze({
...lastProxyConfiguration,
configurationAvailable: true,
scopeCatalog: PROXY_SCOPE_CATALOG,
})
} catch (error) {
if (error?.code === 'PROXY_MANAGER_UNAVAILABLE') return unavailableProxyConfiguration()
throw error
}
}
void proxyConfiguration().catch(error => logs.diagnostic('platform-management', 'proxy.configuration.cache.failed', { error }))
logs.on('error', error => { void logs.diagnostic('log-manager', 'capture.failed', { error }) })
await logs.diagnostic('platform-management', 'management.starting', {
imageBuildId: imageInventory.imageBuildId,
targetSequence: imageInventory.targetSequence,
})
const metadata = new MetadataClient({
baseUrl: process.env.DSH_UPDATE_METADATA_URL,
trust,
fetchImpl: updateFetch,
retryRequestTimeoutMs: 15_000,
})
const preparer = new TargetPreparer({ untrustedRoot: paths.downloadsRoot, trust, fetchImpl: updateFetch })
const activator = new PlatformActivator({ dataRoot, runRoot, stage0: trust })
const journal = new UpdateJournal(join(paths.updaterStateRoot, 'transaction.json'))
const snapshots = new SnapshotClient(snapshotApi, 'dsh')
const completeRecovery = new CompleteStateRecovery({ journal, snapshots, activator })
const automaticChecks = new AutomaticCheckStateStore(join(paths.updaterStateRoot, 'automatic-check.json'))
const coordinator = new UpdateCoordinator({
metadata,
preparer,
activator,
state: new UpdateStateStore(join(paths.updaterStateRoot, 'status.json')),
npm: new NpmRegistryClient({ fetchImpl: updateFetch, retryRequestTimeoutMs: 15_000 }),
journal,
snapshots,
probationSeconds: Number(process.env.DSH_EXPERIMENTAL_PROBATION_SECONDS ?? 120),
channelState: new ChannelStateStore(join(paths.updaterStateRoot, 'channel.json')),
completeRecovery,
automaticChecks,
allowUnavailableMetadata: imageInventory.authority === 'development',
report: (message, fields) => logs.diagnostic('updater', message, fields),
})
const settingsDocument = new SettingsDocumentStore(dshHome)
const listBundledPlugins = async () => (await bootstrap.request('GET', '/v1/system-plugins')).plugins
const listSystemSkills = async () => (await bootstrap.request('GET', '/v1/system-skills')).skills
const userSkills = new UserSkillManager({
dshHome,
agentsHome: process.env.DSH_AGENTS_HOME ?? '/home/node/.agents',
})
const userPluginInventory = new UserPluginInventory({
dshHome,
selectionPath: paths.userPluginStatePath,
systemPluginNames: async () => (await listBundledPlugins()).map(plugin => `@dsh-docker/${plugin.id}`),
})
const userPluginTransactions = new UserPluginTransactionManager({
inventory: userPluginInventory,
selectionStore: new UserPluginSelectionStore(paths.userPluginStatePath),
snapshots: new SnapshotClient(snapshotApi, 'user-plugin'),
journal: new UserPluginJournal(paths.userPluginJournalPath),
pauseDsh: () => bootstrap.request('POST', '/v1/components/dsh-runtime/pause'),
restartDsh: () => bootstrap.request('POST', '/v1/components/dsh-runtime/restart'),
report: (message, fields) => logs.diagnostic('user-plugin-manager', message, fields),
})
const initialUserPluginTransaction = await userPluginTransactions.recoverBeforeDshStart()
if (initialUserPluginTransaction !== undefined) {
await logs.diagnostic('user-plugin-manager', 'user-plugin.startup-state.reconciled', {
taskId: initialUserPluginTransaction.taskId,
phase: initialUserPluginTransaction.phase,
recoveryResult: initialUserPluginTransaction.recoveryResult,
})
}
let loadedUserPluginInventory
const markUserPluginsLoaded = async () => {
loadedUserPluginInventory = await userPluginInventory.read()
return loadedUserPluginInventory
}
const listUserPlugins = async () => markUserPluginRestartState(
await userPluginInventory.read(),
loadedUserPluginInventory,
)
const waitForBootstrapStartup = async () => {
for (;;) {
try {
const status = await bootstrap.status()
if (status.startupComplete === true) return status
} catch (error) {
await logs.diagnostic('user-plugin-manager', 'bootstrap.startup-status.retrying', {
error,
level: 'warning',
})
}
await new Promise(resolve => setTimeout(resolve, 100))
}
}
const scheduler = new UpdateScheduler({
check: () => coordinator.check('automatic'),
onError: error => logs.diagnostic('updater', 'update.automatic-check.failed', { error }),
})
let server
const restrictedCliToken = randomBytes(32).toString('base64url')
server = createManagementServer({
coordinator,
logs,
authorizeInternal: async request => {
const token = request.headers['x-dsh-internal-capability']
const audience = request.headers['x-dsh-internal-capability-audience'] === 'plugin' ? 'plugin' : 'management'
const restrictedToken = request.headers['x-dsh-restricted-cli']
if (restrictedToken === restrictedCliToken
&& restrictedCliRoute(request.method ?? 'GET', request.url ?? '/')) return true
if (typeof token !== 'string') return false
if (['/_dsh_platform/api/v1/auth-settings', '/_dsh_platform/api/v1/auth-totp/enrollments', '/_dsh_platform/api/v1/auth-totp/enrollments/confirm', '/_dsh_platform/api/v1/auth-totp/enrollments/cancel', '/_dsh_platform/api/v1/auth-totp/disable-confirmations', '/_dsh_platform/api/v1/auth-totp/disable-confirmations/confirm', '/_dsh_platform/api/v1/auth-totp/disable-confirmations/cancel', '/_dsh_platform/api/v1/management-origin/transitions', '/_dsh_platform/api/v1/management-origin/transitions/commit', '/_dsh_platform/api/v1/management-origin/transitions/failure', '/_dsh_platform/api/v1/auth-sessions/revoke'].includes(request.url)) return true
return consumeInternalCapability(access, {
token, audience, method: request.method ?? 'GET', target: request.url ?? '/',
})
},
createManagementOriginTransition: async (value, request) => {
return access.request('POST', '/v1/management/transitions', {
...value,
internalCapability: request.headers['x-dsh-internal-capability'],
method: request.method,
target: request.url,
})
},
commitManagementOriginTransition: async (value, request) => {
return access.request('POST', '/v1/management/transitions/commit', {
...value,
internalCapability: request.headers['x-dsh-internal-capability'],
method: request.method,
target: request.url,
})
},
reportManagementOriginTransitionFailure: async value => {
if (value?.stage !== 'browser-probe' || !['compat', 'isolated'].includes(value?.mode)) {
throw Object.assign(new Error('Management origin transition failure report is invalid'), { statusCode: 400 })
}
await logs.diagnosticRateLimited('management-origin.probe.failed', 'platform-management', 'management-origin.probe.failed', {
level: 'warning', stage: value.stage, mode: value.mode,
})
return { reported: true }
},
getAuthenticationSettings: async request => {
return access.request('POST', '/v1/management/settings', {
internalCapability: request.headers['x-dsh-internal-capability'],
method: request.method,
target: request.url,
origin: requestOrigin(request, { requireHeader: false }),
})
},
updateAuthenticationSettings: async (value, request) => {
return access.request('POST', '/v1/management/auth-settings', {
...value,
internalCapability: request.headers['x-dsh-internal-capability'],
method: request.method,
target: request.url,
})
},
beginTotpEnrollment: async (value, request) => {
const enrollment = await access.request('POST', '/v1/management/totp/enrollments', {
...value,
internalCapability: request.headers['x-dsh-internal-capability'],
method: request.method,
target: request.url,
})
return { ...enrollment, qrCode: qrCodeDataUri(enrollment.uri), uri: undefined }
},
confirmTotpEnrollment: async (value, request) => {
return access.request('POST', '/v1/management/totp/enrollments/confirm', {
...value,
internalCapability: request.headers['x-dsh-internal-capability'],
method: request.method,
target: request.url,
})
},
cancelTotpEnrollment: async (value, request) => {
return access.request('POST', '/v1/management/totp/enrollments/cancel', {
...value,
internalCapability: request.headers['x-dsh-internal-capability'],
method: request.method,
target: request.url,
})
},
beginTotpDisableConfirmation: async (value, request) => {
return access.request('POST', '/v1/management/totp/disable-confirmations', {
...value,
internalCapability: request.headers['x-dsh-internal-capability'],
method: request.method,
target: request.url,
})
},
confirmTotpDisable: async (value, request) => {
return access.request('POST', '/v1/management/totp/disable-confirmations/confirm', {
...value,
internalCapability: request.headers['x-dsh-internal-capability'],
method: request.method,
target: request.url,
})
},
cancelTotpDisableConfirmation: async (value, request) => {
return access.request('POST', '/v1/management/totp/disable-confirmations/cancel', {
...value,
internalCapability: request.headers['x-dsh-internal-capability'],
method: request.method,
target: request.url,
})
},
revokeAuthenticationSessions: async (value, request) => {
return access.request('POST', '/v1/management/sessions/revoke', {
...value,
internalCapability: request.headers['x-dsh-internal-capability'],
method: request.method,
target: request.url,
})
},
platformStatus: async () => ({
...await readDeploymentStatus(paths.deploymentStatusPath),
trust: await trust.status(),
dshLifecycle: (await bootstrap.status()).dshLifecycle,
}),
startDsh: () => bootstrap.request('POST', '/v1/components/dsh-runtime/resume'),
stopDsh: () => bootstrap.request('POST', '/v1/components/dsh-runtime/pause'),
restartDsh: () => bootstrap.request('POST', '/v1/components/dsh-runtime/restart'),
resetRuntime: () => bootstrap.request('POST', '/v1/deployments/runtime/reset'),
listBundledPlugins,
configureBundledPlugin: (id, action) => bootstrap.request('POST', '/v1/system-plugins/action', { id, action }),
recoverBundledPlugin: (id, action) => bootstrap.request('POST', '/v1/system-plugins/recovery-action', { id, action }),
discardBundledPluginChanges: () => bootstrap.request('POST', '/v1/system-plugins/discard'),
listSystemSkills,
configureSystemSkill: (skillId, action) => bootstrap.request('POST', '/v1/system-skills/action', { skillId, action }),
listUserSkills: () => userSkills.list(),
validateUserSkillAction: value => userSkills.validate(value),
configureUserSkill: value => userSkillApi.request('POST', '/v1/action', value),
listUserPlugins,
markUserPluginsLoaded,
validateUserPluginActions: value => userPluginTransactions.validate(value),
applyUserPluginActions: value => userPluginTransactions.apply(value),
initialUserPluginTransaction,
recoverUserPluginTransaction: initialUserPluginTransaction?.phase === 'restarting' ? async () => {
const status = await waitForBootstrapStartup()
try {
const health = await bootstrap.request('GET', '/v1/health')
const dsh = health.components?.find(component => component.id === 'dsh-runtime')
return userPluginTransactions.completeDshStartup({
healthy: status.recoveryMode === null && dsh?.healthy === true,
error: status.recoveryMode,
})
} catch (error) {
return userPluginTransactions.completeDshStartup({
healthy: false,
error: error instanceof Error ? error.message : String(error),
})
}
} : undefined,
settingsDocument,
privilegedMutationActive: () => existsSync(paths.maintenanceLeasePath),
updateAutomaticCheck: async value => {
const state = await automaticChecks.configure(value)
scheduler.configure(state.automaticCheck)
return state.automaticCheck
},
getProxyConfiguration: proxyConfiguration,
updateProxyConfiguration: async value => Object.freeze({
...await outboundProxy.updateConfiguration(value),
scopeCatalog: PROXY_SCOPE_CATALOG,
}),
listProxyProviders: async () => {
const configuration = await proxyConfiguration()
return providerInventory.list(proxySnapshot(configuration))
},
startProxyTest: value => outboundProxy.startTest(value),
getProxyTest: taskId => outboundProxy.test(taskId),
cancelProxyTest: taskId => outboundProxy.cancelTest(taskId),
})
await listenManagement(server, paths.managementSocket)
const restrictedCliServer = createRestrictedCliServer({
managementSocketPath: paths.managementSocket,
token: restrictedCliToken,
})
await listenManagement(restrictedCliServer, paths.managementCliSocket)
void waitForBootstrapStartup()
.then(markUserPluginsLoaded)
.catch(error => logs.diagnostic('user-plugin-manager', 'user-plugin.loaded-state.capture.failed', { error }))
scheduler.configure((await automaticChecks.read()).automaticCheck)
let recovered
try {
recovered = await reconcileRecoveredState({ journal, state: coordinator.state })
} catch (error) {
await logs.diagnostic('updater', 'update.recovery-state.failed', { error })
throw error
}
const { transaction, persisted } = recovered
const journalOwnsState = transaction !== undefined && persisted.taskId === transaction.transactionId
const resumeUpdate = !journalOwnsState && !['idle', 'success', 'failed'].includes(persisted.status)
if (resumeUpdate) {
setImmediate(() => {
try {
resumeInterruptedReconcile({
coordinator,
persisted,
report: (message, fields) => logs.diagnostic('updater', message, fields),
audit: (message, fields) => logs.diagnostic('audit', message, { stream: 'audit', ...fields }),
})
} catch (error) {
void logs.diagnostic('updater', 'update.resume.failed', { error })
}
})
}
await logs.diagnostic('platform-management', 'management.ready', { resumedUpdate: resumeUpdate })
let stopping = false
const stop = signal => {
if (stopping) return
stopping = true
scheduler.stop()
void logs.diagnostic('platform-management', 'management.stopping', { signal }).then(async () => {
await updateFetch.close().catch(error => logs.diagnostic('platform-management', 'proxy.dispatcher.close.failed', { error }))
let closed = 0
const finish = () => {
closed += 1
if (closed === 2) void logs.diagnostic('platform-management', 'management.stopped').finally(() => process.exit(0))
}
restrictedCliServer.close(finish)
server.close(finish)
})
}
process.once('SIGINT', () => stop('SIGINT'))
process.once('SIGTERM', () => stop('SIGTERM'))
upstream/container/control-plane/services/management/outbound-proxy-client.mjs
+81
View File
@@ -0,0 +1,81 @@
import { request } from 'node:http'
const MAX_RESPONSE_BYTES = 256 * 1024
const UNAVAILABLE_CODES = new Set(['ENOENT', 'ECONNREFUSED', 'ECONNRESET', 'EPIPE', 'ETIMEDOUT', 'ABORT_ERR'])
export class OutboundProxyControlError extends Error {
constructor(error, statusCode, path) {
super(error?.message ?? `Outbound Proxy returned HTTP ${String(statusCode)}`)
this.name = 'OutboundProxyControlError'
this.statusCode = statusCode
this.code = error?.code ?? 'PROXY_MANAGER_UNAVAILABLE'
this.stage = error?.stage ?? 'unknown'
this.retryable = error?.retryable === true
this.localApiPath = path
this.proxyError = Object.freeze({
code: this.code,
message: this.message,
stage: this.stage,
retryable: this.retryable,
})
}
}
export class OutboundProxyControlClient {
constructor(socketPath, { timeoutMs = 10_000 } = {}) {
this.socketPath = socketPath
this.timeoutMs = timeoutMs
}
request(method, path, body) {
return new Promise((resolve, reject) => {
const bytes = body === undefined ? undefined : Buffer.from(JSON.stringify(body))
const outgoing = request({
socketPath: this.socketPath,
method,
path,
headers: bytes === undefined ? {} : {
'content-type': 'application/json',
'content-length': bytes.byteLength,
},
signal: AbortSignal.timeout(this.timeoutMs),
}, response => {
const chunks = []
let size = 0
response.on('data', chunk => {
size += chunk.byteLength
if (size > MAX_RESPONSE_BYTES) outgoing.destroy(new Error('Outbound Proxy response is too large'))
else chunks.push(chunk)
})
response.on('end', () => {
let value
try { value = JSON.parse(Buffer.concat(chunks).toString('utf8')) } catch {
reject(new Error('Outbound Proxy response is invalid'))
return
}
const statusCode = response.statusCode ?? 500
if (statusCode >= 400) reject(new OutboundProxyControlError(value.error, statusCode, path))
else resolve(value)
})
})
outgoing.once('error', error => {
if (UNAVAILABLE_CODES.has(error?.code) || error?.name === 'TimeoutError' || error?.name === 'AbortError') {
reject(new OutboundProxyControlError({
code: 'PROXY_MANAGER_UNAVAILABLE',
message: 'Outbound Proxy control service is unavailable',
stage: 'control',
retryable: true,
}, 503, path))
} else reject(error)
})
outgoing.end(bytes)
})
}
configuration() { return this.request('GET', '/v1/configuration') }
updateConfiguration(value) { return this.request('PUT', '/v1/configuration', value) }
startTest(value) { return this.request('POST', '/v1/test', value) }
test(taskId) { return this.request('GET', `/v1/test/tasks/${encodeURIComponent(taskId)}`) }
cancelTest(taskId) { return this.request('DELETE', `/v1/test/tasks/${encodeURIComponent(taskId)}`) }
status() { return this.request('GET', '/v1/status') }
}
Loaded 100 of 302 files, more files were not shown because too many files have changed in this diff. Show more