# Tonbo Artifacts — full documentation > A shared filesystem on S3, for AI agents, for any sandbox you already run. BYO bucket, multi-mount, daemonized FUSE. Source: https://artifacts.tonbo.io --- # Overview > When an agent workload outgrows the local disk, and what Tonbo Artifacts does about it. AI agents do their best work on a filesystem. Coding agents like Claude Code and OpenCode spend most of their tool calls on `read`, `glob`, and `grep`, and almost any agent you write reaches for `fs.readFile` before it reaches for a database. The problem is where that filesystem lives: a local disk belongs to one machine, and the machines agents run on are increasingly disposable. Tonbo Artifacts gives your agents a workspace that mounts as a regular Linux directory. The agent keeps using plain file operations. The workspace persists after the sandbox is gone, stays in sync across every machine that mounts it, and stores its data in S3 (ours by default, or your own bucket). **CLI on Linux** ```sh # Install once in your VM or container image, then run anywhere curl -fsSL https://artifacts.tonbo.io/install.sh | bash artifacts workspace create my-workspace /mnt/work # creates "my-workspace", mounts it cd /mnt/work && claude ``` > [!TIP] > **Agent-friendly docs.** Paste > [`https://artifacts.tonbo.io/llms.txt`](https://artifacts.tonbo.io/llms.txt) into Claude Code, > Cursor, or any coding agent. It can read these docs end to end and wire itself up. ## When you need this Every server already has a disk. These are the situations where the disk stops being the answer. ### The sandbox is disposable; the agent's work isn't Sandbox VMs, CI runners, and serverless machines are built to be torn down, and everything the agent wrote to local disk goes with them. So you end up writing copy-in/copy-out glue around every run. With a mounted workspace, outputs and intermediate state are durable the moment they're written; the next run mounts the same workspace and picks up where the last one stopped. Start with the [Quickstart](/docs/quickstart). ### More than one machine needs the same files, live The agent writes in its sandbox while your editor reads on your laptop. Ten workers fan out over one dataset. An ingestion daemon writes while an analyzer reads. Syncing copies through S3 or rsync turns your agent loop into a sync engine. Every mount of the same workspace sees the same files, writes appear on other machines within tens of milliseconds, and `flock` works across hosts. See [Share files across machines](/docs/multi-mount). ### The data lives in S3, but your agent speaks files Your corpus sits in a bucket; `grep -r` doesn't take an S3 URL. Downloading the bucket at boot makes cold starts scale with dataset size, and rewriting the agent against an object-store SDK means touching code that already works. A workspace presents that data as files and pulls content on demand. Agent traffic is dominated by `stat`, `readdir`, and `open`, which is exactly the path that generic S3 mounts are slow on and this one is built for. See [Import data](/docs/import-data) and [Performance](/docs/performance). If the data must stay in a bucket you control, [bring your own bucket](/docs/storage-credentials); credentials never leave your hosts. ### You provision storage for your customers' agents You run a sandbox or agent platform, every customer task needs isolated storage, and your platform credentials must never enter a customer-facing runtime. Tenants, per-runtime grants, and short-lived signed mount tickets are the primary API shape here, not a bolt-on. See the [Quickstart for platform users](/docs/platform-quickstart). ## When a local disk is still the right answer - The agent runs on one durable machine and nothing needs to outlive it or be shared. Use the disk; archive to S3 if you want backups. - The workload is streaming multi-GB sequential reads of a few large files (media processing, model weights). A direct S3 client with ranged reads beats any general-purpose filesystem, including this one. - You need a native mount on macOS or Windows. The CLI mounts on Linux amd64 and arm64; a Mac can still [stage data in](/docs/import-data#from-a-mac-folder). ## What your agent sees A regular directory. `ls`, `grep -r`, `fs.readFile`, Claude Code, OpenCode, and anything else that opens files works against `/mnt/work` unchanged. There is no SDK in the agent's code, and the agent doesn't need to know Tonbo Artifacts exists. Under the hood it's a real kernel mount served by the `artifacts` CLI; how that works, and what it costs, is in [Architecture](/docs/architecture) and [Performance](/docs/performance). ## Get started - [**Quickstart**](/docs/quickstart) — Sign up, mount, use it. SDK and CLI shown side by side at every step. - [**Quickstart for platform users**](/docs/platform-quickstart) — End-to-end recipe if you're a platform provisioning workspaces and tickets for your customers' VMs. - [**Share files across machines**](/docs/multi-mount) — Mount one workspace from many machines and let the writes converge. --- # Quickstart > Zero to a mounted workspace with an agent working in it, in five minutes: create and mount, point your agent at durable storage. Give your coding agent a filesystem that survives its sandbox. In the next five minutes you'll mount a workspace at `/mnt/work` and point an agent at durable storage. No object-store SDK, and no changes to the agent's code. Building a platform that provisions isolated storage for your own customers (sandbox runtimes, agent platforms)? Start with [Quickstart for platform users](/docs/platform-quickstart) instead. > [!NOTE] > Prerequisites: a Linux x86_64 host to mount on (sandbox VM, EC2, Fly machine, dev box). No Linux > box handy? Any throwaway cloud VM works. macOS can't mount in v0. ## 1. Sign up and authenticate Register or sign in at [tonbo.dev](https://tonbo.dev/login) with GitHub or Google. **CLI on Linux** ```sh # Install the CLI: curl -fsSL https://artifacts.tonbo.io/install.sh | bash # Then sign in via browser OAuth (GitHub or Google): artifacts login ``` On a host that can't open a browser (CI, a sandbox shell), use an API key instead; see [Log in](/docs/login#headless-hosts-no-browser). ## 2. Create a workspace and mount it A **workspace** is the shared filesystem your agent works in. Think of it as a network drive for your agents: it exists independently of any machine, and any Linux host you authorize can mount it, several at the same time. Pick any name you like; these docs use `my-workspace` as the example name throughout. One command creates the workspace and mounts it: **CLI on Linux** ```sh # "my-workspace" is the workspace name; /mnt/work is where it appears. artifacts workspace create my-workspace /mnt/work ``` **What you should see:** ```text Creating workspace my-workspace (provisioning storage credentials)… Formatting workspace my-workspace… Activating workspace my-workspace… Workspace my-workspace is active. Mounting /mnt/work ... ``` The prompt returns and the mount stays up in the background. Verify it with `mount | grep /mnt/work`. For runtime-specific setup, see [Mount on Linux](/docs/mount), [Mount in Kubernetes](/docs/kubernetes), or [Mount in a container](/docs/mount-container). ## 3. Point your agent at it `/mnt/work` is a regular directory, so an agent uses it natively. Nothing in its code or tools changes. **Coding agent** ```sh cd /mnt/work && claude # or opencode, codex — anything that works on files # Try: "Look around this directory, then write notes/findings.md # describing what you see." ``` **Your own agent (Node.js)** ```ts import fs from "node:fs/promises"; // Persist whatever your agent should remember across runs. await fs.mkdir("/mnt/work/memory", { recursive: true }); await fs.writeFile("/mnt/work/memory/run-001.md", "# Notes from run 001\n"); ``` Every `read`, `glob`, and `grep -r` the agent runs now goes through the workspace. **The agent doesn't need to know Tonbo Artifacts exists.** ## Next steps - [**Persist agent memory across runs**](/docs/agent-memory) — Turn step 3 into a production loop: sandbox boot scripts and a memory contract for your agent. - [**Share files across machines**](/docs/multi-mount) — Mount the same workspace from a second machine; the files are already there. - [**Quickstart for platform users**](/docs/platform-quickstart) — Provision isolated storage for your own customers: tenants, runtimes, grants, and mount tickets. - [**Bring your own bucket**](/docs/storage-credentials) — Keep file data in your own S3 bucket; Tonbo never sees the credentials. --- # Mount on Linux > Mount a Tonbo Artifacts workspace on a Linux host or sandbox VM, and unmount cleanly. For a bare Linux host, an EC2 instance, or a sandbox VM. Running in a container? See [Mount in a container](/docs/mount-container). On Kubernetes? See [Mount in Kubernetes](/docs/kubernetes). The mount surfaces a workspace as a regular Linux directory. Programs use it like a local path without knowing it's remote. Mounting is done by the CLI on the host that needs the filesystem; the TypeScript SDK manages workspaces over HTTP and can't mount. ## Install the client **CLI on Linux** ```sh curl -fsSL https://artifacts.tonbo.io/install.sh | bash ``` See [Install](/docs/install) for offline, pinned-version, or non-root install options. ## Authorize **CLI on Linux** ```sh artifacts login # Browser opens; sign in with GitHub or Google. # Credentials cache at ~/.config/tonbo/artifacts/credentials.json. ``` **TypeScript** ```ts // Generate an API key at tonbo.dev (Profile → API keys), then: export TONBO_API_KEY= // The SDK reads it automatically: const artifacts = new Artifacts(); // or: new Artifacts({ apiKey: '' }) ``` > [!NOTE] > **Running a sandbox platform that mounts on behalf of your customers?** Don't ship your API key > into customer-facing runtimes. Sign short-lived per-task tokens on your backend instead. See > [Quickstart for platform users](/docs/platform-quickstart) for the end-to-end recipe, or > [Temporary mount credentials](/docs/identity-access#mount-sessions) for the conceptual reference. ## Mount a workspace **CLI on Linux** ```sh mkdir -p /mnt/work artifacts mount my-workspace /mnt/work ``` **What you should see:** the CLI prints `Mounting /mnt/work ...` then the prompt returns. The FUSE-serving process detaches into the background; your shell, container entrypoint, or agent runtime can keep going. Verify the mount: ```sh mount | grep /mnt/work # Tonbo:my-workspace on /mnt/work type fuse.juicefs (rw,...) ``` `ls /mnt/work` shows your data plus four system files: `.accesslog`, `.config`, `.stats`, `.trash`. `.stats` is a Prometheus-format counters file you can `cat` or scrape for cache hit rates and request counts. ## Unmount **CLI on Linux** ```sh artifacts unmount /mnt/work ``` Fallbacks if the supervisor process is gone: ```sh fusermount3 -u /mnt/work # or pgrep -af "artifacts mount my-workspace /mnt/work" kill -TERM ``` All paths trigger the same graceful unmount: in-flight operations drain, the kernel mount goes away within ~0.5 seconds. > [!WARNING] > **Don't `kill -9` the mount.** SIGKILL bypasses the unmount handler and leaves a stale FUSE mount. > `ls /mnt/work` then hangs until cleared with `sudo umount -f /mnt/work` (or `umount -l` lazy > fallback). That's the short path. Everything below is opt-in. ## Advanced ### Read-only mode **CLI on Linux** ```sh artifacts mount my-workspace /mnt/work --read-only ``` The kernel mount is registered with `ro`; write syscalls return `EROFS` immediately. The mount session itself is also signed read-only at issue time: for managed workspaces the underlying S3 credentials deny `PutObject` / `DeleteObject`; for BYO workspaces the metadata service rejects write operations server-side. A read-only mount can't be escalated to read-write by bypassing the kernel `ro` flag. ### Cache profile **CLI on Linux** ```sh artifacts mount my-workspace /mnt/work --cache-profile benchmark ``` `interactive` (default), `benchmark`, and `custom` are supported. `benchmark` allocates more buffer and prefetch, which suits a `find`-and-read-heavy agent workload. `custom` accepts additional flags; see `artifacts mount --help`. ### Daemon vs foreground The default is daemon (the shell returns within a second). Pass `--foreground` (`-f`) to keep FUSE attached: ```sh artifacts mount my-workspace /mnt/work --foreground ``` Foreground is useful for ops or debug when you want live mount logs. systemd units use it so the unit can supervise the process. ### Mount automatically on startup There's no `mount.tonbo` fstab helper. To bring a mount up at boot, supervise the foreground process. Two common options. **Via nohup.** For a quick boot-script entry that survives the shell exiting: ```sh nohup artifacts mount my-workspace /mnt/work --foreground \ >> ~/.cache/tonbo/artifacts/mount.log 2>&1 & ``` **Via systemd.** For finer control over lifecycle (logging, restart policy, unit dependencies): **/etc/systemd/system/artifacts-my-workspace.service** ```text [Unit] Description=Tonbo Artifacts mount: my-workspace at /mnt/work After=network-online.target Requires=network-online.target [Service] Type=simple ExecStart=/usr/local/bin/artifacts mount my-workspace /mnt/work --foreground ExecStop=/usr/local/bin/artifacts unmount /mnt/work Restart=on-failure [Install] WantedBy=multi-user.target ``` ```sh sudo systemctl enable --now artifacts-my-workspace.service ``` For runtime templates managed by a cloud platform, inject the signed mount ticket as a file and put the mount command directly in your boot script: ```sh artifacts mount my-workspace /mnt/work \ --ticket-file /run/tonbo/artifacts/mount-ticket.json ``` ### Mount session lifetime Each mount provisions a fresh permission scope on the metadata service. For a CLI-based mount authorized by `artifacts login`, the session is long-lived and revoked when you log out or an admin revokes it. Mounts never auto-expire mid-flight. Cleanup is always explicit via `artifacts unmount`. For platforms issuing short-lived mount tickets to customer runtimes, the lifecycle is different (TTL-bounded, per-session revoke). See [Temporary mount credentials](/docs/identity-access#mount-sessions). ## Command reference See [CLI reference](/docs/cli) for the full subcommand listing. The flags relevant to `mount` and `unmount`: | Flag | Meaning | | ------------------------ | --------------------------------------------------- | | `--read-only` | Mount in read-only mode | | `--foreground` / `-f` | Keep FUSE attached to terminal | | `--cache-profile ` | Cache profile: `interactive`, `benchmark`, `custom` | --- # Mount in a container > Run Tonbo Artifacts in Docker, Podman, or any OCI runtime, bind-mount from the host or run FUSE inside the container. Two patterns. Pick by your runtime: - **From a host mount** (recommended for Docker / Podman / containerd / nerdctl): mount the workspace once on the host, bind it into every container with `-v`. Containers stay fully unprivileged. - **FUSE inside the container** (microVMs / gVisor / Kata): the container runs the CLI itself and mounts inside the guest. Requires `SYS_ADMIN` inside the container. If your runtime shares the host kernel, use the first pattern. Only use the second if you're running a sandbox runtime that doesn't. ## From a host mount (recommended) Mount the workspace on the host once. Bind it into as many containers as you want via `-v`. The containers don't need `SYS_ADMIN`, `/dev/fuse`, or any FUSE awareness, they just see a regular directory. Host setup: ```sh curl -fsSL https://artifacts.tonbo.io/install.sh | bash artifacts login mkdir -p /mnt/work artifacts mount your-org/my-workspace /mnt/work ``` Run containers against the bind: ```sh docker run --rm -it -v /mnt/work:/data your-image ``` For Podman, swap `docker run` for `podman run`. ### docker-compose **docker-compose.yaml** ```yaml services: agent: image: your-image volumes: - /mnt/work:/data ``` ### Why this is the recommended path - Containers stay completely unprivileged, pass any hardened image-scan or admission policy. - One host mount serves N containers simultaneously; no per-container FUSE setup. - Update the CLI on the host once; container images don't carry it. - The container image doesn't need to know Tonbo Artifacts exists. ## FUSE inside the container (microVMs, gVisor, Kata) Use this only when the runtime doesn't share the host kernel, the FUSE-on-host pattern above can't reach into the guest. The CLI runs inside the guest and mounts there. > [!WARNING] > `SYS_ADMIN` (or `--privileged`) inside the container is required for the kernel `mount()` syscall. > It applies inside the container's namespace, not on the host like `--privileged` would, but treat > it as an elevated capability: avoid combining with other broad permissions, and don't grant it to > untrusted images. ### Dockerfile ```dockerfile FROM ubuntu:24.04 RUN apt-get update -qq && \ apt-get install -y -qq curl ca-certificates fuse3 && \ rm -rf /var/lib/apt/lists/* RUN curl -fsSL https://artifacts.tonbo.io/install.sh | bash RUN artifacts --version ``` ### Platform worker mount ticket For Daytona-style platform workers, your backend issues a signed mount ticket. Inject the config into the worker as a secret-backed file, then run the mount config entrypoint inside the guest: ```sh mkdir -p /run/tonbo/artifacts chmod 0700 /run/tonbo/artifacts cp mount-ticket.json /run/tonbo/artifacts/mount-ticket.json chmod 0600 /run/tonbo/artifacts/mount-ticket.json artifacts mount --ticket-file /run/tonbo/artifacts/mount-ticket.json /mnt/work ``` ### Run ```sh docker run --rm -it \ --cap-add SYS_ADMIN \ --device /dev/fuse \ --security-opt apparmor=unconfined \ -v "$PWD/mount-ticket.json:/run/tonbo/artifacts/mount-ticket.json:ro" \ your-image bash ``` Inside the container: ```sh artifacts mount --ticket-file /run/tonbo/artifacts/mount-ticket.json /mnt/work ``` ### docker-compose **docker-compose.yaml** ```yaml services: agent: image: your-image cap_add: - SYS_ADMIN devices: - /dev/fuse security_opt: - apparmor=unconfined volumes: - ./mount-ticket.json:/run/tonbo/artifacts/mount-ticket.json:ro command: > bash -c "artifacts mount --ticket-file /run/tonbo/artifacts/mount-ticket.json /mnt/work && exec your-agent" ``` ### Cleanup Short-lived containers don't need explicit unmount, the kernel tears the mount down when the container exits. For long-running containers rotating tokens: ```sh artifacts unmount /mnt/work cp new-mount-ticket.json /run/tonbo/artifacts/mount-ticket.json chmod 0600 /run/tonbo/artifacts/mount-ticket.json artifacts mount --ticket-file /run/tonbo/artifacts/mount-ticket.json /mnt/work ``` ## Troubleshooting ### Empty /data in container (bind-mount pattern) The host mount didn't come up before `docker run`. Verify `ls /mnt/work` on the host shows your workspace contents first. ### fusermount: permission denied (FUSE-in-container) The container doesn't have `SYS_ADMIN`. Re-check `--cap-add SYS_ADMIN` (Docker / Podman) or `cap_add` (Compose). ### /dev/fuse: no such device (FUSE-in-container) The host kernel doesn't have FUSE loaded, or `--device /dev/fuse` is missing. Confirm `ls -la /dev/fuse` on the host before launching the container. ### mount() blocked by AppArmor or SELinux (FUSE-in-container) On hardened hosts the `mount()` syscall can be denied even with SYS_ADMIN. `--security-opt apparmor=unconfined` (Docker) or labeling the container `unconfined_t` (Podman / SELinux) lets the mount proceed. --- # Mount in Kubernetes > Mount Tonbo Artifacts inside a Kubernetes pod on EKS, GKE, AKS, or kind. For pods on EKS, GKE, AKS, or kind. A bare Linux host has its own [page](/docs/mount); generic Docker / Podman containers have [one too](/docs/mount-container). Tonbo Artifacts mounts via FUSE. Inside Kubernetes that needs one of: - a **privileged** pod, or - the **`SYS_ADMIN`** capability plus `/dev/fuse` exposed as a device. Default Pod Security Standard `restricted` blocks both. `baseline` allows the SYS_ADMIN path. `privileged` allows either. A CSI-driver path that would let fully `restricted` clusters mount without privileged workloads is not shipped in v0 (see the [note](#csi-driver) at the bottom of this page). The rest of this page walks an EKS cluster start-to-mount end-to-end. The pod spec and command sequence work unchanged on GKE, AKS, and kind; only the cluster-creation step differs. ## 1. EKS cluster Smallest viable cluster, one managed node: ```sh eksctl create cluster \ --name tonbo-eks-test \ --region us-west-1 \ --node-type t3.medium \ --nodes 1 \ --node-volume-size 20 \ --managed ``` This takes ~15 minutes for the control plane plus ~5 for the node group. The default Amazon Linux 2023 node ships with the FUSE kernel module already loaded and `/dev/fuse` present. `eksctl` writes a kubeconfig entry. Verify: ```sh kubectl get nodes # NAME STATUS ROLES AGE VERSION # ip-192-168-23-209.us-west-1.compute.internal Ready 2m v1.34.7-eks-7fcd7ec ``` Confirm `/dev/fuse` is on the node and the kernel ships FUSE: ```sh kubectl debug node/$(kubectl get nodes -o name | head -1) -it --image=busybox -- ls -la /host/dev/fuse # crw-rw-rw- 1 root root 10, 229 May 16 04:43 /host/dev/fuse ``` ## 2. Pod spec `artifacts-pod.yaml`: ```yaml apiVersion: v1 kind: Pod metadata: name: artifacts-test spec: restartPolicy: Never containers: - name: artifacts image: ubuntu:24.04 command: ["sleep", "infinity"] securityContext: privileged: true volumeMounts: - name: fuse-device mountPath: /dev/fuse volumes: - name: fuse-device hostPath: path: /dev/fuse type: CharDevice ``` Two non-default bits: - `securityContext.privileged: true`, needed for the `mount()` syscall the FUSE driver calls. Without this the install proceeds but `artifacts mount` fails with `fusermount: permission denied`. - The `hostPath` device volume pulls `/dev/fuse` from the node into the container at the same path. The container's own `/dev` filesystem is a tmpfs by default and doesn't expose `/dev/fuse` automatically even under `privileged`. Apply and wait: ```sh kubectl apply -f artifacts-pod.yaml kubectl wait --for=condition=Ready pod/artifacts-test --timeout=120s ``` ## 3. Install + login + mount inside the pod ```sh kubectl exec -it artifacts-test -- bash ``` Then follow the standard [Quickstart](/docs/quickstart): ```sh apt-get update -qq && apt-get install -y -qq curl ca-certificates curl -fsSL https://artifacts.tonbo.io/install.sh | bash # ==> Release: 0.2.0 (linux-amd64) # ==> fuse3 already installed # ==> sha256 ok: 07e264d209c99c957ee8812ce1a3eb8cc5b28acbb9b67e52a9fad8cf80fd875e # ==> Installed: /usr/local/bin/artifacts artifacts --version # artifacts version 0.2.0 artifacts login --manual-code # see Headless hosts below artifacts workspace create eks-demo --backend managed # Workspace eks-demo is active. mkdir -p /mnt/work artifacts mount eks-demo /mnt/work # Mounting /mnt/work ... ``` Verify: ```sh mount | grep tonbo # Tonbo:tzu-3833a78-eks-demo-0ed546c4dc38 on /mnt/work type fuse.juicefs # (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) ls /mnt/work # .accesslog .config .stats .trash plus your data echo "hello from EKS pod on AL2023 kernel 6.12" > /mnt/work/eks.txt cat /mnt/work/eks.txt mkdir -p /mnt/work/agent-memory/2026/05/case-a/notes echo "deep" > /mnt/work/agent-memory/2026/05/case-a/notes/intake.txt find /mnt/work -type f # /mnt/work/.accesslog # /mnt/work/.stats # /mnt/work/.config # /mnt/work/eks.txt # /mnt/work/agent-memory/2026/05/case-a/notes/intake.txt ``` ## 4. Headless login EKS pods don't have a browser, so the loopback OAuth callback can't reach the CLI. Use one of: - **API key** (preferred for non-interactive flows): mint a `tbo_*` key in your User Center account, ship it to the pod as a Kubernetes secret, and export it. The CLI auto-detects it, so no `artifacts login` is needed: ```sh export TONBO_API_KEY=tbo_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ``` See [API keys](/docs/service-tokens). - **`--manual-code`**: the CLI prints an auth URL; open it in any browser, sign in, paste the one-time code back into the pod terminal. ## 5. Tear down From inside the pod, unmount and delete the workspace first: ```sh artifacts unmount /mnt/work artifacts workspace delete eks-demo ``` Then exit and clean up the cluster: ```sh kubectl delete pod artifacts-test eksctl delete cluster --name tonbo-eks-test --region us-west-1 ``` The cluster delete tears down the eksctl-managed VPC, NAT gateway, nodegroup, and control plane. Allow ~10 minutes. ## Troubleshooting ### fusermount: permission denied The pod isn't actually privileged. Re-check `securityContext.privileged` and confirm your namespace doesn't have a `restricted` Pod Security Standard label blocking it: ```sh kubectl get namespace default -o jsonpath='{.metadata.labels}' ``` ### mount: /mnt/work: special device Tonbo:... does not exist `/dev/fuse` isn't reaching the container. Confirm the `hostPath` volume is on the spec and the node has `/dev/fuse` (`kubectl debug node/ -it --image=busybox -- ls -la /host/dev/fuse`). ### Pod stuck in CreateContainerConfigError Usually a Pod Security Standard violation. Check: ```sh kubectl describe pod artifacts-test | tail -20 ``` If your cluster runs `restricted` PSS on default namespace, create a dedicated namespace with the `privileged` label: ```sh kubectl create namespace tonbo-test kubectl label namespace tonbo-test pod-security.kubernetes.io/enforce=privileged kubectl apply -n tonbo-test -f artifacts-pod.yaml ``` ## CSI driver (not in v0) A CSI driver, Helm chart, and `StorageClass` that would let workload pods consume workspaces through a `PersistentVolumeClaim` (with the kubelet performing the FUSE mount, so the workload stays unprivileged) do **not** ship in v0. There is no Helm chart to install today. In v0, mounting in Kubernetes requires the `privileged` or `SYS_ADMIN` + `/dev/fuse` path shown above. If you need unprivileged mounts on a hardened `restricted` cluster, reach out and tell us about your requirements. --- # Persist agent memory across runs > Give an agent in a disposable sandbox a memory that survives: mount at boot, a directory contract the agent follows, and notes the next run can trust. An agent that boots in a fresh sandbox starts from zero: yesterday's research, half-finished work, and lessons learned died with yesterday's VM. This guide wires a workspace in as the agent's memory, so each run picks up where the last one stopped. ## When you need this - Your agent runs in throwaway sandboxes (Daytona, Modal, Firecracker, CI jobs) and re-learns the same things every run. - You copy results out of the sandbox by hand, or lose them when it exits. - Several runs collaborate on one long task over days. Skip this guide if your agent runs on one durable machine; a local directory already persists there. ## 1. Mount at sandbox boot Install the CLI in the sandbox image once (see [Install](/docs/install)), then mount in the boot script: **Sandbox boot script** ```sh artifacts mount my-workspace /mnt/work ``` Authenticate with an API key if the sandbox is your own infrastructure (`TONBO_API_KEY`, see [API keys](/docs/service-tokens)). If the sandbox runs code you don't fully trust, inject a short-lived mount ticket instead of any account credential: see [Temporary mount credentials](/docs/identity-access). ## 2. Give the agent a memory contract Agents follow instruction files. Point yours at the mount: **CLAUDE.md / AGENTS.md in the sandbox** ```md ## Memory Your memory lives at /mnt/work/memory and survives this sandbox. - Read memory/summary.md before starting. - Append what you learned to memory/runs/.md before finishing. - Working files go in /mnt/work/scratch; final results in /mnt/work/outputs. ``` Nothing else changes. The agent reads and writes plain files; it doesn't need to know the directory is remote. ## 3. Write notes the next run can trust Two habits keep memory clean when runs overlap or die mid-write: - **One file per run.** `memory/runs/2026-06-10-a.md` never collides with another run's notes, even if two sandboxes run at once. - **Publish summaries by rename.** Rewrite the rolling summary to a hidden file, then `mv /mnt/work/memory/.tmp.summary.md /mnt/work/memory/summary.md`. A run that boots mid-update sees the old summary or the new one, never half. ([Concurrent writes](/docs/consistency#publishing-results-readers-can-trust) explains why this is safe.) ## 4. Watch it work Run the agent, destroy the sandbox, boot a new one: ```sh ls /mnt/work/memory/runs/ # 2026-06-09-a.md 2026-06-10-a.md — the new run reads what the old runs left ``` ## Related - [Quickstart](/docs/quickstart): the five-minute version of this loop - [Fan out agents over one dataset](/docs/fan-out-agents): many sandboxes, one task - [Share files across machines](/docs/multi-mount): reading the agent's memory live from your laptop --- # Share files across machines > Mount one workspace from several machines at once: when you need it, what each machine sees, and the write patterns that stay collision-free. A workspace can be mounted by any number of machines at the same time. There is no limit and nothing extra to configure; the only operation is the mount command itself, repeated on every machine that needs the workspace: ```sh artifacts mount my-workspace /mnt/work ``` Every machine sees the same files, and a write on one machine is readable on the others within tens of milliseconds. ## When you need this - **You read results while the agent is still working.** The agent writes in its sandbox; the editor on your laptop, or a reviewer agent elsewhere, reads the same paths live. - **You fan one task out to many workers.** N machines mount the same workspace and each writes its own subpath. - **Producers and consumers run on different machines.** A long-running ingestion daemon writes while short-lived analyzers read. If only one machine has the workspace mounted at a time, skip this page: mount, work, unmount, mount elsewhere later. The files carry over by default; nothing here needs configuring. ## What each machine sees - **The directory tree is exact and immediate.** Create, rename, or delete a file on host A, and a listing on host B reflects it right away. There is no window where two machines disagree about which files exist. - **File contents follow within tens of milliseconds.** Each mount caches data it has read; when another machine changes a file, the next read returns the new bytes. - **Simultaneous writes to the same file don't merge.** If two machines write the same file at the same moment, the result interleaves in 4 MB blocks instead of combining cleanly. Give each writer its own file (patterns below). Exact guarantees: [Concurrent writes](/docs/consistency). - **File locks work across machines.** `flock` held on host A blocks host B, the same way it blocks another process on the same machine. Details: [File locks and coordination](/docs/file-locks-coordination). ## Patterns that stay collision-free - **One file per writer.** Each writer claims its own path, e.g. `outputs/agent-7/result.json`. Nothing needs to coordinate because nothing collides. - **Read-only fan-out.** One machine writes; every other machine mounts with `--read-only`. Reads never conflict. - **Write hidden, rename to publish.** Write to `outputs/.tmp.result.json`, then `mv` it to `outputs/result.json`. The rename is atomic across machines, so a reader sees the old file or the new one, never half of either. ## One thing to watch for - **A result made of several files can be seen half-published.** Files land one by one, so a reader on another machine can list a directory after the first file arrived but before the last. If readers need all-or-nothing, write everything into a hidden directory first and rename it into place as the final step; [Concurrent writes](/docs/consistency#publishing-results-readers-can-trust) shows the recipe. --- # Fan out agents over one dataset > Run many agents on separate machines against one corpus: a work queue they claim from without double-processing, collision-free outputs, and a gather step. One corpus, one big task, N agents on N machines. This guide turns a workspace into the coordination layer: a shared corpus, a work queue agents claim from without double-processing, and an output layout that never collides. ## When you need this - The task is too big for one agent: thousands of documents to analyze, a large repo to migrate, a dataset to label. - Workers are ephemeral and may crash, and no work item may be processed twice or lost. - Results must land in one place for a final aggregation. Skip this guide for a single agent on a single machine; plain directories already do everything you need. ## The layout ```text /mnt/work/ corpus/ the dataset, read by every worker queue/ one file per work item, written once up front claimed/ workers move items here to take ownership outputs/ one directory per worker ``` ## 1. Seed the workspace once From any machine with the workspace mounted: ```sh # The corpus; for bulk imports see Import data. cp -r ./documents /mnt/work/corpus/ # The queue: one file per work item. Here, batches of 50 filenames each. mkdir -p /mnt/work/queue ls /mnt/work/corpus | split -l 50 - /mnt/work/queue/batch- ``` ## 2. Boot the workers Every worker mounts the same workspace with the same command and loops: **Worker loop** ```sh WORKER=worker-$(hostname) mkdir -p /mnt/work/claimed/$WORKER /mnt/work/outputs/$WORKER for item in /mnt/work/queue/*; do # Claim by rename: exactly one worker wins each item. if mv "$item" /mnt/work/claimed/$WORKER/ 2>/dev/null; then name=$(basename "$item") run-agent --task /mnt/work/claimed/$WORKER/$name \ --out /mnt/work/outputs/$WORKER/$name.json mv /mnt/work/claimed/$WORKER/$name /mnt/work/claimed/$WORKER/$name.done fi done ``` The `mv` out of `queue/` is atomic across machines: if two workers grab the same item, one succeeds and the other's `mv` fails, so it moves on to the next item. No lock is held while the agent works, so a crashed worker never blocks the fleet. ## 3. Recover crashed workers A worker that died mid-item leaves a claimed file without the `.done` suffix. Requeue those before the gather step: ```sh find /mnt/work/claimed -type f ! -name '*.done' \ -exec mv {} /mnt/work/queue/ \; ``` ## 4. Gather When `queue/` is empty and every claimed item is `.done`, aggregate and publish: ```sh jq -s '.' /mnt/work/outputs/*/*.json > /mnt/work/.tmp.report.json mv /mnt/work/.tmp.report.json /mnt/work/report.json ``` The final rename publishes the report atomically: anything watching the workspace sees `report.json` appear complete or not at all. ## Related - [Share files across machines](/docs/multi-mount): what each worker's mount sees and when - [File locks and coordination](/docs/file-locks-coordination): claim-by-rename and when you'd want a real lock - [Import data](/docs/import-data): moving a large corpus in efficiently --- # Workspaces > What a workspace is, how it's named and addressed, and the create / list / delete lifecycle with what each step actually does. A workspace is a filesystem you create once and mount anywhere: think of it as a network drive for your agents. Its files outlive any single machine, and any Linux host you authorize can mount it, several at the same time. Each workspace is fully isolated from your other workspaces, and its file data lives in S3: Tonbo's managed bucket by default, or a bucket you bring yourself ([BYO mode](/docs/storage-credentials)). ## Names and handles Every account has a **handle**, a slug like `acme`, chosen at signup. Workspaces live under it: the workspace you call `my-workspace` is fully addressed as `acme/my-workspace`, or `artifacts://acme/my-workspace` in URI form. The bare name works everywhere in the CLI and resolves to your own handle; the longer forms matter when scripting or reading logs. ## Create ```sh artifacts workspace create my-workspace ``` That's the whole command. Tonbo Artifacts allocates a per-workspace prefix in its managed bucket, mints scoped credentials for it, and the CLI runs the in-process format step automatically. Add a mountpoint to create and mount in one step: ```sh artifacts workspace create my-workspace /mnt/work ``` ### Flags | Flag | Default | Notes | | --------------------- | ------------- | ------------------------------------------------- | | `` (positional) | | required | | `--backend` | `managed` | `managed` (default) or `byo` | | `--cache-profile` | `interactive` | `interactive` / `benchmark` / `custom` | | `--bucket` | | required for `--backend byo`; ignored for managed | | `--endpoint` | | required for `--backend byo`; ignored for managed | | `--region` | | byo only | The workspace name has to satisfy `^[a-zA-Z0-9][a-zA-Z0-9_-]{0,62}$`: letters (upper or lower case), digits, hyphens, and underscores, up to 63 characters, starting with a letter or digit. ### Backends - **Managed (default)** — Tonbo Artifacts provisions one workspace-scoped S3 prefix and one IAM credential pair. Customer's environment is not consulted; credentials never leave the Tonbo Artifacts managed bucket. - [**BYO**](/docs/storage-credentials) — Bring your own S3-compatible bucket. AK/SK come from `ARTIFACTS_S3_*` env vars on your host and never travel to the Tonbo Artifacts orchestrator. ### What happens during create **Phase 1: reserve** CLI calls `POST /workspaces` on the orchestrator. The server allocates a per-workspace metadata-namespace prefix `{u_X_w_Y}:` and provisions a 15-minute format-scope permission. **Managed mode also mints an IAM user + access key scoped to the workspace's bucket prefix and returns the AK/SK in the response.** Workspace status: `pending_format`. **Phase 2: format** The bucket is formatted *in-process on your machine*. For managed workspaces the CLI uses the AK/SK from the Phase 1 response; for BYO it reads `ARTIFACTS_S3_*` from the host environment. Either way, the volume's `setting` key is written to Tonbo Artifacts' metadata service under your prefix and the namespace is bootstrapped in the bucket. **Phase 3: confirm** CLI calls `POST /workspaces///format-completed`. The orchestrator verifies the `setting` key exists, flips status to `active`, revokes the format permission. Output on success (managed). The three phases show as progress spinners, then the final line: ```text Creating workspace my-workspace (provisioning storage credentials)… Formatting workspace my-workspace… Activating workspace my-workspace… Workspace my-workspace is active. ``` ### If create fails mid-phase Re-run the same command. Each phase is idempotent: the orchestrator returns the same hashtag prefix on resume, the format token is re-issued, and the namespace bootstrap is re-pointed at the existing keys. Run with `--verbose` to see `Resuming pending_format workspace my-workspace` on the second run. To start over cleanly: ```sh artifacts workspace delete my-workspace --yes artifacts workspace create my-workspace ... ``` ## List ```sh artifacts workspace list # Or machine-readable: artifacts --format json workspace list | jq . ``` ```text IDENTIFIER STATUS BACKEND BUCKET CREATED artifacts://tzu/my-workspace active managed tonbo-managed-storage 2026-05-02T17:37:04Z ``` ## Show ```sh artifacts workspace show my-workspace ``` Includes the storage block (bucket / endpoint / region / prefix), the `status`, the `volume_name` (server-derived volume identifier), and the `metadata_key_prefix` for debugging. ## Delete ```sh artifacts workspace delete my-workspace --yes ``` Returns `202 Accepted`. The orchestrator: 1. Acquires a `FOR UPDATE` lock on the workspace row, flips status to `deleting`. 2. Revokes any active mount sessions (drops each session's metadata-service permissions, marks `revoked_at`). 3. SCANs and DELs every metadata key under `{u_X_w_Y}:*` (your workspace's metadata). 4. **Managed only**: revokes the per-workspace IAM user and best-effort-deletes its prefix in `tonbo-managed-storage`. 5. Flips status to `deleted`. > [!WARNING] > **BYO bucket chunks are not touched.** For BYO workspaces, the Tonbo > Artifacts delete flow only cleans per-workspace state in our metadata > service. To free bucket storage on a BYO bucket, list and delete from > the bucket yourself. > > For managed workspaces Tonbo Artifacts owns the bucket, so the chunks under the workspace's prefix are best-effort-deleted as part of the same flow. A `deleted` workspace name is reusable; the next `workspace create my-workspace` provisions a fresh row with a fresh hashtag. --- # Concurrent writes > What happens when several machines or agents write one workspace at once: what readers see, when same-file writes collide, and how to publish results readers can trust. This page is about what happens when more than one writer works in the same workspace at the same time: agents on different machines, a fleet of workers, or an agent writing while you edit. If your workspace has one writer at a time, skip this page; everything behaves like a local directory, and other machines see finished writes within tens of milliseconds. ## Two machines, one directory tree File and directory operations (create, `mkdir`, `rename`, delete, `chmod`) take effect for every mount the moment they succeed. Two machines never see different directory listings: ```sh # On machine A: mkdir /mnt/work/outputs # On machine B, immediately after: ls /mnt/work # outputs ``` ## Two writers, one file: don't If two machines write the same file at the same time, the file ends up interleaved: blocks of up to 4 MB from one writer mixed with blocks from the other. No bytes are lost or corrupted, but the file as a whole is neither writer's version. This is the one behavior to design around, and the fix is cheap: - **One file per writer.** `outputs/agent-7/result.json` instead of every agent appending to `outputs/all.log`. Collisions become impossible. - **Merge at read time.** The reader concatenates `logs/*.log` when it needs the combined view. One writer per file is always safe, regardless of how many machines write to the workspace. ## When other machines see your write A finished write is readable everywhere within tens of milliseconds. Each mount caches data it has read, and a read after a remote change returns the new bytes. You don't manage this; it's the default. ## Publishing results readers can trust Two situations let a reader catch work in progress: - **A file is read mid-write**, showing only the part written so far. - **A result spans several files** (a report plus its attachments, a directory of shards), and a reader lists the directory after the first file landed but before the last. Files land one by one; there is no way to make several files appear simultaneously. One pattern fixes both: write into a hidden path, then rename into place as the last step. Rename is atomic across machines, so every reader sees the old state or the new state, never something in between. ```sh # Single file: echo "result" > /mnt/work/outputs/.tmp.result.json mv /mnt/work/outputs/.tmp.result.json /mnt/work/outputs/result.json # A result made of several files: stage the directory, rename it last. mkdir -p /mnt/work/outputs/.staging # ... write report.md, chart.png, data.csv into .staging/ ... mv /mnt/work/outputs/.staging /mnt/work/outputs/run-001 ``` A reader polling `outputs/` sees `run-001` appear complete, with everything inside it, or not at all. ## If you need exactly one writer When two machines must not do the same work at the same time (claiming a queue item, rebuilding an index), use a lock or a claim-by-rename: both work across machines. See [File locks and coordination](/docs/file-locks-coordination). --- # File locks and coordination > Cross-machine mutual exclusion for agent fleets: flock that works between machines, claim-by-rename for work queues, and when you don't need a lock at all. This page is for when two machines must not do the same thing at the same time: exactly one agent may claim a work item, exactly one process may rebuild an index, a cleanup job must not run twice. The tools are file locks and atomic renames, both of which work across machines. If your writers each write their own files and nothing contends for a shared resource, skip this page; that design needs no coordination at all (see [Concurrent writes](/docs/consistency)). ## Locks work across machines `flock` and `fcntl` byte-range locks behave between machines the way they behave between two processes on one machine: a lock held on machine A blocks machine B until it's released. This is on by default; nothing to configure. ```sh # On machine A: hold the lock for 60 seconds. flock /mnt/work/jobs.lock -c "sleep 60" ``` ```sh # On machine B, meanwhile: flock --nonblock /mnt/work/jobs.lock -c "process-jobs" # Exits immediately: the lock is held elsewhere. ``` Standard pidfile-style tooling works unchanged. ## What a network lock can and can't promise The lock is held through the workspace, not inside one machine's kernel. If the holder disappears without releasing it (crash, kill, network cut), the lock is reclaimed when that machine's mount session is cleaned up, not instantly. Two consequences: - Keep critical sections short. - Don't build on "the holder can never lose the lock". If stale holders must be fenced out on a timeout you control, use a lock service you operate (Redis `SETNX` with a TTL, a Postgres advisory lock) instead of a file lock. ## Coordination without locks Most agent workflows can make contention impossible instead of managing it: - **One file per writer.** Each writer claims its own path, e.g. `outputs/agent-7/result.json`. Nothing collides, nothing coordinates. - **Claim by rename.** To hand work items to exactly one worker each, let workers claim by moving the item into their own directory: ```sh mv /mnt/work/queue/item-42 /mnt/work/claimed/worker-3/item-42 ``` Rename is atomic across machines: when two workers grab the same item, one `mv` succeeds and the other fails with "no such file" and moves on to the next item. No lock is held while the work runs, so a crashed worker never blocks the queue (its claimed-but-unfinished items are visible under `claimed/` for a janitor to requeue). - **Publish by rename.** Write results to a hidden path and `mv` into place as the last step; readers see complete results or nothing. The multi-file variant is in [Concurrent writes](/docs/consistency#publishing-results-readers-can-trust). --- # Quickstart for platform users > Provision isolated storage for your customers' runtimes end to end: tenant, grant, signed mount ticket, inject, mount, revoke. For platforms whose backends provision storage on behalf of their own customers: sandbox runtimes (Daytona, Modal, your own), AI agent platforms, ML training services, anything where each of your customers gets isolated storage they don't sign up to Tonbo for. End-to-end: sign up, model a tenant, grant a runtime id workspace access, create a short-lived mount ticket, inject the config into the customer runtime, mount, run, clean up. Read this if your backend is provisioning storage per customer task. For the single-machine "I just want to mount my workspace" path, see [Quickstart](/docs/quickstart) instead. > [!NOTE] > Prerequisites: a Node.js backend, an API key from your profile at > [tonbo.dev](https://tonbo.dev/login). Your sandbox VM image needs the `artifacts` CLI installed > (one line at image-build time). ## 1. Sign up and get an API key Register at [tonbo.dev](https://tonbo.dev/login). From your profile, generate an API key. ```sh export TONBO_API_KEY= ``` This key authenticates your backend. **Never inject it into customer-facing runtimes.** ## 2. Install the SDK ```sh pnpm add @tonbo/artifacts ``` **TypeScript** ```ts import { Artifacts } from "@tonbo/artifacts"; const artifacts = new Artifacts(); ``` ## 3. The mental model A short chain: your **root user** (this API key) owns **tenants** (your customers); each tenant scopes **runtimes** (your tasks or sandboxes, identified by an opaque id you choose); an **access grant** lets a tenant or runtime mount a **workspace**; and a **mount ticket** is the short-lived signed file a runtime mounts with. Your root credential stays on your backend; runtimes receive tickets only. See [Multi-tenancy model](/docs/tenancy-model) for the full diagram and lifecycle, and [Temporary mount credentials](/docs/identity-access) for the authorization detail. ## 4. Choose a runtime id for the task **TypeScript** ```ts const tenantSlug = "acme"; const runtimeId = taskId; ``` The tenant is your customer or account boundary; the runtime id is the access scope for one container, VM, task, or agent run. Both are plain strings you choose. You do not pre-create a runtime row. ## 5. Provision or choose a workspace Workspace provisioning runs through the CLI today: the create step initializes the workspace's storage from the machine that runs it, so it isn't available over the plain HTTP API. Run the CLI from your backend host with the same API key (no browser login involved), or pre-provision a pool of workspaces ahead of task time: **Backend host** ```sh export TONBO_API_KEY= artifacts workspace create task-4711 ``` Workspaces are the filesystem namespaces. Some platforms create one workspace per task; others reuse a workspace across several runtimes under the same tenant. If your platform needs to provision workspaces over the HTTP API, reach out. ## 6. Create a short-lived mount ticket This is the one call that matters. `createMountTicket` auto-creates the tenant (if new) and a runtime-scoped grant, then signs a short-lived ticket — all in one request. **TypeScript** ```ts const { mount_ticket } = await artifacts.platform.createMountTicket(tenantSlug, runtimeId, { workspace: `/task-${taskId}`, mode: "rw", ttlSeconds: 3600, }); ``` **What you should see:** a signed mount ticket expiring one hour from now. It authorizes this runtime to mount that one workspace only, in `rw` mode. Your account API key never leaves your backend. > [!NOTE] > That single call is the fast path. You only need explicit `createTenant` / `createAccessGrant` > calls for tenant metadata, a tenant-wide grant shared by every runtime, or to authorize a runtime > without issuing a ticket yet. See [Tenants and grants](/docs/tenants-grants). ## 7. Inject the mount ticket into your runtime The runtime needs the mount ticket available as a secret-backed file before `artifacts mount` runs: ```sh /run/tonbo/artifacts/mount-ticket.json ``` How you inject depends on your orchestration. The common shapes are a Kubernetes secret volume, an init-time file write, or a microVM/containerd secret mount. The `mount-ticket.json` file must contain the JSON-serialized mount ticket returned by Orchestrator. **Kubernetes pod spec** ```yaml volumes: - name: tonbo-artifacts secret: secretName: tonbo-mount defaultMode: 0400 items: - key: mount-ticket.json path: mount-ticket.json containers: - name: agent volumeMounts: - name: tonbo-artifacts mountPath: /run/tonbo/artifacts readOnly: true ``` **Firecracker / Kata / containerd** ```sh # Whatever your launcher is, place the mount ticket file before it runs `artifacts mount`. mkdir -p /run/tonbo/artifacts chmod 0700 /run/tonbo/artifacts cp mount-ticket.json /run/tonbo/artifacts/mount-ticket.json chmod 0600 /run/tonbo/artifacts/mount-ticket.json ``` The Orchestrator returns `{ mount_ticket }` to your backend. Your platform control plane delivers the config to the worker; Orchestrator does not relay or supervise the worker command. ## 8. Mount and use it in the VM In your VM template, install the CLI once at image-build time (see [Install](/docs/install)). At VM boot, mount the workspace: ```sh artifacts mount --ticket-file /run/tonbo/artifacts/mount-ticket.json /mnt/work ``` The customer's agent code uses `/mnt/work` like a regular local directory. **No SDK in their code, no Tonbo awareness, no special API:** **TypeScript** ```ts import fs from "node:fs/promises"; await fs.writeFile("/mnt/work/output.txt", "hello"); ``` This is the value-prop your platform delivers: persistent shared filesystem across your customers' runtimes, using their existing code. ## 9. Tear down when the task ends Revoke the runtime's grant, and delete the workspace if it was task-scoped: **TypeScript** ```ts await artifacts.platform.revokeAccessGrants({ tenantSlug, runtimeId }); await artifacts.workspaces.delete(`/task-${taskId}`); ``` Revocation propagates within about one second; the mount starts erroring on the next syscall. For revoking by grant id, tearing down a whole tenant, tenant-wide grants, and authorize-without-issue, see [Tenants and grants](/docs/tenants-grants). ## Next steps - [**Tenants and grants**](/docs/tenants-grants) — The explicit path: pre-create tenants, tenant-wide grants, authorize without issuing, and revoke. - [**Concurrent writes**](/docs/consistency) — What happens when multiple customer VMs mount the same workspace or write the same file. - [**File locks and coordination**](/docs/file-locks-coordination) — Patterns to give your customers when they need cross-mount mutual exclusion. --- # Multi-tenancy model > How Artifacts models a platform that provisions isolated storage for its own customers: root user, tenant, runtime, workspace, access grant, and mount ticket. You need multi-tenancy when one Tonbo account serves many downstream customers whose storage must stay isolated: a sandbox platform, an agent product with per-customer data, an ML service running customer jobs. The model gives each customer a **tenant** you can grant, meter, and revoke as a unit, and your customers never sign up to Tonbo: your platform holds one Tonbo identity (the root user) and maps your own customer and task ids onto tenants and runtimes. If you mount your own workspaces as yourself, none of these objects exist for you; skip this page. The end-to-end recipe is the [platform quickstart](/docs/platform-quickstart); this page is the model underneath it. ```text ┌─────────────────────────────────┐ │ Root user: your API key │ └────────────────┬────────────────┘ │ owns ▼ ┌─────────────────────────────────┐ │ Tenant: "acme" │ one of your customers └────────────────┬────────────────┘ │ scopes ▼ ┌─────────────────────────────────┐ │ Runtime: "task-123" │ their sandbox / VM / task └────────────────┬────────────────┘ │ you issue it a ▼ ┌─────────────────────────────────┐ │ Mount ticket: signed, short │ the only thing the runtime gets └────────────────┬────────────────┘ │ runtime mounts it at /mnt/work ▼ ┌─────────────────────────────────┐ │ Workspace: "my-workspace" │ a real POSIX filesystem └─────────────────────────────────┘ access grant = the rule that authorizes the ticket: "acme / task-123 may mount my-workspace as ro | rw" ``` Read it top to bottom: your **root user** owns a **tenant** (`acme`), which scopes a **runtime** (`task-123`); your backend issues that runtime a **mount ticket**, which it mounts at `/mnt/work`. The **access grant** is the rule that authorizes the ticket, and the **workspace** (`my-workspace`) is the storage on the other end. ## Root user Your platform's Tonbo identity: a User Center account or an API key (`tbo_…`). It is the **only** identity that signs up to Tonbo, and it stays in your control plane. It owns your tenants, workspaces, grants, and issues every ticket. Never ship it into a customer-facing runtime. ## Tenant One of your customers, projects, or account boundaries. A tenant has a stable `slug` and an optional `external_id` (your own customer id, e.g. `cust_42`) and `metadata`. It is the unit for grouping and for bulk revocation: delete a tenant and every grant and session under it is revoked. The end customer behind a tenant has no Tonbo account. ## Runtime The thing that actually mounts: a task, container, VM, agent run, or sandbox (Daytona, Modal, your own). You identify it with an opaque **runtime id** that you choose. It is a scope value, not a resource you pre-create. Artifacts records the runtime id directly on grants and tickets the first time you use it. ## Workspace The storage namespace being shared, addressed as `artifacts:///`. Managed (Tonbo-issued storage credentials) or BYO (your own S3 bucket). See [Workspaces](/docs/workspaces). ## Access grant The durable authorization edge that connects identity to storage: _this tenant (tenant-wide) or this specific runtime may mount this workspace in `ro` or `rw`_. A grant with no runtime is tenant-wide and applies to any runtime under the tenant; a runtime-scoped grant applies only to requests carrying that runtime id. Grants are what you revoke. See [Temporary mount credentials](/docs/identity-access). ## Mount ticket The short-lived signed file a runtime presents to mount. It wraps the workspace's mount manifest (endpoint, scoped credentials, mode, expiry) and is the **only** material that reaches the customer runtime. The runtime mounts with `artifacts mount --ticket-file /mnt/work`. It carries no Tonbo account and cannot widen its own access. ## How they compose Issuing a ticket is one shot. `createMountTicket(tenant, runtime, { workspace, mode })` (or `artifacts platform ticket create --to /`) creates a missing tenant and a missing runtime-scoped grant in the same call, then issues the ticket. Create tenants and grants explicitly only when you need tenant metadata, a tenant-wide grant, or to authorize without issuing yet. Walk it end to end in [Quickstart for platform users](/docs/platform-quickstart). For grant shapes, revocation, and session lifetime, see [Temporary mount credentials](/docs/identity-access). --- # Tenants and grants > Explicit tenant and grant management for platform backends: pre-create tenants, tenant-wide vs runtime-scoped grants, authorize without issuing, and revoke. The one-shot path covers most platforms: `createMountTicket` auto-creates a missing tenant and a runtime-scoped grant on first use, so you never manage either explicitly. This page is for when you need more than that happy path, and the table below is the trigger list. For what tenants, grants, and tickets are, see [Multi-tenancy model](/docs/tenancy-model). ## When you need explicit calls | You want to | Use | | --- | --- | | Link a tenant to your own customer id, or attach metadata | `createTenant` | | Let every runtime under a tenant mount a workspace (one grant, many runtimes) | tenant-wide `createAccessGrant` | | Authorize a runtime now, issue the ticket later (or many over time) | `createAccessGrant`, then `createMountTicket` per task | | Expire access automatically | `ttlSeconds` on the grant | Otherwise the one-shot `createMountTicket` is enough. ## Create a tenant explicitly **TypeScript** ```ts const tenant = await artifacts.platform.createTenant("acme", { externalId: "cust_42", // your own customer id metadata: { plan: "pro" }, }); ``` `externalId` links the Tonbo tenant to your customer record; `metadata` is free-form. The slug must match `^[a-z0-9][a-z0-9-]{0,62}$`. Creating a tenant that already exists returns a conflict; the one-shot path reuses an existing tenant instead of failing. ## Grant access explicitly A grant is the durable authorization edge. Scope it to one runtime, or make it tenant-wide: **TypeScript** ```ts // Runtime-scoped: only this runtime id can mount, read-only. await artifacts.platform.createAccessGrant("/my-workspace", { tenantSlug: "acme", runtimeId: "task-123", mode: "ro", ttlSeconds: 3600, // optional auto-expiry }); // Tenant-wide: omit runtimeId. Any runtime under acme can mount. await artifacts.platform.createAccessGrant("/my-workspace", { tenantSlug: "acme", mode: "rw", }); ``` With a grant in place you can issue tickets repeatedly for that scope without re-authorizing each task. A pre-existing `ro` grant is never widened to `rw` by a later ticket request. ## Revoke **TypeScript** ```ts // Precise: one grant whose id you tracked. await artifacts.platform.revokeAccessGrant(grantId); // By scope: all grants for a tenant + runtime (tenantSlug is required). await artifacts.platform.revokeAccessGrants({ tenantSlug: "acme", runtimeId: "task-123" }); // Everything under a tenant (the customer churned): tears down the tenant and all its grants and sessions. await artifacts.platform.deleteTenant("acme"); ``` Revocation propagates within about one second; a live mount starts erroring on the next syscall. See [Temporary mount credentials](/docs/identity-access) for session lifetime and the full authorization reference. --- # Temporary mount credentials > Hand a runtime a short-lived mount ticket instead of your API key: issuing, narrowing scope, revocation, and what a ticket can never do. You need a mount ticket when a machine should mount a workspace but must not hold your account credentials: a customer's sandbox, a short-lived worker VM, a CI job you'd rather not trust with a root API key. A ticket is a short-lived signed file that authorizes exactly one mount; if the runtime leaks it, the blast radius is one workspace, one mode, one TTL. This page explains the **mount session** behind every mount, the **mount ticket** that delegates one, and exactly what a ticket can and cannot do. Skip this page if every machine that mounts is one you trust with your own login or API key. Sessions are issued automatically for those mounts, and none of this needs managing. ## Mount sessions A grant says a runtime _may_ mount. Issuing turns that permission into a **mount session**: an authentication record scoped to one workspace, one tenant, optionally one runtime id, one grant, one mode, and one TTL. The **mount ticket** is the signed file that carries that session's credential to the runtime. One runtime id can hold many sessions over its lifetime. ### Why short-lived instead of passing the root credential A customer runtime is an environment you may not fully control. Handing it a long-lived API key means a compromise of the runtime compromises your platform account. A mount session caps the blast radius: bounded in time, bounded to one workspace, bounded by grant mode, and revocable on its own. ## Issuing a ticket **TypeScript** ```ts const { mount_ticket } = await artifacts.platform.createMountTicket("acme", taskId, { workspace: `/task-${taskId}`, mode: "ro", ttlSeconds: 3600, }); ``` `ttlSeconds` is required. There's no implicit long default; pick the smallest TTL that comfortably outlives the runtime task. If the tenant or a runtime-scoped grant doesn't exist yet, this call creates them. Not a platform, just running your own sandbox fleet? Issue a ticket against your own workspace directly, with no tenant objects involved; the ticket behaves identically on the runtime side: **TypeScript** ```ts const { mount_ticket } = await artifacts.workspaces.createMountTicket( "/my-workspace", { mode: "ro", ttlSeconds: 3600, }, ); ``` The Orchestrator returns `{ mount_ticket }` to your backend. The ticket wraps the mount manifest and includes the cleartext session token; you see that token exactly once at create time, and the Orchestrator stores only a hash. Use the session `id` inside the manifest for subsequent lookups and revocation. Your control plane writes the JSON-serialized ticket to the worker as a file before `artifacts mount` runs, e.g. `/run/tonbo/artifacts/mount-ticket.json`. Orchestrator does not inject files, relay commands, or supervise the worker. ## Scoping: narrow, never expand A session can only narrow the grant it's signed from, never widen it: | Active grant | Session request | Session can mount | | ------------------------ | --------------- | ---------------------------------------------- | | tenant-wide `rw` on A | A, `rw` | A at `rw` | | tenant-wide `rw` on A | A, `ro` | A at `ro` | | runtime-scoped `ro` on A | A, `rw` | Rejected; session mode can't exceed grant mode | | runtime-scoped `rw` on A | B, `rw` | Rejected; grant is for A only | | revoked grant on A | A, `ro` | Rejected; grant is not active | The mount-time check runs against current grant state. If you revoke a grant between signing the session and the actual mount, the mount is rejected even if the session token is otherwise valid. ## Revocation Revoke with the calls in [Tenants and grants](/docs/tenants-grants): `revokeAccessGrant(grantId)`, `revokeAccessGrants({ tenantSlug, runtimeId })`, or `deleteTenant(slug)` to tear down everything for a customer. Revocation propagates within about one second; an active mount starts returning errors on the next syscall, and the unmount path is the same as `artifacts unmount`. A delegated mount ticket is **not refreshable** in place. When one approaches its TTL, your backend issues a new ticket and the runtime re-mounts. ## What this isn't - **Not POSIX-based.** POSIX permissions inside the mount are separate. Don't rely on them as a security boundary between mutually untrusted runtimes (see [Concurrent writes](/docs/consistency)). - **Not per-path.** Grants are workspace-level. There's no per-directory scoping. - **Not coordinated writes.** Two runtimes with `rw` on the same workspace can write concurrently; what happens then is [Concurrent writes](/docs/consistency)'s problem, not the grant system's. - **Not transferable.** A session token issued for runtime A can't be used to act as runtime B. ## API summary **TypeScript** ```ts const runtimeId = taskId; // Workspace provisioned ahead of time via the CLI: // artifacts workspace create task-4711 // One-shot: auto-creates the tenant + a runtime-scoped grant, then signs the ticket. const { mount_ticket } = await artifacts.platform.createMountTicket("acme", runtimeId, { workspace: `/task-${taskId}`, mode: "ro", ttlSeconds: 3600, }); // Your control plane writes mount_ticket to the worker as a file, e.g. // /run/tonbo/artifacts/mount-ticket.json, before `artifacts mount` runs. ``` --- # Install the CLI > One-liner installer for the Tonbo Artifacts client. ```sh curl -fsSL https://artifacts.tonbo.io/install.sh | bash ``` Verify: ```sh artifacts --version # artifacts version 0.2.0 ``` > [!NOTE] > Running inside Kubernetes (EKS / GKE / AKS / kind)? The pod needs a privileged container with > `/dev/fuse` exposed before the installer can mount. See [Run in Kubernetes](/docs/kubernetes) for > the full pod spec. ### What the installer does **Platform detection** Selects the native `linux-amd64` or `linux-arm64` release. The Mac can still be used as a *staging source* for [Import data](/docs/import-data#from-a-mac-folder). **fuse3 install** Detects `fusermount3` in `PATH`. If missing, installs `fuse3` via the host's package manager (`apt-get`, `dnf`, `yum`, `apk`). Uses `sudo` if available; falls back to error if not root and no sudo. **Binary download + verification** Downloads the tagged release from the public release bucket, then verifies it against the adjacent `.sha256` file before installation. **Install to PATH** Drops to `/usr/local/bin/artifacts` (root) or `~/.local/bin/artifacts` (non-root). Prints a `PATH` hint if `~/.local/bin` isn't already on `$PATH`. ### Pin a specific version (for CI / reproducible builds) ```sh ARTIFACTS_VERSION=0.2.0 \ curl -fsSL https://artifacts.tonbo.io/install.sh | bash ``` Use this for reproducible CI builds or to match what's running in production. ### Manual install (locked-down hosts that can't pipe to bash) For audit policies or hardened hosts that disallow piping a remote script to `bash`: ```sh VERSION=0.2.0 ARCH=amd64 # use arm64 on aarch64 hosts BASE=https://tonbo-artifacts-builds.s3.us-east-1.amazonaws.com/release curl -fsSL "$BASE/$VERSION/linux-$ARCH/artifacts" -o /tmp/artifacts curl -fsSL "$BASE/$VERSION/linux-$ARCH/artifacts.sha256" -o /tmp/artifacts.sha256 echo "$(cat /tmp/artifacts.sha256) /tmp/artifacts" | sha256sum -c - chmod +x /tmp/artifacts sudo mv /tmp/artifacts /usr/local/bin/artifacts sudo apt-get install -y fuse3 ``` > [!WARNING] > Always sha256-verify before installing. The `.sha256` file is > generated by CI alongside the binary and lives at the same release > prefix. The one-liner installer above does this for you; the manual > flow makes it explicit. --- # Log in > Sign in to the CLI with GitHub or Google, or use an API key for headless hosts. ```sh artifacts login ``` The CLI opens your default browser at `https://tonbo.dev/login`, you sign in, and the browser hands a short-lived authorization back to the CLI on a local callback. Subsequent commands read the saved credentials at `~/.config/tonbo/artifacts/credentials.json` (mode `0600`); you don't need to re-authenticate per shell. ## Verify ```sh artifacts workspace list # {"workspaces":[]} on a fresh login ``` ## Headless hosts (no browser) Some environments can't receive a loopback browser callback, a Fly machine shell, a CI runner, a container with no display. Two options: **Use manual-code login for an interactive remote shell** Manual-code mode still uses the normal OAuth flow, but the browser shows a short one-time code that you paste back into the terminal. ```sh artifacts login --manual-code ``` **Use an API key for fully non-interactive hosts** API keys come from your User Center account, not the CLI. Mint a `tbo_*` key in the dashboard, then export it on the headless host. Both the CLI and the SDK auto-detect it, so no `artifacts login` step is needed: ```sh export TONBO_API_KEY=tbo_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx artifacts mount my-workspace /mnt/work ``` See [API keys](/docs/service-tokens) for the full pattern. ## Logout ```sh artifacts logout ``` Removes `~/.config/tonbo/artifacts/credentials.json`. Doesn't touch the BYO credentials cache (separate file at `~/.config/tonbo/artifacts/byo-credentials`); use `artifacts storage clear` for that. --- # Bring your own bucket > Point a workspace at your own S3 bucket; credentials stay on the mounting host. Every workspace keeps its file data in an S3 bucket, and the one storage decision you make is whose bucket: - **Managed (the default).** Tonbo provisions storage inside its own bucket and handles credentials; nothing to set up, nothing on this page applies. Right for getting started and for teams that don't care which bucket holds the bytes. - **BYO (`--backend byo`).** The workspace points at a bucket you own (AWS S3, Tigris, R2, MinIO, any S3-compatible). Right when data residency, audit visibility, or paying for storage on your own bill matters. Skip the rest of this page unless you're passing `--backend byo`. In BYO mode the orchestrator never sees your S3 credentials. Your AK/SK live only on the machines that mount the workspace. For what objects appear in the bucket once Tonbo Artifacts starts writing, see [What's in your bucket](/docs/storage-layout). ## Where credentials live | Purpose | Directory | | ----------------------- | -------------------------------- | | Config and credentials | `~/.config/tonbo/artifacts` | | Mount cache and logs | `~/.cache/tonbo/artifacts` | | Sensitive runtime state | `~/.local/state/tonbo/artifacts` | | Location | When it's used | | ------------------------------------------------------------ | -------------------------------- | | `ARTIFACTS_S3_ACCESS_KEY_ID` / `_SECRET_ACCESS_KEY` env vars | Always wins; per-shell override | | `~/.config/tonbo/artifacts/byo-credentials` (mode `0600`) | Fallback when env vars are unset | Both `artifacts workspace create` and `artifacts mount` walk this chain: env first, cache file second, error last. > [!NOTE] > Setting `XDG_CONFIG_HOME`, `XDG_CACHE_HOME`, or `XDG_STATE_HOME` overrides the corresponding > `~/.config`, `~/.cache`, or `~/.local/state` root for the CLI. The CLI prints the resolved > absolute path on first write. ## Auto-cache on `workspace create` After a successful `artifacts workspace create`, the CLI saves the current `ARTIFACTS_S3_*` env vars to `~/.config/tonbo/artifacts/byo-credentials` automatically. You'll see the resolved path: ```text Cached BYO credentials at /home//.config/tonbo/artifacts/byo-credentials (mode 0600); future mounts will use them automatically. ``` So a typical first-time BYO setup looks like: ```sh export ARTIFACTS_S3_ACCESS_KEY_ID=... export ARTIFACTS_S3_SECRET_ACCESS_KEY=... artifacts workspace create my-workspace \ --backend byo \ --bucket your-bucket \ --endpoint https://fly.storage.tigris.dev # (cache populated) # New shell, no exports needed: artifacts mount my-workspace /mnt/work ``` ## Manual management **storage set** ```sh # Persist current ARTIFACTS_S3_* env vars to the cache. artifacts storage set # Or rotate just specific fields without touching others. artifacts storage set --access-key-id AKIA... --secret-access-key SECRET artifacts storage set --region us-west-2 # Or set the cache from scratch via flags only (env not needed). artifacts storage set \ --access-key-id AKIA... \ --secret-access-key SECRET \ --region us-west-2 ``` **storage show** ```sh # Print the cache path + masked field values. artifacts storage show # Path: /home/ubuntu/.config/tonbo/artifacts/byo-credentials # ACCESS_KEY_ID: AKIA***726M # SECRET_ACCESS_KEY: LFq0***g03S # SESSION_TOKEN: (unset) # REGION: us-west-1 ``` **storage clear** ```sh # Remove the cache. Future mounts will require ARTIFACTS_S3_* env. artifacts storage clear ``` ## Rotating credentials When you rotate the underlying AK/SK (scheduled rotation, suspected exposure), update the cache with the new pair: ```sh artifacts storage set --access-key-id --secret-access-key ``` Existing mounts on this host keep using the in-memory credentials they were issued at mount time, so they don't break mid-flight. New mounts pick up the new credentials. For a forced re-issue: ```sh artifacts unmount /mnt/work artifacts mount my-workspace /mnt/work ``` ## Per-shell override Env vars always win. So if you have AK/SK cached for the default region but want one shell to hit a different region: ```sh export ARTIFACTS_S3_REGION=eu-west-2 artifacts mount my-workspace /mnt/work ``` The cache's `REGION` field is ignored for this command; `ACCESS_KEY_ID` and `SECRET_ACCESS_KEY` still come from the cache. ## Multiple buckets / workspaces The cache is **global** in v0: one set of credentials across all BYO workspaces on this host. If different workspaces use different buckets with different credentials, manage that via env vars in the shell/container that's mounting each. Managed workspaces don't share the cache; each has its own server-issued credentials. A `--profile` model (mirroring AWS CLI profiles) is on the v1 roadmap. ## Security model > [!NOTE] > The BYO credentials cache is client-side only. It's not synced anywhere; an operator with > `ssm:GetCommandInvocation` permission on the orchestrator EC2 cannot read your customer-side > cache. The orchestrator never holds these values, never proxies S3 reads, and never sees a request that includes them. Customer-side leaks (e.g. you commit `byo-credentials` to a git repo by accident, or the host gets compromised) are handled by: - File mode `0600` (`rw-------`); no other local user can read it. - The cache lives outside the source tree. `~/.config/tonbo/artifacts/` is not git-tracked by default. - Rotate via `artifacts storage set` whenever you suspect exposure. Tonbo Artifacts doesn't need to be told. --- # API keys > Long-lived API keys for headless agents and CI pipelines. API keys are for machines that can't open a browser: CI runners, your backend, a sandbox VM template. `artifacts login` authenticates by bouncing through a browser; an API key is a long-lived `tbo_*` string you mint once in your **User Center** account and export on the host instead, so no login step ever runs there. Skip this page if you only mount from machines where `artifacts login` works; interactive hosts don't need API keys. ## Mint API keys are issued by User Center, not the CLI. Sign in to your account dashboard at `https://tonbo.dev`, open the API keys section, and create a key. The value is shown **once**: ```text tbo_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ``` Save it in your secret store (Fly secrets, GitHub Actions secrets, AWS SecretsManager, 1Password, etc.). ## Use Export the key in the headless environment. Both the CLI and the SDK auto-detect it, so there is no `artifacts login` step: ```sh export TONBO_API_KEY=tbo_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx artifacts mount my-workspace /mnt/work ``` Any command the CLI runs while `TONBO_API_KEY` is set authenticates with that key. The orchestrator accepts both a browser-issued JWT and a `tbo_*` API key as the bearer. ## Scope and permissions An API key authenticates as your account, so it reaches the workspaces your account owns. Access is enforced by the Orchestrator on workspace ownership, not via OAuth scopes (Supabase issues only standard OIDC scopes). Mount mode (`ro` / `rw`) is chosen per mount, not baked into the key. Per-key workspace or mode scoping is not in v0; reach out if you need it. > [!NOTE] > Revoking a key does NOT terminate any mount sessions that the key had already issued. Mount > sessions outlive their issuing token by design; the per-mount metadata-service permissions hold > independently. To force-tear an in-flight mount, the customer has to `artifacts unmount` or you > can delete the workspace. ## Rotation Mint a new key in User Center, deploy it to your CI / secret store, then revoke the old one. There's no built-in rolling-rotation; the same pattern as AWS access keys works fine. --- # Import data > Move existing data into a Tonbo Artifacts workspace from a Mac folder, an S3 bucket, or local disk. A native `artifacts migrate` command is on the v1 list; for v0, use the existing battle-tested S3 / rsync tooling. Pick the recipe that matches where your data lives now. > [!NOTE] > **These recipes are backend-agnostic.** `/mnt/work` is a FUSE mount, > so any tool that writes a regular Linux filesystem works. The chunks > flow to wherever the workspace points, Tonbo Artifacts' managed bucket > by default, or your own BYO bucket. You don't have to think about chunk > storage at import time. > > Any **staging** bucket in the recipes below is independent from the workspace's chunk storage. It's wherever your data has to live temporarily so a Linux host can pull from it. Tigris, R2, AWS S3, MinIO, any S3-compatible bucket you can read works. ## From a Mac folder The v0 mount client is Linux-only, so Mac → Tonbo Artifacts is a **two-hop**: stage to any S3-compatible bucket, then pull from a Linux host that has the workspace mounted. **(if your data is an archive) extract it first** The next step expects a directory tree. If you have an archive, expand it locally: ```sh # tar.zst (zstd), fast, common from `tar --zstd` mkdir -p ~/dataset && zstd -dc dataset.tar.zst | tar -xC ~/dataset # tar.gz tar -xzf dataset.tar.gz -C ~/dataset # zip unzip dataset.zip -d ~/dataset ``` **Pick (or create) a staging bucket and credentials** Any S3-compatible bucket the Linux mount host can read works. Tigris, R2, AWS S3, MinIO, pick whatever you already have keys for. If you're starting from scratch, AWS S3 is one curl away: ```sh aws s3 mb s3://my-staging-bucket --region us-west-1 # Lock down public access on the new bucket: aws s3api put-public-access-block --bucket my-staging-bucket \ --public-access-block-configuration \ BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true ``` Mint an IAM user with read-write on `my-staging-bucket` and copy its access key + secret. The same key pair gets used by `rclone` on the Mac (next step) and by `aws s3 cp` on the Linux host (step after). **On the Mac: rclone stage to the bucket** ```sh brew install rclone ``` Configure rclone. The example uses AWS S3 us-west-1; for Tigris swap `provider=Tigris` and set `endpoint=https://fly.storage.tigris.dev`: ```sh rclone config create staging s3 \ provider=AWS \ access_key_id= \ secret_access_key= \ region=us-west-1 ``` Upload: ```sh rclone copy ~/dataset staging:my-staging-bucket/ \ --transfers 32 --progress ``` On home broadband expect roughly **uplink-Mbps × 0.1 GB/min**; a 5–6 GB / 40k-file dataset over a ~50 Mbps uplink lands in about 15 minutes. **On the Linux mount host: pull from staging into /mnt/work** The Linux box needs read access to the staging bucket. Export the same access key you used for `rclone`: ```sh export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= export AWS_REGION=us-west-1 ``` **aws cli** ```sh aws s3 cp s3://my-staging-bucket/ /mnt/work/ \ --recursive --quiet ``` `--quiet` avoids one log line per file (matters at 40k+ files). For non-AWS providers add `--endpoint-url https://`. Same-region intra-AWS this typically runs at ~10 MB/s per stream; a 5 GB pull is ~10 minutes wall-clock. **Validate, then drop the staging bucket** Pick a representative file and verify cold + warm reads (see [Validation](#validation) below). Once you're satisfied, drop the staging bucket, the workspace owns the data now via its chunk backend (managed or BYO) plus Tonbo Artifacts' metadata service. ```sh aws s3 rm s3://my-staging-bucket --recursive aws s3 rb s3://my-staging-bucket aws iam delete-access-key --user-name --access-key-id aws iam delete-user-policy --user-name --policy-name staging-rw aws iam delete-user --user-name ``` ## From an S3 source If your data already lives in another S3-compatible bucket (Tigris, R2, AWS S3, MinIO), single hop. Make sure the Linux host has read credentials for the source bucket: ```sh export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= export AWS_REGION= ``` **aws cli** ```sh aws s3 cp s3:/// /mnt/work/ \ --recursive --quiet \ --endpoint-url https:// ``` **rclone** ```sh rclone copy : /mnt/work/ \ --transfers 32 --progress ``` Drop `--endpoint-url` if you're on AWS S3 (the SDK figures it out from `AWS_REGION`). Writes flow through `/mnt/work` (FUSE) to the workspace's chunk backend; metadata (inode tree, chunk pointers) lands in Tonbo Artifacts' metadata service. You don't see the chunk-side bucket from inside the mount. ## From a local disk / NFS / etc. If your data already sits on the same Linux host as the mount (or you can `scp` / `rsync` it there), this is the shortest path: ```sh rsync -avP --info=progress2 /local/source/ /mnt/work/ ``` For directory trees with deep parallelism opportunities: ```sh find /local/source -type f \ | parallel -j 32 rsync -aR --info=progress2 {} /mnt/work/ ``` (Requires GNU `parallel`. The `-aR` flag preserves relative paths so the structure mirrors under `/mnt/work`.) If your data is a Mac-side tarball and you don't want to set up a staging bucket, `scp` it to the Linux host first, expand under `/tmp/...`, and use this recipe. Same total cost as the Mac-folder path, one fewer service to configure. ## Validation Always sanity-check after a bulk import: ```sh # Pick a file that's in your real workload (not synthetic). TARGET=/mnt/work/ ls -la "$TARGET" # Cold read: first time pulls the chunk. time cat "$TARGET" >/dev/null # Warm read: should be single-digit ms. time cat "$TARGET" >/dev/null ``` After 60 s of idle, confirm zero object-storage errors via the mount's stats file: ```sh sleep 60 grep -E 'object_request_errors|staging_blocks' /mnt/work/.stats # expected: # juicefs_object_request_errors 0 # juicefs_staging_blocks 0 ``` Quick spot-check that the chunks landed in the workspace's bucket (managed example shown; for BYO substitute your bucket + creds): ```sh VOL=$(artifacts --format json workspace show | jq -r .volume_name) aws s3 ls --recursive --summarize \ s3://tonbo-managed-storage/$VOL/ --region us-west-1 | tail -2 # Total objects + total bytes should roughly match what you imported. ``` ## What about an `artifacts migrate` command? A first-class wrapper that handles all three patterns above with progress, validation, and resume in one CLI is on the v1 list. v0 intentionally leans on the existing tools because they're mature and your Linux host already has them. If your migration ergonomics are blocking your benchmark or production cutover, ping us. We'll prioritise based on what's actually painful. --- # CLI reference > Every Tonbo Artifacts subcommand and flag. ```text artifacts [options] ``` Global flags (apply to every command): | Flag | Meaning | | ------------------ | ----------------------------------------- | | `--verbose` / `-v` | Debug logging | | `--quiet` / `-q` | Warnings + errors only | | `--trace` | Trace logging | | `--format json` | Machine-readable output (where supported) | | `--no-color` | Disable colour output | | `--version` / `-V` | Print version | ## Commands | Command | Purpose | | --- | --- | | `login` | Authenticate against the orchestrator | | `logout` | Forget saved credentials | | `whoami` | Print the currently authenticated identity | | `workspace create` | Create a workspace; mount it too if a mountpoint is given | | `workspace list` | List your workspaces | | `workspace show` | Print one workspace's details | | `workspace delete` | Soft-delete a workspace | | `inspect` | Inspect a workspace's metadata keyspace | | `platform` | Platform-operator subcommands (`tenant`, `grant`, `ticket`) | | `mount` | Mount a workspace at a local path | | `unmount` | Cleanly unmount a workspace | | `storage set` | Persist BYO bucket credentials to local cache | | `storage show` | Print masked cache contents | | `storage clear` | Remove the BYO cache | | `status` | Per-mount health & metrics (Phase 2; prints "not yet implemented") | | `stats` | Workspace-wide telemetry aggregates (Phase 2; prints "not yet implemented") | | `migrate` | Bulk import (Phase 2; prints "not yet implemented"). Use rclone / `aws s3 cp`. | ## `login` ```text artifacts login [--manual-code] ``` Opens a browser to `https://tonbo.dev/login` by default. Stores credentials at `~/.config/tonbo/artifacts/credentials.json` (mode 0600). Use `--manual-code` for SSH sessions, cloud shells, and containers where the browser cannot reach the CLI callback. ## `logout` ```text artifacts logout ``` Removes the credentials file. Doesn't touch the BYO storage cache. ## `workspace create` ```text artifacts workspace create [mountpoint] \ [--backend managed|byo] \ [--bucket ] \ [--endpoint ] \ [--region ] \ [--cache-profile interactive|benchmark|custom] ``` `` matches `^[a-zA-Z0-9][a-zA-Z0-9_-]{0,62}$` (letters, digits, `-`, `_`; up to 63 characters). Each flag has an `ARTIFACTS_S3_*` env var fallback (see the [Workspaces guide](/docs/workspaces#create) for the full table). Pass a `[mountpoint]` to mount the new workspace in the same command, equivalent to `workspace create` followed by `mount`. The mountpoint directory is created if missing and the mount daemonizes as usual: ```text artifacts workspace create my-workspace /mnt/work ``` Output on success: ```text Workspace my-workspace is active. ``` Re-running the same command on a `pending_format` workspace resumes provisioning from the last successful step. ## `workspace list / show / delete` ```text artifacts workspace list artifacts workspace show artifacts workspace delete [--yes] ``` `--yes` skips the interactive confirmation prompt. The workspace argument accepts three forms in every command (`create`, `show`, `delete`, `mount`): | Form | Example | When to use | | --- | --- | --- | | `` | `my-workspace` | Default; resolves to your own handle | | `/` | `tzu/my-workspace` | Unambiguous, useful when scripting | | `artifacts:///` | `artifacts://tzu/my-workspace` | Fully-qualified URI; same effect as the slash form | ## `mount` ```text artifacts mount \ [--read-only] \ [--foreground|-f] \ [--cache-profile ] \ [--cache-root ] [--cache-size ] artifacts mount --manifest \ [--read-only] \ [--foreground|-f] \ [--cache-profile ] \ [--cache-root ] [--cache-size ] ``` Default is daemonized (background). `--foreground` / `-f` keeps FUSE attached to the terminal, useful for debugging. `--read-only` mounts the workspace read-only; there is no `-r` short form. `--manifest ` mounts from supplied manifest material instead of issuing a new mount session. Platform worker agents use this form when they inject a manifest file into the worker machine; ops can also use it to replay a saved manifest. Only the mountpoint is positional in that case. Worker flags may narrow access with `--read-only`, but cannot widen a read-only manifest to writable. For platform-issued mount tickets (see `platform ticket`), pass the ticket and the material used to verify its signature: | Flag | Meaning | | -------------------------------------- | ----------------------------------------------- | | `--ticket ` | Mount-session ticket value | | `--ticket-file ` | Read the ticket from a file instead of the flag | | `--ticket-public-key ` | Public key to verify the ticket signature | | `--ticket-jwks-url ` | JWKS endpoint to fetch verification keys | | `--ticket-skip-signature-verification` | Skip signature verification (dev/testing only) | ## `unmount` ```text artifacts unmount ``` Calls `fusermount3 -u`. Equivalent to `kill -TERM` on the daemon. ## `key` API keys (`tbo_*`) are issued and managed in your User Center account at `https://tonbo.dev`, not from the CLI. Export the key as `TONBO_API_KEY` and the CLI and SDK pick it up automatically; no `artifacts login` is needed. See [API keys](/docs/service-tokens). ```sh export TONBO_API_KEY=tbo_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx artifacts mount my-workspace /mnt/work ``` > [!NOTE] > The `artifacts key` subcommand group is not yet wired to the orchestrator. Mint and manage keys in > User Center for now. ## `storage set / show / clear` ```text artifacts storage set [--access-key-id ...] [--secret-access-key ...] [--session-token ...] [--region ...] artifacts storage show artifacts storage clear ``` Manages the local BYO credential cache at `~/.config/tonbo/artifacts/byo-credentials` (mode 0600). `set` with no flags captures the current `ARTIFACTS_S3_*` env vars. ## Environment variables the CLI reads | Variable | Used by | Effect | | --- | --- | --- | | `TONBO_API_KEY` | all commands | User-Center-issued `tbo_*` API key; auto-detected for headless auth | | `ARTIFACTS_S3_ACCESS_KEY_ID` | `workspace create`, `mount` | BYO bucket access key | | `ARTIFACTS_S3_SECRET_ACCESS_KEY` | same | BYO bucket secret | | `ARTIFACTS_S3_SESSION_TOKEN` | same | (optional) STS session token | | `ARTIFACTS_S3_REGION` | same | bucket region | | `ARTIFACTS_S3_BUCKET` | `workspace create` | default for `--bucket` | | `ARTIFACTS_S3_ENDPOINT` | `workspace create` | default for `--endpoint` | | `AWS_REGION` / `AWS_DEFAULT_REGION` | fallback for region | | | `ARTIFACTS_VERSION` | `install.sh` | pin a specific build | --- # Error codes > API error responses, with hints for what to do next. The orchestrator returns errors as JSON: ```json { "code": "", "message": "" } ``` The CLI surfaces both, and adds a one-line hint when applicable. ## Authentication | Code | HTTP | What it means | Action | | --- | --- | --- | --- | | `unauthorized` | 401 | Missing / invalid bearer token | Re-run `artifacts login`; check the JWT hasn't expired. | | `forbidden` | 403 | Token valid but lacks permission | The user (or service token) doesn't own the workspace. Check `artifacts workspace list`. | ## Workspaces | Code | HTTP | What it means | Action | | --- | --- | --- | --- | | `workspace_not_found` | 404 | Identifier doesn't match an existing workspace | Confirm the workspace name is right; `artifacts workspace list` shows what you own. The CLI accepts the short form `` (resolves to your own handle), `/`, or the full `artifacts:///`. | | `workspace_exists` | 409 | A workspace with that name already exists for this user | Workspace names are unique per user across all of your Agents. Pick a different name, or `artifacts workspace delete ` first. When the existing workspace is `pending_format` and belongs to the Agent you are creating for, the 409 body includes an `existing` field with the live state, which `artifacts workspace create` uses to resume automatically. A name held by a different Agent returns no `existing` field. | | `workspace_not_active` | 409 | Tried to mount or `format-completed` against a workspace that's not `active` | If `pending_format`, re-run `artifacts workspace create ` to resume. If `deleting` / `deleted`, recreate. | | `workspace_not_completable` | 409 | `/format-completed` called on something other than `pending_format` | Almost always means the workspace is already `active`, in which case the orchestrator returns 200 idempotently via the active short-circuit. If you're seeing this 409, it's likely a stale CLI binary; reinstall. | | `invalid_request` | 422 | `/format-completed` called with no `setting` key under the workspace prefix in the metadata service | Phase 2 didn't actually run, or it ran against a different bucket. The `message` tells you to re-run `juicefs format`; in practice re-run `artifacts workspace create`. | ## Platform tenants | Code | HTTP | What it means | Action | | --- | --- | --- | --- | | `tenant_exists` | 409 | A tenant with that slug already exists for this user | Reuse the existing tenant slug in subsequent calls, pick a different slug, or delete the tenant before recreating it. | ## Mount sessions | Code | HTTP | What it means | Action | | --- | --- | --- | --- | | `metadata_acl_provisioning_failed` | 500 | The orchestrator couldn't reach the metadata service to set up your mount-session permissions | **Transient.** Retry in a few seconds. If it persists past 60 s, contact us; there's an alert on our side too. | ## Validation | Code | HTTP | What it means | Action | | --- | --- | --- | --- | | `invalid_request` | 422 | Request body / params don't satisfy the schema | The `message` field has the specific complaint, e.g. `storage.bucket is required` or `ttl_seconds must be between 60 and ...`. | ## Internal | Code | HTTP | What it means | Action | | --- | --- | --- | --- | | `internal_error` | 500 | Unexpected server-side failure | Retry once. If it repeats, contact us with the timestamp + your workspace identifier. | ## Client-side errors These never have an HTTP code; the CLI surfaces them before the request hits the orchestrator. | Symptom | Hint | | --- | --- | | `path /mnt/work is already a mount point` | `artifacts unmount /mnt/work` first. The CLI reads `/proc/self/mountinfo` to detect this. | | `BYO mode requires ARTIFACTS_S3_ACCESS_KEY_ID and ARTIFACTS_S3_SECRET_ACCESS_KEY` | Export them in your shell or run `artifacts storage set --access-key-id ... --secret-access-key ...`. | | `byo_env requires ARTIFACTS_S3_ACCESS_KEY_ID and ARTIFACTS_S3_SECRET_ACCESS_KEY (set them in the environment or run \`artifacts storage set\`)` | Same as above; the cache is also empty. | ## Bucket / S3 errors during `workspace create` Phase 2 These come straight from the storage backend; the message includes the underlying S3 error code. For **BYO** workspaces: | What you'll see | Likely cause | | --- | --- | | `Object storage test failed: ... AccessDenied` | Bucket policy doesn't grant your AK list/get/put. | | `Object storage test failed: ... NoSuchBucket` | Bucket name typo or wrong endpoint. | | `Object storage test failed: ... InvalidAccessKeyId` | AK/SK pair is wrong or expired. | | `Object storage test failed: ... SignatureDoesNotMatch` | Region mismatch (Tigris uses `auto`; AWS regions are `us-west-1` etc.). | Reproduce locally with `aws s3 ls s3:/// --endpoint-url ` to confirm before re-running `workspace create`. For **managed** workspaces: | What you'll see | Likely cause | | --- | --- | | `Object storage test failed: ... InvalidAccessKeyId` on a fresh `workspace create` | AWS IAM eventual consistency. The orchestrator just minted credentials and they haven't propagated globally yet (~5–10s). Re-run `artifacts workspace create ` after a short wait, it resumes the pending_format workspace and reuses the same creds, which by then are valid. | | `internal_error` with `managed_provisioning_failed: ...` | The orchestrator couldn't talk to AWS IAM (transient AWS issue, or an orchestrator-side IAM permissions regression). Retry; if it persists, contact ops. | | `internal_error` with `managed_storage is malformed; delete + recreate` | The workspace's `managed_storage` jsonb is missing required fields (rw / ro / prefix). Should never happen organically; usually means a hand-edit or a partial migration. `artifacts workspace delete --yes ` (cron will retry the IAM teardown) and recreate. | --- # Troubleshooting > Common errors during install, mount, and import, and how to clear them. ### error: bucket access denied, check ARTIFACTS_S3_* env vars or bucket policy The orchestrator never sees your S3 credentials, so we can't help debug from the server side. Reproduce locally first: ```sh aws s3 ls s3:/// --endpoint-url ``` If that fails, fix the bucket / key first; if it succeeds, re-export the env vars in the same shell that's running `artifacts` and re-run. ```sh env | grep ARTIFACTS_S3 ``` A common mistake: `set` without `export`, or exporting in a parent shell that doesn't pass through to a sudo'd or container-shelled child. ### metadata_acl_provisioning_failed (transient; retry) Tonbo Artifacts' metadata service connection blipped. Re-run the failing command (`workspace create`, `mount`, etc.). If it persists past 60 s, ping us; we have an alert on our side too. ### `InvalidAccessKeyId` on the first managed `workspace create` AWS IAM is eventually consistent. The orchestrator mints your workspace's IAM credentials a few hundred milliseconds before juicefs format runs against S3, and the new key isn't always globally available yet (~5–10 seconds). Re-run the same command: ```sh artifacts workspace create ``` The CLI sees the workspace already exists in `pending_format` and resumes from the same persisted credentials, which by then have propagated. No state changes between attempts. ### path /mnt/work is already a mount point; run 'artifacts unmount /mnt/work' first Self-explanatory. The CLI reads `/proc/self/mountinfo` to detect this before issuing a new mount session, so you'll see this clean error instead of a confusing `EBUSY` deep in the FUSE library. Don't `kill -9` the existing mount. SIGKILL bypasses the unmount handler and leaves a stale FUSE mount that needs `umount -f` to clear. ### Mount stuck or `ls` hangs at the mount point The FUSE supervisor died ungracefully (SIGKILL, OOM, container kill) and left a stale mount. Force-unmount as root: ```sh sudo umount -f /mnt/work # try force first sudo umount -l /mnt/work # lazy fallback if force fails ``` Then `artifacts mount my-workspace /mnt/work` to issue a fresh session. If a workspace's mount gets repeatedly into this state under normal operation, let us know; we want to investigate. ### workspace_not_active A previous `workspace create` died mid-format. Rerunning the same command resumes from the last successful phase: the orchestrator returns the same hashtag prefix, the format token is re-issued, and the namespace bootstrap is re-pointed at the existing keys. To start over cleanly: ```sh artifacts workspace delete my-workspace --yes artifacts workspace create my-workspace --bucket ... --endpoint ... ``` Bucket chunks stay; only the per-workspace state in Tonbo Artifacts' metadata service is cleared. ### WRONGPASS invalid username-password pair (in mount logs) The mount-session permissions were revoked between the orchestrator issuing them and your client connecting. This almost always means an operator restarted the metadata service or rotated credentials. Run: ```sh artifacts unmount /mnt/work artifacts mount my-workspace /mnt/work ``` to issue a fresh session. If you see this without an operator notice, ping us; it shouldn't happen mid-flight. ### Unimplemented opcode POLL in mount logs Cosmetic. Your kernel sent a FUSE_POLL opcode (some app called `poll()` / `select()` / `epoll()` on a file under the mount); the mount returned ENOSYS, the kernel doesn't ask again. Logged once per mount, never repeated. No action needed. ### install.sh refuses an unsupported platform The CLI ships native Linux amd64 and arm64 binaries. On other platforms: - **macOS**: use the Mac as a _staging source_ only. See [Import data → from a Mac folder](/docs/import-data#from-a-mac-folder). Mount happens on a Linux host that pulls from your bucket. - **Windows**: WSL2 works when its Linux architecture is amd64 or arm64. Run `install.sh` inside the WSL distro. ### install.sh fails to install fuse3 The installer tries `apt-get`, `dnf`, `yum`, `apk` in order. On a host without any of those (extremely minimal containers, custom distros), install fuse3 manually: ```sh # Ubuntu / Debian apt-get install -y fuse3 # Fedora / RHEL / Amazon Linux dnf install -y fuse3 # Alpine apk add fuse3 ``` Then re-run `install.sh`. ### `artifacts mount` returns immediately but `mount | grep` shows nothing The daemon parent reported ready but the FUSE child died right after. Check the per-mount log: ```sh ls ~/.cache/tonbo/artifacts/ # mount-/ tail -50 ~/.cache/tonbo/artifacts/mount-/artifacts.log ``` The most common causes are bad bucket credentials (the format phase failed at chunk-write time) and a workspace that got into `pending_format` from a previous half-run. Re-run `artifacts workspace create` to resume; if the bucket creds are wrong, fix and `artifacts storage set` with the new ones. ## Still stuck? Open a Slack ticket with: - `artifacts --version` - The command you ran and its full output - `tail -50 ~/.cache/tonbo/artifacts/mount-*/artifacts.log` (for mount issues) - The workspace identifier `artifacts:///` (for orchestrator issues, so we can correlate on our side) --- # Architecture > Tonbo Artifacts splits into three planes that fail independently. Knowing the model makes the failure behavior easy to reason about. Tonbo Artifacts splits into three planes. Each fails independently; a problem in one doesn't propagate to the others. ## The three planes | Plane | Where it lives | What it holds | | --- | --- | --- | | **Data plane** | Your S3 bucket (BYO) or a Tonbo-managed S3 prefix | Immutable file chunks (≤4 MiB blocks) | | **Metadata plane** | Tonbo-operated metadata store | Per-workspace directory tree, file index, per-mount permissions | | **Control plane** | Tonbo Orchestrator API | Auth, workspace lifecycle, mount-session issuance | On the customer host, a **sidecar agent process** (the `artifacts` binary, started by `artifacts mount`) serves the FUSE mount. It owns the on-disk metadata + chunk cache and talks to the metadata and data planes directly. The control plane is **not in the read/write path**. It shows up only on login, workspace create/delete, and the initial mount-session handshake. ```text Agent VM Tonbo S3 -------- ----- -- +-----------------+ +--------------------+ | FUSE mount | | Orchestrator | | | <--> | (control plane) | | sidecar agent | +--------------------+ | process | | | +--------------------+ | local cache | <--> | Metadata store | +-----------------+ +--------------------+ | | chunks (read / write) +-------------------------------------------------> +--------+ | chunks | +--------+ ``` ## Failure isolation | If this is unavailable | Existing mounts | New operations | | --- | --- | --- | | **Orchestrator** | Reads and writes continue; metadata flows directly between the sidecar and the metadata store | `login`, `workspace create/delete`, `mount` blocked | | **Metadata store** | Cached lookups still serve from the sidecar; uncached lookups and metadata writes return errors | Mounts that need fresh metadata fail to start | | **Chunk store (S3)** | Cached chunks read normally; uncached reads block until S3 returns | Writes buffer to local cache, then back-pressure | | **Sidecar agent process** | That host's mount stops responding; `artifacts mount` re-establishes it. Other hosts unaffected. | Other hosts unaffected | | **Agent VM** | Nothing on the workspace is lost. Chunks remain in S3, metadata in the metadata store. | None | ## Durability **No customer data ever lives only on the agent VM.** The local cache is a performance artifact; losing it is a latency event, not a data event. **Chunks (data plane).** Chunks are written to S3 on file `close` and on explicit `fsync`. S3 provides eleven nines of durability. In BYO mode, your bucket's existing properties (versioning, replication, lifecycle) apply unchanged. **Metadata (metadata plane).** Metadata updates are committed to the metadata store before the originating operation (or `fsync`) returns. The metadata store rotates a full snapshot to a separate S3 region at least daily. Recovery from process crash is log replay on the metadata store; recovery from host loss is snapshot restore. **Chunks and the index.** A chunk's object key is a sequential slice id assigned by the metadata store, not a content hash; the same bytes written twice land in different chunks. The file-to-chunk index lives in the metadata store, so backup, restore, and migration move that metadata snapshot alongside the chunks rather than reconstructing the index from the bucket. ## Isolation Each workspace is a separate namespace inside the metadata store. A mount-session token is scoped to exactly one workspace; it cannot enumerate or read any other workspace's metadata, even within the same organization. In managed mode, the data plane is scoped the same way. The IAM credentials issued to a mount session can only read or write within that workspace's S3 prefix. In BYO mode, S3 credentials never leave the customer host. The orchestrator never holds them and the metadata store never holds them. A Tonbo operator with full operational access to either plane cannot read your bytes. ## Exit In BYO mode the customer owns the data path end-to-end. Chunks live under documented prefixes in your bucket as stable, immutable objects readable by any S3-aware tool (`aws s3 sync`, `rclone`, `s3cmd`) without Tonbo running and without Tonbo's permission. Reconstructing files (names, tree, permissions) additionally needs the metadata index, which Tonbo holds; the bytes themselves are never locked to Tonbo. --- # Performance > The latency tiers that govern a Tonbo Artifacts mount, and why metadata-heavy agent workloads run an order of magnitude faster on this architecture. Tonbo Artifacts is tuned for **metadata-heavy** workloads: directory trees walked by recursive `find` and `grep`, many small file opens, frequent attribute lookups, write-edit-reread loops. Most of the wall clock in these workloads is spent on metadata round-trips, not on moving bytes. The architecture optimizes accordingly. ## A representative workload The pattern this design is tuned for looks like: - A workspace with tens of thousands of files, single-digit GB total - Deep, sparse directories, such as a code repository, a hierarchical document store, or a structured dataset - An AI agent that runs broad recursive searches, opens many small files in sequence, then writes a handful of outputs This shape is the common case for AI coding and analyst agents. It is dominated by `getattr`, `readdir`, and `open`. Each of these moves a few bytes, but they happen thousands of times per session. ## Three latency tiers | Tier | What's resolved from | Typical latency | When it dominates | | --- | --- | --- | --- | | **Hot** | The mount client's local cache | sub-millisecond | Re-walking paths the agent has already touched | | **Warm** | The metadata store; chunks already in the local chunk cache | low single-digit milliseconds | First time a path is touched in this mount session | | **Cold** | Chunks fetched from S3 on demand | tens of milliseconds | First read of a file's contents | On its first traversal, a typical agent session is mostly **warm**. The metadata store answers from memory and the mount client populates its local cache as it goes. Every traversal after that is mostly **hot**. Cold latency applies only to file content that has not yet been pulled from S3 in this mount session; metadata is never on the cold path. ## Why agent workloads favor this architecture Agent workloads have two properties that the architecture exploits: 1. **The metadata working set fits in memory.** Tens of thousands of files of metadata is megabytes, not gigabytes. The metadata store keeps it resident; the mount client's local cache keeps the actively-walked subset even closer to the FUSE syscall. 2. **The same paths are revisited.** Agents loop: read-think-edit-read-think-edit. Each loop revisits much of the same directory tree. Aggressive metadata caching turns these revisits into in-process lookups. The cold-path cost of a first metadata touch is a single network round-trip from the mount client to the metadata store. After that, the path is warm; on the next pass it is hot. ## Compared to per-client metadata caching Designs that cache metadata only inside each mount client have a different cost profile on deep recursive operations: - Every mount starts with a cold cache. A second mount on the same workspace re-walks the tree from scratch. - A miss on a deep path is a chain of round-trips out of the client back to whatever upstream holds the truth. - Concurrent agents on different VMs cannot share warm state with each other. Tonbo Artifacts caches metadata at the **metadata store** itself, not only inside each mount. The store is the source of truth, kept warm by every mount that touches it. A second mount on the same workspace hits a pre-warmed view on its first walk; concurrent agents on different VMs converge on the same cached state. The gap is largest on operations that walk **deep, sparse trees**, which is what agents do constantly. ## What we don't optimize for The same architectural choices imply trade-offs. Workloads where this is not the design point: - **Streaming multi-GB sequential reads** of a single file. The S3 data path serves these adequately, but a direct S3 client with range-read parallelism will be faster. - **High-rate concurrent writers** to the same file from multiple mounts. The metadata plane serializes metadata updates; on the data plane, last write wins at the chunk level. Agents rarely hit this; pipelines that fan out writes to one file from many machines do. See [Concurrent writes](/docs/consistency) for the exact semantics. ## Benchmark methodology The characteristics above are validated against AI-coding-agent workloads on customer datasets. Detailed benchmark methodology, dataset shapes, and head-to-head numbers against alternative remote filesystems are available on request. --- # What's in your bucket > What Tonbo Artifacts writes into a BYO bucket, what never appears there, and how to estimate object counts for a security or cost review. If you brought your own bucket, you probably want to know what we write into it, both to verify the product is doing what it says and for security review. ## What's in the bucket When you mount a workspace and write to it, what lands in your bucket is immutable chunks. Under the workspace's prefix: ``` your-bucket/ your-org/my-workspace/ chunks/ 0/0/1_0_4194304 0/0/2_0_4194304 0/0/3_0_1234567 0/1/4_0_4194304 0/1/5_0_4194304 ... ``` Path convention: `chunks/{slice_id / 1_000_000}/{slice_id / 1_000}/{slice_id}_{block_index}_{block_size}`. The two intermediate directories spread chunks for S3 listing performance; they're not semantically meaningful. File naming: - `slice_id`: monotonic integer assigned by the metadata service. - `block_index`: position of this block within the slice (`0` for the first ≤4 MiB block, `1` for the next, and so on). Small files have a single block, so you'll mostly see `0`. - `block_size`: actual byte size of this chunk. Default block is 4 MiB (`4_194_304`); the trailing chunk of a file can be smaller. Inspect from your host: ```sh aws s3 ls s3://your-bucket/your-org/my-workspace/chunks/ --recursive --summarize ``` ```text 2026-05-22 18:00:01 4194304 your-org/my-workspace/chunks/0/0/1_0_4194304 2026-05-22 18:00:01 4194304 your-org/my-workspace/chunks/0/0/2_0_4194304 2026-05-22 18:00:01 1234567 your-org/my-workspace/chunks/0/0/3_0_1234567 ... Total Objects: 247 Total Size: 1023421238 ``` You can read the total size and object count from the bucket directly. Each 4 MiB chunk is one object, so a roughly 1 GiB workspace shows ~250 objects. ## What's not in the bucket Everything except raw chunk bytes lives in the metadata service. You won't find any of this in your bucket: - File and directory **names**, **paths**, and **tree structure** - **`mtime`**, **`atime`**, **`ctime`**, file size - **POSIX permissions**, owner UID/GID - **Symlinks**, extended attributes (xattr) - **Workspace name, handle, labels** - **Grants** (who can mount this workspace) - **Mount sessions** (active tokens and their scopes) What follows from this: - **You can't reconstruct a file from the bucket alone.** A 4 MiB chunk is anonymous bytes with no association to a file path. The metadata service has to interpret it. - **You can't rename or restructure files by manipulating S3.** Renaming `foo.txt` to `bar.txt` is a metadata-only operation; the chunks don't move. - **Chunks aren't deduplicated by content.** Each slice gets a fresh id, so writing the same bytes twice (even within one file) produces separate chunks. This is the structural difference from a "files as S3 objects" approach (Archil, s3fs, goofys): those keep bytes readable by any S3-aware tool with nothing else running, at the cost of turning metadata operations (listing a deep tree, statting many files) into S3 API calls. ## Estimating bucket size | Workspace logical size | Object count | | ---------------------- | ------------ | | 100 MB | ~25 | | 1 GB | ~250 | | 10 GB | ~2,500 | | 100 GB | ~25,000 | | 1 TB | ~250,000 | There is no cross-workspace dedup: 20 workspaces holding the same data hold 20 copies of the chunks. Within a workspace, the metadata service tracks which slices are still referenced; a chunk becomes eligible for garbage collection once nothing references its slice, after overwrites, deletes, or slice compaction. ## Cleanup on workspace delete - **Managed workspaces**: deleting the workspace triggers a best-effort delete of every chunk under its prefix. Failures during the bulk delete don't block the workspace from transitioning to `deleted`; orphan chunks get reaped by a background GC. - **BYO workspaces**: deleting the workspace clears metadata-service state but **doesn't touch your bucket**. Chunks remain under the prefix. To free storage, list the prefix and delete from your bucket yourself: `aws s3 rm s3://your-bucket/your-org/my-workspace/ --recursive`. ## Relation to other concepts - [Workspace](/docs/workspaces): the unit whose contents live under one prefix in the bucket. - [Bring your own bucket](/docs/storage-credentials): how the bucket gets attached in the first place. - [Concurrent writes](/docs/consistency): why writes show up as chunks, not files.