Deploying Immiscible

One Node 22 process, no runtime dependencies, one SQLite file. This page covers the image, the three ways to run it, backups (online copies and Litestream), the health endpoints, logs, shutdown, smoke testing a live deployment, the demo seed, and every environment variable the code reads.

With DATABASE_URL=postgres://... the server runs on Postgres instead; see postgres.md, and self-host.md for the Helm chart.

Operator commands on this page (backup, restore, integrity, db, seed-demo, rekey) belong to immiscible-server, the operator CLI in this repository. In a checkout, run it as npm run admin -- <command>; inside the image, as node src/cli/immiscible.js <command>. It is not the developer CLI (npx immiscible login, init, doctor), which talks to a server rather than running one.

#The image

Dockerfile builds a two-stage image on node:22-alpine:

• runs as an unprivileged user with a fixed uid and gid (10001), never root;
• the code under /app is owned by root and cannot be changed by the process; the only writable path is the /data volume, so the root filesystem can be mounted read-only (docker run --read-only --tmpfs /tmp, or read_only: true in compose);
• HEALTHCHECK calls /readyz on $PORT;
• STOPSIGNAL SIGTERM, and the process drains on it (see Shutdown);
• no npm install step: nothing is downloaded at build or run time.

#Upgrading the data volume

Images before this one created the user without a fixed uid. A volume written by an older image is owned by that uid, and the new user (10001) cannot write to it. Once, before starting the new image:

docker compose run --rm --user root --entrypoint sh immiscible -c "chown -R 10001:10001 /data"
# Fly: fly ssh console -C "chown -R 10001:10001 /data"

#Three ways to run it

#docker compose (one host, your VPC)

cp .env.example .env        # fill in the required values below
docker compose up -d                                          # the app on 127.0.0.1:8787
docker compose --profile https --profile litestream up -d     # production

Needs Docker Compose 2.20 or later (optional depends_on, used so the app waits for a Litestream restore only when that profile is on).

The app container runs read-only, with every Linux capability dropped and no-new-privileges, init: true for signal handling, and a 30 second stop grace period. Caddy waits until the app reports healthy.

#Fly.io

fly.toml runs one machine with a volume at /data and checks /readyz. Fly snapshots volumes daily; for a recovery point better than a day, run Litestream (below) or a scheduled immiscible-server backup shipped off the machine. Keep to one machine: SQLite is a single writer. For billing, run node scripts/stripe-setup.mjs and paste the fly secrets set line it prints (setup-stripe.md).

Deploys of the hosted service run from CI, not from a laptop: see Operating the hosted service.

#Regions

The hosted service runs as one deployment per region: EU (Fly app immiscible, fra, fly.toml) and US (Fly app immiscible-us, iad, fly.us.toml). Each region has its own machine, volume, SQLite file, master key, backup bucket and Ed25519 signing key. No customer data is copied between regions; a workspace lives where its owner created it.

• Which region a deployment is: IMMISCIBLE_REGION (eu or us). Unset, the deployment is a single region of its own (self-hosting, development) and none of this applies.
• The hosts: config/regions.json lists each region with its URL, Fly app and backup prefix. When the immiscible.ai domain arrives, change the two url values there, or set IMMISCIBLE_REGION_URLS=eu=https://eu.immiscible.ai,us=https://us.immiscible.ai on both apps, and nothing else.
• Sign-up: the sign-up page says where the new workspace’s data will live and links to the other region’s own sign-up page; a sign-up posted for another region is refused with that region’s URL (other_region), so no account or workspace is made in the wrong place. A region is offered only when it is live ("live": true in config/regions.json, or IMMISCIBLE_REGIONS_LIVE=eu,us).
• Where it shows: /trust, trust.json (deployment.region, regions) and the console under Settings, General, Data region.
• Backups: each region writes under its own prefix (immiscible/backups for EU, immiscible-us/backups for US) in its own bucket. Each manifest records its region, and a restore refuses a backup from another region unless --allow-other-region is given (only to move a workspace at its owner’s written request).
• Receipt keys: a key minted in a region has the region in its id (eu-..., us-...); a key minted before regions keeps the id it was published under. Each region fetches the other’s public keys from /.well-known/immiscible-keys.json?scope=own every six hours and publishes them beside its own, each tagged with its region, so a receipt from either region verifies offline against either key set. POST /v1/verify on one region recognises a genuine receipt from the other and answers with that region’s verify URL (otherRegion: true): single use and revocation are kept where the receipt was issued, and the receipt is never forwarded.

#Setting up the US region

Approved by the founder on 7 October 2026. That day the app, the 3 GB volume and the Tigris bucket were created, and the founder set the secrets with scripts/ops/set-us-region-secrets.sh (a new master key via the clipboard for the password manager, the Resend key read hidden). The launch itself, the first deploy and "live": true, is postponed until the founder says go; no machine runs yet. The commands below are the record and the rest of the steps.

fly apps create immiscible-us
fly volumes create immiscible_us_data --app immiscible-us --region iad --size 3
fly storage create --app immiscible-us --name immiscible-us-backups     # Tigris; sets BUCKET_NAME and AWS_* secrets on immiscible-us only
fly secrets set --app immiscible-us \
  IMMISCIBLE_MASTER_KEY="$(node -e "console.log(require('crypto').randomBytes(32).toString('base64'))")" \
  PUBLIC_URL=https://immiscible-us.fly.dev \
  RESEND_API_KEY=... MAIL_FROM=... FOUNDER_EMAIL=...
# Optional, the same values as the EU app: STRIPE_*, GOOGLE_*, MICROSOFT_*, SLACK_* (each OAuth app needs
# https://immiscible-us.fly.dev/... added as a redirect URI), plus any operator apps from the tables below.
fly deploy --config fly.us.toml
node scripts/smoke.mjs https://immiscible-us.fly.dev

The US master key is new and separate: never reuse the EU key. Keep it in the password manager beside the EU one. Then make the US region live and redeploy both, so each offers the other at sign-up and fetches its keys:

# in config/regions.json set "live": true for us, commit, then
fly deploy                                  # EU
fly deploy --config fly.us.toml             # US
# optional, so the EU key id also says its region:
fly ssh console --app immiscible -C "node src/cli/immiscible.js rotate-signing-key --reason 'region key id'"

Estimated cost of the US region, at Fly’s published prices on 7 October 2026 (iad is Fly’s base price):

A dedicated IPv4 address adds $2 a month; a custom certificate for us.immiscible.ai is free up to Fly’s certificate allowance.

#Render

render.yaml is the same shape: a Docker web service with a persistent disk at /data and /readyz as its health check.

#Backups and restore

The evidence ledger lives in the database, so a backup is evidence too.

#Online copy

node src/cli/immiscible.js backup backups/immiscible-2026-10-04.db

VACUUM INTO takes a transactionally consistent copy while the server keeps serving, then the copy is integrity-checked before the command reports success. A backup never overwrites a file. npm run backup (scripts/backup.mjs) does the same into $DATA_DIR/backups and keeps the newest BACKUP_KEEP (default 14). Copy backups off the machine.

#Integrity check

node src/cli/immiscible.js integrity                # the live database
node src/cli/immiscible.js integrity backups/x.db --json

Checks the file (PRAGMA integrity_check), foreign keys, that the schema is not newer than this build, and every workspace’s evidence chain: sequence, links, and each record’s hash recomputed from its content. Exit code 1 on any problem.

#Restore

# stop the server first
node src/cli/immiscible.js restore backups/immiscible-2026-10-04.db

The backup is integrity-checked first; the database it replaces is kept as immiscible.db.pre-restore-<time>; the old -wal and -shm are removed so they cannot be replayed onto the restored copy. If the live -wal is not empty the command assumes a server is still running and refuses (--force if you are sure). Start the server afterwards: it migrates an older copy forward.

#Litestream (continuous)

litestream.yml streams the database to any S3-compatible bucket about once a second, snapshots daily, and keeps 30 days. With the litestream compose profile, an empty volume is restored from the newest replica before the app starts, so recovering a lost host is: provision a new one, copy .env, run docker compose --profile https --profile litestream up -d.

Point-in-time restore by hand:

docker compose run --rm litestream restore -config /etc/litestream.yml \
  -timestamp 2026-10-04T09:00:00Z -o /data/restored.db /data/immiscible.db

Give the bucket versioning or object lock, and the key write access without delete, where the provider allows it.

#Built-in off-machine backups (Fly and anywhere without Litestream)

On Fly a volume attaches to one machine at a time, so a scheduled machine cannot mount immiscible_data while the app runs. The server backs itself up instead when IMMISCIBLE_BACKUP_BUCKET (or Fly’s BUCKET_NAME) and credentials are set:

fly storage create --app immiscible       # Tigris; sets BUCKET_NAME and AWS_* secrets

Every IMMISCIBLE_BACKUP_INTERVAL_MINUTES (default 60), in a worker thread so requests are not held up: VACUUM INTO a copy, run the integrity check and recompute every workspace’s evidence chain on it, gzip it, encrypt it with AES-256-GCM under a key derived from IMMISCIBLE_MASTER_KEY, upload it with a manifest (sizes, SHA-256 of both files, schema, every chain’s head), and never delete anything: old copies are expired by a lifecycle rule on the bucket, or pruned by the operator with a key of their own (immiscible-server backups prune), so the server’s key needs no delete permission (see Backups a stolen key cannot delete). A run is skipped if the disk lacks room for the copy. /readyz reports the last good backup’s age; backup_failed is logged at error level, and a failed or stale backup emails the operator (see Backup alerts).

immiscible-server backups                          # what is in the bucket
immiscible-server backup --upload                  # one now, from the machine
immiscible-server restore --from-bucket latest --to /data/drill.db   # a restore drill: fetched, decrypted, every chain checked; live data untouched
immiscible-server restore --from-bucket latest     # the real thing (server stopped)

In a hosted region the prefix is that region’s own (see Regions), and a restore refuses a backup whose manifest names another region.

A restore checks the download’s hash, the GCM tag, the decrypted file’s hash, the integrity of the file and every evidence chain, and that each chain ends exactly where the manifest says, before anything touches the live database. Without the master key the backups cannot be read: keep it in a password manager. Recovery point: up to one interval (60 minutes by default) of writes. Recovery time: the download and checks (about a minute per GB) plus a restart.

A drill (--to FILE) records its result, pass or fail, with how long it took, as restore-drill.json beside the backups. The server reads it with each backup, and trust.json (backup.lastRestoreDrill), /readyz and Security posture in the console report it. Delete the scratch file afterwards.

Database settings the app applies on every open: WAL journal, synchronous = NORMAL, foreign keys on, busy_timeout = 5000 ms. The app never truncates the WAL itself, which is what Litestream requires.

#Health, readiness, logs, shutdown

Logs: with LOG_FORMAT=json (the production default) every line on stdout is one JSON object with t (ISO time), level and either msg or request fields (rid, method, path, status, ms, ip). Start-up, config errors, shutdown and crashes are JSON lines too. IMMISCIBLE_LOG=off turns off request lines only.

Shutdown: on SIGTERM or SIGINT the process marks itself not ready (so a load balancer stops sending traffic), stops accepting connections, closes idle keep-alive connections, lets in-flight requests and streams finish, flushes every workspace, logs stopped, and exits 0. If that takes longer than IMMISCIBLE_SHUTDOWN_TIMEOUT_MS (default 25 s), open connections are cut and it exits anyway. An uncaught exception or unhandled rejection is logged and starts the same orderly stop with exit code 1, so the supervisor restarts it.

Keep-alive: idle connections stay open 65 s (headers timeout 66 s), longer than the idle timeout of the usual proxies (Caddy, Fly, cloud load balancers at 60 s), so the server never closes a pooled socket just as the proxy reuses it. If your proxy keeps upstream connections longer than 65 s, lower its idle timeout.

#Smoke testing a live deployment

node scripts/smoke.mjs https://immiscible.example --keep

Walks the whole product through the public API: sign-up and email verification, a workspace, a second owner, a provider, a service token, an agent from a blueprint with standing and allowlists, a gateway model call, a payment approved by the other owner, the card charge for it through the issuer webhook, an MCP tool call, a freeze and lift, a drill, the trace view, and the evidence bundle verified offline against keys pinned at the start. The same journey runs in CI against a server booted on a temporary database (test/e2e-journey.test.js).

Email links (verification, invitation) need mailbox access. Outbox mode returns the invitation link in the API answer; otherwise run the smoke on the server host and pass the database file:

fly ssh console -C "node scripts/smoke.mjs http://127.0.0.1:8787 --db /data/immiscible.db"

Steps that cannot run are reported as skipped, never as passed. The workspace is deleted at the end unless --keep. Variables: SMOKE_URL (instead of the argument), SMOKE_PROVIDER_KEY (an OpenAI key to connect a real provider), SMOKE_MCP_URL and SMOKE_MCP_SECRET (a real MCP server; against a loopback URL the script starts its own), SMOKE_EMAIL_DOMAIN (default smoke.test). --json prints a machine-readable report.

#The demo workspace

node src/cli/immiscible.js seed-demo --password 'choose-one'

Creates Amethyst, a London design and research studio: Jules Moreau (jules@amethyst.example, owner, the demo login), Priya Shah (second owner) and Sam Okafor (member); five agents with sponsors, kill owners, purposes, end dates and mandates; model calls through the gateway (the labelled mock, so no provider key is needed); about twenty payment requests that are allowed, held or refused with reasons; approvals decided by Priya; a freeze and lift; a drill; a signed checkpoint; single sign-on set up for amethyst.example against a sample identity provider, not required, so the demo logins sign in with a password. --crypto adds a sixth agent that pays x402 services in USDC on Base; the default demo has no crypto in it. It writes to the database the server would open and is idempotent. Without --password (or SEED_PASSWORD) a password is generated and printed.

#Operating the hosted service

How the hosted service at immiscible.fly.dev (Fly app immiscible, region fra, one machine) is changed, watched and recorded. The scripts named here live in scripts/ops; each one records who ran it, what, when and why in the operator log before it acts.

#Deploying

The normal path is CI. A push to main runs .github/workflows/ci.yml (tests, preflight, image build and scan, npm audit, CodeQL, TruffleHog, gitleaks). Only when that whole run succeeds does .github/workflows/deploy.yml deploy the same commit:

flyctl deploy --remote-only --ha=false --build-arg GIT_SHA=<commit> --build-arg BUILT_AT=<time>

The Dockerfile turns the two build arguments into IMMISCIBLE_COMMIT and IMMISCIBLE_BUILT_AT; /healthz, /version, trust.json and /trust report them. The workflow then reads /version until the live service reports the commit it deployed, and fails if it never does. The Actions run history and fly releases -a immiscible are the deploy log. Anyone can check what is live:

curl -s https://immiscible.fly.dev/version     # {"version":"4f1c2a9e0b7d","commit":"4f1c2a9e0b7d...","builtAt":"2026-10-07T10:12:00Z"}

The workflow needs one repository secret, FLY_API_TOKEN: a deploy token limited to the one app. To set or rotate it (yearly; the script suggests a one-year expiry):

scripts/ops/set-fly-deploy-token.sh --reason "first CI deploy token"

It asks you to run fly tokens create deploy -a immiscible in another terminal, reads the token at a hidden prompt and pipes it straight into gh secret set FLY_API_TOKEN. The token is never echoed, written to a file or put on a command line.

A laptop deploy is break-glass only: GitHub Actions is down, or an incident cannot wait for the pipeline.

scripts/ops/breakglass-deploy.sh --reason "Actions outage; fix for the 503s on /v1/actions"

It refuses a working tree with uncommitted changes, runs the tests (unless --skip-tests, which is recorded), records the reason in the operator log (or, if the server cannot be reached, in a GitHub issue labelled operator-log), and deploys with the same build arguments as CI. Push the commit and open a pull request within one working day.

#Branch protection

Branch protection and rulesets are not available for a private repository on GitHub’s free plan; the API answers “Upgrade to GitHub Pro or make this repository public”. The options:

• GitHub Team (an organisation, about $4 per user a month): rulesets on main requiring a pull request and the test, secrets, gitleaks and codeql checks, no force pushes or deletion. The repository then belongs to the company and organisation-wide two-factor can be required. GitHub Pro on the personal account (about $4 a month) also allows protection on private repositories.
• Make the repository public: protection and rulesets become free, and so do Actions minutes. The code would be visible to everyone.

Until then there is a local hook, which is not a real control:

git config core.hooksPath scripts/hooks    # once per clone

scripts/hooks/pre-push runs npm test before a push to main and refuses the push if it fails. git push --no-verify, another clone or the web editor skip it, and nothing on GitHub enforces it. What does hold today is the deploy gate: nothing reaches production through CI unless the whole ci run passed.

#Watching from outside

.github/workflows/uptime.yml runs every hour on GitHub’s runners, outside Fly.io (scripts/ops/uptime-check.mjs):

• /healthz must answer 200 with ok: true and a backup that is not stale;
• the console sign-in page /login must answer 200 with a page.

After two failed runs in a row it opens an issue labelled incident, assigned to the repository owner; GitHub emails the assignee (keep email notifications on for the repository). While it stays down each run comments on the issue, and the first good run comments and closes it.

Limits, plainly. GitHub’s schedule is best effort: runs start late, by several minutes or more at busy times, and some are dropped, so with an hourly run expect detection within roughly two hours. A GitHub Actions outage is also an outage of this monitor. Each run is billed as at least one minute, so hourly is about 720 of the 2,000 minutes a free private repository includes; with a spending limit of zero an overflow would also stop CI and deploys. For faster detection, make the repository public (Actions are then free) or move to a plan with more minutes and set the schedule to */5, or add a free external monitor alongside this one.

#Backup alerts

A failed backup, or no good backup for IMMISCIBLE_BACKUP_STALE_MINUTES (default 120, two missed hourly runs), emails OPERATOR_EMAIL (else FOUNDER_EMAIL) through the configured mail sender (Resend on the hosted service): at most one email an hour about failures, and one every six hours while it stays stale. /healthz says "backup": "stale", which fails the uptime check, and /trust shows the last good backup and the build.

fly secrets set -a immiscible OPERATOR_EMAIL=you@example.com

#Security log

Besides stdout, the server keeps these in the security_log table:

Never kept: query strings, bodies, headers, prompt or answer text, audit details (they stay in the workspace’s own audit log), stack traces, or anything shaped like a key or token; long opaque path segments become :token. The table is in the database, so the hourly encrypted backups carry it off the machine; there is no log shipper and no extra vendor. It holds user ids and addresses, so the retention policy covers it. Export, for an auditor or an investigation:

immiscible-server logs export --out /data/security-log-2026-10.jsonl --since 2026-10-01 --until 2026-11-01 --reason "monthly evidence"
# then copy it off: fly ssh sftp get /data/security-log-2026-10.jsonl

One JSON object a line; the command prints the file’s SHA-256, and the export itself is recorded in the operator log.

#Operator actions

Every immiscible-server operations command (backup, backups, restore, integrity, rebuild-totals, db, seed-demo, rekey, rotate-signing-key, incident, logs) writes a row to the operator_log table before it acts: who (--by, else IMMISCIBLE_OPERATOR, else the login name), what (the command and its arguments, with anything secret redacted), when, where (FLY_MACHINE_ID or the host name) and why (--reason). A restore of the live database writes it afterwards, into the restored copy. In production a command that changes data refuses to run without --reason. scripts/backup.mjs and scripts/smoke.mjs --db record themselves too, and the shell scripts in scripts/ops record through immiscible-server note over fly ssh console.

The table is append-only: triggers refuse any change or deletion, and each row carries the SHA-256 of the one before, so editing the raw file shows.

immiscible-server operator-log                 # who did what, when and why
immiscible-server operator-log --verify        # recompute the chain
immiscible-server operator-log --json --since 2026-10-01
immiscible-server note --what "rotated Resend key" --reason "yearly rotation"

A shell on the machine, with the reason recorded first:

scripts/ops/breakglass.sh --reason "quarterly restore drill"

What cannot be captured. Inside an SSH session, anything not done through immiscible-server (sqlite3, editing files) leaves no record; Fly.io gives customers no transcript of SSH sessions. Actions in the Fly.io or Tigris dashboards, who read a secret, and anything Fly.io staff do are not recorded here either. The record is honour-based for a person with a shell; the chain makes later tampering visible, not impossible.

Fly.io’s own records. Fly.io does not offer customers an organisation audit log (checked 7 October 2026: nothing in flyctl or the documentation, and Fly’s community forum says it is not yet available). What it does expose is saved, dated and hashed, by:

scripts/ops/export-fly-activity.sh            # ops-evidence/fly-<date>/ (not committed)

fly releases --json --image (each release: who triggered it, when, the image), fly machine list --json, fly secrets list --json (names, digests and when each was set, never values), fly tokens list and fly orgs show. Run it monthly and after any incident, and keep the folder with the compliance evidence.

#Backups a stolen key cannot delete

The server never deletes a backup, so its key needs no delete permission. What Tigris offers (checked 7 October 2026 against its documentation and CLI 3.15.0):

• Scoped access keys with IAM policies. Supported actions include s3:PutObject, s3:GetObject, s3:ListBucket and, separately, s3:DeleteObject, so a key can write without deleting.
• Lifecycle rules that expire objects after a number of days, by prefix.
• Soft delete: deleted objects stay recoverable for 7 to 90 days.
• Snapshots of a whole bucket, immutable once taken.
• Object lock and bucket versioning are not documented among Tigris’s supported S3 operations or IAM actions. For write once, read many storage, use a second provider that has it (AWS S3, Backblaze B2 and Cloudflare R2 all do).

One gap remains even with a write-only key: s3:PutObject can overwrite an object under an existing name. Backups have unique names and the server never reuses one, but a stolen key could. Soft delete does not cover an overwrite; snapshots and a locked second copy do.

scripts/ops/backup-hardening.sh holds the exact commands. Run with no arguments it prints the plan and changes nothing; each step asks for “yes” and a reason. In order:

npm install -g @tigrisdata/cli && tigris login

# 1. A key that can write and read the backup prefix and never delete, set as
#    IMMISCIBLE_BACKUP_ACCESS_KEY_ID / _SECRET_ACCESS_KEY on the app.
#    Policy: Allow s3:PutObject, s3:GetObject, s3:AbortMultipartUpload,
#    s3:ListMultipartUploadParts on <bucket>/immiscible/backups/*, and
#    s3:ListBucket, s3:ListBucketMultipartUploads on <bucket>; Deny
#    s3:DeleteObject, s3:DeleteObjectVersion, s3:DeleteBucket,
#    s3:PutLifecycleConfiguration, s3:PutBucketAcl, s3:PutObjectAcl.
tigris iam policies create immiscible-backup-write-only --document policy.json
tigris access-keys create immiscible-server-backups --env <private temp file> --for aws
tigris access-keys attach-policy <key id> --policy-arn <policy arn>
#    then the id and secret piped into: fly secrets import -a immiscible
scripts/ops/backup-hardening.sh scoped-key --reason "..."

# 2. Prove it: one backup with the new key.
scripts/ops/backup-hardening.sh verify --reason "..."

# 3. Expiry, now that the server does not prune.
tigris buckets lifecycle create <bucket> --prefix immiscible/backups/ --expire-days 35

# 4. Deleted objects recoverable for 30 days.
tigris buckets set <bucket> --soft-delete enable --retention-days 30

# 5. Take the key that can delete off the server (fly storage create set it
#    as AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY). Keep one in the password
#    manager for `immiscible-server backups prune` and emergencies.
fly secrets unset -a immiscible AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY

# 6. IRREVERSIBLE, the founder's decision: a second, locked copy at another
#    provider, in the EU. Compliance mode means nobody, the account's root
#    user included, can delete a copy or shorten its lock for 35 days, and
#    object lock can never be turned off on that bucket.
aws s3api create-bucket --bucket immiscible-backups-locked-eu --region eu-central-1 \
  --create-bucket-configuration LocationConstraint=eu-central-1 --object-lock-enabled-for-bucket
aws s3api put-object-lock-configuration --bucket immiscible-backups-locked-eu --region eu-central-1 \
  --object-lock-configuration '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"COMPLIANCE","Days":35}}}'

Step 6 also needs code that does not exist yet (each run uploading to a second bucket), an IAM user for it with put, get and list only, and AWS added to the sub-processors with the notice the DPA promises.

Pruning by hand, with the operator’s own key (never set on the server):

IMMISCIBLE_BACKUP_PRUNE_ACCESS_KEY_ID=... IMMISCIBLE_BACKUP_PRUNE_SECRET_ACCESS_KEY=... \
  immiscible-server backups prune --reason "lifecycle rule not yet set" [--dry-run]

#Scanning

• .github/dependabot.yml: weekly update proposals for npm (the server and each published package), the GitHub Actions in the workflows (all pinned by commit SHA) and the Docker base image. Dependabot alerts and automated security fixes are on in the repository settings.
• ci.yml: npm audit --omit=dev (any advisory fails), a Trivy scan of the built image that fails on a critical vulnerability with a fix available, gitleaks on every push beside TruffleHog, and a weekly scheduled run.

#Operations settings

#Environment variables

Every variable the code reads. Required in production are marked required; the server refuses to start without them when NODE_ENV=production. test/e2e-ops.test.js fails if code reads a variable this table does not list.

#Core

#Rotating the master key

1. Make a new key: node -e "console.log(require('crypto').randomBytes(32).toString('base64'))". Keep the old one; you need both.
2. Take a backup (immiscible-server backup <file>) and stop the server.
3. Re-seal every stored secret under the new key, with both keys supplied:
   ShellCopy

   IMMISCIBLE_MASTER_KEY=<new> IMMISCIBLE_MASTER_KEY_PREVIOUS=<old> immiscible-server rekey

   It lists, table by table, what it re-sealed. Sign-in and connection flows in progress (they live ten minutes) are cleared. If anything could not be opened it says so and exits non-zero; nothing is deleted.

4. Start the server with IMMISCIBLE_MASTER_KEY=<new> and, to be safe, IMMISCIBLE_MASTER_KEY_PREVIOUS=<old>. Both keys open values, and new ones use the new key.
5. Once it runs well and rekey reported nothing left, remove IMMISCIBLE_MASTER_KEY_PREVIOUS and destroy the old key.

Receipts and checkpoints already signed stay verifiable: the signing key is re-sealed, not replaced.

#Rotating the signing key

The Ed25519 key that signs receipts, checkpoints and erasure statements is separate from the master key. To replace it (on a schedule, or if it may be exposed):

IMMISCIBLE_MASTER_KEY=<the server's key> immiscible-server rotate-signing-key --reason "annual rotation"

The new key signs from then on; a running server picks it up within 30 seconds, no restart needed. The retired key stays in /.well-known/immiscible-keys.json (and /.well-known/assay-keys.json), marked "retired": true, so every receipt and checkpoint it signed still verifies. Every live workspace’s evidence ledger gets a signing_key_rotated record naming both key ids.

#Incidents on /status

/status shows the incident history from the database and 90 days of uptime the server measures on itself (IMMISCIBLE_HEALTH_SAMPLES). Write incidents on the server:

immiscible-server incident add --title "Approvals delayed" --impact minor --note "Looking at the queue."
immiscible-server incident update inc_1a2b3c4d5e --status monitoring --note "Fix deployed."
immiscible-server incident resolve inc_1a2b3c4d5e --note "Back to normal."
immiscible-server incident list

Impacts: minor, major, critical, maintenance. There is no web route that writes an incident.

#Data

#Behaviour and limits

#Accounts and sign-in

#Workspace security and retention

Workspace session limits (sessionIdleMinutes, sessionMaxHours) can only tighten IMMISCIBLE_SESSION_IDLE_MINUTES and IMMISCIBLE_SESSION_MAX_DAYS, never loosen them. See Security administration.

#Directory sync

SCIM needs no variables: each workspace makes its own token. Google Workspace and Microsoft Entra sync use an OAuth app the operator registers once (setup-directory.md). Microsoft reuses MICROSOFT_CLIENT_ID and MICROSOFT_CLIENT_SECRET.

#Crypto payments

Crypto payments are priced at the moment they are decided from two public, keyless price APIs: Coinbase first, Kraken if Coinbase does not answer. The server needs outbound HTTPS to api.coinbase.com and api.kraken.com; nothing else is configured. With no answer in time, the payment waits for a person (“Couldn’t price this payment right now”); a rate is never guessed. See docs/site/guides/crypto-rates.md.

#Self-hosted bootstrap

#Email

Without a provider every message is kept in the outbox table (and the log), which is how tests and the smoke read verification links.

#Billing

To set these, run node scripts/stripe-setup.mjs. It creates the Stripe products, prices and webhook from config/plans.json (or reuses them on a rerun) and prints the fly secrets set line with every value below except your secret key, which you paste in yourself. See setup-stripe.md.

#Chat and mobile

#Finance and operations integrations

Each one-click card appears only when its app is configured. Register each app once; the steps and redirect URIs are in the guides under docs/site/guides (xero, quickbooks, pagerduty).

The agent inventory’s Microsoft source reuses MICROSOFT_CLIENT_ID and MICROSOFT_CLIENT_SECRET, and its Google Workspace source the directory or sign-in client; the permissions and redirect URIs they need are in docs/site/guides/agent-inventory.md.

Opsgenie, Datadog and Splunk need nothing from the operator: none of them offers OAuth for sending events, so each workspace pastes a key once and Immiscible checks it live.

#Reports and dashboards

The CSV downloads, the board links (/board/...), /openapi.json, the import files under /downloads/ and action callbacks need nothing from the operator. Callbacks go only to https on public addresses in production.

#Model providers

In self-hosted mode a provider’s key may come from the environment instead of the console. In cloud mode each workspace stores its own keys, sealed with IMMISCIBLE_MASTER_KEY.

#Smoke test

#Older names

The product was called Assay before it was Immiscible. A deployment set up under the old name keeps working without changes; everything below is read silently, and wherever both names are present the new one wins.

Some names are part of signed or hashed formats and never changed, so that evidence and receipts issued before the rename verify byte for byte: the receipt typ (assay-receipt+jwt), the record schema ids (assay.evidence.v1, assay.evidencepack.v1, assay.mandate.v1, assay.snapshot.v1), and the labels that derive keys from IMMISCIBLE_MASTER_KEY. They are identifiers, not branding, and stay as they are.