Deploy CosmicAC

Deploy the CosmicAC stack on your host machine with Docker Compose. For how CosmicAC connects to your cluster and runs jobs, see Deployment architecture.

Stack

The stack runs seven services, plus Redis and Caddy:

The CosmicAC services use private images published under ghcr.io/tetherto/, in the form ghcr.io/tetherto/<repo>:<tag>. Redis and Caddy use their official public images.

Prerequisites

The recommended host is Ubuntu 22.04 or 24.04 on x86_64. On that host, you need the following:

Docker Engine and Docker Compose v2.
Task.
The jq, node, and kubectl command-line tools.
Access to the private CosmicAC deployment repository, which holds the Compose files and deploy scripts. Request access from the CosmicAC team.
GitHub Container Registry (GHCR) credentials for private images.
- Your GitHub username.
- A classic GitHub personal access token with the read:packages scope so it can pull the private ghcr.io/tetherto images. If the tetherto org enforces single sign-on, approve the token for the org. See Managing your personal access tokens.
A GPU Kubernetes cluster that already meets the Requirements. CosmicAC connects to this cluster but doesn't set it up.
A valid, readable kubeconfig for the GPU Kubernetes cluster.

Verify the prerequisites

Enable Docker, then confirm you have everything listed in Prerequisites:

sudo systemctl enable --now docker
docker compose version
task --version
jq --version
node --version
kubectl version --client

Set up the environment

Clone the deployment repository and change into it:

git clone <deployment-repo-url>
cd <deployment-repo>

Create the .env file from the example:

cp .env.example .env

In .env, set the variables your deployment needs. See Required before the first deployment for the ones you must set, and the deployment configuration for the rest.

If GITHUB_PAT and GITHUB_USER aren't set in .env, bootstrap prompts for them so it can log in to GHCR and pull the private images.

Add the kubeconfig

Ask your cluster administrator for the kubeconfig. It must meet the Kubeconfig requirements.

Open a file on the host, paste the kubeconfig, and save it:

nano ~/kubeconfig

Print the file's absolute path:

realpath ~/kubeconfig

Set KUBECONFIG_SRC in .env to that path:

KUBECONFIG_SRC=/home/<user>/kubeconfig

Confirm the kubeconfig is readable and reaches your cluster:

test -s ~/kubeconfig && echo ok
kubectl --kubeconfig ~/kubeconfig config current-context
kubectl --kubeconfig ~/kubeconfig cluster-info

Run all later task commands as root. Bootstrap and the running services write the deployment's config and state files as root, so commands such as task backup and task update fail for any other user. See Task deployment commands.

Verify the deployment

Check the services and the API:

task ps
curl -s4 -o /dev/null -w '%{http_code}\n' http://127.0.0.1:5173/
curl -s4 "http://127.0.0.1:5173/api/auth/servers?overwrite_cache=true"
curl -s4 "http://127.0.0.1:5173/api/auth/jobs?page=1&pageSize=10"

The calls carry no token because a default deployment runs with authentication turned off.

The deployment is healthy when task ps shows every service Up and http://127.0.0.1:5173/ returns 200. The servers call lists your GPUs, and the jobs call returns an empty list on a fresh deployment.

Open the web interface

Open http://<server-ip>:5173 in your browser. If you changed UI_PORT, use that port instead.

A default deployment runs with authentication turned off, so anyone who reaches this port has full access without signing in. Restrict network access to the deployment. See Deployment configuration for the authentication defaults.

Help and troubleshooting

Bootstrap fails with rm: cannot remove ...: Permission denied

Bootstrap resets the HyperMQ stores under ./services/*/store before it deploys. The stack's containers create those files as root, so a non-root user can't delete them.

Print the deployment directory and copy the path:

pwd

Open a root shell:

sudo -i

Return to the copied path:

cd <copied-path>

Rerun the bootstrap:

task bootstrap

Bootstrap fails with error from registry: denied

Error response from daemon: error from registry: denied

[ERROR] Required image is unavailable: ghcr.io/tetherto/cosmicac-wrk-ork:release-1.1.0

Docker stores GHCR credentials separately for each user, in that user's ~/.docker/config.json. Bootstrap logs in only when the user running it has no stored credentials, so it can't replace credentials that are present but no longer work. See task ensure-login.

The error appears in two cases. The user running the bootstrap has never logged in to GHCR, or an earlier deployment left credentials that have since expired or been revoked.

To cover both cases, run the following commands as root, in the deployment directory.

Clear the stored GHCR credentials:
```
docker logout ghcr.io
```
Name the registry. A bare docker logout signs you out of Docker Hub instead.
Log in again:
```
task login
```
This command reads GITHUB_PAT and GITHUB_USER from .env, so it doesn't prompt you and it keeps your token out of your shell history.
Run the bootstrap again:
```
task bootstrap
```

If the pull still fails, check the Docker config file, ~/.docker/config.json by default, for a credsStore or credHelpers entry. With either entry, Docker keeps the credential in a separate helper program instead of in the file, and a missing or broken helper fails the same way an expired token does.

Next steps

With CosmicAC running, create your first job or operate the deployment: