Skip to main content

Manifest Reference

Complete field-by-field reference for manifest.yaml.

See the Manifest Examples page for complete, per-operator manifest.yaml files.

kind#stringrequiredAlways AIchorManifest

Always AIchorManifest.

apiVersion#stringrequiredSchema version — current: 0.2.3

Schema version. Set to 0.2.3 for new manifests.

name#stringoptionalHuman-readable manifest name

Name for this manifest.

builder#objectrequired
buildArgs#mapoptionalDocker build arguments

Key/value map of Docker build arguments (passed as --build-arg). Values may be strings or booleans.

buildArgs:
  PYTHON_VERSION: "3.11"
  USE_CUDA: true
skipBuild#objectoptionalReuse a previous experiment's image instead of building

To set whether the building step uses a previous experiment's image instead of building a new one.

enabled#booleanoptionalEnable skipping the build

Whether to skip the build and reuse a previous experiment's image. When false, the rest of this block is ignored and the build runs as usual.

Default: false

experimentID#stringoptionalUUID of the previous experiment to reuse

UUID of a previous experiment whose image should be reused.

failIfNotFound#booleanoptionalFail instead of falling back to a normal build

When true, fail the build if the previous experiment's image can't be found (e.g. a mistyped UUID, or the image was purged by the registry's cleanup policy). When false, ignore it and build a new image

Default: false

context#stringoptionalDocker build context path

Docker build context path.

Default: "."

dockerfile#stringoptionalPath to the Dockerfile

Path to the Dockerfile, relative to the repository root (e.g. ./Dockerfile).

image#stringoptionalDocker image name

Name of the Docker image to build and use. Must match spec.image.

registry#stringoptionalDocker registry prefix

Registry prefix used to push and pull the image (e.g. europe-west4-docker.pkg.dev/my-project/my-repo).

target#stringoptionalDockerfile target stage

Target stage for multi-stage Dockerfiles (passed as --target).

clone#objectoptional
depth#integeroptionalShallow clone depth

Number of commits to fetch. 1 produces a shallow clone. Increase when git log history is needed. Minimum: 1.

Default: 1

includeDotGit#booleanoptionalKeep .git directory in clone

When true, the .git directory is kept in the cloned repository. Required for submodule initialisation.

Default: false

submodules#stringoptionalSubmodule init mode

Controls submodule initialisation. Only processed when includeDotGit is true.

ValueBehaviour
"false"No submodules cloned
"true"Non-recursive clone
"recursive"Recursive clone

Default: "false"

spec#objectrequired
activeDeadlineSeconds#integeroptionalMax experiment duration in seconds

Maximum running duration for the experiment in seconds. Superseded by timeout when both are set.

command#stringrequiredCommand to run in the container

The command or script to execute inside the container.

spec:
  command: "python3 -u main.py"
debug#object
jupyter#objectoptionalJupyter server — kuberay only

Starts a Jupyter server on the Ray head node. Only available when operator is kuberay.

enabled#booleanoptionalEnable the Jupyter server

Whether to start the Jupyter server.

Default: false

path#stringoptionalPath to Jupyter binary

Path to the Jupyter binary. Defaults to $PATH resolution.

vscode#objectoptionalVSCode tunnel — all operators

Attaches a VSCode tunnel to the experiment. Available for all operators.

enabled#booleanoptionalEnable the VSCode tunnel

Whether to start the VSCode tunnel.

Default: false

path#stringoptionalPath to VSCode CLI binary

Path to the VSCode CLI binary. Defaults to $PATH resolution.

provider#stringoptionalTunnel auth provider

VSCode tunnel authentication provider. Values: "github", "microsoft".

Default: "github"

engineName#stringoptionalTarget engine name

Name of the engine to run the experiment on. When omitted, the project's default engine is used. Useful in multi-engine projects.

env#mapoptionalExperiment-level environment variables

Key/value map of environment variables scoped to this experiment. Each entry is injected into the containers as a plain literal value, which makes env suited to per-run values such as hyperparameters without touching the shared project configuration.

env:
  LEARNING_RATE: "0.001"
  BATCH_SIZE: "32"
  NUM_EPOCHS: "100"

When the same key is also defined as a project-level variable, the spec.env value takes precedence. Predefined variables — the AICHOR_* variables and other protected names — cannot be set here; attempting to do so fails validation when the experiment is submitted. See Environment Variables for the full list.

experimentConfig#stringoptionalExperiment-level configuration

configurations that are passed to all the experiment pods, no specific requirement on the structure as this field accepts all strings. one way you can set up these configs could be as follows

  experimentConfig: |
    config1: this-is-a-config
    config2: this-is-another-config
    parent-config:
      nested-config: this-is-nested
extraLabels#mapoptionalExtra Kubernetes pod labels

Additional Kubernetes labels applied to the experiment pods.

extraLabels:
  team: research
  project: my-project
gangScheduling#object

Schedules the experiment's pods as a single atomic unit: either all of them can be placed at once, or none are scheduled yet. Available on AIchor-managed engines, backed by the KAI scheduler; on imported engines, availability depends on whether the underlying cluster has a compatible scheduler plugin installed. See Gang scheduling for the full behavior per operator.

spec:
  gangScheduling:
    enabled: true
enabled#booleanoptionalEnable gang scheduling

Whether to schedule this experiment's pods as a gang. For pytorch, jax, xgboost, and jobset, the pod count requested is the same either way — this only controls whether the experiment waits for all of them to be placeable together before any of them start. For kuberay, this controls whether a worker pool's count is requested upfront (true) or treated as a ceiling that Ray's own autoscaler grows into on demand (false).

Default: false

autoScalingScaleDown#integeroptionalIdle seconds before a worker scales down — kuberay only

How long, in seconds, an idle kuberay worker waits before Ray's autoscaler scales it back down. Only relevant for kuberay experiments — jobset, jax, pytorch, and xgboost request all their pods upfront and never scale up or down, gang scheduling enabled or not. Only takes effect while enabled is false, since that's the only case where kuberay workers scale up on demand in the first place. Minimum: 0.

Default: unset, which keeps idle workers around indefinitely (no automatic scale-down)

gracefulTermination#object

Runs a command when a pod is being terminated, so the experiment can shut down cleanly, most commonly to save a checkpoint before a spot instance is reclaimed.

The command is installed as a Kubernetes preStop hook. On termination (a spot-instance reclaim, a cancel, or any pod deletion) it runs first; once it returns, the container receives SIGTERM. The command therefore does not need to trap signals itself.

When this block is present, both shutdownCommand and terminationGracePeriodSeconds are required, and the grace period must be greater than 0. When the block is omitted entirely, no shutdown command runs and Kubernetes applies its own default grace period of 30 seconds.

gracefulTermination:
  shutdownCommand: ["python", "save_checkpoint.py"]
  terminationGracePeriodSeconds: 60
Spot instances

Cloud providers give only a short interruption notice for spot/preemptible instances. For example, AWS offers around 120 seconds. On a spot reclaim the node is drained within that window, so a terminationGracePeriodSeconds larger than the provider's notice will not be fully honoured. Keep the shutdown command well within ~120 seconds for checkpoints to complete reliably. On a normal cancel or deletion the full grace period is honoured. For more information please check the documentation of your cloud provider

shutdownCommand#array[string]requiredCommand run before the pod is killed

Command to execute inside the container before termination (e.g. to save a checkpoint). Provided as an argument vector — the executable followed by its arguments.

shutdownCommand: ["python", "save_checkpoint.py"]
terminationGracePeriodSeconds#integerrequiredGrace period before force-kill

How long, in seconds, Kubernetes waits for the shutdown sequence (the shutdownCommand followed by SIGTERM handling) to complete before force-killing the pod with SIGKILL. Must be greater than 0.

The countdown starts when termination begins, so this value must cover the full time the shutdownCommand needs to run.

image#stringrequiredDocker image to run

The Docker image to run. Must match builder.image.

operator#stringrequiredWorkload operator

The operator that manages the workload.

ValueDescription
kuberayRay cluster managed by KubeRay
jaxJAX distributed training
pytorchPyTorch distributed training
xgboostXGBoost distributed training
jobsetGeneric Kubernetes JobSet
restartPolicy#object
backoffLimit#integeroptionalMax restarts before failure

Number of allowed restarts before the experiment is marked as failed. Minimum: 0.

On jobset and kuberay, this covers both eviction (spot preemption) and software failures. On other operators it covers only software failures.

Default: 0

securityContext#object
spec:
  securityContext:
    runAsUser: 1000     # run as a non-root user...
    runAsGroup: 1000
    fsGroup: 1000       # ...and give that group ownership of mounted volumes
  storage:
    sharedVolume:
      mountPoint: /mnt/shared
      sizeGB: 50
      accessMode: ReadWriteMany
      storageClass: standard-rwo
fsGroup#integeroptionalGID applied to mounted volumes

Sets a supplementary group ID (GID) that owns all volumes mounted into the pod. Kubernetes changes the group ownership of mounted volumes to this GID and adds it to every process in the pod, so files created on those volumes are group-owned and group-writable by this GID.

This is mainly useful when the container does not run as root. By default a mounted volume (such as a PVC or a shared volume) is owned by root, so a non-root process cannot write to it. Setting fsGroup makes the volume writable by the chosen group, and lets several containers in the same pod share the same files.

In the example at the top of this section, the process runs as UID 1000 and can read and write /mnt/shared because the volume is group-owned by GID 1000. Without fsGroup, that same non-root process would be denied write access to the volume.

perfmon#booleanoptionalGrant CAP_PERFMON

Grants CAP_PERFMON to allow performance-monitoring syscalls. Required by profilers such as perf.

Default: false

ptrace#booleanoptionalGrant CAP_SYS_PTRACE

Grants CAP_SYS_PTRACE to allow process tracing. Required by debuggers and profilers such as gdb and nsys.

This capability is gated by an organisation-level flag, which is disabled by default. When the organisation does not have it enabled, the setting is silently ignored (no CAP_SYS_PTRACE is granted and no error is raised). Contact the AIchor team to have it enabled for an organisation.

Default: false

runAsGroup#integeroptionalPrimary GID for all processes

Sets the primary group ID (GID) that every process in the container runs as, overriding the group defined in the image. This controls the group ownership of files the processes create and which group-restricted files they can read. It is the group counterpart to runAsUser and is commonly paired with it (and with fsGroup) to run as a specific non-root identity.

runAsUser#integeroptionalUID for all container processes

Sets the user ID (UID) that every process in the container runs as, overriding the user defined in the image's Dockerfile. Use it to run as a specific non-root user.

A value of 0 means root. When omitted, the user baked into the image is used. To write to mounted volumes as a non-root user, combine runAsUser with fsGroup so the volumes are group-writable.

storage#object

Attaches volumes to every pod in the experiment. Use attachExistingPVCs to mount Persistent Volume Claims that already exist in the project, and sharedVolume to have AIchor provision a temporary volume shared by all pods for the duration of the run.

spec:
  storage:
    attachExistingPVCs:
      - name: datasets-pvc
        mountPoint: /mnt/datasets
    sharedVolume:
      mountPoint: /mnt/shared
      sizeGB: 100
      accessMode: ReadWriteMany
      storageClass: standard-rwo
attachExistingPVCs#array[object]optionalAttach existing PVCs to all pods

List of existing Persistent Volume Claims to attach to all pods.

[].mountPoint#stringrequiredMount path inside the container

Mount path inside the container.

[].name#stringrequiredPVC name

Name of the PVC to attach.

sharedVolume#objectoptionalEphemeral shared volume across all pods

AIchor creates a temporary shared volume, mounts it on all pods, and deletes it when the experiment ends.

accessMode#stringrequiredKubernetes PV access mode

Kubernetes PV access mode, e.g. ReadWriteOnce or ReadWriteMany.

mountPoint#stringrequiredMount path inside the container

Mount path inside the container (e.g. /mnt/shared).

sizeGB#integerrequiredVolume size in GiB

Volume size in gigabytes.

Storage classes enforce a minimum size

Each cloud provider applies a minimum volume size per storage class. A request below that floor is rejected or silently rounded up by the CSI driver, and billing is based on the provisioned size, not the requested size. The value of sizeGB must respect the minimum of the chosen storageClass.

Common minimums:

Storage classProviderMinimum sizeGB
gp2, gp3 (EBS)AWS1
io1, io2 (EBS)AWS4
pd-standard, pd-balanced, pd-ssd (Persistent Disk)GCP10
standard / premium (Filestore, NFS)GCP1024 / 2560

For example, requesting sizeGB: 1 against a pd-balanced storage class on GCP still provisions a 10 GiB disk, and a Filestore-backed class is billed for at least 1 TiB regardless of the requested value.

storageClass#stringrequiredKubernetes storage class

Kubernetes storage class name (e.g. gp2, standard-rwo, ceph-rbd, cephfs, longhorn).

tensorboard#object
enabled#booleanrequiredEnable TensorBoard

Whether to run a TensorBoard sidecar for the experiment. When enabled, the TensorBoard dashboard is accessible from the experiment's page.

usePVC#objectoptionalRead logs from a PVC

When set, TensorBoard reads logs from a PVC instead of object storage. Requires spec.storage.attachExistingPVCs to be configured.

enabled#booleanrequiredUse PVC for TensorBoard logs

Whether to mount a PVC for TensorBoard log storage.

name#stringoptionalPVC name from attachExistingPVCs

Name of a PVC from spec.storage.attachExistingPVCs. When omitted, the first PVC in the list is used.

timeout#stringoptionalMax duration — e.g. "1d 6h"

Maximum duration as a human-readable string. Takes precedence over activeDeadlineSeconds.

Accepts combinations of w (weeks), d (days), h (hours), m (minutes), s (seconds).

timeout: "1w 2d 5h 6m 20s"
types#objectrequired

Defines worker pools — required field names depend on the operator. Select an operator below to see the structure of spec.types for that operator.

Across all operators, spec.types is required and contains the worker-pool specifications. The exact shape depends on the operator:

  • jax, pytorch, xgboost — single worker pool.
  • jobset — named replicated jobs (free-form job keys).
  • kuberay — one Head pod plus one or more Workers pools. For all Kubernetes-based operators (jax, pytorch, xgboost, jobset, kuberay), each pool's resources accepts the same fields — select an operator tab to see the full schema inline.