Commands

Command	Description
aichor auth	Authentication commands
aichor context	Manage the CLI local context
aichor local-repo	Manage the local repository
aichor engines	Operations on engines
aichor experiments	Operations on experiments
aichor projects	Operations on projects
aichor storage	Operations on project storage

aichor auth

aichor auth key

Login using API key; requires an API key to authenticate.

The key can be obtained from https://web.aichor.ai/credentials

You can provide it in different ways:

• Interactive Login (Prompt for API key): Simply run the following command and enter the API key when prompted:

aichor auth key

• Passing the API key as an Argument:

aichor auth key --apikey YOUR_API_KEY

• Using an Environment Variable (Recommended for CI/CD): Set the AICHOR_API_KEY environment variable and run the login command:

export AICHOR_API_KEY=YOUR_API_KEY
aichor auth key

If the API key is valid, the CLI will store it for future use. Otherwise, an error message will be displayed.

Options:

• --apikey API key for login (will prompt if not provided).

aichor context

Manage the CLI context. Allows setting default projects and engines used by the CLI.

Command	Description
aichor context list	Print the current CLI context
aichor context reset all	Reset all customizable fields in local context
aichor context reset engine	Reset default engine in local context
aichor context reset project	Reset default project in local context
aichor context set engine	Set default engine in local context
aichor context set project	Set default project in local context

aichor context list

Print the current CLI context as JSON, showing the active default project and engine.

aichor context list

aichor context reset all

Remove all customizable fields from the CLI context.

aichor context reset all

aichor context reset engine

Remove the current engine from the CLI context.

aichor context reset engine

aichor context reset project

Remove the current project from the CLI context.

aichor context reset project

aichor context set engine

Define the default engine to be used in commands.
Alternatively, an engine name must be specified each time a command is executed.

Arguments:

• engine_name
  The engine name.

Example:

aichor context set engine <engine_name>

aichor context set project

Define the default project.
Alternatively, a project name must be specified each time a command is executed.

Arguments:

• project_name
  The project name.

Example:

aichor context set project <project_name>

aichor local-repo

Commands to generate templated files to help getting started with AIchor.

Command	Description
aichor local-repo generate-files	Generate specific templated files for the local repository
aichor local-repo init	Generate templated AIchor-ready repo structure

aichor local-repo generate-files

Generate specific templated files for the local repository.

⚠️ This will overwrite files with the same name in the directory where the CLI is being executed.

aichor local-repo generate-files [--force] <file_type>...

Arguments:

• <file_type>...
  One or more file types to generate.
  Options: dockerfile, readme, pyproject, uv_directory, manifest

Options:

• --force
  Skip confirmation if existing files are found.

Example:

aichor local-repo generate-files uv_directory dockerfile

The generated directory tree for the uv_directory option will look as follows:

src/
└── project_name/
    ├── __init__.py
    └── main.py

aichor local-repo init

Generate the required directory structure and files to start running experiments with AIchor, using uv to package the project.

aichor local-repo init

The generated files include:

• UV compliant directory structure: uv requires certain files and directories to exist, which AIchor will automatically generate.
• Pyproject.toml: This is the file where the required packages for the project can be defined. Depending on the chosen ML operator, some packages may already be included.
• Dockerfile: A basic Dockerfile that uses uv to create a ready-to-run image of the project.
• README.md: Simple README.md that explains the generated files and any changes that might be needed to suit custom needs and start running experiments.
• Manifest.yaml: An AIchor manifest file adapted to the selected ML operator and computing resource requests.

The generated directory structure will look like this:

<project_name>/
├── Dockerfile
├── README.md
├── manifest.yaml
├── pyproject.toml
└── src/
    └── project_name/
        ├── __init__.py
        └── main.py

aichor engines

Command	Description
aichor engines list	List available engines

aichor engines list

Returns the list of engines, or a specific engine if engine-name is provided.

aichor engines list [engine-name] [options]

Arguments:

• engine-name
  Optional. Name of a specific engine to show details for.

Options:

• -o, --output
  Output format.
  Accepted values: table, json.
  Default is table.

Example:

aichor engines list <engine_name> --output json

aichor experiments

Command	Description
aichor experiments cancel	Cancel an experiment
aichor experiments list	List experiments
aichor experiments list-pods	List pods for an experiment
aichor experiments logs query	Retrieve logs from a completed experiment
aichor experiments logs stream	Stream logs for a running experiment
aichor experiments resubmit	Resubmit an experiment
aichor experiments status	Get the status of an experiment
aichor experiments step	Get the current step of an experiment
aichor experiments submit local	Submit an experiment from your local repository
aichor experiments submit commit-sha <commit-sha>	Submit an experiment by specifying a branch and commit SHA

aichor experiments cancel

Cancel an experiment.

aichor experiments cancel <experiment-id> [options]

Arguments:

• experiment-id
  ID of the experiment to cancel.

Options:

• --project-name
  Override the default project.
  Required if no default project in CLI context exists.

aichor experiments list

Returns the list of experiments, or a specific experiment if experiment-id is provided.

aichor experiments list [experiment-id] [options]

Arguments:

• experiment-id
  Optional. ID of a specific experiment to show.

Options:

• --engine-name
  Override the default engine.
  Used only when listing multiple experiments.

• -o, --output
  Output format.
  Accepted values: table, json.
  Default is table.

• --page-number
  Page number for pagination.
  Default is 1.

• --page-size
  Number of experiments per page.
  Default is 50.

• --project-name
  Override the default project.
  Required if no default project in CLI context exists.

Example:

• JSON output for a single experiment using project from CLI context:

aichor experiments list <experiment_id> --output json

• JSON output for all experiments with a specific project and engine:

aichor experiments list --project-name <project_name> --engine-name <engine_name> --output json

aichor experiments list-pods

Lists all pods for an experiment, or just a specific one if --pod-id is specified.

aichor experiments list-pods <experiment-id> [options]

Arguments:

• experiment-id
  ID of the experiment.

Options:

• -o, --output
  Output format.
  Accepted values: table, json.
  Default is table.

• --pod-id
  ID of the pod.

• --project-name
  Override the default project.
  Required if no default project in CLI context exists.

Examples:

• JSON output for a single pod:

aichor experiments list-pods <experiment_id> --pod-id <pod_id> --output json

• JSON output for all pods in an experiment:

aichor experiments list-pods <experiment_id> --output json

aichor experiments logs query

Retrieve the last 2000 logs from a specific step in an experiment.

aichor experiments logs query <experiment-id> [options]

Arguments:

• experiment-id
  ID of the experiment to retrieve logs from.

Options:

• --engine-name
  Override the default engine.
  Required if no default engine in CLI context exists.

• -o, --output
  Output format.
  Types: terminal, dir.
  dir creates a directory at the current working path named after the experiment-id. It contains .txt files, each with logs from a specific pod.
  Default is terminal.

• --pod-id
  Retrieve logs only from the specified pod.
  If not provided, logs from all pods are shown.
  Can only specify pod-id when --step is run

• --project-name
  Override the default project.
  Required if no default project in CLI context exists.

• --step
  Experiment step to fetch logs from.
  Possible values: clone, build, submit, run.
  Default is run.

• --oldest
  Fetch the oldest log entries instead of the most recent.
  By default, the most recent 2000 entries are returned.

Example:

• Send logs to a directory named after the experiment ID and with a .txt file per pod with the corresponding logs.

aichor experiments logs query <experiment-id> --project-name <project_name>  --engine-name <engine_name> -o dir

aichor experiments logs stream

Stream experiment logs starting from the current step.
If the experiment is older than 30 days, the logs will no longer be available.

aichor experiments logs stream <experiment-id> [options]

Arguments:

• experiment-id
  ID of the experiment to stream logs from.

Options:

• --engine-name
  Override the default engine.
  Required if no default engine in CLI context exists.

• --pod-id
  Stream logs only from the specified pod.
  If not provided, logs from all pods are shown.

• --project-name
  Override the default project.
  Required if no default project in CLI context exists.

Example:

• Capturing the final experiment status in a bash script:

STATUS=$(aichor experiments logs stream <experiment-id> | jq -r '.experiment_status')

aichor experiments resubmit

Resubmit an experiment.

aichor experiments resubmit <experiment-id> [options]

Arguments:

• experiment-id
  ID of the experiment to resubmit.

Options:

• --project-name
  Override the default project.
  Required if no default project in CLI context exists.

aichor experiments status

Get the status of an experiment.

Outputs {"experiment_status": <status>} as JSON to stdout. Possible status values: Created, Processing, Cancelled, Succeeded, Failed.

aichor experiments status <experiment-id> [options]

Arguments:

• experiment-id
  ID of the experiment.

Options:

• --project-name
  Override the default project.
  Required if no default project in CLI context exists.

Examples:

aichor experiments status <experiment_id>

• Capturing the experiment status in a bash script:

STATUS=$(aichor experiments status <experiment-id> | jq -r '.experiment_status')

aichor experiments step

Get the current step of an experiment.

Outputs {"experiment_step": <step>} as JSON to stdout. Possible step values: Waiting, Cloning, Building, Submitting, Running, Completed.

aichor experiments step <experiment-id> [options]

Arguments:

• experiment-id
  ID of the experiment.

Options:

• --project-name
  Override the default project.
  Required if no default project in CLI context exists.

Examples:

aichor experiments step <experiment_id>

Capturing the experiment step in a bash script:

STEP=$(aichor experiments step <experiment-id> | jq -r '.experiment_step')

aichor experiments submit commit-sha

Submits an experiment by specifying a branch and a commit SHA.

aichor experiments submit commit-sha <commit_sha> --branch <branch> [options]

Arguments:

• commit_sha
  Commit SHA to use.

Required options:

• --branch
  Branch to trigger the experiment from.

Options:

• --engine-name
  Override the default engine.

• --manifest-path
  Override the manifest in the root by providing the path to another manifest.

• --project-name
  Override the default project.
  Required if no default project in CLI context exists.

Example:

• Submitting an experiment from a specific branch and commit SHA, using project and engine from the CLI context:

aichor experiments submit commit-sha <commit-sha> --branch <branch>

Capturing the experiment ID in a bash script:

EXPERIMENT_ID=$(aichor experiments submit commit-sha <commit-sha> --branch <branch> | jq -r '.experiment_id')

aichor experiments submit local

Submits an experiment from your local repository.
This will zip a local directory, send it to AIchor and trigger an experiment using this code.

aichor experiments submit local --message <message> --repo-dir <repo_dir> [options]

Required options:

• --message
  Commit message for the experiment.

• --repo-dir
  Path to the local repository.

Options:

• --engine-name
  Override the default engine.

• --project-name
  Override the default project.
  Required if no default project in CLI context exists.

Example:

• Submitting an experiment with contents of the current directory with commit message "First experiment", using project and engine from the CLI context:

aichor experiments submit local --message "First experiment" --repo-dir .

Capturing the experiment ID in a bash script:

EXPERIMENT_ID=$(aichor experiments submit local --message "First experiment" --repo-dir . | jq -r '.experiment_id')

Support for .aichorignore

An .aichorignore file in the project's root directory can be used to exclude files and directories from being included in the repo when an experiment is submitted via the CLI.

Create an .aichorignore file in the project's root directory with .gitignore-style patterns:

# Ignore all files in test_ignore directory
test_ignore/

# Ignore temporary files
*.tmp

aichor projects

Command	Description
aichor projects costs	List costs for a project
aichor projects list	List available projects
aichor projects linked-engines	List engines linked to a project

aichor projects costs

Returns the costs of all projects, or a specific project if project-name is provided.
By default shows just total cost for the project.
It is also possible to select specific resources such as cpu, memory, pvc, buckets, as well as total and itemized gpu costs.

aichor projects costs [project-name] [options]

Arguments:

• project-name
  Optional. Name of a specific project.

Options:

• --accelerators
  Show accelerators usage costs per gpu name.

• --buckets
  Show buckets usage costs.

• --cpu
  Show cpu usage costs.

• --gpu
  Show gpu usage costs.

• --memory
  Show memory usage costs.

• -o, --output
  Output format.
  Accepted values: table, json.
  Default is table.

• --pvc
  Show PVC usage costs.

• --total
  Show all costs.

Example:

• JSON output for total gpu and pvc costs

aichor projects costs <project_name> --gpu --pvc --output json

aichor projects linked-engines

Returns the engines linked to a project.

aichor projects linked-engines [project-name] [options]

Arguments:

• project-name
  Optional. Name of a specific project.

Options:

• -o, --output
  Output format.
  Accepted values: table, json.
  Default is table.

Example:

aichor projects linked-engines <project_name> --output json

aichor projects list

Returns the list of projects, or a specific project if project-name is provided.

aichor projects list [project-name] [options]

Arguments:

• project-name
  Optional. Name of a specific project to show.

Options:

• -o, --output
  Output format.
  Accepted values: table, json.
  Default is table.

Example:

aichor projects list <project_name> --output json

aichor storage

Operations on project storage buckets.
⚠️ The CLI is working under the following assumption:

• Directory paths end with trailing backslash, e.g. ./Path/To/Directory/
• File paths do not end with a trailing backslash, e.g. ./Path/To/File.txt

Command	Description
aichor storage cloud cp	Copy files between two remote storage buckets
aichor storage cloud download	Download a file or directory from a remote storage bucket to a local path
aichor storage cloud list	List buckets for a project
aichor storage cloud list-contents	List contents inside a project bucket
aichor storage cloud mkdir	Create a directory in a project bucket
aichor storage cloud rm	Delete a file or directory in a project bucket
aichor storage cloud storage-credentials	Fetch storage credentials for project buckets
aichor storage cloud upload	Upload a local file or directory to a remote storage bucket

Filter patterns

Several storage commands accept --include and --exclude options that filter files using glob patterns (e.g. *.txt, data/*.csv). Each option can be specified multiple times to provide more than one pattern. Patterns are matched against both the full path and the filename.

Priority rules:

1. Exclude is evaluated first. A file matching any --exclude pattern is always removed, even if it also matches an --include pattern.
2. Include is evaluated second. If at least one --include pattern is given, only files matching one of those patterns are kept.
3. No filters — all files are returned when neither option is provided.

Scenario	Result
--include "*.txt"	Only .txt files are kept
--exclude "*.tmp"	All files except .tmp files are kept
--include "*.txt" --exclude "*.txt"	No files (exclude has priority)
--include "data/*.csv" --exclude "data/raw*"	.csv files in data/ except those whose name starts with raw

aichor storage cloud cp

Copy files between two remote storage buckets.

aichor storage cloud cp --dst-path <dst_path> --dst-storage-id <dst_storage_id> --src-path <src_path> --src-storage-id <src_storage_id> [options]

Required options:

• --dst-path
  Destination path within the bucket.

• --dst-storage-id
  ID of the destination storage bucket.

• --src-path
  Source path within the bucket.

• --src-storage-id
  ID of the source storage bucket.

Options:

• --engine-name
  Override the default engine.

• --project-name
  Override the default project.
  Required if no default project in CLI context exists.

Example:

aichor storage cloud cp --src-storage-id <src_storage_id> --src-path /mydir/ --dst-storage-id <dst_storage_id> --dst-path /mydir/ --project-name <project_name> --engine-name <engine_name>

aichor storage cloud download

Download a file or directory from a remote storage bucket to a local path.

aichor storage cloud download --remote-path <remote_path> --storage-id <storage_id> [options]

Required options:

• --remote-path
  Source path within the bucket to download from.

• --storage-id
  ID of the source storage bucket.

Options:

• --engine-name
  Override the default engine.

• --local-path
  Destination local path.
  Defaults to current directory.

• --max-workers
  Number of parallel download workers.
  Default is 1.

• --overwrite
  Overwrite existing local files.

• --project-name
  Override the default project.
  Required if no default project in CLI context exists.

Examples:

• Download a directory:

aichor storage cloud download --remote-path /mydir/ --storage-id <storage_id> --project-name <project_name> --engine-name <engine_name> --local-path ./mydir/

• Download a remote directory to a local directory with a different name:

aichor storage cloud download --remote-path /mydir/ --storage-id <storage_id> --project-name <project_name> --engine-name <engine_name> --local-path ./another_dir/

• Download a single file to the ./downloads directory:

aichor storage cloud download --remote-path /mydir/file.txt --storage-id <storage_id> --project-name <project_name> --engine-name <engine_name> --local-path ./downloads/

• Download a single file:

aichor storage cloud download --remote-path /mydir/file.txt --storage-id <storage_id> --project-name <project_name> --engine-name <engine_name> --local-path ./file.txt

aichor storage cloud list

List buckets for a project.

aichor storage cloud list [options]

Options:

• --engine-name
  Override the default engine.

• -o, --output
  Output format.
  Accepted values: table, json.
  Default is table.

• --project-name
  Override the default project.
  Required if no default project in CLI context exists.

Example:

aichor storage cloud list --project-name <project_name> --engine-name <engine_name>

aichor storage cloud list-contents

List contents inside a project bucket.

aichor storage cloud list-contents --storage-id <storage_id> [options]

Required options:

• --storage-id
  ID of the storage bucket.

Options:

• --engine-name
  Override the default engine.

• --exclude
  Exclude items matching the given glob pattern (e.g. *.tmp).
  Can be specified multiple times.
  See Filter patterns for priority rules.

• --include
  Include only items matching the given glob pattern (e.g. *.txt).
  Can be specified multiple times.
  See Filter patterns for priority rules.

• --max-depth
  Maximum recursion depth when using --recursive.
  If not set, depth is unlimited.

• -o, --output
  Output format.
  Types: [tree, size, simple].
  Default is tree.

• --path
  Path within the bucket to list. Defaults to root (.).

• --project-name
  Override the default project.
  Required if no default project in CLI context exists.

• -r, --recursive
  List subdirectories recursively.

The different output work as follows:

• tree shows the directory structure of the selected path. For example:

📂 Root
├── 📄 file1.txt
├── 📄 file2.txt
└── 📁 example_folder
    └── 📄 file3.txt

• size shows a file or directory per line along with the corresponding size. For example:

22.0 MiB 📄 file1.txt
24.0 MiB 📄 file2.txt
15.0 MiB 📁 example_folder/
15.0 MiB 📄 example_folder/file3.txt
----------------------------------------
Total unique data: 75.0 MiB across 4 items.

• simple shows the list of files, one per line. For example:

file1.txt
file2.txt
example_folder/
example_folder/file3.txt

Examples:

• List all contents recursively:

aichor storage cloud list-contents --storage-id <storage_id> --project-name <project_name> --engine-name <engine_name> -r

• List direct children of a path only (depth 0):

aichor storage cloud list-contents --storage-id <storage_id> --project-name <project_name> --engine-name <engine_name> -r --max-depth 0 --path /mydir/

• List only .txt files:

aichor storage cloud list-contents --storage-id <storage_id> --project-name <project_name> --engine-name <engine_name> -r --include "*.txt"

• Exclude .txt files:

aichor storage cloud list-contents --storage-id <storage_id> --project-name <project_name> --engine-name <engine_name> -r --exclude "*.txt"

aichor storage cloud mkdir

Create a new directory in a project bucket.

aichor storage cloud mkdir --path <path> --storage-id <storage_id> [options]

Required options:

• --path
  Path of the directory to create within the bucket.

• --storage-id
  ID of the storage bucket.

Options:

• --engine-name
  Override the default engine.

• --project-name
  Override the default project.
  Required if no default project in CLI context exists.

Example:

aichor storage cloud mkdir --storage-id <storage_id> --project-name <project_name> --engine-name <engine_name> --path ./mydir/

aichor storage cloud rm

Delete a file or directory in a project bucket.

aichor storage cloud rm --path <path> --storage-id <storage_id> [options]

Required options:

• --path
  Path of the file or directory to delete.

• --storage-id
  ID of the storage bucket.

Options:

• --dry-run
  Preview what would be deleted without performing any deletions.

• --engine-name
  Override the default engine.

• --exclude
  Exclude items matching the given glob pattern (e.g. *.tmp).
  Can be specified multiple times.
  See Filter patterns for priority rules.

• --force, -f
  Delete without confirmation prompt.

• --include
  Include only items matching the given glob pattern.
  Can be specified multiple times.
  See Filter patterns for priority rules.

• --max-print
  Maximum number of items to display during the dry-run.
  Default is 20.

• --max-workers
  Number of parallel deletion workers.
  Default is 3.

• --project-name
  Override the default project.
  Required if no default project in CLI context exists.

Examples:

• Delete a specific file:

aichor storage cloud rm --storage-id <storage_id> --project-name <project_name> --engine-name <engine_name> --path /mydir/file.txt --force

• Preview deletion of a directory:

aichor storage cloud rm --storage-id <storage_id> --project-name <project_name> --engine-name <engine_name> --path /mydir/ --dry-run

aichor storage cloud storage-credentials

Fetch storage credentials for the project's buckets.

aichor storage cloud storage-credentials --engine-name <engine_name> [options]

Options:

• --engine-name
  Override the default engine.
  Required if no default engine in CLI context exists.

• -o, --output
  Output format.
  Accepted values: table, json.
  Default is table.

• --project-name
  Override the default project.
  Required if no default project in CLI context exists.

Example:

aichor storage cloud storage-credentials --project-name <project_name> --engine-name <engine_name> --output json

aichor storage cloud upload

Upload a local file or directory to a remote storage bucket.

aichor storage cloud upload --remote-path <remote_path> --storage-id <storage_id> [options]

Required options:

• --remote-path
  Destination path within the bucket.

• --storage-id
  ID of the destination storage bucket.

Options:

• --engine-name
  Override the default engine.

• --local-path
  Source local path.
  Defaults to current directory.

• --max-workers
  Number of parallel upload workers.
  Default is 1.

• --overwrite
  Overwrite existing files at the destination.

• --project-name
  Override the default project.
  Required if no default project in CLI context exists.

Examples:

• Upload a directory:

aichor storage cloud upload --local-path ./mydir/ --storage-id <storage_id> --project-name <project_name> --engine-name <engine_name> --remote-path /mydir/

• Upload a local directory to a remote directory with another name:

aichor storage cloud upload --local-path ./mydir/ --storage-id <storage_id> --project-name <project_name> --engine-name <engine_name> --remote-path /another_dir/

• Upload a file to a remote directory:

aichor storage cloud upload --local-path ./file.txt --storage-id <storage_id> --project-name <project_name> --engine-name <engine_name> --remote-path /mydir/

• Upload a file:

aichor storage cloud upload --local-path ./file.txt --storage-id <storage_id> --project-name <project_name> --engine-name <engine_name> --remote-path ./file.txt

• Upload and overwrite existing files:

aichor storage cloud upload --local-path ./mydir/ --storage-id <storage_id> --project-name <project_name> --engine-name <engine_name> --overwrite