Logs

Experiment logs are available both in the AIchor dashboard and through the AIchor CLI.

warning

Logs are displayed in UTC. AIchor retains logs for 30 days — anything you need to keep longer must be saved manually.

Logs viewer

Selecting an experiment in the dashboard opens its detail view, where logs are displayed for each step of the experiment. The viewer is scrollable, and a pod selector lets you focus on a specific pod's output.

Logs are also streamed while the experiment is running.

info

When you open a step's logs, the viewer loads the last 1,000 lines the step printed, counted across all of its pods, then adds new lines as they are printed while the page stays open. For the complete logs, use Download logs: the download icon in the logs toolbar, or the ⋮ menu when the panel is narrow.

Logs from the CLI

The CLI exposes two commands for accessing logs:

• aichor experiments logs stream — streams logs in real time from the experiment's current step. Intended for active experiments.
• aichor experiments logs query — retrieves up to the last 2000 log entries from a specific step (clone, build, submit, or run). Intended for inspecting completed experiments. Output can be printed to the terminal or written to a directory with one .txt file per pod.

For a running experiment, logs can be streamed in real time:

aichor experiments logs stream <experiment-id>

For a completed experiment, the last 2000 log entries can be retrieved:

aichor experiments logs query <experiment-id>

To save logs to a directory (one .txt file per pod):

aichor experiments logs query <experiment-id> -o dir

To retrieve logs from a specific step (clone, build, submit, run):

aichor experiments logs query <experiment-id> --step build

See the CLI Reference for the full list of options.