Skip to main content

Tips and tricks

Different types of output​

If running the cli in scripts it might be beneficial to use the -o json output to parse information from the cli, instead of parsing the default table that gets shown in the terminal.

Leveraging Docker build caching for similar experiments​

Lets say you would like to run multiple experiments on AIchor where the differences between each experiment are not significant, e.g., the Dockerfile is exactly the same but there are small changes in resources for the experiment, the data being used or the command being executed.

To benefit from Docker image caching you can use aichor experiments submit commit-sha command with the --manifest-path option. This way you can have multiple manifest files in your repo, one per experiment type, each with their own command/data/resources. The workflow would be as follows:

  1. Create multiple manifest files in your repo, one per experiment variant; each with a different command:
  manifests/baseline.yaml
  manifests/model-encoder-v2.yaml
  manifests/model-encoder-v3.yaml

  1. Push a single commit with all of them and save the commit-sha (e.g. d1faca3)

  2. Submit each experiment against the same commit SHA:

  aichor experiments submit commit-sha d1faca3 --branch main --project-name <project> --manifest-path manifests/baseline.yaml

  aichor experiments submit commit-sha d1faca3 --branch main --project-name <project> --manifest-path manifests/model-encoder-v2.yaml

  aichor experiments submit commit-sha d1faca3 --branch main --project-name <project> --manifest-path manifests/model-encoder-v3.yaml

Why this is better than submitting using local repo:

  • Unlike with local submit (aichor experiments submit local), there is no zipped project upload each time; the code is fetched from git using the commit SHA
  • All builds share the same cache tag (your branch name), so after the first build completes, subsequent ones are fast cache hits
  • No need to edit and re-save manifest.yaml between submissions
  • Cleaner audit trail; each experiment points to a specific commit + manifest file

⚠️ One note: submit the first experiment and wait for its build step to finish before submitting the rest. That way the first build populates the registry cache and the rest will be near-instant.