Skip to main content

Debug Tools

AIchor offers some tools and features for debug purposes.

Terminal​

AIchor offers the possibility to users to access running pods in order to debug the experiments or any other purpose that they see fit.

alt text alt text

Interactive debugging with Visual Studio Code and Cursor​

Although the terminal feature allows users to access running pods and debug potential issues in the model from AIchor UI, options are limited and the terminal does not offer options that are made available on IDEs.

AI Engineers need to access running pods to debug their code in a user friendly environment while accessing tools that would allow them to be efficient and fast.

In this context, accessing running pods from the local computer of the AI engineer through VS Code or Cursor is an option in AIchor.

User requirements​

The only requirement to be able to take advantage of this feature is to have either:

  • a GitHub account
  • or a Microsoft account

This account will be needed for authentication only to VCS.

Code Requirements​

Visual Studio Code binary must be present in the experiment's image. For example with this additional Dockerfile stage:

FROM alpine/curl AS vscode-installer

RUN mkdir /aichor
RUN curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output /aichor/vscode_cli.tar.gz
RUN tar -xf /aichor/vscode_cli.tar.gz -C /aichor


# Project's Dockerfile content, for example
FROM pytorch/pytorch:2.4.1-cuda12.4-cudnn9-runtime
# ...
# Project's Dockerfile content

# Copy the vscode binary on the final Dockerfile stage at '/aichor'
# binary won't be findable in $PATH, it'll just be located at '/aichor/code'
COPY --from=vscode-installer /aichor /aichor

Similarly in Cursor, changes in the Dockerfile should be as follows:

FROM alpine/curl AS cursor-installer

RUN mkdir /aichor
RUN curl -L 'https://api2.cursor.sh/updates/download-latest?os=cli-alpine-x64' --output /aichor/cursor.tar.gz
RUN tar -xf /aichor/cursor.tar.gz -C /aichor

# Project's Dockerfile content, for example
FROM pytorch/pytorch:2.4.1-cuda12.4-cudnn9-runtime
# ...
# Project's Dockerfile content

# Copy the vscode binary on the final Dockerfile stage at '/aichor'
# binary won't be findable in $PATH, it'll just be located at '/aichor/code'

COPY --from=cursor-installer /aichor /aichor

Enabling vscode debug mode:

...
spec:
  ...
  debug:
    vscode:
      enabled: true # required
      path: "/aichor/code" # Optional, defaults to "code". Refer the absolute path of the binary if not findable in $PATH
      provider: "github" # Optional, vscode tunnel provider. Options are "github" or "microsoft", defaults to "github"
...

As per Cursor, changes in the manifest are as follows:

...
spec:
  ...
  debug:
    vscode:
      enabled: true # required
      path: "/aichor/cursor" # cursor binary path, setup in the Dockerfile
      provider: "github" 
...

For example, using GitHub, wait for the experiment to be running, then check the logs you should find a log line similar to

To grant access to the server, please log into https://github.com/login/device and use code XXXX-XXXX

Follow the link, enter the given code and allow GitHub. After that, the vscode's tunnel should be opened with GitHub, more logs should appear.

*
* Visual Studio Code Server
*
* By using the software, you agree to
* the Visual Studio Code Server License Terms (https://aka.ms/vscode-server-license) and
* the Microsoft Privacy Statement (https://privacy.microsoft.com/en-US/privacystatement).
*
[2025-02-24 14:25:24] info Creating tunnel with the name: 8e68f47f-6e08-mhTrZ
Open this link in your browser https://vscode.dev/tunnel/8e68f47f-6e08-mhTrZ/opt/app

Click the vscode.dev link to enter vscode, select GitHub to connect to the tunnel

select_provider

Then you should be able to access the experiment files and terminal

show_nvidiasmi

You can also access the tunnel from the vscode app. To do that run the app and click the button to setup a remote vscode:

setup_remote_vscode

Select "Connect to tunnel":

select_connect_to_tunnel

Login with GitHub and select the tunnel, an "Online" flag should be present:

select_experiment_tunnel

Then vscode will be started at ~ which probably won't contain the experiment's files, you can access them by selecting "Open Folder", then select the path.

open_folder

Finally, you should be able to interact with the code and the requested compute:

exec_sw_show_nvtop

Also, you can install remote extensions like the official Python one to run the debugger with the code:

show_py_debugger

Known issues​

Cursor Extension​

Recently Cursor removed the "Remote - Tunnels" extension from their marketplace breaking the Cursor setup. We have a few testimonies from our users that managed to install it anyway:

Solution #1, Manually install the extension.​
  • Download the extension pkg:
curl -L "https://ms-vscode.gallery.vsassets.io/_apis/public/gallery/publisher/ms-vscode/extension/remote-server/latest/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage" --output remote-server-latest.vsix
  • Then install it manually:
cursor --install-extension ./remote-server-latest.vsix
  • Open cursor, ctrl+shift+P -> type remote-tunnel: you should now see the Remote-Tunnel commands
Solution #2, Import from vscode​

In extensions panel, use the option to import extensions from vscode

If none of the 2 solutions work for you, you can try to completely uninstall and reinstall Cursor ref.

Additional notes​

Additional notes:

  • The Visual Studio Code server will run for 12 hours max. When you are done debugging your code, cancel it like a regular experiment.
  • The traffic between the vscode server running in the experiment and the client (web, or vscode app) is tunneled via GitHub or Microsoft.
  • When this spec.debug.vscode.enabled is true, AIchor will overwrite the manifest's command with a vscode server start command. That means the manifest's command won't be interpreted.