Submit jobs

You can submit jobs via the web UI or SDK. A job can run using either a Studio environment or a Docker image as its environment.

Use a Studio environment

A Studio environment contains everything your code needs to execute, including code, data, dependencies, and environment variables. When you submit a job using the Studio, the environment is forked at submission, allowing you to continue editing without affecting the submitted job.

Jobs are non-interactive, parallel executions of your Studio.

Submit via the web UI

Click the Jobs icon in the Studio.
Select the file to execute.
Hit Submit

How to submit a job

Submit via code (SDK)

Use the Lightning SDK to submit a job programmatically:

from lightning_sdk import Studio, Machine, Job

studio = Studio(name='my-studio', teamspace='my-teamspace', user='my-user')

# Define the command to run and submit the job
submitted_job = Job.run(command="echo Hello, Lightning!", name="echo-example", machine=Machine.CPU, studio=studio)

print(f"Job submitted: {submitted_job}")

Submit via terminal (CLI)

After installing the latest version of the command line interface (CLI; pip install --upgrade lightning-sdk), submit a job by running

lightning job run \
  --command="python train.py" \
  --studio=my-studio \
  --teamspace=my-teamspace \
  --org=my-org

For additional information on submitting a job from the CLI, run

lightning job run --help

Use a Docker image

A Docker image can be used as the job environment, providing a fully isolated and consistent setup.

Submit via code (SDK)

Submit a job with a Docker image:

from lightning_sdk import Job, Teamspace, Machine

teamspace = Teamspace(name="my-teamspace", user="my-user")

submitted_job = Job.run(
    name="my-job",
    command="python /scripts/train.py",
    image="litcr.io/lit-container/<my_lightning_org>/<my_teamspace>/<my_image>:<tag>",
    machine=Machine.L40s,
    teamspace=teamspace
)

print(submitted_job.status)

Private DockerHub image

For private images, add your credentials as a secret:

For accessing, private images on DockerHub, make sure to add your credentials as a secret to your account. These will then automatically be used to pull your specified image. On the SDK, these can then be specified as

from lightning_sdk import Job, Teamspace, Machine, Status

# Get the teamspace you want to submit your Job in
teamspace = Teamspace(name="my-teamspace", user="my-user")

# Submit the Job
submitted_job = Job.run(
    name="my-job",
    command="python /scripts/train.py",
    image="my-image",
    machine=Machine.L40s,
    teamspace=teamspace,
    image_credentials="my-secret-name"
)

with "my-secret-name" being whatever you specified as the name of your credential secret.

Private image with AWS ECR

To use private ECR images, add a private cloud account first. Once that's done, use the cloud account to authenticate to ECR:

from lightning_sdk import Job, Teamspace, Machine, Status

# Get the teamspace you want to submit your Job in
teamspace = Teamspace(name="my-teamspace", user="my-user")

# Submit the Job
submitted_job = Job.run(
    name="my-job",
    command="python /scripts/train.py",
    image="my-image",
    machine=Machine.L40s,
    teamspace=teamspace,
    cloud_account_auth=True
)

Submit via terminal (CLI)

After installing the latest version of the command line interface (CLI; pip install --upgrade lightning-sdk), submit a job by running

lightning job run \
  --image="my-image:latest" \
  --command="python /train.py" \
  --teamspace=my-teamspace \
  --org=my-org

Was this page helpful?