Scheduling

Pipelines can be scheduled either through the Lightning UI or directly via the Lightning SDK.

From the SDK

This script demonstrates how to set up an automated pipeline using the Lightning SDK. The pipeline, named 'cron', executes a Python script, "main.py", within a Studio environment. It is configured to run repeatedly, with a schedule set to trigger the execution every minute using a cron expression. The setup process involves initializing the pipeline, defining the job execution steps, and establishing a schedule for the pipeline to automate its run at the specified intervals.

Initialize the PipelineWe start by importing necessary classes from the lightning_sdk and create an instance of the Pipeline class with the name 'cron'.Define Pipeline StepsNext, we define the steps that the pipeline should execute. In this example, we are running a single Studio Job. The Job executes the command "python main.py" within the Studio environment.Schedule Pipeline ExecutionFinally, we schedule the pipeline to run at specified intervals. The schedule used here is "* * * * *", which configures the pipeline to run every minute.

1/3

2/3

3/3

from lightning_sdk.pipeline import Pipeline, JobStep, Studio, Schedule

pipeline = Pipeline(name='cron')

pipeline.run(
    steps=[
        JobStep(command="python main.py", studio=Studio()),
    ],
    schedules=[Schedule("* * * * *")]
)

Notes:

For the time being, Lightning pipelines can be created only the SDK. However, you can configure your scheduling from the UI once created.
From the SDK, the schedule timezone defaults to UTC.

Notes:

For the time being, Lightning pipelines can be created only the SDK. However, you can configure your scheduling from the UI once created.
From the SDK, the schedule timezone defaults to UTC.

From the UI

In the UI, navigate to the Schedules Tab, where you'll find the New Schedule button in the top right corner. Clicking this button allows you to easily configure your schedule through Lightning AI's user-friendly interface. Alternatively, you can use the traditional cron format for specifying schedules.

Video showing how to create a pipeline schedule

Notes:

From the UI, you can configure the scheduling timezone. By default, it uses the browser's timezone but you can manually change it.
From the UI, you can stop or delete a schedule whenever you need to.

Was this page helpful?