Overview

The Lightning data platform has been designed by AI researchers for high-performance AI research. We understand the different storage solutions needed to make team collaboration efficient, high-performance training, etc...

Lightning AI offers various data solutions that solve everything from team collaboration with granular access controls, to high-performance training, cloud agnostic datasets and high-performance cluster file systems.

Cloud folders

Cloud Folders let you organize data across teams and cloud providers with granular, team-based access controls. You can create folders that live on AWS S3, Google Cloud Storage, or other providers from a simple interface and access them from the Lightning AI teamspace.

Data connections

Data connections let you bring data from other providers such as AWS, GCP, Cloudflare, Snowflake, Databricks, etc...

Teamspace Drive

Lightning Drive is a unified cloud filesystem shared by all users in a Teamspace. It allows every type of workload - Studios, Jobs, Deployments, Pipelines, etc... - to share a single, unified filesystem that can be accessed by any workload.

Think of a teamspace Drive like a* permissioned* Google drive folder (or data lake) for any type of file (ie: unstructured object store).

High-performance training

To help you achieve the highest data speeds for high-performance training, Lightning AI mixes a few different technologies with enterprise-grade storage solutions, mixed with our data optimization tools that allow frontier-grade data loading speeds for massive 5,000+ GPU multi-node training.

Clusters

We can set up different types of filesystem solutions depending on your speed needs and budgets. Everything from a simple S3-backed cluster, to high-performance Lustre and more performant, enterprise-grade data solutions.

Artifacts

Models, checkpoints, logs, and other outputs generated during training or evaluation are automatically stored on the Teamspace Drive which can be accessed by anyone in the teamspace.

Artifacts are:

Files created by Jobs, Pipelines, logs, etc...
Automatically saved during Jobs, Pipelines, and Studio workflows
Stored in Drive or cloud folders depending on the environment
Versioned to support experiment tracking and reproducibility
Usable across other components - such as for evaluation, deployment, or fine-tuning

Artifacts can be manually uploaded, downloaded, or referenced in future workflows without needing to reconfigure paths or re-run previous jobs.

Model weights

Lightning AI has a native storage-solution for model weights which can be individually versioned, shared and more.

Storing models on Lightning AI allow you to reach the highest performance and reliability for applications like inference.

Bring your own cloud

Some organizations require full control over where data is stored and how it's accessed. Lightning AI supports Bring Your Own Cloud (BYOC) deployments to meet these needs, allowing data to stay within a customer’s own cloud environment.

With BYOC:

Data never leaves the organization’s cloud account
Teams can comply with internal policies, security standards, or regional regulations
Lightning AI runs in the customer’s infrastructure, while still providing the same Studio, Job, and Deployment experience

BYOC is only supported for enterprise customers. Please contact us to set this up.

Storage speed vs cost

Lightning Drive offers different storage options. There is always a trade-off between cost and speed. Lightning's storage options are backed by the respective cloud provider. Here's how to think about the trade-offs.

Lightning storage option	AWS Equivalent	GCP Equivalent	Speed	Cost	Best For
High-performance folders	EFS	Filestore	⭐️⭐️⭐️	$$$	Training, jobs, finetuning
High-performance volumes	EFS	Filestore	⭐️⭐️⭐️	$$$	Training and jobs on one Lightning Cloud location
Cloud folders	S3	GCS	⭐️	$	Archive, budget workloads
Optimized storage	S3 + LitData	GCS + LitData	⭐️⭐️⭐️	$$	Balanced speed + cost
Fastest storage	EFS + LitData	FileStore + LitData	⭐️⭐️⭐️⭐️	$$$
Studio disks	—	—	⭐️⭐️⭐️	$	Temporary prep, exploration

Best practices

These are best practices to speed up data loading/movement on Lightning AI.

Optimize with LitData

AI training often stalls not due to compute limits, but because data is slow to load. This is especially true when reading from cloud storage like S3 or GCS, or when working with millions of small files. LitData solves this by converting datasets into an optimized format optimized for streaming and high-throughput training.

LitData compresses and chunks raw files into sharded blocks that are fast to read and portable across environments. The optimized datasets can be stored in:

S3 or GCS for low-cost, high-speed access
EFS or Filestore for maximum performance
Training or jobs are slow due to object store latency
Datasets contain thousands or millions of small files
Faster performance is needed without switching off S3 or GCS

Once a dataset is converted with LitData, it can be used in any Studio, Job, or Deployment - no code changes required.

Store checkpoints efficiently

Model checkpoints - especially from libraries like Hugging Face Transformers - can grow large and consume significant disk space. These files are often cached locally in Studios, which can lead to slow startup and shutdown times or impact job performance.

To avoid this:

Store checkpoints in the Lightning model registry instead of keeping them on the Studio
Use the registry to load or download checkpoints as needed, without bloating local storage

This keeps Studios lightweight and ensures faster start/stop times and faster Job start times.

Use studios to prep data

Data preparation is often fastest when done interactively. Lightning Studios provide a persistent, cloud-based environment for downloading, inspecting, cleaning, and transforming datasets before training.

Each Studio includes terabytes of persistent storage and full access to Drive and cloud folders. Processed datasets can either be left on the Studio or moved to shared storage, depending on performance and team access needs.

Use a Studio to prepare data interactively
Move the final dataset to a shared cloud folder (S3, GCS, EFS, or Filestore) for faster access across Studios and Jobs

Leaving large datasets on a Studio is convenient, but comes with trade-offs:

Startup/shutdown times grow with data size. A Studio with 500GB of files may take minutes to boot. If this becomes an issue, move datasets to a cloud folder (S3, GCS, EFS, or Filestore).
Jobs started from a Studio copy all files from that Studio. If your Studio has 500GB of data, your job will also start slower.

Best practice: Prep data in a Studio, then move it to a shared cloud folder to avoid slowdowns and enable reuse across the Teamspace.

Data performance trade-offs

The biggest data loading bottlenecks aren't just about total size - they’re about file count and format.

Example	Performance
5GB with 400M tiny files	🐢 Extremely slow (even though small)
500GB with large files	⚡ Faster (because fewer files)
2TB with 1-2MB files	🐢 Still slow unless optimized

Solution: Use LitData to compress and chunk the data. It won’t help in every case, but when file count is high or dataset is large, the speedups can be massive.

FAQs & best practices

When should I use LitData?

When your dataset has millions of files or training is slow.

Should I keep data on a Studio?

Yes for prep and exploration. No for long-term storage, move to cloud folders.

What's faster: EFS/Filestore or S3/GCS?

EFS/Filestore can be 100x faster or more, but also more expensive.

Can I bring my existing S3/GCS buckets?

Yes. Use Lightning data connections for zero-setup integration.

What’s the best setup for training?

Use LitData + EFS (or Filestore) for the fastest performance.

Was this page helpful?