Features

Use these features to optimize datasets for fast AI model training at scale.

Work with large cloud datasets

• Stream large cloud datasets: Avoid the overhead of downloading large datasets locally by streaming data directly from the cloud, allowing you to work with data immediately.
• Streams on multi-GPU, multi-node: Automatically scale data streaming across multiple GPUs and nodes, ensuring efficient data handling during distributed model training.
• Stream from multiple cloud providers: Easily access and stream data from AWS, GCS, Azure, or other cloud storage systems, giving you flexibility in where and how you store your data.
• Pause, resume data streaming: Seamlessly pause and resume data streams, enabling long-running training tasks to recover from interruptions without restarting.
• Load a subset of the remote dataset: Work with smaller, relevant sections of large datasets, reducing load times and resource consumption during development and experimentation
• Modify optimized cloud datasets: Append new data or overwrite existing datasets in the cloud without reoptimizing the entire dataset, allowing you to keep your data up-to-date
• Access samples without full data download: Directly retrieve specific samples from a cloud-hosted dataset for analysis or debugging without needing to download the entire dataset.

Save disk space

• Limit local cache space: Control how much disk space is used for caching temporary data, preventing storage issues during large data operations.
• Change cache directory path: Customize where cached data is stored, allowing you to optimize for performance or storage availability based on your environment.

Speed up model training

• LLM Pre-training: Preprocess and tokenize large language model datasets quickly and efficiently, allowing faster model iteration and experimentation.
• Combine datasets: Merge datasets from multiple sources into a single stream for richer, more diverse training data, improving model robustness
• Split datasets for train, val, test: Automatically partition datasets for training, validation, and testing, eliminating manual splits and ensuring consistent dataset management.
• Use compression: Reduce dataset sizes with built-in compression algorithms while maintaining fast loading speeds, saving both storage space and cloud costs.
• Profile data loading speed: Profile the data loading process and optimize bottlenecks, ensuring maximum throughput during model training, especially for large datasets
• Optimize dataset in distributed environment: Scale dataset optimization across multiple machines, cutting down processing time dramatically, particularly in high-performance or enterprise settings.

Speed up data processing

• Parallelize data transformations (map): Run data transformations in parallel across multiple machines, dramatically reducing processing times for large datasets.
• Support S3-Compatible cloud object storage: Integrate with S3-compatible storage solutions for cost-effective and scalable data management, making it easier to use on-premise or hybrid setups.
• Reduce memory use for large files: Efficiently handle large files by reducing memory overhead during processing, ensuring smooth operation even with limited system resources.

Data security

• Supports encryption and decryption of data at chunk/sample level: Encrypt datasets at the chunk or sample level, ensuring that sensitive data remains secure during storage and processing, critical for compliance and security-focused applications.