Finetune models

Lightning AI offers various ways of finetuning models.

Low code AI: AI Hub

The Lightning AI Hub enables individuals and organizations to easily find and run prebuilt APIs and AI services with minimal setup, offering low-code solutions that are ready to deploy on the cloud or private VPCs. See below for a few examples, and check out more here.

Full code AI

Lightning AI also offers Studio templates for a full code experience. Build and share fully reproducible environments preloaded with everything needed for AI projects like finetuning. Templates have already installed dependencies, model weights, data, code and more. Pick a GPU and instantly customize from your browser or any local IDE with zero setup. See below for a few examples, and check out more here.

About Finetuning

Use finetuning to improve the quality of a model's outputs by specializing on data it did not have during training. If you're looking to improve model performance then you're likely looking for finetuning!

Lightning AI offers various finetuning options for everything from LLMs to computer vision models.

What is finetuning

Finetuning is the process of specializing a base model to your data. Finetuning allows you to take a model that was trained for a task or a large dataset and transfer that knowledge to a different task or dataset.

For example, LLMs are typically trained on almost all the data found on the internet. This makes the LLM a strong foundation on which to build specialized models. These foundation models encode good high-level representations of the data it was trained on. However, it may not have strong, specialized knowledge about a particular domain (like finance), or you may need to get the model to behave in a specialized way.

The process of specializing the foundation model is called finetuning.

Without a strong base model, you would need to train a model from scratch (called pretraining), which often requires more data and more compute.

Compute requirements

Counter to popular belief, finetuning may not require huge compute resources. Many factors play into the compute requirements needed, such as model type, application domain, amount of data and strength of the foundation model. However, the main two factors are size of model and size of data to finetune on.

This plot summarizes at a high-level the cost, compute and time requirement estimates depending on model and data size.

Time requirements

Fundamentally, finetuning is about how many parameters to change in a model to achieve the desired specialization. On one extreme you can change zero parameters and simply change the input to bias it closer to the desired specialization, in LLMs this is called prompt engineering. On another extreme, you can tweak every parameter in the model; this is called pretraining. Finetuning typically falls somewhere in between these two extremes.

Popular techniques like LoRA, freezing/unfreezing certain layers, and so on are all concerned with which parameters you change and when. Finetuning is an active area of research, and new techniques come out weekly because it is an open and active area of research. We recommend experimenting with different techniques that appropriately match the data and task you are working on.

Finetune vision models

Many techniques exist around finetuning compute vision models. Most techniques boil down to using a strong pretrained model and layering in another model that is finetuned on top of it. The pretrained model will generate a low-dimensional representation of the input, and the final layers will turn that into an output, such as a label corresponding to content of the image.

Finetune LLMs

Large language models (LLMs), have seen a rise in popularity in the past few years largely due to their ability to transfer knowledge well via finetuning. The research community has introduced many ways of finetuning LLMs. Many of these are experimental because this is an active area of research.

Lightning AI allows developers to run arbitrary code on cloud GPUs. This means that any LLM finetuning technique is supported (including ones that don't yet exist), because ultimately they all just run a model under a specific setup that decides what parameters to update or not.

However, the Lightning AI community has created Studio templates for a lot of the latest techniques that require no manual implementation from a user. You can either use one of these templates or bring in your own code that finetunes LLMs in your own particular, customized way.

Here are a few of these techniques

Prompt engineering

Perhaps the simplest, most popular way to get an LLM to perform a given task is to specify it as part of the prompt. In this extreme case, no model weights are changing. Instead, the knowledge (e.g. input-output pairs, or a textual description of the task) is provided as part of the input. LLMs, especially larger ones, will tend to follow instructions and examples given in the prompt through a phenomenon called in-context learning, which requires no updates in the weights.

In-context learning is often referred to as few-shot learning when one or more sets of input-output pairs are provided in the prompt. For instance, examples in the form

hello -> olleh
party -> ytarp
dance -> ecnad

will inform the model about the task to execute (reverse letters in the input).

Prompt-engineering tends to work better on larger models (parameters in the tens of billions). Context size, the amount of tokens (words or fragments of words) than can fit in the prompt, is also an important factor, as a shorter context will limit the amount of knowledge that can be specified.

Continued pretraining

Continued pretraining is the process of continuing to update a pretrained model using new data. Take for example an LLM that is trained on news articles: There is a knowledge cutoff at the date the data for training was collected. One could add all new articles to the dataset and retrain the model from scratch, but this is very expensive. Instead, continued pretraining allows us to continue training on the recent data without discarding the previously acquired knowledge.

Run continued pretraining here.

Instruction finetuning

LLMs are pretrained to predict the next word (not to answer questions). Instruction finetuning teaches the model to follow instructions (such as "write me a list from 1-10"), and answer questions.

>> Prompt: What do llamas eat?
>> Reply: Grass and chocolates.

Without instruction finetuning, a pretrained LLM will have answers like this:

>> Prompt: What do llamas eat?
>> Reply: What do llamas smell like? How do you get along with llamas? Are llamas good pets?
These are questions you might have if you ...

Learn more and run instruction finetuning in this guide.

Alignment to human preferences

While instruction finetuning enables a model to follow instructions and produce responses in line with the requests, LLMs may undergo a more complicated process called alignment. Alignment is the process of inducing a LLM to produce output that has specific qualities, like being ethical and safe, fluent, accurate, not verbose, etc.

All these qualities are hard to express in mathematical terms. To overcome this, the LLM is finetuned using a method called reinforcement learning with human feedback (RLHF). RLHF typically has two stages. In the first stage, a model called the reward model is trained to predict how a human would rank a set of sentences based a set of core values. In the second stage, the reward model is used to provide feedback to the LLM using reinforcement learning, so that the LLM produces outputs that align with the same core values.

A simpler alternative, that has quickly become the standard for aligning LLMs to human preferences, is Direct Preference Optimization (DPO). With DPO there is no need for training a separate reward model, and RLHF is reduced to a simpler classification task (preferred vs not). The resulting alignment process is more stable and less computationally expensive.

LoRA finetuning

LoRA, which stands for Low-Rank Adaptation, is a popular technique to finetune LLMs more efficiently. Instead of adjusting all the parameters of a deep neural network, LoRA focuses on updating only a small set of low-rank matrices. This results in much lower cost to finetune because fewer parameters need to be adjusted

Use this in-depth guide to finetune using LoRA.

LoRA can be used beyond language models. For instance, image generation models like Stable Diffusion, can be finetuned using LoRA. This allows to steer the model to generate images with given styles (cartoonish, vintage) or genre (minimalistic design, baroque).

Given the fact that the amount of weights finetuned on top of a base model is small, LoRA layers can be trained inexpensively and be saved in small files that are easy to exchange during inference. This has empowered the image generation community to train and distribute large amounts of LoRA layers with distinct creative styles.