Preparing data for supervised fine-tuning Part 2: Advanced data strategies
The advanced side of supervised fine-tuning data prep. This second post in a two-part series covers evaluating data readiness with learning curves, selecting high-value data subsets, augmenting data with synthetic and distilled examples, and mixing data sources to prevent catastrophic forgetting.
Once a dataset is ready and formatted for supervised fine-tuning, the next step is to analyze and optimize its quality. Assess whether the dataset contains enough high-quality samples to train effectively. For typical tasks, around 2,000 samples are a good starting point, but the optimal number depends on task difficulty and current model performance.
Consider factors like dataset size, data distribution, and the need for a clearly defined evaluation benchmark. Perform a learning curve analysis to empirically determine the ideal dataset size. Train a model on subsets of the data, evaluate checkpoints, and plot performance against data scale. Identify the saturation point where additional data yields minimal improvement.
If gains plateau, focus on improving data quality instead of simply collecting more samples. Data selection techniques such as DEITA, DELIFT, and coreset selection can help identify a smaller, high-quality subset that retains the model's capabilities while reducing training time and preventing catastrophic forgetting.
Written by urgent.news from AWS Machine Learning's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
