Synthetic Data for LLM Training: When AI Teaches AI

Synthetic data—data generated by AI models—has become essential for training state-of-the-art language models, raising questions about quality, diversity, and the risks of model collapse.

Why Synthetic Data?

Human-generated data is:

Expensive: Expert annotation costs $$$
Limited: Finite high-quality sources
Biased: Reflects human limitations
Private: Many valuable datasets can't be shared

Synthetic data addresses these:

Scalable: Generate arbitrary amounts
Controllable: Specify exact properties
Diverse: Cover rare scenarios
Safe: No privacy concerns

Techniques

1. Self-Instruct

Generate instruction-following data from a seed set:

def self_instruct(model, seed_tasks, n_generate=10000):
    tasks = seed_tasks.copy()
    
    for _ in range(n_generate):
        # Sample diverse seed tasks
        examples = random.sample(tasks, k=5)
        
        # Prompt model to generate new task
        prompt = format_generation_prompt(examples)
        new_task = model.generate(prompt)
        
        # Filter low quality
        if quality_check(new_task):
            tasks.append(new_task)
    
    return tasks

2. Evol-Instruct (WizardLM)

Evolve instructions to be more complex:

EVOLUTION_PROMPTS = {
    "add_constraints": "Add more constraints to this task: {task}",
    "deepen": "Make this task require deeper reasoning: {task}",
    "concretize": "Make this task more specific and concrete: {task}",
    "increase_steps": "Require more reasoning steps: {task}",
}

def evol_instruct(model, task, n_evolutions=3):
    evolved = task
    for _ in range(n_evolutions):
        evolution_type = random.choice(list(EVOLUTION_PROMPTS.keys()))
        prompt = EVOLUTION_PROMPTS[evolution_type].format(task=evolved)
        evolved = model.generate(prompt)
    return evolved

3. Rejection Sampling

Generate many, keep the best:

def rejection_sampling(model, prompt, reward_model, n_samples=64, k=4):
    # Generate many responses
    responses = [model.generate(prompt) for _ in range(n_samples)]
    
    # Score each
    scores = [reward_model.score(prompt, r) for r in responses]
    
    # Keep top-k
    top_indices = np.argsort(scores)[-k:]
    return [responses[i] for i in top_indices]

4. Distillation

Transfer knowledge from large to small model:

def distill_dataset(teacher, prompts, temperature=1.0):
    dataset = []
    
    for prompt in prompts:
        # Get teacher's response
        response = teacher.generate(prompt, temperature=temperature)
        
        # Get teacher's token probabilities for soft labels
        soft_labels = teacher.get_probs(prompt + response)
        
        dataset.append({
            "prompt": prompt,
            "response": response,
            "soft_labels": soft_labels
        })
    
    return dataset

5. Constitutional AI Data

Generate via self-critique and revision:

def constitutional_generation(model, prompt, principles):
    # Initial response
    response = model.generate(prompt)
    
    for principle in principles:
        # Critique
        critique = model.generate(
            f"Does this response follow '{principle}'? Response: {response}"
        )
        
        # Revise if needed
        if "no" in critique.lower():
            response = model.generate(
                f"Revise to follow '{principle}': {response}"
            )
    
    return response

Quality Control

Diversity Metrics

def measure_diversity(dataset):
    embeddings = embed_all(dataset)
    
    return {
        "self_bleu": compute_self_bleu(dataset),
        "embedding_variance": embeddings.var(dim=0).mean(),
        "unique_ngrams": count_unique_ngrams(dataset),
        "topic_coverage": measure_topic_coverage(dataset)
    }

Contamination Detection

def check_contamination(synthetic_data, test_sets):
    """Check if synthetic data memorizes test data"""
    for test_set in test_sets:
        for test_item in test_set:
            for synth_item in synthetic_data:
                if similarity(test_item, synth_item) > threshold:
                    print(f"Potential contamination: {test_item}")

Model Collapse

The risk: Training on model-generated data loses diversity over generations.

Generation 1: Full distribution
Generation 2: Training on Gen1 output → Narrower distribution
Generation 3: Training on Gen2 output → Even narrower
...
Generation N: Collapsed to single mode

Mitigations:

Mix with real data: Always include human data
Diverse generation: High temperature, diverse prompts
Quality filtering: Remove repetitive/low-quality samples
Fresh models: Don't only train on own outputs

Case Studies

Phi Models (Microsoft)

Trained primarily on synthetic "textbook-quality" data
Small models with outsized capabilities
Demonstrates synthetic data can improve efficiency

Orca

Distilled from GPT-4 explanations
Focused on reasoning traces
Smaller model learns how to think

WizardCoder

Evol-Instruct applied to code
Generated complex programming problems
Improved coding benchmarks significantly

Best Practices

Do	Don't
Diversify sources and prompts	Use single model/prompt
Filter aggressively	Keep all outputs
Validate against held-out data	Train blindly
Mix with real data	Go fully synthetic
Monitor for collapse	Iterate blindly

References

Synthetic data is not a replacement for human data—it's an amplifier that lets us extract more value from the human data we have.