The Complete Production SOP: How to Deploy and Run Synthetic Training Data & Instruction Generator in Autonomous AI Agents

---
name: "synthetic-data-generation-pipeline"
description: "Generate diverse, high-quality instruction tuning datasets using frontier LLMs."
metadata:
  author: "Stanford CRFM"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "generate synthetic instruction dataset"
  - "evolve prompts self-instruct"
  - "filter low quality synthetic data"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# Synthetic Training Data & Instruction Generator

Builds synthetic data pipelines (Self-Instruct, Evol-Instruct): generating complex instructions, applying constraint mutations, filtering low-quality samples, and formatting JSONL datasets.

## Execution SOP & Rules
1. Seed the generation pipeline with 100+ diverse, human-verified example tasks.
2. Apply prompt mutations: increase reasoning depth, add domain constraints, or require specific schemas.
3. Filter generated pairs: remove near-duplicate items with sentence embedding cosine similarity > 0.85.
4. Validate code outputs by running automated unit test execution before accepting samples.

from pydantic import BaseModel, Field
from typing import List, Optional
import json

class InstructionSample(BaseModel):
    instruction: str = Field(..., min_length=20)
    input_context: Optional[str] = None
    expected_output: str = Field(..., min_length=10)
    mutation_type: str
    complexity_score: int = Field(ge=1, le=5)

def format_jsonl_record(sample: InstructionSample) -> str:
    return json.dumps({
        "messages": [
            {"role": "system", "content": "You are an expert autonomous assistant."},
            {"role": "user", "content": f"{sample.instruction}\n{sample.input_context or ''}".strip()},
            {"role": "assistant", "content": sample.expected_output}
        ]
    })

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')

def deduplicate_instructions(instructions: list[str], threshold: float = 0.85) -> list[str]:
    embeddings = model.encode(instructions, normalize_embeddings=True)
    unique_indices = []
    for i, emb in enumerate(embeddings):
        if not unique_indices:
            unique_indices.append(i)
            continue
        similarities = np.dot(embeddings[unique_indices], emb)
        if np.max(similarities) < threshold:
            unique_indices.append(i)
    return [instructions[idx] for idx in unique_indices]

import json
import pytest
from pathlib import Path

def test_train_jsonl_schema():
    dataset_path = Path('data/train.jsonl')
    assert dataset_path.exists(), 'train.jsonl does not exist'
    with open(dataset_path, 'r', encoding='utf-8') as f:
        for line_num, line in enumerate(f, 1):
            record = json.loads(line)
            assert 'messages' in record, f'Line {line_num} missing messages key'
            assert len(record['messages']) == 3, f'Line {line_num} invalid message count'