Synthetic Data Generation for Financial AI Research with NVIDIA NeMo

Fine-tuning LLMs for financial natural language processing (NLP) is constrained by limited, imbalanced data. Real-world financial news overrepresents earnings and stock movements, while rarer events such as credit-rating changes, product approvals, and labor issues are harder to capture at scale. Synthetic generation can help fill those gaps for trading research, risk modeling, and surveillance, but diversity requires more than a single large generation run.

This post walks through an iterative pipeline for generating 500,000 unique financial news headlines across 12 topics plus an “Other” category. The workflow combines NVIDIA NeMo Data Designer for structured generation, NVIDIA NeMo Curator for semantic deduplication, and NVIDIA Nemotron models for high-throughput headline synthesis. It also adds a farthest-from-centroid few-shot strategy to steer each round toward novel outputs.

In a naive run of 50,000 headlines, 65% were removed as near-duplicates. Instead of scaling a single batch, the pipeline iterates: generate, filter, deduplicate globally, select distinctive few-shot examples, correct category weights, and repeat until the corpus reaches the target size. Our full run produced 502,536 unique headlines across 13 categories in 82 iterations, using approximately 6 days of compute on a single 8-way NVIDIA B200 node with checkpointing, crash recovery, and SLURM job chaining.

By the end, you’ll understand how to reproduce the generation-deduplication loop, adapt it for your own financial research workflows, or start directly from the open-sourced dataset.

Table 1. Software and model versions used in this guide

Hardware: Single 8-way NVIDIA B200 node. GPUs 0–3 dedicated to vLLM inference (4-way tensor parallelism, 448 concurrent requests); GPUs 4–7 ran NeMo Curator semantic deduplication with Ray.

Pipeline parameters: 35K headlines per batch (50K in early iterations), 90% cosine-similarity deduplication threshold, 500 K-means clusters, 3 few-shot examples per category, 80% cross-iteration similarity cutoff for example selection.

Checkpointing: Each iteration persisted corpus state, category weights, few-shot examples, metrics, and used-example embeddings for crash recovery and SLURM job chaining.

Financial headline generation fails as a single-pass problem because real-world feeds are imbalanced: earnings reports and stock movements dominate, while credit-rating changes, product approvals, and other rarer events are sparse. In a 50K-headline baseline, semantic deduplication retained only 17,348 unique headlines, showing that larger batches add many near-duplicates rather than proportionally more coverage.

The iterative pipeline addresses this with a closed loop. Each iteration generates a category-weighted batch, filters malformed outputs, deduplicates against the full accumulated corpus, selects diverse few-shot examples, and adjusts category weights before the next iteration.

Figure 1, below, illustrates the overall architecture.

Figure 1. Iterative headline generation pipeline. Each batch is generated, filtered, deduplicated against the accumulated corpus, sampled for few-shot examples, and reweighted by category before the next iteration

The key design decision is global deduplication. Each batch is compared with all previously retained headlines, not just with itself, preventing cross-batch duplicates and preserving semantic uniqueness across the 500K corpus.

Step 1. Generate headlines with NeMo Data Designer

We serve Nemotron 3 Nano (30B parameters, 3B active) via vLLM with 4-way tensor parallelism and 448 concurrent requests. The model’s Mixture-of-Experts (MoE) architecture balances throughput and quality, using 3B active parameters per forward pass within a 30B-parameter model.

NeMo Data Designer orchestrates generation through a declarative configuration: a weighted category sampler across 12 topics plus “Other,” and an LLM column that generates headlines conditioned on the sampled category and current few-shot examples.

Category weights are updated after each iteration through distribution correction (Step 5).

We use high-diversity sampling (temperature=0.95, frequency penalty 0.3, presence penalty 0.2) and rely on downstream filtering and semantic deduplication to remove malformed or redundant outputs.

The prompt includes a role-setting system prompt, few-shot examples grouped by category, and an instruction for the sampled category. Data Designer injects the category at runtime through Jinja templating, and the few-shot examples are refreshed each iteration (Step 4).

Step 2. Quality filtering

Before semantic deduplication, a lightweight rule-based pass removes malformed outputs such as truncated, runaway, or garbled headlines.

Table 2. Quality filters applied to generated headlines, with thresholds and rationale for each

In practice, these filters removed fewer than 1% of headlines per batch, so most quality control comes from semantic deduplication in the next step.

Step 3. Semantic deduplication with NeMo Curator

This is the core quality mechanism. After each batch, the pipeline combines newly generated headlines with the accumulated corpus and runs NeMo Curator’s TextSemanticDeduplicationWorkflow. Headlines are embedded with all-MiniLM-L6-v2, clustered with K-means, and compared within clusters using pairwise cosine similarity. Any headline above the 90% similarity threshold is removed.

Why 500 clusters?

Pairwise similarity is expensive because comparisons scale quadratically within each cluster. At the 500K-headline endpoint, 500 clusters keep the average cluster near ~1,000 headlines and total comparisons near ~500M. Using 13 clusters, one per category, would push total comparisons toward ~19B. The higher cluster count keeps global deduplication tractable as the corpus grows.

After each iteration, the pipeline plots an ECDF (Empirical Cumulative Distribution Function) of pairwise cosine similarities. The red line marks the 0.90 deduplication threshold: a steep rise near 1.0 signals many near-duplicates, while a smoother left-shifted curve indicates broader coverage.

Figure 2, below, overlays representative iterations 5, 20, 40, 60, and 82. Early curves skew right because the model repeats common patterns from limited few-shot examples. Later curves shift left and stabilize, showing that updated few-shot examples continue to push generation toward novel content even as the corpus approaches 500K headlines.

Figure 2. Combined ECDF of cosine similarity distributions across iterations 5, 20, 40, 60, and the final iteration 82

After deduplication, the pipeline selects few-shot examples for the next batch. The goal is to steer the model away from patterns it has already produced.

K-means clustering from the deduplication step gives each headline a cluster centroid. We rank headlines by cosine distance from that centroid and select farthest examples, which are atypical within their semantic neighborhood and useful for prompting novel outputs.

Distance alone is not enough. The pipeline also compares each candidate against all previously used few-shot examples and rejects any candidate with >=80% cosine similarity. This prevents semantically repeated prompt signals across iterations.

Table 3, below, summarizes the selection priority. Three examples are selected per category, for 39 total per iteration, and the used-example set plus embeddings are checkpointed for recovery.

2. Semantic filter

Table 3. Few-shot example selection priority

Table 4, below, shows how selected examples become more specific by the final iteration.

Google Parent Alphabet Posts $1.84 EPS, $63.5B Revenue, Exceeds Forecasts

Macy’s to Acquire Nordstrom’s Home Division in $1.8B Deal

Siemens Agrees to Acquire Mentor Graphics in $3.5B Deal, EU Demands Divestiture of Overlapping EDA Tools

Table 4. Few-shot examples from iteration 1 (early) and iteration 82 (final). Later examples feature more specific entities, niche financial instruments, and longer, more detailed structures, reflecting the pipeline’s drift toward frontier content.

Step 5. Dynamic distribution correction

Weighted sampling alone doesn’t keep categories on target because the model favors easier classes such as “Other” and “Stock Movement.” After each iteration, the pipeline compares target and actual category proportions, boosts underrepresented categories, clamps extremes, and normalizes weights for the next batch.

By iteration 82, rare categories such as “Credit Ratings” and “Product Approval” closely match their 1% targets, while “Other” remains overrepresented. The correction loop still reduces model bias enough to preserve coverage across all categories.

Figure 3. Target vs. actual category distribution at iteration 82 (502,536 headlines)

Pipeline run summary: The 500K target is slightly overshot because generation proceeds in fixed 35K batches and halts only after the threshold is crossed; the excess 2,536 examples are trimmed from the over-represented “Other” category.

Total generated (pre-deduplication): ~2.87M headlines

Semantic deduplication was the dominant filter, and its role shifted over the run. In iteration 1, seeded by only one manual example per category, the pipeline retained 17,348 of 50K headlines; the remaining 32,652 (65%) were near-duplicates. Iterations 2–3 used farthest-from-centroid examples from previous outputs and maintained higher yields of 15K–17K new headlines per 50K batch, though cross-batch collisions began rising as the corpus grew.

From iteration 4 onward, we reduced batch size from 50K to 35K once the deduplication drop rate exceeded ~80%. Larger SLURM jobs were mostly producing headlines discarded downstream, while smaller 35K batches delivered similar net-new yield with less wasted compute.

After the corpus passed ~100K headlines, each new headline had to be distinct from a much larger pool. By iteration 40+, yield stabilized at ~5K–6K new headlines per 35K batch, or roughly 15%, with about 30K discarded per batch. Table 4 shows the transition: early duplicates are mostly within-batch repetition, while later duplicates are mostly collisions with the accumulated corpus. The steady nonzero yield near 500K shows that evolving few-shot examples continued to steer generation into novel regions.

Table 5. Deduplication yield across representative iterations. Yield drops from ~35% (early, small corpus) to ~15% (late, large corpus) as cross-batch collisions dominate

The generated dataset supports the AI Model Distillation for Financial Data developer example, where a 49B–70B teacher model labels synthetic headlines and smaller 1B, 3B, and 8B student models are fine-tuned with LoRA. With 25K labeled examples, a 3B student reaches 95% of the teacher’s F1-score on this financial headline classification task.

Dataset diversity matters here: semantic uniqueness and category balance expose the student to rare financial events and edge cases that are difficult to collect from real-world sources.

Iterate, don’t batch. A single 500K generation pass produces mostly duplicates. Eighty-two smaller iterations with evolving prompts yield a far more diverse corpus.

Deduplicate globally, not locally. Each iteration must deduplicate against the full accumulated dataset to prevent cross-batch duplicates.

Few-shot examples are the steering wheel. Farthest-from-centroid selection with semantic similarity filtering is the most impactful lever for generation diversity.

Distribution correction is essential. Without dynamic weight adjustment, models over-generate popular categories and starve rare ones.

Use the dataset — Download FinHeadlineMix on Hugging Face and use it directly for headline classification, fine-tuning, or as training data for your own distillation pipeline.

Run the recipe — Reproduce or extend this pipeline using NeMo Data Designer for structured synthetic generation and NeMo Curator for scalable semantic deduplication.

Train a distilled model — Follow the AI Model Distillation for Financial Data developer example to fine-tune a compact student model on this dataset.