.png)
Dharma AI released a post on using Direct Preference Optimization (DPO) beyond chatbots, centered on its DharmaOCR structured OCR system. The key result is practical: a second training stage on top of SFT cut repetition-loop failures in every tested model family, with an average degeneration reduction of 59.4% and a best case of 87.6%.
The interesting move here is not “DPO for alignment” in the usual chatbot sense. It is DPO as a failure-mode suppressor for an objective extraction task.
DharmaOCR’s benchmark tracked text degeneration rate, meaning the model falls into a repetition loop instead of producing a transcription. That failure mode is especially ugly in production OCR because it can silently turn a bad sample into a much larger downstream incident: malformed JSON, stalled pipelines, or garbage that still looks syntactically plausible enough to survive shallow validation.
SFT helped, but not enough. Across the evaluated open-source families, vanilla degeneration rates ranged from below 1% to above 33%, and supervised fine-tuning reduced those rates without reliably getting them into production-safe territory. That is the core limitation: SFT rewards correct outputs, but it does not explicitly penalize degenerate continuations.
DPO gives the model an explicit preference signal. The chosen response is the correct transcription; the rejected response is the repetition loop. That is a clean fit for OCR because the task has a binary notion of success at the sequence level, even if the output surface is structured text rather than a single label.
The result suggests a useful pattern for production LLM systems: if a failure mode is recognizable, enumerable, and frequently observed in your own outputs, you can often turn it into preference data instead of trying to brute-force it away with more SFT.
That matters because a lot of real deployment pain comes from behaviors that are not “wrong answer” in the narrow supervised sense. Degeneration, verbosity drift, invalid formatting, refusal leakage, partial schema collapse, and other generation pathologies often survive SFT longer than expected. They are not always correlated with token-level loss in the way you would want.
DPO is attractive in these cases because it directly changes relative preferences between two completions from the same base model. In practice, that means you can mine rejected samples from model failures, pair them against successful outputs, and apply a second stage without needing a new annotation program for subjective human preference. For structured OCR, this is especially appealing because the preferred output is often mechanically derivable from the document itself.
The reported numbers also make the argument concrete. Every tested family improved after DPO, with no exceptions. The magnitude varied, but the direction did not. That consistency is more useful than a single benchmark win because it suggests the method is robust to model family and baseline quality.
Most published DPO work focuses on helpfulness and harmlessness because chat is where preference data is easiest to collect. But OCR shows that DPO is not inherently a chatbot technique. Any task with a recognizable “better vs. worse” completion pair can use the same machinery.
That extends beyond OCR. If a model’s bad behaviors are self-reproducible, you can often build rejection pairs from:
The main requirement is that the rejected example be meaningfully worse for the task, not just different. In OCR, a repetition loop is unambiguously worse than a faithful transcription, so the preference signal is straightforward.
That said, the paper’s result does not imply DPO is a universal replacement for SFT. It is a second-stage correction tool. SFT still establishes the base task behavior; DPO then sharpens the model away from a known failure mode. In a production training stack, that sequencing is the point.
For teams shipping structured generation systems, the actionable lesson is to inspect your own model failures as a data source. If the bad outputs are consistent enough to pair against good ones, DPO can turn those failures into a targeted alignment layer instead of treating them as an evaluation-only nuisance.
