Neural Machine Translation

The problem

Machine translation, on its own, translates each sentence in isolation. It doesn't know the company has translated a near-identical clause a hundred times before in a specific way, for a specific regulatory reason — or that a product name has exactly one approved Chinese rendering and every other one is wrong. Point a generic translator at a new document and the output is fluent but inconsistent with everything already published.

The brief: build a pipeline that pulls from the company's own translation memory (TMX) and glossaries before generating anything, so a document translated today reads consistently with one translated a year ago — not like it went through a different translator each time.

Retrieval before translation

This is a RAG pipeline where the thing being retrieved isn't a knowledge-base passage, it's a prior translation decision. Four retrieval strategies run per chunk, each covering a gap the others miss, before the translation engine (Google NMT or an LLM, depending on which the accuracy assessment favors for that content type) ever sees the text.

Four strategies, one precedence rule

Fuzzy TM match

Top-n candidates from the translation memory, ranked by similarity to the source sentence — reuses a past translation that's close but not identical.

NER exact match

Named entities — product names, regulatory terms — matched exactly rather than fuzzily. A slightly-off phrase is recoverable; a wrong entity name isn't.

Longest-common-substring tie-break

When a span falls between two glossary terms, the candidate sharing the longest overlapping text with the source wins — keeps phrasing consistent with what's already used nearby.

Full glossary term-pair retrieval

Every glossary pair present anywhere in the source text is pulled in, not just the single best match, so the model sees all the relevant terminology at once.

✓ TMX match found

Translation memory wins

An existing, previously-used translation unit takes precedence over any glossary match covering the same span.

— no TMX match

Glossary fills the gap

Glossary term pairs only get used where the translation memory has nothing to offer.

Difflib in practice

Both steps use Python's difflib.SequenceMatcher: ratio() for the fuzzy top-n ranking, find_longest_match() for the tie-break.

# top-n fuzzy match against the translation memory
scores = [
    (tu, difflib.SequenceMatcher(None, source_sentence, tu.source).ratio())
    for tu in translation_memory
]
top_n = sorted(scores, key=lambda x: x[1], reverse=True)[:5]

# tie-break between two glossary candidates by longest overlapping span
match = difflib.SequenceMatcher(None, source_sentence, candidate.source)
overlap = match.find_longest_match(0, len(source_sentence), 0, len(candidate.source))
    

Chunking a .docx without touching its structure

Each chunk is one paragraph's full text — enough context for the model to produce coherent English→Chinese grammar, since word order shifts across the two languages in ways that per-run fragments can't capture. The write-back stays inside that paragraph's own runs, so the document's XML — styles, tables, headers — is never rebuilt, only the text inside existing runs changes.

# chunk = one paragraph's full text, across body and table cells
for para in iter_all_paragraphs(doc):
    source_text = "".join(run.text for run in para.runs)
    if not source_text.strip():
        continue

    context = retrieve_context(source_text)   # TMX + glossary hits, above
    translated = translate(source_text, context=context)

    para.runs[0].text = translated
    for run in para.runs[1:]:
        run.text = ""              # keep the run object, clear its text

doc.save(output_path)
    

Trade-off: collapsing a paragraph's runs into one keeps the document's structure and paragraph-level styling intact, but character-level formatting inside a sentence — one bold word mid-clause, say — isn't preserved. The whole paragraph inherits the first run's style.

The flow, end to end

Load docx → chunk per paragraph → retrieve context → translate → write back into the original runs → repeat to the end of the document.

Load .docx

python-docx opens the file; paragraphs across the body and table cells are enumerated in document order.

→

Chunk per paragraph

Each paragraph's full text becomes one translation unit — enough context for coherent grammar.

→

Retrieve context

Fuzzy TM match, NER exact match, glossary LCS tie-break, and full term-pair retrieval — TMX wins on overlap.

→

Translate

The chosen engine translates the chunk with the retrieved context injected alongside it.

→

Write back

Translated text goes into the paragraph's first run; the rest are cleared, not removed.

→

Repeat to end

Every paragraph draws on the same shared TMX/glossary store, so terminology stays consistent across even a very long document — without chaining context between chunks.

Evaluating translation quality

Building the retrieval pipeline was one problem; proving which translation approach actually performs best for this document type was another. Alongside it, I helped build an evaluation framework that scores candidate translation engines head-to-head on a held-out set of real documents, across five metrics that each catch something the others miss.

BLEU score

N-gram overlap vs. human reference

LLM-as-a-judge

FluencyAccuracy

Glossary compliance

Automated term extractionManual verification

Terminology adherence, by paragraph

Required terms actually used / expected

Translation memory evaluation

Similarity & coverageQualitative error analysis

LLM-as-a-judge, roughly

The judge gets the original, the human reference, and the candidate translation side by side, and scores the candidate on accuracy and fluency independently — deliberately not just "does it match the reference," since a translation can be faithful and fluent while still wording things differently than the reference did.

# illustrative — not the exact prompt in use
SYSTEM_PROMPT = """
You are a bilingual medical-translation reviewer (English -> Chinese).
Score the CANDIDATE against the ORIGINAL and REFERENCE, 0-100 on:

  accuracy — meaning preserved, no facts added, dropped, or altered
  fluency  — reads as natural professional Chinese, not a literal
             word-for-word rendering

A candidate that differs from the reference in wording but is still
accurate and fluent should score well — don't penalize for that alone.

Return only JSON: {"accuracy": <int>, "fluency": <int>, "notes": <str>}
"""
    

Ranking with the same Bradley-Terry approach

Rather than average scores across documents, each pair of translation methods is compared head-to-head on the same document, per metric — the same Bradley-Terry ranking approach used for LLM benchmarking, reused here to turn per-document scores into a single relative ranking per metric.

Reading the ranking: a win score is only meaningful within its own metric — a higher score means a method beats others more consistently on that metric, but scores aren't comparable across metrics (BLEU vs. fluency, say), since each has its own scoring scale and variance.

Where it's headed

This isn't a one-time comparison — as new translation approaches and newer LLMs show up, the plan is to re-run the same evaluation framework against them, so a model's fit for this use case stays a question that gets re-asked, not one answered once and left to age.

From POC to production

The retrieval pipeline and evaluation above settled which approach to use — turning that into something other systems can rely on was a separate step. Once the POC proved out, it handed off to MSD's Central AI team, who own productionizing it: hosting, scaling, and exposing it as a shared API rather than something every consuming app has to embed on its own.

POC validated

The retrieval pipeline and evaluation framework above prove the approach out on real documents.

→

Handoff to Central AI

Productionizing — hosting, scaling, versioning — becomes the Central AI team's responsibility, not ours.

→

Exposed as an API

A shared translation endpoint any internal app can call, instead of each team embedding its own copy of the pipeline.

→

Called directly from our app

Our app, linked to the AE intake CRM (Veeva), calls the endpoint directly to translate case narratives as part of intake.

Why hand it off: a translation API used by more than one team is infrastructure, not a project — centrally owning it means one team versions and maintains it consistently for every consumer, instead of each app carrying its own copy of the pipeline.

Extending the proof of concept

A self-evolving translation memory. Exploring automatically updating TMX entries from post-translation edits, so a human correction today improves retrieval for every document translated after it — instead of the same mistake recurring.
RLHF into the pipeline. Investigating how reinforcement learning from human feedback fits into the translation step itself, including whether DPO is a feasible route given the volume of edit data available.
Open-source models as an alternative engine. Assessing feasibility and accuracy of models such as TranslateGemma, with RLHF integration, against the Google NMT / LLM comparison already done.

Stack

RAG-based retrieval difflib (fuzzy + LCS) python-docx TMX / glossary matching LangGraph Google NMT & LLM comparison RLHF / DPO (in progress)