General-purpose large language models (LLMs) are increasingly competitive with dedicated machine translation services. A September 2026 benchmark published on GitHub offers an interesting example: GPT-6 Luna, Gemma 4, and DeepSeek V4 Flash all achieved lower translation error scores than DeepL API and Google Cloud Translation Basic.
GPT-6 Luna ranked first, followed by several other general-purpose language models. The results suggest that models developed for a broad range of tasks can also perform exceptionally well at translation, particularly when conversational context matters.
There is one important distinction: the benchmark evaluated Google Cloud Translation Basic, not the consumer-facing Google Translate application, which has incorporated additional Gemini-powered translation capabilities.
What Did the Benchmark Test?
The Korean Multi-turn Context Translation Benchmark compared 15 translation systems on their ability to translate conversational Korean into English, Japanese, and Simplified Chinese.
The dataset contained 216 Korean conversational examples, producing up to 648 translations per system.
The tests focused on aspects of language that can be difficult to translate without context:
- Omitted information: Korean frequently leaves out subjects and objects that are understood from previous sentences.
- Ambiguous references: Words and expressions may refer to people or objects mentioned earlier.
- Tone and politeness: Translations need to account for formality and relationships between speakers.
- Conversational intent: Short replies, implied meanings, and incomplete sentences often depend on preceding messages.
- Irrelevant context: A translation system must also recognize when earlier information should be ignored.
Translation quality was evaluated using a methodology based on GEMBA-MQM. An automated AI evaluator identified errors and assigned penalties according to their severity.
A lower mean penalty indicates fewer or less severe errors under this evaluation.
Translation Benchmark Results
The September 2026 comparison produced the following results for selected text-based systems:
| Translation system | Mean error penalty |
|---|---|
| GPT-6 Luna | 0.130 |
| Gemma 4 31B | 0.353 |
| Gemma 4 26B | 0.387 |
| DeepSeek V4 Flash | 0.571 |
| Gemma 4 12B QAT Q4 | 0.855 |
| Gemma 4 E4B fp16 | 1.353 |
| Papago Web | 2.699 |
| DeepL API | 3.914 |
| Google Cloud Translation Basic | 5.731 |
Source: 15-System Integrated Comparison, September 2026. Lower is better. DeepL had 642 valid evaluations; the other listed systems had 648. The full comparison also included speech translation systems, omitted here for consistency.
GPT-6 Luna had the lowest mean error penalty at 0.130. Gemma 4 31B and Gemma 4 26B followed with scores of 0.353 and 0.387, while DeepSeek V4 Flash scored 0.571.
The commercial translation services performed less well on this particular dataset. Papago scored 2.699, DeepL API scored 3.914, and Google Cloud Translation Basic scored 5.731.
One notable finding is that several relatively small models also performed well. Gemma 4 12B QAT Q4, for example, achieved a mean penalty of 0.855.
These results are interesting because GPT-6 Luna, Gemma, and DeepSeek are general-purpose language models rather than dedicated translation products. They are designed to handle many different tasks, including writing, reasoning, coding, and analysis.
However, the comparison is not entirely like-for-like. The LLM configurations could use previous conversation turns, while the commercial translation services were evaluated without that history. The results therefore reflect the performance of the tested translation workflows, not simply the underlying model architectures.
Why Context Matters for Translation
Translating individual sentences accurately does not always guarantee that their intended meaning is preserved.
For example, imagine a Korean conversation in which someone mentions buying sandals and slippers at a store. In the following sentence, they explain that the items were out of stock and had to be delivered.
The second sentence may omit the objects being discussed because they are already understood from the conversation.
A system translating only that sentence may produce an ambiguous result. A model with access to the earlier message can identify the missing references.
Similar challenges appear in many languages.
Idioms and informal expressions may require interpretation rather than literal translation. Politeness and formality can depend on the relationship between speakers. A short reply may have different meanings depending on the preceding question.
LLMs can use surrounding information to resolve these ambiguities. They can also follow instructions about terminology, writing style, and intended audience.
This flexibility is one reason general-purpose language models have become increasingly useful for translation.
Adding Context and Instructions Reduced Errors by 31.5%
The benchmark included a separate experiment examining how additional information affects translation quality.
Researchers tested Gemma 4 E4B QAT Q4 under two conditions: translating a sentence on its own, and translating it with additional instructions and the full conversation history.
| Configuration | Mean error penalty |
|---|---|
| Sentence only | 2.118 |
| Instructions + conversation history | 1.452 |
The second configuration achieved a 31.5% lower mean error penalty.
Across 642 paired evaluations, the context-and-instructions configuration improved 236 translations, worsened 134, and produced equivalent scores for 272.
The improvement was observed both when earlier context was needed and when the model had to recognize that previous information was irrelevant.
An important limitation is that the experiment changed both the instructions and the conversation history. It therefore measures their combined effect rather than isolating the benefit of context alone.
Still, the results show that translation quality can vary considerably even when the underlying AI model remains unchanged.
Read the full context experiment.
Does This Mean LLMs Are Always Better at Translation?
No. The benchmark focused on a specific type of translation: Korean conversations requiring contextual understanding.
Its findings do not necessarily extend to every language pair, subject, or document type.
The evaluation also used an automated AI judge rather than professional human translators. Although this makes large-scale comparisons practical, automated judgments can introduce biases and may not capture every aspect of translation quality.
Another consideration is that commercial translation services are evolving.
DeepL has developed its own translation-specialized LLMs, while Google has introduced Gemini-powered improvements to Google Translate. Google Cloud also offers dedicated translation models beyond the Basic API tested in this benchmark.
In other words, the distinction between general-purpose LLMs and specialized translation services is becoming less clear.
Different models may perform better on different language pairs, technical subjects, or translation tasks. Accuracy is also only one consideration: speed, cost, and consistency can be equally important depending on the use case.
Rather than identifying one universally superior translation system, the benchmark illustrates how competitive general-purpose LLMs have become.
What This Means for AI Translation Services
As general-purpose models improve, translation services have more options than relying exclusively on a single proprietary translation engine.
A service can instead provide access to models from multiple developers, selecting or recommending different options depending on the task.
This is the approach we take at Lexibird.
Lexibird provides access to a range of language models from different AI providers. Users can select a model themselves or use an automatic option for everyday translation.
We regularly evaluate available models with three considerations in mind:
- Accuracy: How well a model preserves meaning, context, tone, and terminology.
- Speed: How quickly it produces translations, particularly for everyday tasks.
- Cost: Whether the improvement in quality justifies the additional processing expense.
There are trade-offs between these factors. A more capable model may produce better results on difficult text but respond more slowly or cost more to run. A smaller model may be sufficient for straightforward translations.
Our goal is to keep improving the available options as new models emerge, rather than assuming that one model will remain the best choice indefinitely.
Lexibird also supports translation glossaries, tone customization, and translation of documents, images, and audio. These features help users adapt translations to particular requirements.
Lexibird itself was not evaluated in the Korean benchmark, so the published scores should not be interpreted as measurements of Lexibird's translation performance.
The Best Translation Model May Not Stay the Best for Long
The most interesting finding from this benchmark is not necessarily GPT-6 Luna's first-place result. It is the strong performance of several general-purpose language models against dedicated translation services.
The results illustrate how much machine translation has changed. Models developed for general language understanding can now handle translation tasks that require interpreting context, recovering omitted information, and preserving conversational intent.
At the same time, the relative performance of models can change quickly. New releases may improve translation quality, reduce response times, or lower costs.
For translation services and their users, this suggests that flexibility may be more valuable than committing to one translation model.
The benchmark offers a useful snapshot of the state of AI translation in 2026, but the broader competition between general-purpose and specialized translation models is far from settled.
