Multilingual Embeddings: Search Quality Must Be Checked Language by Language

One model does not create one uniform experience

A search service can accept questions in several languages while delivering very different levels of usefulness in each. The model may represent many languages, but the collection can vary in quality, terminology, and coverage. A query written in one language may retrieve a document in another, which can be helpful or frustrating depending on the user’s needs. Multilingual capability should therefore be evaluated as a set of user journeys rather than a single checkbox beside a model name.

The multilingual E5 model card is a primary example of documentation that describes supported usage, training context, and limitations. pgvector provides the database-side mechanisms for storing and searching vectors, independently of the language quality of the embeddings. These references illustrate two different layers of the system. A faster index does not repair a representation that misses a crucial term, and a multilingual model does not decide which source language the reader can comfortably use. The product has to connect those layers deliberately.

Define same-language and cross-language goals separately

A hypothetical international support portal may have English manuals, translated quick-start guides, and local-language troubleshooting notes. A person searching in Spanish might prefer a Spanish result when one exists, but accept an English manual for an advanced issue. Another person may need results only in the language they can read. These are different retrieval objectives. The system should know whether it is finding semantically relevant material regardless of language or selecting the best usable material for a particular audience.

Test both same-language and cross-language cases. A query and document in the same language can reveal terminology and coverage problems. A query in one language and a relevant document in another tests the cross-language relationship. Keep the judgments distinct so a strong result in one category does not conceal weakness in the other. Include language preference in the interface where it helps the user. Silent cross-language retrieval can look like a search failure when the result is relevant but unreadable to the person asking.

Identifiers and domain terms deserve their own tests

Technical search often depends on product codes, model numbers, abbreviations, and terms that do not translate cleanly. An embedding can capture broad meaning while overlooking a small identifier that changes the answer. A manual for one device generation may be semantically close to a manual for another. Combine semantic retrieval with metadata or other appropriate matching methods when exact identity matters. The application should not expect a general representation to enforce every domain constraint by itself.

Create examples with local terminology, borrowed words, and mixed-language queries. People do not always use the language conventions of a formal translation. A user may combine an English product name with a local-language description of a symptom. Evaluate those ordinary patterns with reviewers who understand the language and domain. Automated translation can help prepare material, but it should not be the only source of relevance judgments when the question is whether the search experience works for actual speakers.

The collection can be uneven before the model sees it

A multilingual portal may contain a current English policy and an older translation. If the system treats both as equally authoritative, it can return a readable but obsolete answer. Store version, effective date, owner, and translation relationship alongside the document. A translated title alone is not enough to establish that two files describe the same release. The retrieval policy should preserve authority and freshness rather than favoring a language match without regard to whether the information remains valid.

Document processing can introduce language-specific damage. Text extraction may mishandle a script, drop accents, flatten tables, or separate headings from the content they qualify. Inspect the indexed text, not only the original document. If the representation is built from corrupted input, model selection will not solve the problem. A small review of actual indexed passages across supported languages can reveal errors that a general benchmark never encounters because its documents were already clean and well structured.

Evaluation should show the distribution of experience

A single average score can be dominated by the language with the largest test set. Report results by language and by task category, while acknowledging where sample sizes are too small for confident conclusions. Include queries with no relevant answer so the system is not rewarded for returning something in every case. Measure whether the chosen results are authoritative, usable, and sufficient for the task. Similarity alone is not the final product objective.

For a retrieval-augmented assistant, separate retrieval quality from answer-language quality. The system may find the correct source and then mistranslate an important qualification. Or it may produce fluent local-language prose from the wrong source. These failures need different corrections. Preserve the evidence path so reviewers can locate the stage that failed. If the answer cites a document in another language, explain that relationship rather than implying that the generated translation is the original source text.

When changing embedding models, plan a representation migration. Vectors from different models or configurations should not be assumed compatible. Keep version metadata, rebuild the relevant collection, and compare the same evaluation cases before switching traffic. A staged rollout can reveal language-specific regressions that disappear in a global average. Retain a rollback path and enough operational identity to connect a reported issue to the representation and index that served it.

Offer a clear limitation instead of a universal promise

A product can support a language at a defined level without pretending that every task is equally reliable. It might offer strong navigation to local-language guides while directing advanced technical questions to an authoritative source in another language. State that limitation clearly and let users choose where possible. A transparent boundary is more useful than a broad multilingual claim that leaves readers to discover uneven coverage through failed searches. The supported experience should reflect the evidence from the collection and evaluation.

Multilingual embeddings can make information more accessible across language boundaries, especially in organizations whose knowledge is distributed unevenly. Their value depends on preserving domain identity, source authority, and the reader’s ability to use the result. Test language journeys separately, inspect the indexed text, and keep metadata with the vectors. The goal is not merely a model that accepts many languages. It is a search service that helps people find the right information in a form they can understand and trust.