Blog

Mandarin Document Review | Tokenization Breaks First

East Asian review fails quietly. A search that returns nothing looks exactly like a search with no responsive documents, and nobody checks which one happened.

Chinese, Japanese and Korean text does not behave like European-language text in a review platform. The failures are not error messages. They are empty result sets that look like good news.

Most of what goes wrong is decided at processing, before anyone reviews anything, and is expensive to fix afterward because it means reprocessing the population.

Three failures that happen before review starts

Tokenization. Chinese and Japanese are written without spaces between words. A platform that tokenizes on whitespace indexes an entire sentence as one term, which means a search term list returns nothing at all. Correct CJK tokenization has to be configured at processing rather than discovered at review.

Encoding. Double-byte characters survive or corrupt depending on encoding. A load file written in one encoding and read in another turns a whole Chinese or Japanese production into unreadable output, and mixed-language documents break first.

Search terms. A term list translated from English into Chinese produces terms nobody in the documents actually wrote. Lists have to be built in the source language, with the translators, and hit counts tested before review begins rather than after.

Mandarin and Chinese material

  • Simplified and Traditional characters both appear, often in the same matter. A search built in one will not match the other unless the platform is configured for variant mapping. Mainland material is generally Simplified; Taiwan and Hong Kong material is generally Traditional.
  • Names are a recurring problem. Chinese names appear surname-first in the original, in more than one romanisation system in English correspondence, and sometimes in a third form on a business card. The same custodian can appear four ways in one production, so name variants belong in the matter glossary from the start.
  • WeChat is frequently the most important source and the hardest to collect. It is mobile-first, it mixes text, voice, images and transfers in one thread, and it does not export like email. Scope it at the beginning of a matter rather than when somebody asks where the chat records are.

Korean material

  • Korean is written in Hangul, but older corporate records, legal documents and name fields often carry hanja, the Chinese characters used in Korean. A set spanning twenty years contains both, and a search built only in Hangul misses the hanja occurrences.
  • Honorific register encodes hierarchy. Korean business language marks the relative seniority of writer and reader in the grammar itself, which means a translated document can indicate reporting lines and authority that no organisational chart was produced to show.
  • Romanisation of Korean names is inconsistent in practice, with the same surname appearing several ways in English correspondence from one company. Name variants go in the glossary for the same reason they do in Mandarin matters.
  • KakaoTalk is widely used in Korean business communication and carries the same collection difficulty as WeChat.

Japanese material

  • Japanese mixes three scripts in a single sentence: kanji, hiragana and katakana. The same word can be written in more than one, and the same person’s name may appear in kanji in a formal document and katakana in a chat message. Term lists have to account for all of it.
  • Keigo, the honorific register, carries the same hierarchy signal as Korean honorifics and matters in review for the same reasons.
  • Scanned Japanese is materially harder than scanned English. Older material may be set vertically, OCR accuracy on dense kanji is lower, and a poor OCR pass produces a document that looks searchable and is not. OCR output on CJK scans has to be verified rather than assumed.

Where these populations come from

The Commission sits in Washington, and its Section 337 docket brings respondents in China, Korea, Taiwan and Japan. In our experience these three languages account for most of the foreign-language volume that reaches this region, alongside cross-border commercial disputes, trade secret matters, international arbitration and multilingual regulatory investigations.

Which means the same population frequently has to serve a Commission filing, a district court production and sometimes a foreign proceeding — and the filing constraints differ. Non-Roman alphabets cannot be filed as text in EDIS, so Chinese, Japanese and Korean exhibits go in as scanned images while their translations are text.

Reviewers who read the documents

Machine translation with post-editing handles triage and volume reduction, and it is the right tool for deciding what matters. It is not the tool for a privilege determination or for a document that will be quoted, because a call made from a machine translation is a call made from an approximation.

Anything relied on, filed or logged in detail needs a certified human translation with a signed certificate of accuracy, and the translator has to be identifiable — a disputed rendering becomes a question about who produced it.

What we do

Pro-Legal Services, Inc. reviews Mandarin, Korean and Japanese populations with ATA-certified staff translators in house rather than a brokered network — CJK tokenization and encoding configured at processing, variant mapping for Simplified and Traditional, source-language term lists with tested hit counts, name variants maintained in a matter glossary, verified OCR on CJK scans, and certified translation for anything that will be used.

From Arlington, Virginia, five miles from the Commission.

Related: foreign language document review, Section 337 translation services, ITC foreign language exhibit filing, foreign language privilege review, multilingual review cost, overseas custodian collection, certified translation for court filing.

Pro-Legal Services, Inc. reviews East Asian language populations from Arlington, Virginia, five miles from the Commission, for counsel managing respondents and custodians across Asia.

Chinese, Korean or Japanese population and a search returning nothing? Send us the term list and a sample. Tokenization is usually the answer. Pro-Legal Services, Inc. · 1550 Wilson Blvd, Suite 250, Arlington, VA 22209 · 202-955-9400 · info@pro-legal.com