Lettering machine-translated manga: fitting text back into speech bubbles

Translating manga text into speech bubbles may seem straightforward, but there's much more to it than just using a translation model. Once the model has translated the text, there are several crucial steps that come into play to ensure the final product resembles a professionally lettered comic.

First, the page goes through a process of finding the text. A detector marks the regions that contain text, excluding page numbers, headers, and artist watermarks, as these elements don't constitute dialogue. Next, the system identifies the speech bubbles, as the shape of each balloon is essential for both cleaning and lettering.

Reading the text presents challenges since Japanese can be vertical, Korean and Chinese mix scripts, and sound effects are often drawn rather than typed. Hand-drawn sound effects outside of balloons are typically best left untouched, as the artist's original style should be preserved.

Removing the original text is another delicate step. While the textbook approach suggests masking the text and letting an inpainting model fill in the gaps, this method often leaves faint gray smudges on plain white balloons or invents unwanted texture on screentone or detailed art. A more effective approach is to treat different backgrounds uniquely, ensuring that the art around a balloon remains exactly as the artist originally drew it.

Lettering the translation into the cleaned page requires careful consideration. The system should fit the text into the balloon, not the opposite. Lines should follow the balloon's shape rather than the rectangular text box. Dialogue should not be filled to the edge of the balloon, as this looks unnatural compared to professionally lettered pages.

Professional dialogue should appear as if it was lettered at a consistent size, while shouted phrases can be made larger for emphasis. It's crucial to keep words whole, as hyphenating dialogue can make it difficult to read. Additionally, the chosen font must be capable of accurately drawing the characters, and the text color should ideally match the original letterer's intention, typically black on white or white on black.

Breaking lines appropriately is another important aspect. English typically breaks at spaces, while Chinese divides between characters with punctuation rules, Korean breaks at spaces with its own ellipsis conventions, and Thai necessitates a dictionary to identify the words.

Throughout the process, most issues become apparent at scale. A demo page may appear fine, but a more extensive chapter can reveal problems like stray gray smudges, misplaced hyphens, or bubbles that are slightly too full. Comparing the translated pages against official English and Korean editions, page by page, provided more valuable insights than any metric could offer.

Manga Translator, the web app built on this pipeline, allows users to upload manga, manhwa, or webtoon pages in their browser and download them lettered in English or 50 other languages. By carefully examining the lettering of their own pages, users may discover areas that still require improvement to achieve a polished, professionally lettered appearance.