Offline Translation (Image Recognition and Translation)

Architectures Compared

The app is implemented in three interchangeable pipelines, all with the same UI:

1. Apple-native pipeline
   A baseline using Apple's own APIs for OCR (Vision), language detection (Natural Language), and translation (Translate). This pipeline serves as the control group to benchmark against.
2. Hybrid pipeline
   Primarily uses Apple frameworks, but selectively falls back to third-party local models in failure cases. For example, if Apple Vision fails to detect text or the language is unsupported, the app invokes Google MLKit for OCR and a local copy of Meta's M2M-100 (418M) for multilingual translation.
3. End-to-end vision-language model
   An experimental pipeline that uses a single compact vision-language model (e.g., MiniGPT-style) to go directly from an image to a translated string, bypassing modular components. Used to explore next-generation model architectures.

Choice of Models

Each pipeline uses models selected for their runtime performance, license compatibility, and support for low-resource languages:

• Apple-native pipeline:
  • OCR: VNRecognizeTextRequest (Vision)
  • Language Detection: NLLanguageRecognizer
  • Translation: Apple Translate API
• Hybrid pipeline:
  • OCR: Primary – Apple Vision; Fallback – Google MLKit (for multilingual printed text)
  • Language Detection: Primary – Apple Natural Language
  • Translation: Primary – Apple Translate; Fallback – Meta M2M-100 (418M), running via Core ML with int8 quantization for near real-time performance
• End-to-end vision-language model:
  • Model: A compact, quantized variant of a MiniGPT-style model (under 3B parameters)
  • Functionality: Direct image-to-translation inference; slower but used to evaluate unified multimodal workflows

Model Choice Tradeoffs

This project considered several open-source alternatives before selecting the final models for each component. The selection was based on technical feasibility, mobile optimization potential, and ability to support low-resource languages:

• M2M-100 (418M) was selected for fallback translation in the hybrid pipeline due to its:
  • Direct many-to-many translation across 100 languages (no pivoting through English)
  • Strong performance on both high- and mid-resource language pairs
  • MIT license (suitable for commercial use)
  • Feasibility for real-time execution with quantization (under 1 GB memory footprint)
• NLLB-200 (600M) was evaluated for translation but ruled out for production use due to its non-commercial license (CC-BY-NC 4.0) and larger memory footprint (~1.3–2.5 GB). It remains a useful benchmark for research purposes.
• MarianMT and OPUS-MT models were also considered for translation. While lighter in size, they often underperform M2M-100.
• Tesseract OCR and PaddleOCR were reviewed as alternatives to Apple Vision and MLKit. However, they were deprioritized due to lower mobile inference performance and less seamless integration with iOS. PaddleOCR remains under consideration for future low-resource script support.
• MiniGPT-4, BLIP-2, and other open multimodal models were explored for the end-to-end pipeline. While promising, these models remain too large (>3–7B params) for smooth mobile inference today. Experimental quantized versions are included to evaluate future viability as devices become more capable.
• Gemini Nano (https://deepmind.google/technologies/gemini/nano/) is one of the most promising, highly capable on-device models. However, at the moment it is not available for use on iOS devices.
• Imp (2024) – A recently released mobile-optimized multimodal language model (https://imp-vl.github.io/) that achieves GPT-4V–level performance with only 3B parameters. This makes it a strong candidate for future on-device image-to-text translation without separate OCR.

Design Doc

Evaluation Methodology

To assess the performance of each pipeline, we will conduct evaluations focusing on both individual components and the overall system:

Component-Level Evaluation

• OCR Accuracy:
  • Metrics: Character Error Rate (CER) and Word Error Rate (WER).
  • Test Set: A diverse set of images containing various scripts (Latin, Cyrillic, Devanagari, etc.) under different lighting conditions.
• Language Detection:
  • Metrics: Accuracy and F1 Score.
  • Test Set: Text snippets in multiple languages, including low-resource languages.
• Translation Quality:
  • Metrics: BLEU and METEOR scores.
  • Test Set: Standard translation benchmarks like WMT datasets, focusing on low-resource language pairs.

System-Level Evaluation

• End-to-End Accuracy:
  • Metrics: Overall translation accuracy from image input to translated text output.
  • Test Set: Real-world images with textual content in various languages.
• Performance Metrics:
  • Latency: Time taken to process an image and produce the translation.
  • Resource Utilization: CPU and memory usage during processing.

Results & Learnings

(In progress)

Next Steps

UI Updates

• Enable real-time camera capture in addition to photo uploads.
• Enable in-place translations within the image (with bounding boxes).

Model Updates

• Explore smaller multimodal models that run efficiently on iOS.
• Evaluate the use of quantization techniques to reduce model size and improve inference speed.