Seamless by Meta

Meta has recently unveiled a groundbreaking series of AI technologies aimed at revolutionizing multilingual and streaming speech translation. In this blog post, I delve into these innovations: SeamlessM4T v2, SeamlessExpressive, and SeamlessStreaming.

SeamlessM4T v2: This foundational model, an upgraded version of its predecessor, sets a new benchmark in semantic accuracy for speech and text translation tasks. It supports nearly 100 languages and integrates the multitask-UnitY2 with a non-auto-regressive unit decoder for enhanced efficiency. Notably, its w2v-BERT 2.0 speech encoder has been pre-trained on a massive 4.5M hours of audio, further fine-tuned for improved performance in low-resource languages.

‍

Key Features and Performance:

• Tasks and Language Support: It has been trained and evaluated across four supervised tasks: Text-to-Text Translation (T2TT), Automatic Speech Recognition (ASR), Speech-to-Text Translation (S2TT), Speech-to-Speech Translation (S2ST), and the zero-shot task of Text-to-Speech Translation (T2ST).
• Decoding and Scoring Methods: For text hypothesis generation, it employs beam search with a width of 5. It uses chrF2++ for T2TT scoring, and SacreBLEU for S2TT in various languages. The model's ASR performance is scored with the Word Error Rate (WER) metric.
• S2ST and T2ST Performance: In these tasks, SeamlessM4T-Large v2 uses a two-pass beam-search decoding technique. Its accuracy is evaluated using ASR-BLEU with the Whisper-Large as the underlying ASR model.
• Comparison with Other Models: SeamlessM4T-Large v2 is compared against its predecessor and cascaded models across various tasks. Notably, it shows significant improvement in S2TT and S2ST tasks, particularly in X–eng (languages to English) translations.
• Performance on Low-Resource Languages: The model demonstrates remarkable improvements on low- and medium-resource languages, thanks to the increased supervised and self-supervised data used in training the w2v-BERT 2.0 speech encoder.
• Ablation Studies: The model underwent ablation studies to investigate the best input and output representations for both autoregressive (AR) and non-autoregressive (NAR) Text-to-Unit (T2U) models. It was found that character input and non-reduced unit output were best for the NAR T2U model.

• Multilingual Char-to-Unit Aligner: UnitY2-based aligner, a component of the model, serves as a universal tool for aligning text-audio pairs across various languages. This feature is especially useful for pseudo-labeling with large, unlabeled audio corpora.

Impact and Implications:

SeamlessM4T-Large v2’s performance showcases the potential for AI to break language barriers more efficiently and effectively, especially in real-time communication scenarios. Its advancements in handling low-resource languages and its ability to maintain vocal style and prosody in translations are particularly noteworthy. This makes it a vital tool for applications ranging from global communication to accessibility services, where accurate and expressive language translation is critical. The model's development aligns with the broader goal of making multilingual AI more accessible and effective for diverse global users.

‍

SeamlessExpressive: A pioneering model in the realm of speech translation, SeamlessExpressive focuses on maintaining the vocal style and prosody of the original speech, such as rhythm and tone. It currently supports translations involving English and five other languages, introducing a new dimension to expressive speech-to-speech translation (S2ST), especially in underexplored aspects like speech rate and pauses.

Key Features and Performance:

• Multilingual Support: It can handle translations between English and five other languages: French, German, Italian, Mandarin, and Spanish.
• Enhanced Prosody Preservation: SeamlessExpressive is designed to maintain the vocal style and prosody (like rhythm and tone) of the original speech, making translations more natural and expressive.
• Advanced Model Integration: The model combines the strengths of Prosody UnitY2 and PRETSSEL for better speech generation. This integration allows it to achieve improved content translation while preserving the speaker's voice style and rhythm.
• Competitive Performance: In tests, SeamlessExpressive showed competitive results in terms of content preservation and expressivity, particularly excelling in datasets focused on expressiveness and spontaneous speech.
• Efficient Speech Synthesis: The model uses PRETSSEL for speech synthesis, noted for its efficiency and effectiveness in maintaining vocal style and prosody.

‍

SeamlessStreaming: Leveraging the Efficient Monotonic Multihead Attention (EMMA) mechanism, this model introduces low-latency, simultaneous many-to-many translations without needing complete source utterances. This model's real-time translation capabilities cover the same language range as SeamlessM4T v2 for various tasks.

Quality-Latency Trade-Off in SeamlessStreaming

• Translation Quality and Latency:

• Speech-to-Text (S2TT) and Speech-to-Speech (S2ST) Translations: SeamlessStreaming processes both modalities simultaneously.
• Decision Threshold (tEMMA): Set by default at 0.5, which can be adjusted to fine-tune latency. The model's performance was evaluated under various tEMMA settings.

• Evaluating Translation Quality:

• Speech-to-Text Results: Evaluated on the Fleurs dataset, with metrics like average lagging (AL) and length-adaptive average lagging (LAAL) assessed using SentencePiece tokens.
• ASR Performance: Table 29 shows that SeamlessStreaming performs Automatic Speech Recognition (ASR) tasks with considerably lower latency compared to SeamlessM4T v2, albeit with a slight increase in Word Error Rate (WER).

• Speech-to-Speech Translation:

• The quality of speech output in SeamlessStreaming is slightly lower than text output when compared to SeamlessM4T v2, attributed partly to discontinuities in generated speech.

Data Resources and Language Pairs

Impact of Training Data:

• Resource Level Influence: The quantity and quality of training data directly affect the translation accuracy and latency. High resource languages show less quality drop and lower latency compared to low resource and zero-shot settings.

Language Family Considerations:

• Performance Variability: The quality of streaming translation varies with language pairs due to linguistic divergence and cultural disparities.
• English-Centered Data Training: SeamlessStreaming shows better performance with languages closely related to English, such as Italic and Germanic languages. In contrast, distant language groups like Sinitic or Japanesic show a more significant drop in translation quality and increased latency.

‍

These systems were rigorously evaluated using a blend of existing and newly developed metrics, including those specifically designed for measuring expressivity and prosody. Moreover, Meta’s commitment to responsible AI is evident through their comprehensive approach, which includes red-teaming for machine translation, toxicity detection, gender bias evaluation, and the innovative SeamlessWM watermarking mechanism.

The culmination of these efforts is the unified Seamless model, combining SeamlessExpressive and SeamlessStreaming. This represents a significant leap towards making the Universal Speech Translator a reality, bridging the gap between science fiction and practical technology. In this blog post, we will explore how Seamless stands as a testament to the technical prowess needed for such a transformative leap in AI-powered communication.

‍

Seamless aims to transform machine-assisted cross-lingual communication, significantly aiding in social integration and personal goal achievement in multilingual societies. This technology represents a major step towards bridging language gaps in an interconnected world.

• Audio/Video Calling: Seamless can be integrated into communication apps for real-time, multilingual conversations with live captions, enhancing global connectivity.
• AR/VR Environments: It enables cross-lingual interactions in augmented and virtual reality settings, such as games or virtual meetings, enriching the immersive experience.
• Online Streaming: Users can use Seamless for real-time language translation in streaming content, broadening the reach of streamers to a global audience.
• Wearable Devices: Integration with devices like earbuds and smart glasses could create a Universal Speech Translator, allowing users to communicate in any language with real-time translation.
• Voice-Messaging Platforms: Seamless can translate audio messages on platforms like WhatsApp while maintaining the speaker’s voice style, making cross-lingual communication more personal.
• Long-Form Audio and Video Content: It can be part of a pipeline to translate long-form content such as lectures, podcasts, or video dubbing, offering a fully expressive, multilingual experience.

‍

Meta research paper: https://ai.meta.com/research/publications/seamless-multilingual-expressive-and-streaming-speech-translation/

Github: https://github.com/facebookresearch/seamless_communication

Hugging face: https://huggingface.co/collections/facebook/seamless-communication-6568d486ef451c6ba62c7724