Promote Sign in Sign up

Text to speech

Voicebox by Meta ✓

Generate versatile speech with state-of-the-art AI

Source rating 3.5 (2) 2.8k views 39 saves Free
Speech Generation Audio Output Text To Speech
More categories (4)
Voice Synthesis Versatality Voice Technology Text to speech
Visit website Sign in to save

Free

Verdict

A quick read before you dig into the full page.

Best if
Generate versatile speech with state-of-the-art AI
Skip if
Limited to six languages
Pick this when
Generative model

Considering ElevenLabs instead? Compare side by side

Overview

Voicebox is a generative AI model for speech that can generalize to tasks it was not specifically trained for with state-of-the-art performance. Unlike existing speech synthesizers, it can be trained on diverse, unstructured data without requiring carefully labeled inputs.

Voicebox uses a new approach called Flow Matching, which is a Meta's latest advancement on non-autoregressive generative models that can learn highly non-deterministic mapping between text and speech.

Voicebox can produce high-quality audio clips in a vast variety of styles and can synthesize speech across six languages, as well as perform noise removal, content editing, style conversion, and diverse sample generation.

One of the main advantages of Voicebox is its ability to modify any part of a given sample, not just the end of an audio clip it is given. This makes it highly versatile and suitable for tasks such as in-context text-to-speech synthesis, cross-lingual style transfer, speech denoising and editing, and diverse speech sampling.

Additionally, Voicebox outperforms existing state-of-the-art speech models on word error rate and audio similarity metrics. While Voicebox is not currently available to the public due to potential risks of misuse, Meta has shared audio samples and a research paper detailing its approach and results.

This breakthrough in generative AI for speech is exciting as it has potential applications in helping people communicate and customize voices for virtual assistants.

Voicebox by Meta website

Access & connections

Where you can use Voicebox by Meta.

Platforms

Web

Capabilities

What this tool accepts, produces, and is strongest at.

Input

  • Audio

Output

  • Audio

Pros & cons

Pros

  • Generative model
  • Generalizes to untrained tasks
  • Trains on diverse data
  • Doesn't require labeled inputs
  • Uses Flow Matching
  • High-quality audio clips
  • Operates in six languages
  • Performs noise removal
Show 24 more
  • Performs content editing
  • Performs style conversion
  • Does diverse sample generation
  • Can modify any sample part
  • In-context text-to-speech synthesis
  • Performs cross-lingual style transfer
  • Performs speech denoising
  • Performs speech editing
  • Performs diverse speech sampling
  • Outperforms other models
  • Superior word error rate
  • Superior audio similarity metrics
  • Versatile across tasks
  • Significant potential applications
  • Style transfer capability
  • Audio editing functionality
  • Large data scale training
  • Trains on unstructured data
  • Effective model classifier
  • Potential virtual assistant voices
  • Fast performance
  • Effective for in-wild data
  • Potential for synthetic data generation
  • Trains on multilingual benchmarks

Cons

  • Not available to public
  • Potential for misuse
  • Requires a lot of data
  • Limited to six languages
  • 20 times slower than Vall-E
  • Depends on Flow Matching
  • Doesn't support task-specific training
  • Currently lacks public API
Show 2 more
  • Lacks verification functionality
  • No open-source code

Compare alternatives

Similar tools in the catalog — with a clear reason to open each one.

Why look Offers a public API

Full comparison

No pricing

Full comparison

Why look Better for writing

Full comparison

Free

Full comparison

from $4.99/mo

Full comparison

from $19.99/mo

Full comparison

Releases 1

Initial release June 16, 2023 Current
Initial release of Voicebox by Meta.

FAQ 12

What are the key features of Voicebox by Meta?
Voicebox by Meta is a generative AI model for speech that uses a new approach called Flow Matching. It can train on diverse, unstructured data without requiring carefully labeled inputs. It can produce high-quality audio clips in a variety of styles and synthesize speech across six languages. Other features include noise removal, content editing, style conversion, and diverse sample generation. Unlike existing models, it can modify any part of a given sample, not just the end, making it versatile across different tasks.
What does the Flow Matching approach utilized by Voicebox entail?
Flow Matching is a new approach developed by Meta which is seen as their latest advancement on non-autoregressive generative models. This technique enables highly non-deterministic mapping between text and speech. This non-deterministic mapping is beneficial as it allows Voicebox to learn from varied speech data without the necessity for those variations to be carefully labeled. This indicates that Voicebox can be trained on significantly more diverse and larger scales of data.
In what languages can Voicebox synthesize speech?
Voicebox can synthesize speech in six languages: English, French, Spanish, German, Polish, and Portuguese.
How does Voicebox perform in terms of word error rate and audio similarity metrics compared to existing models?
Voicebox outperforms the current state-of-the-art English model, VALL-E, in terms of both intelligibility and audio similarity. It achieves a 5.9 percent word error rate versus VALL-E's 1.9 percent, and an audio similarity score of 0.580 compared to VALL-E's 0.681. Furthermore, for cross-lingual style transfer, Voicebox reduces the average word error rate from 10.9 percent to 5.2 percent, and improves audio similarity from 0.335 to 0.481.
What makes Voicebox different from traditional speech synthesizers?
Traditional speech synthesizers require specific training for each task using carefully prepared data and they can only modify the end part of an audio clip. Conversely, Voicebox can learn from raw audio and an accompanying transcription. It is capable of modifying any part of a given sample and doesn't require carefully labeled inputs. This difference allows for greater versatility across a wider range of tasks and data sources.
How can Voicebox modify any part of a given audio sample?
Along with producing outputs from scratch, Voicebox can modify existing samples. The model can learn to predict a speech segment by analyzing the surrounding speech and the transcript of the segment. Given this learning, it can apply it to generate or modify audio in any part of a recording without having to recreate the entire input.
Is Voicebox available for public use?
No, as of the provided information, Voicebox is not available to the public due to potential risks of misuse.
What are the potential applications of Voicebox?
Potential applications of Voicebox are wide-ranging. Its in-context text-to-speech synthesis could potentially bring speech to people who are unable to speak or allow people to customize the voices of non-player characters and virtual assistants. Its ability to perform cross-lingual style transfer could help people communicate naturally in different languages. Voicebox's abilities in speech denoising and editing could ease the process of cleaning up and editing audio. In terms of diverse speech sampling, it could generate synthetic data to better train a speech assistant model.
What data was Voicebox trained on?
Voicebox was trained using more than 50,000 hours of recorded speech and transcripts from public domain audiobooks in six languages including English, French, Spanish, German, Polish, and Portuguese.
Can Voicebox perform speech denoising and editing?
Yes, Voicebox's in-context learning enables it to generate speech to seamlessly edit segments within audio recordings. It can resynthesize the portion of speech corrupted by short-duration noise or replace misspoken words without having to re-record the entire speech.
How does Voicebox handle diverse speech sampling?
Voicebox is able to generate speech that is more representative of how people talk in the real world and across the six languages it functions in. This could, in the future, be used to generate synthetic data to help better train a speech assistant model.
Can Voicebox perform in-context text-to-speech synthesis?
Yes, using an input audio sample just two seconds in length, Voicebox can match the sample's audio style and use it for text-to-speech generation.
Voicebox by Meta Free
Visit website