Meta's Data2vec 2.0: Second time around is faster

Meta's Data2vec is an example of a generalist neural network that can use the same exact code to crunch examples of data in different modalities -- in this case, speech, text, and images -- and make predictions about that data.

Baevski et al.

What do you do when you've proven your point in neural networks?

Do it faster is one answer. 

On Tuesday, Meta, the owner of Facebook, Instagram, and WhatsApp, unveiled Data2vec 2.0, a revamp of a neural network introduced earlier this year that behaves as a sort of generalist, performing on tasks that involve text, image, and speech data with the same basic approach to all three. 

Featured

• Garmin wins big at CES, launches new products and updates Garmin Connect
• From indoor solar to light-based internet, photonics offers a brighter future
• I saw Samsung and LG's new transparent TVs at CES, and there's a clear winner
• Have 10 hours? IBM will train you in AI fundamentals - for free

The authors this time made two key changes to Data2vec to make it faster: using "convolutions," and "amortizing" the compressed representations of the teacher network.

On the first score, the student network that has to predict the representations of the teacher is no longer using the part of the Transformer called a decoder to do so. 

AI in 2023: A year of breakthroughs that left no human thing unchanged These are the jobs most likely to be taken over by AI AI at the edge: 5G and the Internet of Things see fast times ahead Almost half of tech executives say their organizations aren't ready for AI or other advanced initiatives AI in 2023: A year of breakthroughs that left no human thing unchanged

These are the jobs most likely to be taken over by AI

AI at the edge: 5G and the Internet of Things see fast times ahead

Almost half of tech executives say their organizations aren't ready for AI or other advanced initiatives

Editorial standards Show Comments