NVIDIA introduces Fugatto as “world’s most flexible sound machine”

NVIDIA has unveiled Fugatto, a generative AI model capable of creating and modifying audio content. The model aims to assist music producers, film creators, and game developers by allowing them to generate novel sounds through text prompts. Fugatto combines various audio generation capabilities, employing advanced algorithms to enhance creative processes in the audio industry.

NVIDIA unveils Fugatto, a generative AI for audio creation

Fugatto, short for Foundational Generative Audio Transformer Opus 1, was introduced by NVIDIA, the world’s leading supplier of chips and software for AI systems. The technology can generate and alter sound from existing audio files, making it distinct from previous models. For instance, it can transform a piano melody into a human voice or modify a spoken recording’s accent and emotional tone. This flexibility allows creators to explore a range of innovative applications across different fields.

The team behind Fugatto consists of over a dozen researchers, including Rafael Valle, NVIDIA’s applied audio research manager. Valle emphasized the goal of the project: “We wanted to create a model that understands and generates sound like humans do.” Key to Fugatto’s design is its ability to integrate multiple tasks related to audio generation and transformation, showcasing emergent properties that arise from its extensive training data.

Users can instruct Fugatto with free-form prompts to create soundscapes, music snippets, or even unique sound effects. For example, a producer could quickly prototype different styles or instruments for a track. Notably, Fugatto features techniques like ComposableART, allowing users to amalgamate varying commands. Testing revealed surprising results, as suggested by Rohan Badlani, an AI researcher involved with the model, who described the experience as artistically rewarding despite his technical background.

Fugatto combines various audio generation capabilities, employing advanced algorithms to enhance creative processes in the audio industry (Image credit)

During its training, Fugatto utilized 2.5 billion parameters and was developed on NVIDIA’s powerful DGX systems featuring 32 H100 Tensor Core GPUs. The model’s training relied on a diverse, blended dataset comprising millions of audio samples, enhancing its multi-accent and multilingual functionality. This ambitious project also took over a year to develop, with the team overcoming several challenges in data generation and model training.

Fugatto offers several potential applications, including for advertising agencies and language learning platforms. It’s been suggested that marketing campaigns could benefit from its ability to tailor voiceovers with different accents or moods. In education, learners might enjoy personalized courses featuring familiar voices. Game developers could adapt in-game audio dynamically, integrating interactive elements that respond to user actions.

While Fugatto’s capabilities are impressive, NVIDIA has not announced immediate plans to release this technology to the public. The company expresses concern over potential misuse of generative AI, with Bryan Catanzaro, NVIDIA’s vice president of applied deep learning research, highlighting the importance of caution given the risks associated with such technology. OpenAI and other firms in the field face similar challenges regarding the responsible deployment of their models, particularly concerning intellectual property rights and misinformation.

Featured image credit: Nvidia