How to Master Your Music with AI

What's in the last mile of music production

After a song is written and recorded, there are four more jobs before anyone hears it.

1. Mixing. Mixing balances the parts of a song against each other: the vocal against the guitars, the kick against the bass. Each part gets a level, a place in the stereo field and a tone, so everything can be heard. When it goes wrong, a great vocal disappears in a crowded mix, or a mix that sounded fine on studio monitors falls apart on earbuds.
2. Denoising and restoration. Home recordings pick up hiss from a cheap interface, the fridge or air conditioner humming in the background, the echo of a bedroom, or distortion from a level set too hot. Cleaning this up early matters, because every later stage makes it louder. More and more music starts as a home or phone recording, so this step keeps getting more important.
3. Mastering. Mastering prepares the finished mix for release. It adjusts tonal balance (too boomy, too harsh, too dull), dynamics, loudness and stereo width, and makes the songs on an EP or album sound like they belong together. There are no fixed right answers here. An engineer makes dozens of small judgement calls based on the genre and what the artist wants.
4. Distribution. Then the track has to reach listeners. It needs to be exported in the right formats, given metadata and artwork, and delivered to the streaming platforms.

Each of these steps traditionally needs its own tools and skills, and often a different person to perform it.

Why “warm” is hard for a computer

Musicians describe sound with metaphors like warm, bright, muddy, airy, punchy and thin. Engineers spend years learning how these words translate into adjustments. Roughly, "muddy" usually means too much energy in the low mids, "bright" means more high-frequency content, and "punchy" is about how the drums hit and decay.

These translations depend on the track. Warming up a dark acoustic ballad is a very different job from warming up a trap beat that is already bass-heavy, and the same word can call for opposite adjustments.

Preset-based tools struggle with this. A preset ties one word to one fixed processing chain and applies it the same way to every track.

A preset gives one answer to “make it warmer.” A good engineer first asks: warmer compared to what?

Audio-language models learn from many examples of audio paired with descriptions and edits. From those, they learn what a word means for the particular sound in front of them, which lets them act on a vague, human instruction.

Fixing vs. regenerating

Classic restoration tools subtract. A noise reducer estimates the hiss and removes it, and a de-reverb tool tries to separate the direct sound from the room. That works for mild problems. Subtracting can't bring back detail that was never recorded, though, such as the high end a cheap phone microphone lost.

Generative models have learned what clean, well-recorded music sounds like, so they can fill in plausible missing detail as well as remove noise. This is how a phone recording can come back sounding as if it was made in a much better room.

The risk is that the model drifts away from the original performance. It has to stay faithful to what the artist played and sang. Getting that balance right is still an open research problem, and one we work on in the lab.

Putting it into practice: Sequential AI

Sequential AI handles the whole last mile in one place. You upload a track, and it automatically balances the vocals against the rest of the mix, cleans up noise and masters the result. If you want changes, you type them: "make the mix warmer," "bring out the bass," "clean up this phone recording." Each request produces a new version, and you keep going until you like it. Then you can release the track on YouTube and on our Discover page, with Spotify coming soon.

A production dashboard checks the result against the things a mastering engineer would check:

• Loudness and peak headroom: integrated LUFS and true peak, flagged when something is off
• Dynamics: how much the sections move, punch, and peak-to-average ratio
• Stereo and mono: stereo width, mono compatibility and the worst phase moments
• Tonal balance: bass weight, brightness and DC offset
• Noise: the background noise level
• Delivery: sample rate, duration and source quality
• Streaming playback: headroom against the loudness targets of Spotify, Apple Music, YouTube, Amazon Music, Tidal and Deezer

There is also a music intelligence dashboard with insights into the music itself.

How our models work

I built my first music generation model over 20 years ago (I'm serious, read about the model here!), when most people thought of AI music as science fiction. My research since then has asked whether machines can understand and create music. The goal has always been to build tools for musicians.

Sequential AI runs on mastering and music intelligence models we built ourselves, rather than on someone else's model. They come out of years of research at the Audio, Music, and AI (AMAAI) Lab on connecting music, language and emotion.

Most automated mastering tools apply a fixed signal-processing chain and adjust a few settings. Our models take your audio and a text instruction and generate the improved audio directly, using a flow-based diffusion model.

Because the models link language and sound, one system covers many tasks through the same text box. These include equalization, dynamics, loudness, stereo width, and restoration problems such as noise and reverb.

We are close to finishing training a much larger next-generation model. It handles diverse genres, phone recordings and noisy audio better, and it follows more detailed instructions.

How to master a track with AI: a practical workflow

These habits help with any tool.

• Leave headroom. Export without a limiter on the master bus, peaking a few decibels below zero. If the mix is already squashed, mastering has little left to work with.
• Fix big problems in the mix first. Mastering can polish a track and rescue a rough recording, but it works best when the arrangement and balance are already close.
• Describe what you want to hear. "Make the vocals sit more forward" tells an AI model more than "boost 3 kHz."
• Change one thing at a time. Make a request, listen, then make the next one. You'll learn what each instruction does to your track.
• Compare at the same loudness. Our ears tend to prefer the louder version, so a master can seem better just because it's louder. Match the levels before deciding.
• Check on several systems. Earbuds, laptop speakers and a car stereo each reveal different problems.
• Aim for streaming targets. Around -14 LUFS integrated, with true peaks below -1 dBTP, plays back cleanly on Spotify and YouTube and needs only a small adjustment on Apple Music. (RoEx)

The loudness war is over (sort of)

For decades, records kept getting louder. Engineers squeezed out every decibel because a louder track seemed more exciting on the radio and in shops, and the music lost dynamics along the way.

Streaming changed this. Platforms measure each track's perceived loudness in LUFS (Loudness Units relative to Full Scale) and turn it up or down to a common level, so listeners don't have to touch the volume between songs. Spotify and YouTube normalize to -14 LUFS, Deezer to -15 and Apple Music to -16.

A track mastered very loud, at -6 LUFS, gets turned down by 8 dB on Spotify. It ends up no louder than the songs around it, and it sounds flatter, because the dynamics that were squeezed out don't come back. (RoEx, MasteringBOX)

On streaming platforms, a louder master doesn't sound louder. It just sounds more squashed.

The aim now is the best possible sound at the level the platform will play it. Sequential AI's production dashboard measures this for you and shows, for each major platform, how many decibels of headroom your master has against that platform's target.

What else a model can hear in a song

Mastering is only part of our lab's work. A lot of our research looks at what a model can learn about a song just by listening to it.

Emotion. Emotion in music is usually labelled in one of two ways: with categories such as happy, sad or tense, or on two scales, valence (how positive it feels) and arousal (how energetic). Most datasets use one system, which makes them hard to combine. Our Music2Emo model trains on both kinds of labels at once, so it can learn from much more data. Since two people can hear the same song and feel different things, a model like this predicts what listeners tend to perceive. (paper, demo)

Popularity and aesthetic quality. I've worked on hit song prediction for years, with my first paper on it in 2014. AI music platforms have changed the problem. Thousands of songs now appear every day with no artist reputation or label behind them.

Our APEX model was trained on more than 211,000 AI-generated songs from Suno and Udio, about 10,000 hours of audio. It predicts engagement (streams and likes) together with five aesthetic qualities: coherence, musicality, memorability, clarity and naturalness. Adding the aesthetic qualities made the model better at predicting which songs people prefer, even for music generators it had never seen in training. (paper)

No model can guarantee a hit. Timing, audience, influencers and promotion all play a big role. We see scores like these as feedback on a track.

Controllable generation. Mustango, our text-to-music model, lets a text prompt set specific musical elements such as chords, tempo and key, as well as the overall style. (paper)

Emotion and hit-potential analysis from this research will come to the Sequential AI dashboard soon.

From lab to company

Sequential AI is incubated at SUTD under the GAP programme, which helps research teams commercialise their work. We also completed the ARISE validation and venture-building programme, led by Prof. Virginia Cha at NUS, which sent us out of the lab to talk to musicians. Two requests came up again and again. Artists wanted to control the sound in their own words, and they wanted to rescue recordings they had given up on because of noise. When we showed them the platform, though, it was the music intelligence features that surprised them most.

The team:

• Dr. Abhinaba Roy (co-founder) is an AI research scientist specialising in machine learning and signal processing for audio analysis.
• Dr. Ambuj Mehrish (co-founder) is an AI research scientist working on deep learning, natural language processing and the multimodal systems behind our core technology.
• Shi Qilun (product engineer) leads UI/UX and product management, gathering user feedback and running marketing.
• Roshan Emmanuel (product engineer) builds the platform full-stack and, as a trained pianist, leads the evaluation of its musical output.
• Assoc. Prof. Dorien Herremans (founder; that's me) leads the AMAAI Lab and sets the research and product direction.

You can meet us on the team page.

What's next, and how you can help

On our roadmap:

• Stem-level mastering: separate control over vocals, drums, bass and instruments
• Reference matching: upload a track you love and get a master in a similar direction
• A larger model: better with diverse genres, difficult recordings and detailed instructions
• More music intelligence in the dashboard, including emotion and hit song potential

Sequential AI is free during the open beta. If you make music, try it on one of your own songs, and tell us what works and what doesn't. We read all the feedback and use it for the next version.

Assoc. Prof. Dorien Herremans, co-founder of Sequential AI and head of the AMAAI Lab at SUTD