Kyungsu Kim¹**, Junghyun Koo¹, Sungho Lee¹, Haesun Jung¹, Kyogu Lee**¹²³
Music and Audio Research Group (MARG), Department of Intelligence and Information, Seoul National University, Seoul, Republic of Korea¹ Interdisciplinary Program in Artificial Intelligence, Seoul National University² Artificial Intelligence Institute, Seoul National University³
Overall pipeline of TokenSynth. (a) TokenSynth takees a timbre embedding and MIDI tokens as input, where the timbre embedding is extracted from reference audio using a pre-trained CLAP encoder. Both reference and target audio are synthesized using the same instrument but with different musical notes. The objective for TokenSynth is to predict the encoded audio tokens from the target audio signal using a pre-trained DAC encoder. (b) During the inference stage, the timbre embedding can be extracted from either reference audio, a text description, or a combination of both. The predicted output audio tokens are then converted into an audio signal using a pre-trained DAC decoder.
TokenSynth is a token-based neural synthesizer that generates polyphonic single-instrument musical audio from MIDI and timbre embeddings, enabling instrument cloning, text-to-instrument synthesis, and timbre manipulation. It uses a decoder-only transformer trained on neural audio tokens with CLAP-based timbre conditioning, allowing for flexible sound design without fine-tuning.
TokenSynth is capable of performing zero-shot instrument timbre cloning. Given short length (5 seconds) of single-instrument reference audio and arbitrary MIDI notes, TokenSynth generates output audio that plays given MIDI notes with the timbre of given reference audio. Note that instruments shown in following examples are unseen during training.
The reference audio is used to extract CLAP embedding. The reference audio is used to extract the CLAP embedding. The ground-truth audio plays the same MIDI notes given to TokenSynth as input, using the same instrument as the reference audio.
Reference
Ground-Truth
TokenSynth
TokenSynth-Aug
