Utilizing Qwen2-Audio to transcribe music into sheet music
Computerized music transcription is the method of changing audio information like MP3 and WAV into sheet music, guitar tablature, and any format a musician might wish to be taught a tune on their instrument.
We’ll go over one of the best present instruments for doing this, which occur to be deep learning-based, and a novel method for it.
The present state-of-the-art for this job comes from Magenta, an open-source analysis venture developed by the now defunct (as of April 2023) Google Mind Group.
They launched a paper Sequence-to-Sequence Piano Transcription with Transformers in 2021 which used a T5-inspired transformer mannequin (just like “t5-small”) with 54 million parameters and the Maestro dataset, attaining nice outcomes. The issue is approached as a sequence-to-sequence job utilizing an encoder-decoder Transformer structure. The encoder processes mel spectrogram frames as enter and produces embeddings, whereas the decoder makes use of these embeddings through cross-attention to autoregressively generate a sequence of MIDI-like tokens. Their vocabulary consisted of 4 kinds of tokens:
- Notice tokens (128 values for MIDI pitches)
- Velocity tokens (128 values together with zero for note-off)
- Time tokens (6,000 values in 10ms bins for absolute timing)
- EOS token (to mark sequence finish)
See the picture beneath for a visualisation of the structure and an instance sequence of their customized MIDI tokens:
Our mannequin is a generic encoder-decoder Transformer structure the place every enter place accommodates a single spectrogram body and every output place accommodates an occasion from our MIDI-like vocabulary. Outputs tokens are autoregressively sampled from the decoder, at every step taking the token with most likelihood.
In 2022, they launched a paper, MT3: Multi-Job Multitrack Music Transcription. This experiment used the identical method because the final one however added further instrument tokens to signify the totally different devices. Once more, they used the same T5 mannequin and achieved nice efficiency towards lots of the datasets educated on, notably Slakh, Maestro and MusicNet.
MR-MT3 was launched the next 12 months as a slight enchancment to MT3.
Compute/GPU assets
Enormous assets had been wanted to coach this from scratch, regardless of being a lot smaller in measurement in comparison with even the smallest language fashions. The 2021 paper famous:
“We educated all fashions on 32 TPUv3 cores, leading to a per-core batch measurement of 8. Primarily based on validation set outcomes, overfitting didn’t appear to be an issue, so we allowed coaching to progress for 400K steps, which took about 2.5 days for our baseline fashions.”
The MT3 paper doesn’t present as particular particulars on coaching, stating they prepare for 1 million steps.
Different limitations
These fashions have some inherent limitations of their output flexibility. Whereas language fashions usually have giant vocabularies (usually 30,000+ tokens) which might be extensively pre-trained on various pure language knowledge, MT3 and related music transcription fashions use a a lot smaller, specialised token vocabulary (only some thousand tokens) centered solely on musical occasions. This specialisation implies that including new tokens, similar to for brand new devices or taking part in strategies like palm muting on guitars or pizzicato on violins, is probably going not simple — it requires vital retraining to combine these new tokens successfully with the prevailing vocabulary, and infrequently requires substantial coaching knowledge demonstrating these strategies. This differs from giant language fashions which may usually describe such musical nuances in pure language with out modification, as they’ve encountered these ideas throughout their broad pre-training.
Switch studying and zero-shot
We are able to leverage switch studying from giant open-source pre-trained audio and language fashions. Examples of music era fashions embrace OpenAI’s Jukebox and Meta’s MusicGen.
GPT-4o is designed to deal with textual content, audio and pictures “natively”. Though OpenAI has not launched the technical particulars on this, it’s assumed that some weights within the community will course of all modalities. It’s doable that the mannequin makes use of a decoder-only structure like language solely GPT fashions with out the necessity for encoder elements to transform totally different modalities to a dense illustration first. This design permits the mannequin to seamlessly course of and interpret inputs like textual content and pictures collectively, doubtlessly providing efficiency advantages each computationally and by way of mannequin understanding.
Many multi-modal fashions take an easier method paying homage to the encoder-decoder structure: they mix two pre-trained fashions — an encoder for the particular enter modality (like ViT for imaginative and prescient or an audio encoder for sound) and a Giant Language Mannequin (similar to LLaMA, Gemma, or Qwen). These fashions are related by projection layers that align their representations in a shared latent house, usually utilizing only a single linear layer. These projection layers be taught to transform the encoder’s output right into a format that matches the LLM’s anticipated enter dimensions and traits. The projection creates new embeddings/tokens from the enter modality that may then be injected into the LLM’s enter sequence. LLaVA is a first-rate instance of this structure for vision-language duties, whereas Spotify’s Llark and Qwen-Audio apply the identical precept utilizing audio encoders as an alternative of imaginative and prescient encoders.
Right here’s some pseudocode on how the fashions are stitched collectively:
# Extract options from remaining layer of audio encoder
# Form: [batch_size, audio_seq_len, encoder_dim=1024]
audio_features = audio_model(audio_input)# Undertaking audio options to match LLM's embedding dimension
# Form: [batch_size, audio_seq_len, llm_embed_dim=4096]
audio_embeddings = projection_layer(audio_features)
# Get textual content embeddings from LLM's embedding layer
# Form: [batch_size, text_seq_len, llm_embed_dim=4096]
text_embeddings = llm.embed_text(text_input)
# Concatenate alongside sequence size dimension
# Form: [batch_size, audio_seq_len + text_seq_len, llm_embed_dim=4096]
combined_input = concatenate([audio_embeddings, text_embeddings], dim=1)
# Feed them into the LLM as regular for era
output = llm(combined_input)
Overview of structure
Llark makes use of OpenAI’s Jukebox and Qwen2-Audio makes use of OpenAI’s Whisper for the audio towers. Jukebox is a music era mannequin however it may additionally absorb audio clips as enter and outputs a continuation of the audio clip. Whisper is used for transcribing voice to textual content.
Given their function, the selection of audio module is evident: Llark specialises in music evaluation, whereas Qwen2Audio primarily focuses on responding to voice directions with some primary audio and music evaluation capabilities.
Figuring out the optimum supply for extracting embeddings from giant pre-trained fashions includes analysis and experimentation. Moreover, deciding whether or not to fine-tune the whole module or freeze components of it’s a essential design selection. As an illustration, LlaVa’s coaching technique includes freezing the imaginative and prescient tower and specializing in fine-tuning the projection layer and language mannequin. We’ll go over this facet of every mannequin beneath.
Llark: why Jukebox? Are these embeddings one of the best as of September 2024?
Figuring out the optimum location to extract embeddings from giant fashions usually requires in depth probing. This includes testing numerous activations or extracted layers of the mannequin on totally different classification duties by a technique of trial and error. For music era fashions, this might embrace duties like style recognition, instrument detection, emotion detection, in addition to evaluation of harmonic buildings and temporal patterns. Many business embedding fashions (like OpenAI’s embedding fashions) are educated particularly for embedding era with specialised architectures and coaching goals, slightly than being fine-tuned variations of present language fashions.
The 2 largest publicly accessible music era and music continuation (i.e.: in a position to absorb audio as enter) fashions are Jukebox and MusicGen. MusicGen is newer and quicker, and subsequently appeared like it could be the apparent option to me. Nevertheless, in accordance with this paper on probing MusicGen, embeddings extracted from Jukebox seem to outperform MusicGen on common in classification duties. The findings from this paper led to the authors of Llark utilizing the next method for extracting embeddings:
- Embeddings are derived from the output of the thirty sixth layer of the Jukebox encoder following the method described in Castellon et al. (2021)
- Unique Jukebox encoding:
* 4800-dimensional vectors at 345Hz
* For a 25s clip: over 4.14 * 10⁷ floating-point values - The authors use a downsampling method: Imply-pooling inside 100ms frames, leading to:
* Downsampled frequency: 10Hz
* Embedding measurement: 1.2 × 10⁶ for a 25s audio clip. Which means a 2D array with form [240, 4800].
* Retains temporal data (not like Castellon et al. who common over the time dimension)
(The downsampled embedding measurement is roughly 6x bigger than CLIP ViT-L14 fashions utilized in many multimodal imaginative and prescient fashions)
Qwen2Audio: Whisper
The embedding extraction for Qwen2Audio isn’t talked about intimately within the paper. Whisper is an encoder-decoder structure the place the encoder generates deeply realized representations of the audio and the decoder decodes the representations to textual content (the transcription). In Qwen2Audio, it seems they extract embeddings from the ultimate layer of Whisper’s encoder, though they don’t point out whether or not they freeze it throughout coaching.
Pre-trained weights, coaching knowledge and datasets
Sadly Spotify has not offered any datasets or their educated mannequin weights to the general public, noting:
“With respect to inputs: the inputs to our mannequin are public, open-source, Inventive Commons-licensed audio and related annotations. Nevertheless, every particular person audio file can have its personal, doubtlessly extra restrictive license. Lots of the audio information embrace “no derivatives” licenses. We encourage customers of the datasets to familiarize themselves with the restrictions of those licenses; in an effort to honor such licenses, we don’t launch any derivatives from the coaching knowledge on this paper (together with query- response pairs or educated mannequin weights).”
They used the next datasets:
- MusicCaps (Agostinelli et al., 2023)
- YouTube8M-MusicTextClips (McKee et al., 2023)
- MusicNet (Thickstun et al., 2017)
- FMA (Defferrard et al., 2017)
- MTG-Jamendo (Bogdanov et al., 2019)
- MagnaTagATune (Legislation et al., 2009)
Llark particulars it’s coaching knowledge era course of within the following extract:
“We use variants of ChatGPT to extract the instruction- tuning knowledge for all experiments. Nevertheless, the precise language mannequin used varies by dataset. We choose the OpenAI mannequin as follows: We use GPT-4 for all reasoning duties. We discovered that GPT-4 was far more adept at following the advanced directions within the Reasoning job household. For datasets with greater than 25k samples, we restrict Reasoning knowledge to a random subsample of 25k tracks.”
This leads to Q&A knowledge like this:
The datasets used for coaching Qwen2Audio will not be shared both, however the educated mannequin is extensively accessible and likewise is applied within the transformers library:
For this venture, fine-tuning off a pre-trained Llark mannequin would have been optimum, given it’s reportedly good efficiency towards the analysis benchmarks Spotify said within the paper.
Nevertheless, given they didn’t launch the weights for it, it’s unfeasible to start out coaching a mannequin like this from scratch with out a good bit of experience and cash. Spotify educated it on:
Our mannequin is educated on 4 80GB NVIDIA A100 GPUs. Coaching takes roughly 54 hours.
This is able to value round $700 utilizing a supplier like LambdaLabs.
Due to the above, I went with Qwen. Nevertheless, Qwen2-Audio doesn’t carry out that properly throughout primary music duties like tempo and instrument detection. I element this beneath within the analysis part. Because of this the mannequin might be not giant sufficient or pre-trained sufficient to attain this job, however my hope is I may at the very least set a place to begin and framework for fine-tuning on this job sooner or later. As Alibaba state of their Qwen2-Audio weblog publish:
We additionally plan to construct bigger Qwen2-Audio fashions to discover the scaling legal guidelines of audio language fashions.
For my very own studying although, I did have a go at re-creating the mannequin utilizing torch and pre-trained fashions with the transformers library.
I additionally created datasets for Q&A knowledge and embeddings. I generated quick type Q&A knowledge for the URMP dataset, e.g.: “What’s the tempo of this monitor”, “What devices are taking part in on this audio”.
Right here’s a pocket book for operating Jukebox in a Colab surroundings to benefit from a budget T4 GPU’s. I uploaded each Q&A and embeddings datasets to HuggingFace right here.
Right here’s a pocket book with Llark replicated.
Transcription format
I selected ABC music notation because the output format that the language mannequin is anticipated to transcribe the music in. Right here’s an instance of it:
X:1
M:4/4
L:1/16
Okay:none
Q:67V:1 title="Electrical Bass (finger)"
%%octave-default C4
GAA^2E3A2<A^2 | D^D^2E2A2A^4 A^2E2 | A2A^4A^2E2 A2A^4 | A^2E2A2A^4A^2E2A2 |
A^4 A^2E2 A2A^4A^2 E2 | A2A^4 |
V:2 title="Brilliant Acoustic Piano"
%%octave-default C5
[E3C3][E3C3][E3C3] [E3C3][A^,2E2A^2] | [E3A^3][E3A^3][E3A^3][E3A^3][E3A^3] |
[E3A^3][E3A^3][E3A^3] [E3A^3][E3A^3] | [E3A^3][E3A^3][E3A^3][E3A^3][E3A^3] |
[E3A^3][E3A^3][E3A^3] [E3A^3][E3A^3] | [E3A^3] |
V:3 title="Electrical Guitar (jazz)"
%%octave-default C5
E'3C'3A^4E'3C'3 | A^4E'3 C'3A^4E'3C'3 | A^4 E'3C'3A^4 E'3C'3 | A^4E'3C'3A^4E'3C'3 |
A^4E'3C'3 A^4E'3C'3 | A^4 |
On this notation we’ve got the time signature and tempo outlined on the prime denoted by ‘M’ and ‘Q’. The ‘L’ signifies the default be aware size of the notation, on this case a sixteenth be aware, which is the norm. We then outline every instrument and the default octave they need to adhere to when writing the notes for every of them. Right here’s a abstract of the important thing syntactical factors for writing notes in ABC music notation:
- Notes are represented by letters A-G, with lowercase letters indicating greater octaves
- Sharps are denoted by ^ earlier than the be aware, flats by _
- Pure indicators are represented by =
- Notice size is indicated by numbers after the be aware (C2 is twice so long as C)
- Dotted notes use a . after the be aware (C. is a dotted quarter be aware)
- Rests are represented by z, with numbers for length (z2 is a half relaxation)
- Chords are enclosed in sq. brackets [CEG]
- Ties are proven with a hyphen –
- Bar strains are represented by |
- Damaged rhythms use > or < between notes (C>D means dotted-C eighth be aware adopted by D sixteenth be aware)
Why ABC?
The explanations for selecting this notation are:
- It’s a minimalist format for writing music
- It’s extensively used and standard; language fashions have already got good comprehension of ABC notation because of in depth pre-training on it.
- It’s versatile and might simply be prolonged to incorporate tempo adjustments, time signature adjustments, further taking part in kinds like talked about above, and so forth…
I transformed the MIDI information offered by the datasets to ABC notation utilizing this library. A pocket book for creating the datasets is right here.
To guage each the unique mannequin and every stage of fine-tuning I carried out thereafter, I randomly chosen 30 samples of various complexity from the URMP dataset and ran the mannequin 3 times on every pattern, manually analyzing all responses.
By means of handbook testing, I discovered the optimum decoding parameters to be a temperature of 0.7 and a top_p of 1.2. The utmost variety of tokens to return was capped at 2048. Adjusting the max appeared to have little distinction on efficiency.
The unique mannequin carried out poorly on this analysis set. Whereas it often predicted the tempo and devices accurately, it largely failed to take action. A textual content file with the analysis outcomes is offered right here.
Given this start line, it’s unlikely that we’ll see sturdy outcomes from this experiment with out a strong pre-trained mannequin. Nevertheless, the objective is to develop methods that may be utilized sooner or later as extra superior pre-trained fashions develop into accessible.
I first tried fine-tuning with primary cross-entropy loss. Supervised fine-tuning with cross-entropy loss is a fast strategy to begin instructing the mannequin however a primary loss perform like this has limitations as we are going to see beneath. The instinct behind this stage of coaching is that it could nudge the mannequin in the proper route and it could decide up any patterns or any customised ABC notation the dataset might have which the mannequin might not have seen earlier than.
Cross-entropy loss with trainer forcing
First, we educated it in a typical supervised fine-tuning method for language fashions. I used the SFTtrainer from the trl library for this, which makes use of cross-entropy loss with trainer forcing outlined step-by-step beneath:
- The mannequin predicts the subsequent token within the sequence.
- The loss is calculated based mostly on the distinction between the anticipated chances (logits) and the precise subsequent token.
- For the subsequent prediction, the mannequin is given the precise appropriate token (floor reality), slightly than its personal prediction. This is named trainer forcing, it helps stabilise coaching and considerably pace it up, particularly within the early phases.
The outcomes from this coaching section had been poor. It degraded the efficiency of the unique mannequin. The mannequin, which beforehand dealt with tempo and instrument recognition properly, now largely obtained these incorrect. It additionally started producing garbled textual content output with countless repetition. This occurred even when setting a low studying price, making use of gradient clipping, and utilizing low LoRA ranks to mitigate giant adjustments to the mannequin. Total, it appeared the mannequin was very delicate to the coaching utilized.
Nevertheless, whereas this coaching section might supply some enhancements, it received’t result in optimum efficiency as a result of limitations of our primary loss perform. This perform struggles to totally seize the mannequin’s efficiency nuances. For instance, when utilizing trainer forcing, instrument predictions can yield deceptively low loss throughout sure token sections. If an instrument title begins with “V”, the mannequin may confidently predict “Violin” or “Viola” based mostly on our dataset, no matter accuracy. Moreover, the loss perform might not precisely replicate near-misses, similar to predicting a tempo of 195 as an alternative of 200 — a small distinction that’s moderately correct however doubtlessly penalised closely depending on the distribution of chances amongst logits. It’s doable that neighbouring numbers even have excessive chances.
RLHF with PPO
Due to these limitations, we will create our personal customized loss perform that may extra precisely rating the response from the mannequin. That’s, given a predicted sequence from the mannequin, the loss perform may give it a rating between 0 and 1 on how good it’s.
Nevertheless, integrating this tradition loss perform into supervised fine-tuning presents a big problem. The difficulty stems from the non-linearity launched by the customized loss perform, which prevents the direct calculation of gradients. Let’s break this down:
In conventional SFT with cross-entropy loss:
- The mannequin outputs logits (uncooked scores) for every token in its vocabulary
- These logits straight signify the mannequin’s prediction chances
- The loss perform compares these chances to the bottom reality
- Gradients may be computed straight by this comparability
- The chain rule of calculus permits us to propagate these gradients again by the mannequin
With our customized loss perform:
- The mannequin should first generate full textual content output
- This era course of includes sampling from likelihood distributions
- Our loss perform then analyses this textual content output (checking tempo, notes, and so forth.)
- This creates a non-differentiable step between the mannequin’s logits and our loss calculation
- The sampling and textual content evaluation steps break the gradient chain wanted for backpropagation
To beat this, reinforcement studying strategies like Proximal Coverage Optimisation (PPO) may be employed. PPO is particularly designed to deal with non-differentiable loss capabilities and might optimise the mannequin by contemplating the whole coverage (the mannequin’s output distribution), slightly than counting on gradient data from logits.
Notice, there’s a lot of nice articles on right here explaining PPO!
The important thing perception of PPO is that as an alternative of attempting to straight backpropagate by the non-differentiable steps, it:
- Treats the mannequin’s outputs as actions in a reinforcement studying framework
- Makes use of the customized loss perform as a reward sign
- Updates the mannequin’s coverage (its likelihood distributions over tokens) to maximise anticipated reward
- Does this whereas guaranteeing the up to date coverage doesn’t deviate too removed from the present one
This method permits us to successfully prepare the mannequin with the customized loss perform, guaranteeing efficiency enhancements with out disrupting the core coaching dynamics. The PPO algorithm’s conservative replace technique helps keep stability throughout coaching, which is especially necessary when working with giant language fashions.
Normally, this scoring perform can be applied as a separate LLM within the type of a “reward mannequin” generally used when fine-tuning fashions through RLHF, which was a breakthrough first launched when ChatGPT got here out. As a result of nature of this job, we will manually write code to attain the responses, which makes use of fewer assets and is faster.
For time signature and tempo recognition that is simple to calculate. We extract all predicted objects with regex, for instance extracting the metre:
def extract_metre(self, abc_string):
return re.search(r'M:(S+)', abc_string).group(1)
The mannequin ought to be taught the syntax and construction we wish it to output within the SFT stage. If it outputs one thing that can trigger our regex to not discover something or error, we will simply skip that pattern, assuming it’s a small minority of the dataset.
We extract the anticipated tempo and write a perform that’s extra forgiving for small errors however penalises bigger errors extra closely:
- For small variations (≤10 BPM), it makes use of linear scaling.
- For bigger variations, it switches to exponential scaling.
- The ultimate loss is capped between 0 and 1.
Let’s break down the important thing elements of this tradition loss:
Code for the customized loss is right here
1. Metre Loss
The metre loss focuses on the time signature of the piece. It compares the anticipated metre with the bottom reality, contemplating each the numerator and denominator individually, in addition to their ratio. This method permits for a nuanced analysis that may deal with numerous time signatures precisely.
The metre loss makes use of a mixture of linear and exponential scaling to penalise variations. Small discrepancies end in a linear enhance in loss, whereas bigger variations result in an exponential enhance, capped at a most worth of 1.
2. Tempo Loss
Tempo loss evaluates the accuracy of the anticipated beats per minute (BPM). Just like the metre loss, it makes use of a mixture of linear and exponential scaling.
For small tempo variations (≤10 BPM), the perform applies linear scaling. Bigger variations set off exponential scaling, guaranteeing that vital tempo mismatches are penalised extra closely.
3. Pitch Loss
The pitch loss is probably essentially the most essential part, because it assesses the accuracy of the transcribed notes. This perform makes use of the Levenshtein distance to match the sequence of notes in every voice.
The pitch loss calculation accounts for a number of voices, matching every predicted voice to the closest floor reality voice. This method permits for flexibility in voice ordering whereas nonetheless sustaining accuracy within the total pitch content material.
4. Instrument Loss
The instrument loss evaluates the accuracy of instrument choice for every voice.
This perform considers actual matches, devices from the identical household, and makes use of string similarity for extra nuanced comparisons. It gives a complete evaluation of how properly the mannequin identifies and assigns devices to every voice.
5. Combining the Losses
The ultimate loss is a weighted mixture of those particular person elements:
total_loss = (0.5 * pitch_loss +
0.15 * metre_loss +
0.15 * tempo_loss +
0.2 * instrument_loss)
This weighting scheme prioritises pitch accuracy whereas nonetheless contemplating different necessary facets of music transcription.
PPO coaching typically requires much more reminiscence than SFT for a number of causes:
- A number of coverage evaluations — PPO wants to keep up each the present coverage (mannequin weights) and an “previous” coverage to compute the likelihood ratio between them. This successfully doubles the mannequin parameters in reminiscence.
- Expertise buffer — PPO shops a buffer of experiences (states, actions, rewards, and so forth.) to carry out updates in mini-batches. This buffer may be fairly giant and takes vital reminiscence.
- Benefit estimation — Computing benefits requires holding monitor of worth estimates and returns throughout trajectories, including one other layer of reminiscence overhead.
- Extra optimisation goals — PPO tracks a number of loss elements (coverage loss, worth loss, entropy bonus) and their gradients, whereas SFT has a single loss.
Due to the above, we’re extra restricted than SFT within the measurement of the fashions we will prepare and the way a lot it prices. Whereas the above coaching I may do on an A100 40GB in Colab, for the PPO coaching I wanted extra reminiscence. I educated on an H100 80GB, which may prepare a LoRA with a rank of 128 and a batch measurement of 8.
My hyperparameter sweep was slim, I went with what appeared most intuitive utilizing batch sizes starting from 1 to 16 and studying charges from 2e-5 to 2e-4.
The mannequin made no enhancements to the duty. The textual content file with the outcomes is right here.
I tracked numerous coaching metrics utilizing Weights & Biases (WandB). Key metrics included the coverage loss, worth loss, complete loss, KL divergence, and the reward mannequin’s rating.
For all hyperparameter runs, the logs no enchancment within the rewards and loss calculated over time. The KL divergence remained inside the pre-defined threshold.
