PitchSTAR: Self-Supervised Pitch Style Transfer with Complementary Temporal and Time-Agnostic Style Representations

Paper Code (Upon Acceptance)

This is an accompanying page for the paper “PitchSTAR: Self-Supervised Pitch Style Transfer with Complementary Temporal and Time-Agnostic Style Representations”, currently under review. PitchSTAR is a self-supervised framework for arbitrary pitch style transfer (PST). The PST task is defined as generating a pitch curve given a reference ornamented pitch (style) and notes (content), represented in the Figure 1.

Figure 1. The (Pitch Style Transfer) PST task.

PitchSTAR is a flow-matching framework for pitch style transfer. It models a full style reference through note-relative pitch modulation and temporal and time-agnostic branches, which capture phrase-level organization and ornament details and limit trivial copying in a self-supervised training setting, as shown on Figure 2.

Figure 2. Training scheme of PitchSTAR. Two optimization steps at low and high noise levels are shown with highlighted sharp and blurred cross-attention matrices, respectively.

Sound Samples

In-Domain

Sample 1

Sample 2

Sample 3

Sample 4

Out-of-Domain

Sample 1

Sample 2

Sample 3

Sample 4

Effect of CFG

In this section we show the effect of the guidance CFG scale, which balances guided by style and unguided generation.

Sample 1

Sample 2

Pitch Style Classifier

In this section we plot the results of the trained Pitch Ornament Classifier, through its confusion matrices.

Test Data

This confusion matrix is obtained on the test set of the GTSinger.

Figure 3. Confusion Matrix of the Pitch Style Classifier on the test set.

Pitch Style Transfer Models

These are the matrices obtained applying on the transfer of each of the models.

PitchSTAR

PitchSTAR w/o Flow

StylePitcher w/ Mod

StylePitcher

Figure 3. Confusion Matrix of the Pitch Style Classifier on the test set.

Synthetic Dataset Samples

Four examples of the synthetic ornament dataset are shown. Two with ornament and two without ornament.

With Ornament 1

Figure 8. Synthetic notes and their corresponding ornamented pitch.

With Ornament 2

Figure 9. Synthetic notes and their corresponding ornamented pitch.

Without Ornament 1

Figure 10. Synthetic notes and their corresponding non-ornamented pitch.

Without Ornament 2

Figure 11. Synthetic notes and their corresponding non-ornamented pitch.