---
title: Robust music signal separation based on supervised nonnegative matrix factorization with prevention of basis sharing
tags:  #nmf #source separation #music  
author: [Daichi Kitamura](https://www.docswell.com/user/d-kitamura)
site: [Docswell](https://www.docswell.com/)
thumbnail: https://bcdn.docswell.com/page/4L73ZW5Z75.jpg?width=480
description: Presented at IEEE International Symposium on Signal Processing and Information Technology (ISSPIT 2013) (international conference) Daichi Kitamura, Hiroshi Saruwatari, Kosuke Yagi, Kiyohiro Shikano, Yu Takahashi, Kazunobu Kondo, &quot;Robust music signal separation based on supervised nonnegative matrix factorization with prevention of basis sharing,&quot; Proceedings of IEEE International Symposium on Signal Processing and Information Technology (ISSPIT 2013), pp.392-397, Athens, Greece, December 2013.
published: March 18, 15
canonical: https://www.docswell.com/s/d-kitamura/K2QW95-2022-03-15-081857
---
# Page. 1

![Page Image](https://bcdn.docswell.com/page/4L73ZW5Z75.jpg)

IEEE International Symposium on Signal Processing and Information Technology
December 12-15, 2013 - Athens, Greece
Session T.B3: Speech – Audio - Music
Robust Music Signal Separation Based on
Supervised Nonnegative Matrix Factorization
with Prevention of Basis Sharing
Daichi Kitamura, Hiroshi Saruwatari,
Kosuke Yagi, Kiyohiro Shikano
（Nara Institute of Science and Technology, Japan）
Yu Takahashi, Kazunobu Kondo
（Yamaha Corporation, Japan）


# Page. 2

![Page Image](https://bcdn.docswell.com/page/V87DRKD4EG.jpg)

Outline
• 1. Research background
• 2. Conventional method
– Nonnegative matrix factorization
– Supervised nonnegative matrix factorization
– Problem of conventional method: basis sharing
• 3. Proposed method
– Penalized supervised nonnegative matrix factorization
• Orthogonality penalty
• Maximum-divergence penalty
• 4. Experiments
– Two-source case
– Four-source case
• 5. Conclusions
2


# Page. 3

![Page Image](https://bcdn.docswell.com/page/4VJPWKXVE8.jpg)

Outline
• 1. Research background
• 2. Conventional method
– Nonnegative matrix factorization
– Supervised nonnegative matrix factorization
– Problem of conventional method: basis sharing
• 3. Proposed method
– Penalized supervised nonnegative matrix factorization
• Orthogonality penalty
• Maximum-divergence penalty
• 4. Experiments
– Two-source case
– Four-source case
• 5. Conclusions
3


# Page. 4

![Page Image](https://bcdn.docswell.com/page/L2EV8VLREQ.jpg)

Background
• Sound signal separation
– decomposes target source from an observed mixed signal.
– Speech and noise, specific instrumental sound, etc.
Extract!
• Typical method for sound signal separation
Frequency
– is treated in the time-frequency domain.
Separation
First tone
Time
Spectrogram
Second tone
4


# Page. 5

![Page Image](https://bcdn.docswell.com/page/Z57G5LX67L.jpg)

Outline
• 1. Research background
• 2. Conventional method
– Nonnegative matrix factorization
– Supervised nonnegative matrix factorization
– Problem of conventional method: basis sharing
• 3. Proposed method
– Penalized supervised nonnegative matrix factorization
• Orthogonality penalty
• Maximum-divergence penalty
• 4. Experiments
– Two-source case
– Four-source case
• 5. Conclusions
5


# Page. 6

![Page Image](https://bcdn.docswell.com/page/V4EQZY82EP.jpg)

Nonnegative matrix factorization [Lee, et al., 2012]
• Nonnegative matrix factorization (NMF)
– is a sparse representation algorithm.
– can extract significant features from the observed matrix.
Frequency
Amplitude
Basis matrix
Activation matrix
(spectral patterns) (Time-varying gain)
Frequency
Observed matrix
(spectrogram)
Time
Amplitude
Time
Basis
Ω: Number of frequency bins
𝑇: Number of time frames
𝐾: Number of bases
• It is difficult to cluster the bases as specific sources.
6


# Page. 7

![Page Image](https://bcdn.docswell.com/page/DKJ43W2P71.jpg)

Supervised NMF (SNMF) [Smaragdis, et al., 2007]
• SNMF utilizes some sample sounds of the target.
– Construct the trained basis matrix of the target sound.
– Decompose into the target signal and other signal.
Training process
Ex. Musical scale
Sample sounds
of target signal
Supervised basis matrix
(spectral dictionary)
Optimize
Separation process
Mixed signal
Target signal
Fixed
Other signal
7


# Page. 8

![Page Image](https://bcdn.docswell.com/page/PLE11YMXEG.jpg)

Problem of SNMF
• Basis sharing problem in SNMF
– There is no constraint between and
.
– Other bases
may also have the target spectral patterns.
If
also have the target basis…
Target signal
Estimated
target signal
Estimated
other signals
– The estimated target signal loses some of the target signal.
– The cost function is only defined as the distance between
and
.
8


# Page. 9

![Page Image](https://bcdn.docswell.com/page/8GEW8G3KJ2.jpg)

Basis sharing problem: example of SNMF
Separated
by SNMF
Mixed signal
Only the target
signal (oracle)
9


# Page. 10

![Page Image](https://bcdn.docswell.com/page/W47Z8LVNE3.jpg)

Basis sharing problem: example of SNMF
Separated
by SNMF
Mixed signal
Only the target
signal (oracle)
10


# Page. 11

![Page Image](https://bcdn.docswell.com/page/KYJ6PW597V.jpg)

Basis sharing problem: example of SNMF
Separated
by SNMF
Separated signal
(estimated)
The estimated signal loses
some of the target components
because of the basis sharing
problem.
11


# Page. 12

![Page Image](https://bcdn.docswell.com/page/5GJ5KMYDE4.jpg)

Outline
• 1. Research background
• 2. Conventional method
– Nonnegative matrix factorization
– Supervised nonnegative matrix factorization
– Problem of conventional method: basis sharing
• 3. Proposed method
– Penalized supervised nonnegative matrix factorization
• Orthogonality penalty
• Maximum-divergence penalty
• 4. Experiments
– Two-source case
– Four-source case
• 5. Conclusions
12


# Page. 13

![Page Image](https://bcdn.docswell.com/page/V9E2W9YM7R.jpg)

Proposed method
• In SNMF, other basis matrix
may have the same
spectral patterns with supervised basis matrix .
Basis sharing problem
• Propose to make
as different as possible from
by introducing a penalty term in the cost function.
Penalized SNMF (PSNMF)
Mixed signal Fixed Target signal
Optimize
Other signal
as different as possible from
.
13


# Page. 14

![Page Image](https://bcdn.docswell.com/page/5D7YL4GPEM.jpg)

Decomposition model and cost function
Decomposition model:
Supervised basis matrix (fixed)
Cost function in SNMF:
Generalized divergence function:
-divergence [Eguchi, et al., 2001]
14


# Page. 15

![Page Image](https://bcdn.docswell.com/page/2VEN4YGM78.jpg)

Decomposition model and cost function
Decomposition model:
Supervised basis matrix (fixed)
Cost function in SNMF:
Cost function in PSNMF:
Introduce a penalty term
We propose two types of penalty terms. 15


# Page. 16

![Page Image](https://bcdn.docswell.com/page/GY79QPWW73.jpg)

Orthogonality penalty
• Orthogonality penalty is the optimization of
that
minimizes the inner product of matrices and
.
– If
includes the similar basis to
larger.
,
becomes
• All the bases are normalized as one.
• Introduce a weighting parameter
.
16


# Page. 17

![Page Image](https://bcdn.docswell.com/page/5G78GDNR7D.jpg)

Maximum-divergence penalty
• Maximum-divergence penalty is the optimization of
that maximizes the divergence between and .
– If
includes the similar basis to
becomes smaller.
, the divergence
• All the bases are normalized as one.
• Introduce a weighting parameter
and sensitivity
parameter .
17


# Page. 18

![Page Image](https://bcdn.docswell.com/page/ML7LGMP2JR.jpg)

Derivation of optimal variables in PSNMF
• Derive the optimal variables
.
• Auxiliary function method
– Optimization scheme that uses the upper bound function.
– Design the auxiliary function for
and
as
and
.
– Minimize the original cost functions by minimizing the
auxiliary functions indirectly.
18


# Page. 19

![Page Image](https://bcdn.docswell.com/page/54EMQY497W.jpg)

Derivation of optimal variables in PSNMF
where
• The second and third terms become convex or
concave function w.r.t. value.
– Convex: Jensen’s inequality
– Concave: tangent line inequality
19


# Page. 20

![Page Image](https://bcdn.docswell.com/page/ZPER89W979.jpg)

Derivation of optimal variables in PSNMF
: auxiliary variable
• Always becomes the convex function
– Convex: Jensen’s inequality
20


# Page. 21

![Page Image](https://bcdn.docswell.com/page/NP7X8QP3JX.jpg)

Derivation of optimal variables in PSNMF
• Auxiliary functions
and
are designed as
• The update rules for optimization are obtained by
, and
.
21


# Page. 22

![Page Image](https://bcdn.docswell.com/page/637KK9DN7D.jpg)

Update rules for optimization of PSNMF
• Update rules with orthogonality penalty
where,
22


# Page. 23

![Page Image](https://bcdn.docswell.com/page/4LJ3ZWRZJ5.jpg)

Update rules for optimization of PSNMF
• Update rules with maximum-divergence penalty
where,
23


# Page. 24

![Page Image](https://bcdn.docswell.com/page/V8JDRKY47G.jpg)

Outline
• 1. Research background
• 2. Conventional method
– Nonnegative matrix factorization
– Supervised nonnegative matrix factorization
– Problem of conventional method: basis sharing
• 3. Proposed method
– Penalized supervised nonnegative matrix factorization
• Orthogonality penalty
• Maximum-divergence penalty
• 4. Experiments
– Two-source case
– Four-source case
• 5. Conclusions
24


# Page. 25

![Page Image](https://bcdn.docswell.com/page/4VEPWK6V78.jpg)

Experimental conditions
• Produced four melodies using a MIDI synthesizer.
• Used the same MIDI sounds of the target instruments
containing two octave notes as a supervision sound.
Training sound
Two octave notes that cover all the notes of the target signal.
• Evaluation in two-source case and four-source case.
– There are 12 combinations in the two-source case, and 4
patterns in the four-source case.
25


# Page. 26

![Page Image](https://bcdn.docswell.com/page/L27V8V3R7Q.jpg)

Experimental conditions
Observed signal Mixed 2 or 4 signals as the same power
The same MIDI sounds of the target signal
Training signal
containing two octave notes
Divergence
All combinations of
criteria
Supervised bases : 100
Number of bases
Other bases : 50
Parameters
Experimentally determined
Methods
Conventional SNMF, Proposed PSNMF
• Evaluation scores [Vincent, 2006]
– Source-to-distortion ratio (SDR)
– SDR indicates the total quality of separated signal.
26


# Page. 27

![Page Image](https://bcdn.docswell.com/page/Z5JG5L36JL.jpg)

Experimental results: two-source-case
• Average scores of 12 combinations
16
16
14
14
PSNMF
(Max.)
8 Conv.
SNMF
6
PSNMF
(Max.)
14
12
12
10
10
8
6 Conv.
SNMF
6
4
2
2
2
0
0
0
1
2
PSNMF
(Ortho.)
Conv.
8 SNMF
4
0
PSNMF
(Max.)
16
SDR [dB]
SDR [dB]
10
PSNMF
(Ortho.)
SDR [dB]
12
PSNMF
(Ortho.)
4
0
1
2
0
1
2
– Conventional SNMF cannot achieve high separation
accuracy because of the basis sharing problem.
– Proposed method outperforms conventional SNMF.
27


# Page. 28

![Page Image](https://bcdn.docswell.com/page/V47QZY527P.jpg)

Experimental results: four-source-case
14
14
12
12
PSNMF
(Max.)
6
14
PSNMF
(Max.)
PSNMF
(Ortho.)
10
Conv.
8 SNMF
6
6
4
4
4
2
2
2
0
0
0
0
1
2
0
1
2
PSNMF
(Max.)
PSNMF
(Ortho.)
10
Conv.
8 SNMF
12
SDR [dB]
PSNMF
10 (Ortho.)
Conv.
8 SNMF
SDR [dB]
SDR [dB]
• Average scores of 4 combinations
0
1
2
– PSNMF outperforms the conventional method.
28


# Page. 29

![Page Image](https://bcdn.docswell.com/page/DKE43W9PJ1.jpg)

Example of separation (Cello &amp; Oboe)
Separated
by SNMF
Mixed signal
Separated
by PSNMF
(Ortho.)
Cello signal
29


# Page. 30

![Page Image](https://bcdn.docswell.com/page/PL711YKX7G.jpg)

Conclusions
• Conventional supervised NMF has a basis sharing
problem that degrades the separation performance.
• We propose to add a penalty term, which forces the
other bases to become uncorrelated with supervised
bases, in the cost function.
Penalized supervised NMF
• Penalized supervised NMF can achieve the high
separation accuracy.
Thank you for your attention!
30


