---
title: Regularized superresolution-based binaural signal separation with nonnegative matrix factorization
tags:  #nmf #source separation #music #direction of arrivals  
author: [Daichi Kitamura](https://www.docswell.com/user/d-kitamura)
site: [Docswell](https://www.docswell.com/)
thumbnail: https://bcdn.docswell.com/page/NP7X8QQZJX.jpg?width=480
description: Presented at 5th International Conference on 3D Systems and Applications (3DSA 2013) (international conference) Daichi Kitamura, Hiroshi Saruwatari, Kiyohiro Shikano, Kazunobu Kondo, Yu Takahashi, &quot;Regularized superresolution-based binaural signal separation with nonnegative matrix factorization,&quot; Proceedings of 5th International Conference on 3D Systems and Applications (3DSA 2013), S10-4, Osaka, Japan, June 2013.
published: March 18, 15
canonical: https://www.docswell.com/s/d-kitamura/K37L85-2022-03-15-082504
---
# Page. 1

![Page Image](https://bcdn.docswell.com/page/NP7X8QQZJX.jpg)

Regularized Superresolution-Based
Binaural Signal Separation
with Nonnegative Matrix Factorization
Daichi Kitamura, Hiroshi Saruwatari,
Yusuke Iwao, Kiyohiro Shikano
(Nara Institute of Science and Technology, Nara, Japan)
Kazunobu Kondo, Yu Takahashi
(Yamaha Corporation Research &amp; Development Center, Shizuoka, Japan)


# Page. 2

![Page Image](https://bcdn.docswell.com/page/637KK99V7D.jpg)

Outline
• 1. Research background
• 2. Conventional method
–
–
–
–
Nonnegative matrix factorization
Penalized supervised nonnegative matrix factorization
Directional clustering
Hybrid method
• 3. Proposed method
– Regularized superresolution-based nonnegative matrix
factorization
• 4. Experiments
• 5. Conclusions
2


# Page. 3

![Page Image](https://bcdn.docswell.com/page/4LJ3ZW9QJ5.jpg)

Outline
• 1. Research background
• 2. Conventional method
–
–
–
–
Nonnegative matrix factorization
Penalized supervised nonnegative matrix factorization
Directional clustering
Hybrid method
• 3. Proposed method
– Regularized superresolution-based nonnegative matrix
factorization
• 4. Experiments
• 5. Conclusions
3


# Page. 4

![Page Image](https://bcdn.docswell.com/page/V8JDRKGW7G.jpg)

Background
• Music signal separation technologies have received much
attention.
Applications
• Automatic music transcription
• 3D audio system, etc.
• Music signal separation based on nonnegative matrix
factorization (NMF) has been a very active area of the
research.
• The extraction performance of NMF markedly degrades for the
case of many source mixtures.
We propose a new method for multichannel signal
separation with NMF utilizing both spectral and spatial
cues included in mixtures of multiple instruments.
4


# Page. 5

![Page Image](https://bcdn.docswell.com/page/4VEPWK3X78.jpg)

Outline
• 1. Research background
• 2. Conventional method
–
–
–
–
Nonnegative matrix factorization
Penalized supervised nonnegative matrix factorization
Directional clustering
Hybrid method
• 3. Proposed method
– Regularized superresolution-based nonnegative matrix
factorization
• 4. Experiments
• 5. Conclusions
5


# Page. 6

![Page Image](https://bcdn.docswell.com/page/L27V8V437Q.jpg)

NMF
Frequency
Frequency
Amplitude
• NMF is a type of sparse representation algorithm that
decomposes a nonnegative matrix into two nonnegative
matrices. [D. D. Lee, et al., 2001]
Time
Observed matrix
(Spectrogram)
𝒀: Observed matrix
𝑭: Basis matrix
𝑮: Activation matrix
Time
Amplitude
Activation matrix
(Time-varying gain)
Basis matrix
(Spectral bases)
Ω: Number of frequency bins
𝑇: Number of frames
𝐾: Number of bases
6


# Page. 7

![Page Image](https://bcdn.docswell.com/page/Z5JG5L1YJL.jpg)

Penalized Supervised NMF (PSNMF)
• In PSNMF, the following decomposition is addressed under
the condition that is known in advance. [Yagi, et al., 2012]
Training process
Supervised bases
of the target sound
Supervision sound
Separation process
Fix trained bases
and update
.
Update
is forced to become uncorrelated with
7


# Page. 8

![Page Image](https://bcdn.docswell.com/page/V47QZYD67P.jpg)

Penalized Supervised NMF (PSNMF)
• In PSNMF, the following decomposition is addressed under
the condition that is known in advance. [Yagi, et al., 2012]
Training process
Supervised bases
of the target sound
Supervision sound
Separation process
Fix trained bases
and update
Update
Problem of PSNMF:
When
the
signal
includes
many
sources,
is forced to become uncorrelated with
the extraction performance markedly degrades.
.
8


# Page. 9

![Page Image](https://bcdn.docswell.com/page/DKE43WZMJ1.jpg)

Directional Clustering
• Directional clustering can estimate sources and their direction
in multichannel signal. [Araki, et al., 2007] [Miyabe, et al., 2009]
• This method can separate sources with spatial information in
an observed signal.
L
R
L-ch input signal
：Source component
：Centroid vector
R-ch input signal
9


# Page. 10

![Page Image](https://bcdn.docswell.com/page/PL711YRY7G.jpg)

Directional Clustering
• Directional clustering can estimate sources and their direction
in multichannel signal. [Araki, et al., 2007] [Miyabe, et al., 2009]
• This method can separate sources with spatial information in
an observed signal.
L
R
L-ch input signal
：Source component
：Centroid vector
R-ch input signal
Problem of directional clustering:
This method cannot separate sources in the same direction.
10


# Page. 11

![Page Image](https://bcdn.docswell.com/page/8G7W8G11E2.jpg)

Hybrid method
• Conventional hybrid method utilizes PSNMF after the
directional clustering. [Iwao, et al., 2012]
• This method consists of two techniques.
– Directional clustering
– PSNMF
L
R
Spatial
separation
Source
separation
Directional
clustering
PSNMF
Conventional Hybrid method
11


# Page. 12

![Page Image](https://bcdn.docswell.com/page/W4JZ8LPX73.jpg)

Problem of hybrid method
1 0 0 0 0 0 0
0 1 1 0 0 1 1
1 0 0 0 0 0 0
0 1 0 1 1 0 1
Frequency
Frequency
Frequency
• The signal extracted by the hybrid method suffers from the
generation of considerable distortion due to the binary
masking in directional clustering.
• The signal in the target direction, which is obtained by
directional clustering, has many spectral chasms.
• The resolution of the spectrogram is degraded.
Directional Clustering
Input spectrogram Binary mask
Separated cluster
1 0 0 0 0 0 0
1 1 1 0 1 1 0
Time
Time
: Target direction Time
: Other direction
：Hadamard product (product of each element)
12


# Page. 13

![Page Image](https://bcdn.docswell.com/page/KYE6PWMPJV.jpg)

Outline
• 1. Research background
• 2. Conventional method
–
–
–
–
Nonnegative matrix factorization
Penalized supervised nonnegative matrix factorization
Directional clustering
Hybrid method
• 3. Proposed method
– Regularized superresolution-based nonnegative matrix
factorization
• 4. Experiments
• 5. Conclusions
13


# Page. 14

![Page Image](https://bcdn.docswell.com/page/5GE5KMZP74.jpg)

Proposed hybrid method
Conventional
hybrid method
Proposed
hybrid method
Input stereo signal
L-ch
Input stereo signal
R-ch
L-ch
R-ch
STFT
STFT
Directional clustering
Directional clustering
Center component
L-ch
R-ch
PSNMF
PSNMF
ISTFT
ISTFT
Mixing
Extracted signal
Index of
center cluster
Center component
L-ch
R-ch
Superresolutionbased SNMF
Superresolutionbased SNMF
ISTFT
ISTFT
Mixing
Extracted signal
Employ a new supervised NMF algorithm as an alternative
to the conventional PSNMF in the hybrid method.
14


# Page. 15

![Page Image](https://bcdn.docswell.com/page/V972W9RZJR.jpg)

Regularized superresolution-based NMF
• In proposed supervised NMF, the spectral chasms are treated
as unseen observations using index matrix. Chasms
Frequency
Separated cluster
: Chasms
Time
Frequency
Index matrix
Treat chasms as
unseen observations.
1 0 0 0 0 0 0
0 1 1 0 0 1 1
1 0 0 0 0 0 0
0 1 0 1 1 0 1
1 0 0 0 0 0 0
1 1 1 0 1 1 0
Time
15


# Page. 16

![Page Image](https://bcdn.docswell.com/page/5DJYL4DN7M.jpg)

Regularized superresolution-based NMF
: Chasms
Time
Superresolution
using supervised
bases
Supervised
bases
Frequency
Frequency
• The spectrogram of the target sound is reconstructed using
more matched bases because chasms are treated as unseen.
• The components of the target sound lost after directional
clustering can be extrapolated using supervised bases.
Separated cluster
Reconstructed spectrogram
Time
16


# Page. 17

![Page Image](https://bcdn.docswell.com/page/2V7N4Y6VJ8.jpg)

Regularized superresolution-based NMF
• Signal flow of the proposed hybrid method
Frequency of
source component
Target source
(a) Observed
spectra
Left
Center
Direction
Right
17


# Page. 18

![Page Image](https://bcdn.docswell.com/page/GYJ9QPLVJ3.jpg)

Regularized superresolution-based NMF
• Signal flow of the proposed hybrid method
Frequency of
source component
Target source
(a) Observed
spectra
Left
Frequency of
source component
Target direction
Center
Direction
(b) After
directional
clustering
Left
Right
Directional
clustering
Center sources lose some
of their
z components
Center
Direction
Right
18


# Page. 19

![Page Image](https://bcdn.docswell.com/page/5GJ8GDX4JD.jpg)

Regularized superresolution-based NMF
Frequency of
source component
• Signal flow of the proposed hybrid method
(b) After
directional
clustering
Left
Center sources lose some
of their
z components
Center
Direction
Right
19


# Page. 20

![Page Image](https://bcdn.docswell.com/page/MLJLGM8VER.jpg)

Regularized superresolution-based NMF
Frequency of
source component
• Signal flow of the proposed hybrid method
(b) After
directional
clustering
Frequency of
source component
Left
Center sources lose some
of their
z components
Center
Direction
(c) After
superresolutionbased SNMF
Left
Center
Direction
Right
Superresolutionbased NMF
Extrapolated
target source
Right
20


# Page. 21

![Page Image](https://bcdn.docswell.com/page/547MQY6NJW.jpg)

Regularized superresolution-based NMF
• The basis extrapolation includes an underlying problem.
• If the time-frequency spectra are almost unseen in the
spectrogram, which means that the indexes are almost zero, a
large extrapolation error may occur.
• It is necessary to regularize the extrapolation.
4
Frequency [kHz]
Frequency
Separated cluster
Extrapolation error
(incorrectly modifying the activation)
Time
Almost unseen frame
3
2
1
0
0
1
2
3
Time [s]
4
21


# Page. 22

![Page Image](https://bcdn.docswell.com/page/ZP7R89P4J9.jpg)

Regularized superresolution-based NMF
• We propose two types of regularizations.
Regularization of the temporal continuity
Previous
frame
Regularization of the norm minimization
𝑰 : Index matrix
𝑖𝜔,𝑡 : Entry of index matrix 𝑰
𝑓𝜔,𝑘 : Entry of matrix 𝑭
𝑔𝑘,𝑡 : Entry of matrix 𝑮
∙ҧ : Binary complement
The intensity of these regularizations are proportional to the
number of chasms in each frame.
22


# Page. 23

![Page Image](https://bcdn.docswell.com/page/NPJX8Q3ZEX.jpg)

Regularized superresolution-based NMF
• The cost function in regularized superresolution-based NMF is
defined using the index matrix as
: Regularization term
: Weighting parameter
: Penalty term to force and
to
become uncorrelated with each other
23


# Page. 24

![Page Image](https://bcdn.docswell.com/page/63JKK9YVJD.jpg)

Regularized superresolution-based NMF
• The update rules that minimize the cost function are obtained
as follows:
24


# Page. 25

![Page Image](https://bcdn.docswell.com/page/4LE3ZWVQE5.jpg)

Outline
• 1. Research background
• 2. Conventional method
–
–
–
–
Nonnegative matrix factorization
Penalized supervised nonnegative matrix factorization
Directional clustering
Hybrid method
• 3. Proposed method
– Regularized superresolution-based nonnegative matrix
factorization
• 4. Experiments
• 5. Conclusions
25


# Page. 26

![Page Image](https://bcdn.docswell.com/page/V8EDRK8WJG.jpg)

Evaluation experiment
• We compared four methods.
– Conventional hybrid method using PSNMF (Conventional method)
– Proposed hybrid method using superresolution-based NMF without
regularization (Proposed method 1)
– Proposed hybrid method using superresolution-based NMF with
regularization of the temporal continuity (Proposed method 2)
– Proposed hybrid method using superresolution-based NMF with
regularization of the norm minimization (Proposed method 3)
Input stereo signal
Input stereo signal
L-ch
L-ch
R-ch
R-ch
STFT
STFT
Directional clustering
Directional clustering
Center component
Index of
center cluster
Center component
L-ch
PSNMF
R-ch
PSNMF
L-ch
Superresolutionbased SNMF
R-ch
Superresolutionbased SNMF
ISTFT
ISTFT
ISTFT
ISTFT
Mixing
Mixing
Extracted signal
Extracted signal
26


# Page. 27

![Page Image](https://bcdn.docswell.com/page/4V7PWK8XJ8.jpg)

Evaluation experiment
• We used stereo-panning signals (
) and binauralrecorded signals (
) containing four instruments, Ob.,
Fl., Tb., and Pf., generated by MIDI synthesizer.
• The sources are mixed as the same power.
• Target source is always located in the center direction (no.1).
• We used the same type of MIDI sounds of the target
instruments as supervision for training process.
Target source
Left
Center
２
４
１
Right
３
Supervision
sound
Two octave notes that cover all notes of the target signal
27


# Page. 28

![Page Image](https://bcdn.docswell.com/page/L2JV8VN3JQ.jpg)

Experimental results (panning signal)
• Average SDR, SIR, and SAR scores for each method, where the 4
instruments are shuffled with 12 combinations.
SDR ：quality of the separated target sound
SIR ：degree of separation between the target and other sounds
SAR ：absence of artificial distortion
Bad
SDR
24
10
20
8
16
6
4
12
8
SIR
10
SAR
8
SAR [dB]
12
SIR [dB]
SDR [dB]
Good
6
4
2
4
2
0
0
0
Proposed method 1 ：no regularization
Proposed method 2 ：regularization of temporal continuity
Proposed method 3 ：regularization of norm minimization
28


# Page. 29

![Page Image](https://bcdn.docswell.com/page/Z5EG5LKYEL.jpg)

Experimental results (binaural signal)
• Average SDR, SIR, and SAR scores for each method, where the 4
instruments are shuffled with 12 combinations.
SDR ：quality of the separated target sound
SIR ：degree of separation between the target and other sounds
SAR ：absence of artificial distortion
10
SDR
20
SIR [dB]
SDR [dB]
Bad
4
6
12
8
SAR
5
16
8
6
SIR
SAR [dB]
Good
4
3
2
2
4
1
0
0
0
Proposed method 1 ：no regularization
Proposed method 2 ：regularization of temporal continuity
Proposed method 3 ：regularization of norm minimization
29


# Page. 30

![Page Image](https://bcdn.docswell.com/page/V4JQZYN6JP.jpg)

Conclusions
• We propose a new supervised NMF algorithm, which is
superresolution-based method, for the hybrid method to
separate stereo or binaural signals.
• The proposed hybrid method can separate the target signal
with high performance compared with conventional method.
• The regularization of norm minimization is effective for the
proposed supervised NMF algorithm.
Thank you for your attention!
30


