This repository contains the code to reproduce the core results from the paper Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data.

To cite this work, please use

@inproceedings{hsu2017learning,
  title={Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data},
  author={Hsu, Wei-Ning and Zhang, Yu and Glass, James},
  booktitle={Advances in Neural Information Processing Systems},
  year={2017},
}

This project uses Python 2.7.6. Before running the code, you have to install

• Tensorflow 1.0
• Scipy
• Numpy
• CFFI
• Soundfile
• Matplotlib
• Kaldi
• Kaldi-Python

The former 6 dependencies can be installed using pip by running

pip install -r requirements.txt

The last one requires Kaldi before a specific commit (d1e1e3b). If you don't have Kaldi before that version, you can install both Kaldi and Kaldi-Python by running

make all

The code structure follows Kaldi's convention. Scripts for each dataset can be found in egs/<dataset> folder. If you have questions, please write an email to wnhsu@csail.mit.edu

To reproduce the experiments for TIMIT, run:

# stage = -1 for data preparation; run only once.
cd egs/timit
./run_fhvae.sh --TIMIT_RAW_DATA <timit_raw_data_dir> --stage -1

Before running FHVAE experiments, you need to have the following Kaldi data directories of Aurora-4 prepared: <aurora4_kaldi_egs_dir>/data/{train_si84_clean, train_si84_multi, dev_0330, test_eval92}.

To reproduce the experiments for Aurora-4, run:

# stage = -1 for data preparation; run only once.
cd egs/aurora4 
./run_fhvae.sh --AURORA4_KALDI_EGS <aurora4_kaldi_egs_dir> --stage -1

We show the combination of 6 latent sequence variables and 6 latent segment variables. The latent sequence variable for each column is the same; likewise, the latent segment variable for each row is the same.

We show replacing S-vector of a target utterance with that of a reference utterance to alter the sequence-level attribute, which can be applied to speaker conversion or denoising. The FHVAE model used for following modifications is trained on Aurora-4 with alpha = 0