Voice2Series-Reprogramming

Voice2Series: Reprogramming / Prompting Acoustic Models for Time Series Classification

• Paper | Colab Demo | Video | Slides

• We provide an end-to-end approach (Repro. layer) to reprogram on time series data on raw waveform with a differential mel-spectrogram layer from kapre.

• No offiline acoustic feature extraction and all layers are differentiable.

• Pytorch version of reprogram layer could be found out in ICASSP 23 Music Reprogramming.

• updated: if you have used the ECG 200 dataset in this code, please git pull and refer to the issue for one reported label loading error. (has been fixed)

Environment

Tensorflow 2.2 (CUDA=10.0) and Kapre 0.2.0.

• PyTorch noted: Echo to many interests from the community, we will also provide Pytorch V2S layers and frameworks, incoperating the new torch audio layers. Feel free to email the authors for further reprogramming collaboration.

• option 1 (from yml)

conda env create -f V2S.yml

• option 2 (from clean python 3.6)

pip install tensorflow-gpu==2.1.0
pip install kapre==0.2.0
pip install h5py==2.10.0
pip install pyts

Training

• Random Mapping

Please also check the paper for actual validation details. Many Thanks!

python v2s_main.py --dataset 0 --eps 20 --mod 2 --seg 18 --mapping 1

• Result

Epoch 14/20
3601/3601 [==============================] - 4s 1ms/sample - l