SPRT-TANDEM

This repository contains the SPRT-TANDEM model proposed in "Sequential Density Ratio Estimation for Simultaneous Optimization of Speed and Accuracy" [ICLR2021 Spotlight] to perform early classification of time-series. The SPRT-TANDEM sequentially estimates log-likelihood ratios of two hypotheses, or classes, for fast and accurate sequential data classification.

Requirements

• Python 3.5
• TensorFlow 2.0.0
• CUDA 10.0
• cuDNN 7.6.0
• (For reproduction: Jupyter Notebook)
• Optuna 0.14.0

Reproduction and Quick Trial

1. Make Nosaic MNIST (make_nmnist.py): Modify user-defined parameters and python make_nmnist.py.
2. Save bottleneck features of Nosaic MNIST (save_featureTFR_nmnist.ipynb): Modify user-defined parameters and run all the cells.
3. Plot the speed-accuracy tradeoff (SAT) curve (plot_SAT_curve.ipynb): Modify user-defined parameters and run all the cells.

Result images

Files and Directories

• train_fe_nmnist.py

  • training script for feature extractor (ResNet) on Nosaic MNIST
  • Modify config_fe.yaml as you see fit and python train_fe_nmnist.py in /SPRT-TANDEM.

• train_ti_nmnist.py

  • training script for temporal integrator (LSTM) on Nosaic MNIST
  • Modify config_ti.yaml as you see fit and python train_ti_nmnist.py in /SPRT-TANDEM.

• demo_nmnist.ipynb

  • Nosaic MNIST is like this. Have a look!
  • See also the image below.

• make_nmnist.py

  • for reproduction
  • You can easily make Nosaic MNIST dataset.

• save_featureTFR_nmnist.ipynb

  • for reproduction
  • extracts bottleneck features and save them to TFRecords

• plot_SAT_curve.ipynb

  • for reproduction
  • for evaluation
  • draws the speed-accuracy tradeoff (SAT) curve and log-likelihood ratio trajectories of a model

• example_ckpts/

  • Pre-trained models for reproduction and quick trial
  • Our trained models, namely, ResNet and two LSTMs, are here.

• data-directory/

  • Nosaic MNIST will be saved here by default parameters of make_nmnist.py.

• configs/config_fe_nmnist.yaml

  • config fiel for train_fe_nmnist.py

• configs/config_ti_nmnist.yaml

  • config file for train_ti_nmnist.py

• datasets/data_processing.py

  • data loader and data processing functions

• models/backbones_fe.py

  • ResNet v1 and v2 models

• models/backbones_ti.py

  • LSTM model

• models/losses.py

  • loss functions and gradient computation.
  • LLLR is in binary_llr_loss_func.

• models/optimizers.py

  • functions to switch optimizers

• utils/misc.py

  • miscellaneous minor functions

• utils/performance_metrics.py

  • performance metrics such as balanced accuracy
  • the SPRT algorithm is in binary_truncated_sprt.

• utils/util_ckpt.py

  • utility functions to save checkpoints (models)

• utils/util_optuna.py

  • utility functions for Optuna

• utils/util_tensorboard.py

  • TensorBoard logger class

Nosaic MNIST Example

Nosaic (Noise + mOSAIC) MNIST is suitable for early classification of time series. You can generate Nosaic MNIST running make_nmnist.py.

An example video

ICLR2021 Poster

Citation

Please cite our paper if you use the whole or a part of our codes.

Bibtex:

@inproceedings{
  ebihara2021sequential,
  title={Sequential Density Ratio Estimation for Simultaneous Optimization of Speed and Accuracy},
  author={Akinori F Ebihara and Taiki Miyagawa and Kazuyuki Sakurai and Hitoshi Imaoka},
  booktitle={International Conference on Learning Representations},
  year={2021},
  url={https://openreview.net/forum?id=Rhsu5qD36cL}
}