No support for multi-channel images

[adamjstewart]

Multi-channel images are common in the fields of satellite remote sensing (GIS) and in medical imaging. Many of the multispectral satellites I work with in my research have 8+ spectral bands, not just RGB. Some hyperspectral satellites have as many as 136 spectral bands. Microscopy often involves 4+ channel images as well.

Currently, torchvision relies on the Python Imaging Library (PIL) for all of its transforms. Unfortunately, pillow does not support multi-channel images: python-pillow/Pillow#3160, python-pillow/Pillow#1888

The way I see it, researchers such as myself have 3 options:

1. Fix PIL (unlikely, as Add support for high bit depth multichannel images python-pillow/Pillow#1888 has been stalled for 3 years now)
2. Fix torchvision (this would involve implementing all of the current transforms by hand using torch Tensors or numpy arrays)
3. Fork torchvision (write all of the transforms we need ourselves)

I'm about to resort to 3 for my research. Do you have any suggestions for users like me?

[sibocw]

I agree that is indeed a common problem in remote sensing, especially hyperspectral imaging.

[pmeier]

This is an ongoing discussion (although for some other reasons), which leans to option 2. Recently @fmassa stated:

[...] I'd rather not rush things just yet

Thus, if I were you, I would go with option 3. for now, but check back regularly for official support.

[jphdotam]

I would love for this to be a focus. Not only is it a problem for multi-channel data, but also it causes issue for target transformations for semantic segmentation where a channel is needed for each target.

[fmassa]

@jphdotam we are going to be following a slightly different approach for semantic segmentation, where we will be using torch operations to perform the transformations.

See https://github.com/pytorch/vision/blob/master/references/segmentation/transforms.py for an example

[fmassa]

By the way, in #1104 we are slowly going to be starting to support PyTorch Tensors for some of the transforms natively in torchvision, so that this won't be an issue anymore

[adamjstewart]

According to the v0.8.0 release notes, multi-channel images are now natively supported! Thanks so much for everyone's hard work!!!

[fmassa]

Thanks @adamjstewart !

Let us know if you find any issues with it

[adamjstewart]

I actually just started a new research project using drone-based hyperspectral imagery (128+ spectral bands), so I'll definitely be testing this feature to the extreme!

[adiren7]

@adamjstewart were you able to train your model with multispectral images ? if yes ,can you please share what did you change in the dataloader ?

[adamjstewart]

I actually pivoted away from torchvision for transforms and used Kornia instead. Kornia had better support for MSI at the time and supports both single samples and batches of images.

According to the torchvision docs:

Most transformations accept both PIL images and tensor images, although some transformations are PIL-only and some are tensor-only.

So it may be possible to do everything you need without ever leaving torchvision.

P.S. If you're working with satellite/plane/drone imagery, you may be interested in TorchGeo. If you're working with medical imagery I can't help you. Those are the two main applications of MSI that I know of.