MultiMix

Ayaan Haque1, Abdullah-Al-Zubaer Imran2,3, Adam Wang2, Demetri Terzopoulos3,4
1Saratoga High School, 2Stanford University, 3University of California, Los Angeles, 4VoxelCloud Inc.
Conference Paper Journal Paper arXiv Blog Poster Slides Code

Overview

Our proposed model performs joint semi-supervised classification and segmentation by employing a confidence-based augmentation strategy for semi-supervised classification along with a novel saliency bridge module that guides segmentation and provides explainability for the joint tasks.

Our paper "MultiMix: Sparingly Supervised, Extreme Multitask Learning From Medical Images" was accepted to IEEE ISBI 2021 as a full paper. The paper was presented in April and is availabe in the conference proceedings. The extension of our ISBI paper, "Generalized Multi-Task Learning from Substantially Unlabeled Multi-Source Medical Image Data" was accepted to the MELBA Journal. Here are some photos of us presenting as ISBI.

Methods

  • Our model performs joint classification and segmentation using semi-supervised and multi-task learning.
  • For semi-supervised classification we use pseudo-label and consistency augmentation. An unlabeled image is augmented twice, weakly and strongly, and pseudo-labels are produced for the weakly augmented versions with confidence.
  • For segmentation, we propose the novel saliency bridge, which uses saliency maps from the encoder to guide segmentation in the decoder.
  • Our model architecture uses the U-Net and incorporates a classification branch after the encoder. In the bottleneck, we generate saliency maps and concatenate them in the decoder.

    • Visualizations

    Segmentation Visualizations prove the effectiveness of our model in multiple domains over baselines.

    Graphs and Metrics

    For classification and segmentation, metrics and graphs demonstrate the superiority of MultiMix against baselines.

    ISBI Poster

    Journal BibTeX

    @article{melba:2021:011:haque,
      title = "Generalized Multi-Task Learning from Substantially Unlabeled Multi-Source Medical Image Data",
      authors = "Haque, Ayaan and Imran, Abdullah-Al-Zubaer and Wang, Adam and Terzopoulos, Demetri",
      journal = "Machine Learning for Biomedical Imaging",
      volume = "1",
      issue = "October 2021 issue",
      year = "2021"
  }

    Conference BibTeX

    @inproceedings{haque2021multimix,
      title={Multimix: Sparingly-Supervised, Extreme Multitask Learning from Medical Images},
      author={Haque, Ayaan and Wang, Adam and Terzopoulos, Demetri and others},
      booktitle={2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI)},
      pages={693--696},
      year={2021},
      organization={IEEE}
}