Simultaneously segmenting and classifying cell nuclei by using multi-task learning in multiplex immunohistochemical tissue microarray sections

Abstract

Quantitative analysis of tumor immune microenvironment (TIME) in immunohistochemical (IHC) tissue microarray (TMA) sections is crucial in diagnosis and treatment recommendations for cancer patients. Nuclei segmentation and classification are the prerequisites for the TIME quantification, but it still lacks of robust nuclear quantification models used for IHC histological slides. In this paper, we design an approach for simultaneously segmenting and classifying cell nuclei in multiplex IHC TMA sections. The large TMA tissue core is first divided into a set of small overlapping patches, where cell nuclei are then simultaneously segmented and classified by using our multi-task learning model. The model has one feature encoder with cascaded separable-ResUnit blocks, and three decoder branches that incorporate the Self-Attention modules and DenseUnit blocks to perform nuclear segmentation, classification and distance map regression, respectively. After processing all patches, the weighted loss map and vote mechanism are applied to seamlessly stitch patch-level predictions to form the tissue core level results. We finally exploit generalized Laplacian of Gaussian (gLoG) filters based algorithm to post-process segmentation results to further split overlapping cell nuclei. Quantitative evaluations have been performed on a IHC stained histological image dataset with 9725 manually identified cell nuclei and a public H&E stained dataset (CoNSep), which show that our model outperforms state-of-the-art nuclei segmentation and classification models. The qualitative evaluations on TMA sections show the potential of using our approach in clinical applications.