Vision Transformer for the Categorization of Breast Cancer from H&E Histopathology Images

Breast Cancer (BC) is the most frequent form of cancer, accounting for 24.5% of all cancer cases worldwide, with projections estimating 364,000 cases by 2040. Accurate diagnosis and effective categorization of BC are essential for proper treatment planning, patient management, and improved survival. Traditionally, pathologists examine histopathology specimens manually using a microscope to categorize the BC, which is labor-intensive, time-consuming, prone to subjectivity and constrained by experts’ availability. An automated approach can address these limitations; however, previous methods, particularly those based on Convolutional Neural Networks (CNNs), often struggle with data imbalance, poor accuracy and poor generalizability across datasets, especially in multiclass BC categorization. This study presents an automated BC categorization method leveraging whole slide histopathology images and a transformer-based deep learning model. The proposed method uses a cascade of transformers to classify BC using 40× histopathology images, following the taxonomy defined by the BRACS dataset, distinguishing between benign, atypical, and malignant cases. First, it classifies BC into three primary categories—benign, atypical and malignant—and subsequently determines the specific sub-types within each category. The proposed method was validated using two widely recognized datasets: BRACS and BreakHis. On BRACS, it achieved 95.6% accuracy in classifying BC into benign, atypical, and malignant categories, with sub-type accuracies of 94.7% for benign, 98.6% for atypical, and 99.1% for malignant cases. On the BreakHis dataset, the model achieved 93% accuracy for binary benign-malignant classification, with sub-type accuracies of 94% and 91% for benign and malignant cases, respectively. The proposed method outperformed existing methods in accuracy and robustness, making it a promising tool for automated BC diagnosis and classification.