Breast cancer heterogeneity investigation: Multiple k-means clustering approach

Breast cancer, a frequent cause of mortality among women, is a heterogeneous disease with multiple subtypes, which identification has a great impact on the prognosis and the choice of therapy. Various attempts are being made worldwide to classify breast tumors and to discover differences between subtypes. The purpose of this study is to divide a diverse group of patients with a multiple k-means clustering approach in order to investigate the complex structure of the dataset. Primary tumor samples collected from 859 females within The Cancer Genome Atlas Breast Invasive Carcinoma project served as the material for an RNA-Seq experiment. Expression levels measured with the RNA-Seq procedure were downloaded from the Genomic Data Commons Data Portal and used for this study. The proposed multilevel clustering algorithm is based on the idea of treating each resulting cluster as an input for a new k-means clustering as long as a given cluster is not too small or integrated. Each clustering is preceded with feature filtration and selection, which are based on the Gaussian Mixture Model decomposition of gene expression level amplitude or variance, respectively. The gap statistic is used for the selection of the optimal number of clusters as well as for deciding whether a group should be divided. Results were compared to the subtype labels either obtained from the PAM50 classifier or based on the HER2 and hormone-receptor markers. The performed multilevel clustering produced a large number of small clusters. During the first division five clusters were obtained, three of them containing almost only hormone-receptor positive (HR+) tumors. One cluster consisted of similar numbers of HER2-enriched, triple-negative breast cancer (TNBC), and HR+ cases. Obtained results confirm a great heterogeneity of breast cancer. However, they also suggest that commonly used breast tumor classification corresponding to the receptor markers may not entirely reflect the complexity of subtypes of this disease. The proposed algorithm gives a comprehensive insight into the hierarchical composition of dataset and allows for identification of genes influencing each division, which may be used for further investigation of the heterogeneity of breast cancer.