Comparative Analysis of Multivariate Mixture Models for Clustering Cancer Expression Data

Clustering gene expression data is essential for understanding tumor heterogeneity but is challenged by high dimensionality, noise, and class imbalance. We evaluated three Gaussian mixture model (GMM) based clustering methods: classical GMM, GMM with variance decomposition, and GMM with feature saliency on transcriptomic data from The Cancer Genome Atlas (TCGA) involving three cancer type pairs with varying molecular similarity and sample balance: breast invasive carcinoma vs uterine carcinosarcoma (BRCA vs UCS), lung adenocarcinoma vs lung squamous cell carcinoma (LUAD vs LUSC), and pancreatic adenocarcinoma vs sarcoma (PAAD vs SARC). Performance was assessed using Adjusted Rand Index, Fowlkes-Mallows Index, and Normalized Mutual Information.Results showed method-specific strengths influenced by dataset characteristics. Feature saliency clustering excelled in the highly imbalanced BRCA vs UCS pair by effectively downweighting irrelevant features. Variance decomposition performed best for the molecularly similar and balanced LUAD vs LUSC pair, capturing subtle expression differences. Classical GMM achieved the highest accuracy for the moderately imbalanced PAAD vs SARC pair. However, no single method consistently outperformed others across all datasets.This study highlights that while latent-variable mixture models are promising for transcriptomic clustering, their performance depends on data-specific factors such as imbalance and molecular similarity. Further work is needed to develop robust, scalable methods capable of adapting to diverse biological datasets.