We applied CITE-Seq to measure >150 cell surface immune markers and checkpoint proteins simultaneous to RNA-Sequencing. We resolve the tumour-immune milieu with high precision and reveal subsets of CD8 T cells that are specific to luminal and triple-negative breast cancers, with unique checkpoint molecule expression. In addition, a number of stromal cell-states were identified across the cohort, including three subsets of cancer-associated fibroblasts (CAFs), two subsets of smooth-muscle cells and four subsets of endothelial cells. Distinct transcription factor networks regulate these polarised states. We derived novel markers for these populations using CITE-Seq and carried out prospective isolation and functional characterisation of stromal subsets. Ligand-receptor signalling predicted a role for CAF subsets in T cell immunosuppression, which was further supported by deconvoution analysis of large bulk RNA_Seq cohorts and by in vitro CAF-T-cell co-culture proliferation assays.