Inferring cell-cell communication from single-cell data

Cells signal to each other through ligands and receptors, and many tools try to read these interactions off single-cell RNA-seq data. There is no gold standard to check them against, so this benchmark uses an indirect ground truth, inferred cytokine activity, and asks which ligands are active towards which target cell types in a breast cancer atlas.

Task

A method receives a single-cell RNA-seq count matrix with a cell-type label per cell, and a prior-knowledge resource of ligand–receptor pairs (provided through the LIANA framework). It must assign a score to every ligand–target cell type pair it can evaluate, where the ligand is in both the data and the resource and the target is any cell type in the data. The ranking is compared with an assumed truth: whether the cytokine corresponding to that ligand is inferred to be active in that target cell type.

Background

Inference of cell-cell communication (CCC) is widely used, but tools differ in preprocessing, scoring and prior knowledge, and they are hard to evaluate. Dimitrov et al. (2022) compared 16 resources and 7 methods, built LIANA as a common interface, and assessed predictions against spatial co-localisation, cytokine activities and receptor protein abundance. The Open Problems task, maintained by Daniel Dimitrov, takes the cytokine-activity proxy as its ground truth. This entry covers the ligand–target subtask; a source–target subtask is released separately on the same page.

Data and evaluation

• Data: one dataset in release v1.0.0-ligand_target: the human breast cancer atlas of Wu et al. (2021), with 42,512 cells, 29 cell types and 10 patients. Cytokine activities were inferred with a multivariate linear model on cytokine-focused signatures and serve as the assumed truth. The data are downloaded by the task's dataset loader (figshare).
• Split: none; methods must not use the assumed truth.
• Metric: two metrics, both higher-is-better. The odds ratio compares true and false positives among the top-ranked 5% of interactions with those among the rest, transformed by $1 - 1/(1 + \mathrm{OR}/2)$; the second is the area under the precision–recall curve. Each is rescaled between the random-events control (0) and the true-events control (1), clipped to $[0, 1]$, and the overall score is their mean, with failed runs counted as 0. The leaderboard shows it to two decimals; methods are listed by their best and worst parameter (aggregation) setting.
• Code: release v1.0.0 lives in the legacy openproblems repository; the task now continues in openproblems-bio/task_cell_cell_communication.

What is known

On the v1.0.0-ligand_target leaderboard (as of 2026-10-20):

• CellPhoneDB leads at 0.19 (0.1882 before rounding, recomputed from the published results) with its best setting.
• Log2FC, SingleCellSignalR and the two rank-aggregate consensus methods follow at 0.17, and Connectome at 0.15.
• Every method is far below the true-events control (1.00), so on this proxy the ligand rankings carry only a weak signal.

What would count as progress

• An overall score above 0.1882 on the ligand–target dataset, computed with the release's evaluation code, with code that reproduces it.
• A method that beats CellPhoneDB on both metrics, odds ratio and precision–recall, rather than on one.
• A gain that carries over to the source–target subtask on the same page, which uses a different proxy of the truth: spatial adjacency of cell types in a mouse brain atlas.

References

1. Dimitrov D, Türei D, Garrido-Rodriguez M, et al. Comparison of methods and resources for cell-cell communication inference from single-cell RNA-Seq data. Nature Communications 13, 3224 (2022). doi:10.1038/s41467-022-30755-0
2. Wu SZ, Al-Eryani G, Roden DL, et al. A single-cell and spatially resolved atlas of human breast cancers. Nature Genetics 53, 1334–1347 (2021). doi:10.1038/s41588-021-00911-1
3. Efremova M, Vento-Tormo M, Teichmann SA, Vento-Tormo R. CellPhoneDB: inferring cell–cell communication from combined expression of multi-subunit ligand–receptor complexes. Nature Protocols 15, 1484–1506 (2020). doi:10.1038/s41596-020-0292-x
4. Luecken MD, Gigante S, Burkhardt DB, Cannoodt R, et al. Defining and benchmarking open problems in single-cell analysis. Nature Biotechnology 43, 1035–1040 (2025). doi:10.1038/s41587-025-02694-w

Source

Imported from Open Problems in Single-Cell Analysis: Cell-Cell Communication (cell_cell_communication), curated by the Open Problems in Single-Cell Analysis consortium. Open according to the source as of 2026-10-20 (leaderboard release v1.0.0-ligand_target). The statement above is written in our own words; see the source for the original wording, history and discussion.