Background

From 2015 to 2020, the joint DOE-NCI Pilot 1 project developed a foundational set of machine learning models, related research, and infrastructure for predicting tumor drug response to single and combination agents. These models have been benchmarked against those in the literature, compared to numerous of alternative formulations (ranging from classical machine learning to deep learning), and evaluated on a broad range of available training data. Some of these models have become benchmarks for the ECP-CANDLE project (https://github.com/ECP-Candle/Benchmarks).

This work yielded valuable insights that inspired the Innovative Methodologies and New Data for Predictive Oncology Model Evaluation (IMPROVE) project. Notably, it highlighted that while the cancer research community has made significant progress in developing machine learning models for cancer, there’s a lack of standardized, well-documented approaches for building, training, and validating these models.

It is difficult to compare new modeling approaches in the literature with those from previous studies due to different choices of data normalization, encoding, filtering, and other data prepration and preprocssing steps. This is compounded by the absence of well-curated and standardized training and testing datasets, as well as a lack of broadly accepted featurization for both tumors and drug. Understanding and recognizing new innovations in drug response modeling are less clear due to ongoing challenges with data quality and data integration.

With the IMPROVE project, we address some of the challenges around model comparisons and dataset standardization and availability through the development and provision of a software framework to make it routine practice for the broader community (cancer research and other areas) to compare new machine learning modeling approaches to previous models in a rigorous and comprehensive fashion.