Heading towards reproducible machine learning research for medical data

While it is crucial that machine learning models are reproducible, the issue is not sufficiently addressed in current research. Models can be irreproducible due to a variety of non-intentional and intentional reasons ranging from time pressure to deliberate fraud. To assure validity of published models, some venues demand public availability of source code or trained models. However, these approaches have shown to be insufficient, for example, because there are often no adequate quality checks on source code and functionality during the publication process.
We discuss the need as well as the challenges of open source research on machine learning. We address frequently occurring problems and offer (also technical) solutions.
These solutions are integrated in a novel open source framework. The applicability of this framework is discussed in the context of data generation and anomaly detection in electrocardiographic data and its specific requirements concerning data samples and model validity. Finally, we present how such frameworks enable more reliable progress in research.