The Fault in our Stars: Designing Reproducible Large-scale Code Analysis Experiments
Large-scale software repositories are a source of insights for software engineering. They offer an unmatched window into the software development process at scale. Their sheer number and size holds the promise of broadly applicable results. At the same time, that very size presents practical challenges for scaling tools and algorithms to millions of projects. A reasonable approach e is to limit studies to representative samples of the population of interest. Broadly applicable conclusions can then be obtained by generalizing to the entire population. The contribution of this paper is a standardized experimental design methodology for choosing the inputs of studies working with large-scale repositories. We advocate for a methodology that clearly lays out what the population of interest is, how to sample it, and that fosters reproducibility. Along the way, we discourage researchers from using extrinsic attributes of projects such as stars, that measure some unclear notion of popularity.
Tue 17 SepDisplayed time zone: Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna change
13:30 - 15:00 | Technical Papers at EI 7 Chair(s): Jonathan Aldrich Carnegie Mellon University, Guido Salvaneschi University of St. Gallen | ||
13:30 30mAwards | ECOOP Awards Technical Papers | ||
14:00 15mTalk | Mutation-based Lifted Repair of Software Product Lines Technical Papers Aleksandar S. Dimovski Mother Teresa University, Skopje | ||
14:15 15mTalk | Refinements for Multiparty Message-Passing Protocols: Specification-agnostic theory and implementation Technical Papers | ||
14:30 15mTalk | CtChecker: a Precise, Sound and Efficient Static Analysis for Constant-Time Programming Technical Papers | ||
14:45 15mTalk | The Fault in our Stars: Designing Reproducible Large-scale Code Analysis Experiments Technical Papers Petr Maj Czech Technical University, Stefanie Muroya Lei Institute of Science and Technology in Austria, Konrad Siek Czech Technical University in Prague, Luca Di Grazia University of Stuttgart, Jan Vitek Northeastern University |