Call for Artifacts

R1 submission deadline extension

The registration deadline is Jan 23 AoE. By this point, all the artefact metadata (such as software/hardware requirements) should be available in HotCRP. We are now moving the final submission deadline (concerning the heart of the artefact, i.e, the Docker/VM image) to Jan 28 AoE. As such, this extension for authors will not change the review process: it allows the AEC to maintain its original bidding/review assignments and review deadlines.

About

Research artifacts denote digital objects that were either created by the authors of a research article to be used as part of their study or generated by their experiments (adopted from ACM’s recommendations).

The about page provides more background on artifacts and their seal of quality. This page describes the call for artifact. We also make available the submission template, and the template form reviewers will use.

Artifact Submission

Submission will open soon via HotCRP: https://ecoop24aec.hotcrp.com.

Artifact Preparation Guidelines

At a high level, we are interested in artifacts that:

• Have no dependencies. Use of docker images is strongly recommended. Virtual machine images in OVF/OVA format containing the artifact can also be provided.
• Have a minimal number of setup steps. Ideally, it should just be importing the docker/VM image.
• Have a short run, that reviewers can try first before carrying the full review (running in at most 10 minutes).
• Have a push-button evaluation. Ideally, the evaluation can be run through a single script, which performs the computation and generates the relevant figures/experimental data presented in the paper. The evaluation should either display progress messages or expected duration should be provided. This fully automated approach may be a bit more costly to setup, but you won’t have any copy/pasting issues for your paper, and regenerating data is heavily simplified.
• Include some documentation on the code and layout of the artifact.
• Use widely supported open formats for documents, preferably CSV or JSON for data.
• Document which outputs are associated with which parts of your paper, if possible, please specify table, figure or sub-sections.

Double-blind artifact review: Please note that artifact evaluation for ECOOP is double-blind. Please make sure you do not reveal author identities, e.g., through user names in VM/Docker file systems.

When submitting the artifact for review, please consider platforms like Zenodo and Figshare for anonymous submission during the review. After acceptance, you can update the metadata to deanonymize your artifact. Note: you must avoid using platforms with tracking access or requiring individual download permissions.

Authors are strongly discouraged from:

• Downloading content over the internet during experiments or tests;
• Using closed-source software libraries, frameworks, operating systems, and container formats; and
• Providing experiments or tests that run for multiple days. If the artifact takes several days to run, we ask that you provide us with the full artifact and a reduced input set (in addition to the full set) to only partially reproduce your results in a shorter time. If the artifact requires special hardware, please get in touch with the AEC chairs, let us know of the issue, and provide us with (preferably SSH) access to a self-hosted platform for accessing the artifact.

Artifact Packaging Guidelines

When packaging your artifact for submission, please consider the following. Your artifact should be as accessible as possible to the AEC members, and it should be easy for the AEC members to quickly make progress on the evaluation of your artifact. Please provide some simple scenarios describing concretely how the artifact is intended to be used; for a tool, this would include specific inputs to provide or actions to take, and expected output or behavior in response to this input. In addition to these very tightly controlled scenarios that you prepare for the AEC members to try out, it may be very useful if you suggest some variations along the way, such that the AEC members will be able to see that the artifact is robust enough to tolerate experiments.

To avoid problems with software dependencies and installation during artifact review, artifacts must be made available either as a Docker image (https://www.docker.com/) or as a virtual machine image in OVF/OVA format containing the artifact already installed. The artifact must be provided as a self-contained archive file, using a widely supported archive format (e.g., *.zip, *.tgz) with instructions on how to start the container or the VM.

Artifact Submission Guidelines

Every submission must include the following.

• A URL for downloading the artifact.
• A Markdown-formatted file providing an overview of the artifact. Please use this template.

Note that if your paper and artifact are accepted, a follow-up Dagstuhl Artifact document will have to describe the artifact.

Please make sure to use hosting platforms for your artifacts that do not track IP addresses, as this would undermine the double-blind review process.

Artifact Review Process

Submitted artifacts will go through a two-phase evaluation.

• Kick-the-tires: Reviewers check the artifact integrity and look for any possible setup problems that may prevent it from being properly evaluated (e.g., corrupted or missing files, VM won’t start, immediate crashes on the simplest example, etc.). Authors are informed of the outcome and will be given a 4-day period to read and respond to the kick-the-tires reports of their artifacts. During the author response period, interactive discussions between reviewers and authors will be possible through HotCRP.

• Artifact assessment: Reviewers evaluate the artifacts, checking if they live up to the claims the authors make in the accompanying documentation. Note: the positive result of the evaluation, along with a judgment of the importance of these claims for the novelty and technical soundness of the submitted papers, will be considered in the paper review process.

Information for reviewers

As a member of the artifact evaluation committee (AEC), your main goal will be to decide which artifact badges should be awarded to the artifacts you review. We have created a reviewing template, aiming at guiding you for each artifact review.

New this year we have asked authors to provide as much automation as possible in their artifacts, through “push-button evaluations”. We hope this will streamline the artifact reviewing process.

Paper reviews: some members may also provide paper reviews as members of the external review committee. In that case, we expect you will review the corresponding artifacts.

The artifact evaluation will have two rounds, each round split into two phases:

• the kick-the-tires phase aims at quickly checking the reviewers can run the artifact. Reviewers will be able to ask questions to authors to fix any issue they may encounter.
• the artifact assessment is the main reviewing phase.

Important Dates

ECOOP 2024 has two rounds of reviews. Artifacts submissions to the ECOOP 2024 AE track are shortly after the deadline of research papers. The first round (R1) takes place from January to February, followed by a second round (R2) in April and May. Please respect the AEC deadlines below. If you foresee any difficulty in doing so, reach out to the AEC chairs as soon as possible.

First round (R1) important deadlines:

• Jan 23: R1 artifact submissions (authors)
• Jan 29: R1 review preferences (AEC)
• Feb 6: R1 kick-the-tires reviews (AEC)
• Feb 12: R1 author responses (authors)
• Mar 5: R1 artifact reviews (AEC)

Second round (R2) important deadlines:

• Apr 23: R2 artifact submissions (authors)
• Apr 28: R2 review preferences (AEC)
• May 6: R2 kick-the-tires reviews (AEC)
• May 12: R2 author responses (authors)
• Jun 4: R2 artifact reviews (AEC)

Nomination forms

Do you want to participate to ECOOP’s artifact evaluation committee? Please fill the form. You will have to review artifacts during two submission periods (mid-January to March and mid-April to June), and may also join paper reviews (i.e., some members of the AEC will be members of the External Review Committee).

For round 1, the co-chairs will process nominations in two rounds in December 2023 and early January 2024. For round 2, the nominations are open and the co-chairs will process them in April.

Background

Artifacts of technical research papers can include supplementary material such as tools, datasets, models, tutorial videos, or substantial items associated with the research or study presented in the paper.

Traditionally, technical research papers are published without including any artifacts, even though the artifacts may serve as crucial and detailed evidence for the quality of the results that the associated paper offers.

These artefacts enhance transparency and support the repeatability of experiments and precise comparison with alternative approaches, thus enabling higher quality in the research area as a whole. Additionally, artefacts enhance the overall understanding of the research by allowing others to review and build upon the work, making it easier for other researchers to perform their experiments, thus helping the original authors disseminate their ideas in detail.

Consequently, artefacts should be taken seriously and recognized separately. We encourage you to submit relevant artefacts along with the manuscript of technical research papers.

The AE process at ECOOP 2024 is a continuation of the AE process at previous ECOOP editions and several other conferences, including ESEC/FSE, OOPSLA, PLDI, ISSTA, HSCC, and SAS: see the authoritative Artifact Evaluation for Software Conferences website.

Badges

• Functional: The artifacts associated with the research are found to be documented, consistent, complete, exercisable, and include appropriate evidence of verification and validation. In particular, all experimental claims made in the paper should be reproducible through the artifact.

• Reusable The artifacts associated with the paper are of a quality that significantly exceeds minimal functionality. That is, they have all the qualities of the Artifacts Evaluated – Functional level. Yet, in addition, they are very carefully documented and well-structured to the extent that reuse and repurposing is facilitated. For the evaluation of the artifact’s reusability, we will rely on the reuse scenarios that the authors describe in their documentation. Providing an open-source implementation and its source code, and relying on public, open-source benchmarks are good steps to ensure the reusability of an artifact.

Irrespective of the artifact evaluation outcome, artifacts may be awarded the “Available” badge.

• Available - Author-created artifacts relevant to this paper have been placed on a publicly accessible archival repository. A DOI for the object is provided. We offer to publish the artifact on DARTS, in which case the available badge will be issued automatically.

A selected number of artifacts going beyond expectations of quality will receive a Distinguished Artifact award. The selection procedure will be based on review scores and feedback from the artifact evaluation committee.

Related resources

The process for evaluation is based on ACM’s Artifact Review and Badging and NISO’s guidelines for reproducibility badging. However, neither ACM nor NISO are involved in the implementation or evaluation process on behalf of ECOOP.

We will soon provide more detailed instructions by publishing the call for artifacts, artifact submission template and the artifact reviewer template. In the meantime, we advise prospective researchers to read the ACM SIGPLAN’s Empirical Evaluation Guidelines.