welcome: please sign in
location: BenchmarkSubmission

Call for Benchmark Problems

Final package submissions are handled by submitting the problem package via Easychair (here).

The 4th Open Answer Set Programming Competition is open to ASP systems and any other system based on a declarative specification paradigm.

Participants will compete on a selected collection of declarative specifications of benchmark problems, taken from a variety of domains as well as real world applications, and instances thereof.

These include but are not limited to:

We encourage to provide help by proposing and/or devising new challenging benchmark problems. The submission of problems arising from applications having practical impact are especially encouraged; problems used in the former ASP Competitions, or variants thereof, can be re-submitted (see Problem Submission).

Benchmark authors are asked to provide the committee with a problem specification and an instance set (or a generator thereof). Problems will be classified and selected in order to obtain a fair balance between several factors (including application domain, modeling difficulty, complexity category, expressiveness, etc).

What is a problem?

Problem Submission

The problems submission procedure will consist of two stages:

  1. Proposal and discussion;
  2. Validation and final submission.

At the first stage, problem descriptions are submitted and made publicly available in the competition website. Everybody can discuss and improve the proposals on our discussion mailing list. At the second stage, the organizing committee will validate a selection of benchmarks; each selected problem will then be completed under the responsibility of the original contributor(s).

Submission, discussion and selection of benchmark problems is handled via email: details are reported below.


How To Submit a Problem

Preliminary Submission (abstract):

To preliminarily submit a new problem just prepare a benchmark package (see Package Description) and send it by email to Benchmark Submission with subject:

  • Preliminary submissions are now over. Complete problem submissions are now reserved to selected problem authors.

Complete problem submission:

Final package submissions are handled by submitting the problem package via Easychair (here).

Package Description

Problem Discussion and Validation

Benchmarks will be published in a specific section of the competition website.

Join the Discussion Group

All Researchers interested in discussing/improving problem proposals can join the Benchmark Discussion mailing list visiting the group page or sending a message to Benchmark Discussion Subscription Request. All communities are encouraged to participate to this crucial stage in which problems are submitted and evaluated.

Validation and final submission stage

Benchmark problems submitted and discussed in the first stage are evaluated by the competition organizing committee. As for paper submissions in a conference, authors will be notified regarding the final acceptance decision.

In this phase the contribution of benchmarks authors is fundamental; submission which results to be incomplete at the end of this stage are unsuitable fot the competition and will be rejected (see Section Final acceptance of problems). The organizing committee takes the right to exclude benchmark problems whose provided instance family turns out to be blatantly too easy or too difficult in terms of expected evaluation time.

Final acceptance of problems

The final inclusion of problems in the competition is subject to to the conclusive approval of the competition organizing committee. The committee takes the right to discard problems which are not compliant with one or more of the following criteria:

  1. The instance set (or the instances generator output) does not comply with the competition input/output format (see Problem I/O and Instance Specification);

  2. The set of provided/generated instances counts less than 50 instances, or the provided generator is missing/failing;
  3. The provided/generated instances are trivial or not provably solvable on the competition hardware.

ASP Competition 2013: BenchmarkSubmission (last edited 2012-10-05 06:55:24 by GiovambattistaIanni)