The regulations of the System Competition are conceived taking into account the following considerations:
Many families of formalisms, which can be considered to a large extent neighbors of the ASP community, have reached a significant level of language standardization, ranging from the Constraint Handling Rules (CHR) family, the Satisfiability Modulo Theories SMT-LIB format, the Planning Domain Definition Language (PDDL), to the TPTP format used in the Automated Theorem Proving System Competition (CASC).
The above experiences witness that the availability of the common ground of a standard language, possibly undergoing continuous refinement and extension, has usually boosted the availability of resources, the deployment of the technology at hand into practical applications, and the effectiveness of systems. Nonetheless, ASP is missing a standard, high-level input language.
We think, however, that the ASP community is mature enough for starting the development of a common standard input format: an ASP system can be roughly seen as composed of a front-end input language processor and a model generator. The first module is usually (but not necessarily) named "grounder", for it produces a propositional program obtained from an higher-level specification of the problem at hand. Incidentally, currently developed ASP grounder systems have recently reached a good degree of maturity, and, above all, they have reached a fairly large degree of overlap in their input formats. This paves the way for taking the very first serious step towards the proposal of a common input language for ASP solvers.
It thus makes sense to play (part of) the Third ASP competition on the grounds of a common draft input format, in order to promote the adoption of a newly devised standard, and foster the birth of a new standardization working group. In order to met the above goals, the competition input format should be large enough to embed all of the basic constructs included in the language originally specified before 1 (and lifted to its non-ground version), yet conservative enough to allow all the participants to adhere to the standard draft with little or no effort.
Performance of a given ASP system S might greatly vary on problem P, if fine-tuning on P is performed either by improving the problem encoding or by tuning internal ad-hoc optimizations techniques. Although, on the one hand, it is important to encourage system developers to fine-tune their systems, and then compete on this basis, on the other hand it is similarly important to put in evidence how a solver performs with its default behavior: indeed, the user of an ASP system has generally little or no knowledge of the system internals, and might not be aware of which program rewritings and system optimization methods pay off in terms of performance.
The System competition should thus put in evidence the performance of a system when used as an "off-the-shelf black box" on a supposedly unknown problem specification. Rankings on the System competition should give a fairly objective measure of what one can expect when switching from a system to another, while keeping all other conditions fixed (problem encoding and default solver settings). The formula of the System competition thus aims at measuring the performance of a given solver when used on a generic problem encoding, and not when this is fine-tuned for the specific problems selected for the current competition.
Given the above considerations, the System competition will be held on the basis of the following principles:
The System competition is open to systems able to parse a language in a fixed format (which are ASP-Core and ASP-RFC, see the File and Language Format document).
The competition is run over a selection of problems. For each problem, a corresponding, fixed encoding in ASP-Core or ASP-RFC, together with a set of benchmarks instances, is chosen by the organizers (see Benchmark problems classification);
- Each participant system will be launched with its default settings on each problem instance.
Syntactic special-purpose solving techniques, specialized on a per problem basis, are forbidden.
Detailed rules can be found in Rules & Scoring.