|
Size: 10059
Comment:
|
← Revision 72 as of 2012-10-05 06:55:24 ⇥
Size: 6319
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 3: | Line 3: |
| The 4th Open Answer Set Programming Competition is open to ASP systems and *any other system* based on a declarative specification paradigm. |
|
| Line 7: | Line 4: |
| Participants will compete on a selected collection of declarative specifications of benchmark problems, taken from a variety of domains as well as real world applications, and instances thereof. |
{{{#!wiki caution Final package submissions are handled by submitting the problem package via Easychair ([[https://www.easychair.org/conferences/?conf=aspcomp2013|here]]). }}} <<TableOfContents>> The 4th Open Answer Set Programming Competition is open to ASP systems and '''any other system''' based on a declarative specification paradigm. Participants will compete on a selected collection of declarative specifications of benchmark problems, taken from a variety of domains as well as real world applications, and instances thereof. |
| Line 13: | Line 18: |
| * Deductive database tasks on large data-sets * Sequential and Temporal Planning * Classic and Applicative graph problems * Puzzles and combinatorics * Scheduling, timetabling and other resource allocation problems * Combinatorial Optimization problems * Ontology reasoning * Automated Theorem Proving and model checking * Reasoning tasks over large propositional instances * Constraint Programming problems * Other AI problems |
* Deductive database tasks on large data-sets * Sequential and Temporal Planning * Classic and Applicative graph problems * Puzzles and combinatorics * Scheduling, timetabling and other resource allocation problems * Combinatorial Optimization problems * Ontology reasoning * Automated Theorem Proving and model checking * Reasoning tasks over large propositional instances * Constraint Programming problems * Other AI problems |
| Line 25: | Line 30: |
| We encourage to provide help by proposing and/or devising new challenging benchmark problems. |
We encourage to provide help by proposing and/or devising new challenging benchmark problems. The submission of problems arising from applications having practical impact are especially encouraged; problems used in the former ASP Competitions, or variants thereof, can be re-submitted (see [[#problem-submission|Problem Submission]]). |
| Line 28: | Line 32: |
| The submission of problems arising from applications having practical impact are strongly encouraged; problems used in the former ASP Competitions, or variants thereof, can be re-submitted. |
Benchmark authors are asked to provide the committee with a problem specification and an instance set (or a generator thereof). Problems will be classified and selected in order to obtain a fair balance between several factors (including application domain, modeling difficulty, complexity category, expressiveness, etc). |
| Line 33: | Line 34: |
| Benchmark authors are expected to produce a problem specification and an instance set (or a generator thereof). | [[BenchmarkProblems|What is a problem?]] <<Anchor(problem-submission-procedure)>> == Problem Submission == The problems submission procedure will consist of two stages: 1. Proposal and discussion; 1. Validation and final submission. At the first stage, problem descriptions are submitted and made publicly available in the competition website. Everybody can discuss and improve the proposals on our discussion mailing list. At the second stage, the organizing committee will validate a selection of benchmarks; each selected problem will then be completed under the responsibility of the original contributor(s). Submission, discussion and selection of benchmark problems is handled via email: details are reported below. |
| Line 36: | Line 49: |
| * ''Problem Proposal and Discussion'' * '''Aug 31th, 2012''' - Problem submission deadline * ''Problem Validation and Final Submission'' * '''Sep. 30th, 2012''' - Final submission deadline |
|
| Line 37: | Line 54: |
| * ''Problem Proposal and Discussion'' * '''Aug 30th, 2012''' - Problem submission deadline |
<<Anchor(problem-submission)>> |
| Line 40: | Line 56: |
| * ''Problem Validation and Final Submission'' | === How To Submit a Problem === --('''Preliminary Submission''' (abstract):)-- |
| Line 42: | Line 59: |
| === Problem submission procedure === | --(To preliminarily submit a new problem just prepare a benchmark package (see [[#package-format|Package Description]]) and send it by email to [[mailto:aspcomp_benchmarks_submission@yahoogroups.com|Benchmark Submission]] with subject:)-- |
| Line 44: | Line 61: |
| The problems submission procedure will consist of two stages: | --(. SUBMISSION: Benchmark Name)-- |
| Line 46: | Line 63: |
| 1. Proposal and discussion; | {{{#!wiki caution Preliminary submissions are now over. Complete problem submissions are now reserved to selected problem authors. }}} |
| Line 48: | Line 67: |
| 2. Validation and final submission. | '''Complete problem submission''': |
| Line 50: | Line 69: |
| At the first stage, problem descriptions are submitted and made publicly available in the competition website. The community is invited to discuss and improve the proposals. At the second stage, the organizing committee will validate a selection of benchmarks; each selected problem will be completed by original contributor(s). |
Final package submissions are handled by submitting the problem package via Easychair ([[https://www.easychair.org/conferences/?conf=aspcomp2013|here]]). |
| Line 53: | Line 71: |
| Submission, discussion and selection of benchmark problems is handled via email, the details are reported below. |
<<Anchor(package-format)>> |
| Line 56: | Line 73: |
| === Problem selection and balance === | === Package Description === * A ''' ''preliminary submission'' ''' (see the [[ProblemIOSpecification|Package Submission Specification]]) simply requires problem name and description/specification only, even if the submission of sample instances is encouraged (sample instances will not be used for the competition). In case you want to propose the problem for the System Competition, an ASP encoding is also strongly suggested. * A ''' ''complete problem submission'' ''' (see the [[ProblemIOSpecification|Package Submission Specification]]), to be provided if the proposal has been accepted, has to be enclosed in a single compressed package ({{{zip}}}). |
| Line 58: | Line 77: |
| Problems will be selected in the Validation stage and classified in order to obtain a fair balance between several factors (including complexity category, modeling difficulty, expressiveness, application domain, etc). | == Problem Discussion and Validation == Benchmarks will be published in a [[BenchmarkDiscussion|specific section of the competition website]]. |
| Line 60: | Line 80: |
| === Problem submission === To submit a new problem just login the [[http://www.easychair.org/conferences/?conf=aspcomp2013|EasyChair]] system taking the ''Author'' role. A problem submission is handled just like a paper submission. The system will require to fill a form, in which ''title'' (conventional problem name), ''abstract'' (problem specification), and ''problem classification'' (see Section [[BenchmarkProblems|Benchmark problem classification]]) have to be provided by checking appropriate form items; [[http://www.easychair.org/conferences/?conf=aspcomp2013|EasyChair]] will require also to fill the keyword fields, please provide us with some. Problem specifications can be either partial (problem specification only) or detailed. A submission can be uploaded in [[http://www.easychair.org/conferences/?conf=aspcomp2013|EasyChair]] (as a paper submission) enclosed in a a single compressed package ({{{zip}}} and {{{{tar.gz}}} are allowed formats), containing: 1. a textual problem description (same as the abstract) where both names and arguments of input and output predicates are clearly specified; 2. a problem encoding; 3. some sample instances, that is some instances for the problem, which comply with the Input specification (see [[#input-output|Problem I/O and Instance Specification]]), added in order to help evaluating the specification (sample instances should not be used for the competition). 4. a correctness checker (optional), that is, a program or a script able to decide whether the output predicates occurring in some answer set form a solution to a given instance of the problem at hand (and in case of an optimization problem, the program should be able to compute the "quality" of the answer). The above information shall be provided according to the [[ProblemPackageFormats|Problem Package Format]] information. ==== Abstract only submissions ==== Abstract-only submission are accepted in the proposal stage. For abstract-only submissions, check the corresponding box at the end of the submission page, and just provide the problem statement in the ''abstract'' textfield. The problem statement must be anyway unambiguous and include the description of input and output predicates according to the problem description format described above. ''In this stage, the submission of correctness checkers, as well as problem encodings and sample instances, are optional and left up to participants; nonetheless, we are grateful if you can already provide us with some. Especially, the submission of problems encodings often helps in disambiguating blurred specifications, so its provision is greatly appreciated.'' === Problem discussion === Benchmarks can be discussed by accessing [[http://www.easychair.org/conferences/?conf=aspcomp2013|Easychair]] taking the ''PC Member'' role. The usual comment tools provided by [[http://www.easychair.org/conferences/?conf=aspcomp2013|EasyChair]] will be used to post comments and discuss the submissions. Unlike the usual conference "submit, then discuss" procedure, the discussion will be ongoing while problems are being submitted. Researchers interested in discussing/improving problem proposals can ask for the ''virtual'' program committee member role in the [[http://www.easychair.org/conferences/?conf=aspcomp2013|EasyChair]] system. In particular, authors will be added as participants to the discussion by default; people interested only in discussing problems can send an e-mail to [[mailto:aspcomp2013__AT__kr.tuwien.ac.at]] with subject: DISCUSSION: First Name - Second Name - Email for obtaining access to the discussion. The organizing committee will take into account comments in order to select problems in the second stage. ''The community is strongly encouraged to participate to this important moment in which problems are submitted and evaluated.'' |
=== Join the Discussion Group === All Researchers interested in discussing/improving problem proposals can join the [[mailto:aspcomp_benchmarks_discussion@yahoogroups.com|Benchmark Discussion]] mailing list visiting [[http://groups.yahoo.com/group/aspcomp_benchmarks_discussion/join|the group page]] or sending a message to [[mailto:aspcomp_benchmarks_discussion-subscribe@yahoogroups.com|Benchmark Discussion Subscription Request]]. All communities are encouraged to participate to this crucial stage in which problems are submitted and evaluated. |
| Line 104: | Line 84: |
| Benchmark problems submitted and discussed in the first stage are evaluated by the competition organizing committee. As for paper submissions in a conference, authors will be notified regarding the final acceptance decision. | |
| Line 105: | Line 86: |
| Benchmark problems submitted and discussed in the first stage are evaluated by the competition organizing committee. As for paper submissions in a conference, authors will be notified regarding the final acceptance decision. Benchmark problems that are accepted by the organizing committee (see [[#acceptance|Final acceptance of problems]]) will be then finalized by the authors. Finalized submission shall be uploaded to [[http://www.easychair.org/conferences/?conf=aspcomp2013|EasyChair]] enclosed in a single compressed package (zip, rar, and tar.gz are allowed formats), containing: 1. the textual problem description (same as the abstract) where both names and arguments of input and output predicates are clearly specified; 1. the problem encoding; 1. a correctness checker; 1. either an instance generator or at least 50 ''hard'' instances of the problem (using only input predicates); 1. a "demonstration" that the benchmark problem can be effectively solved on the class of instances provided, e.g. a report about tests carried out on a system of choice. The above information shall be provided according to the [[ProblemPackageFormats|Problem Package Format]] information. At least one among item 4 and 5 must be provided. In this phase the contribution of benchmarks authors is fundamental, submission which are incomplete at the end of this stage are unsuitable for usage in the final competition and will be rejected (see Section [[#acceptance|Final acceptance of problems]]). The organizing committee takes the right to exclude benchmark problems whose provided instance family turns out to be blatantly too ''easy'' or ''difficult'' in terms of expected evaluation time. <<Anchor(input-output)>> === Programs with Function symbols and integers === Programs with function symbols and integers are in principle subject to no restriction. For the sake of Competition, and to facilitate implementors of '''ASP-RFC''' and '''ASP-Core''' it is prescribed that * each selected problem encoding ''P'' must provably have finitely many finite answer sets for any of its benchmark instance ''I'', that is ''AS(P U I)'' must be a finite set of finite elements. "Proofs" of finiteness can be given in terms of membership to a known decidable class of programs with functions and/or integers, or any other formal mean. * a bound ''k,,P,,'' on the maximum nesting level of terms, and a bound ''m,,P,,'' on the maximum integer value appearing in answer sets originated from ''P'' must be known. That is, for any instance ''I'' and for any term ''t'' appearing in ''AS(P U I)'', the nesting level of ''t'' must not be greater than ''k,,P,,'' and, if ''t'' is an integer it must not exceed ''m,,P,,''. The values ''m,,P,,'' and ''k,,P,,'' will be provided in input to participant systems, when invoked on ''P'': problem designers are thus invited to provide such values accordingly. |
In this phase the contribution of benchmarks authors is fundamental; submission which results to be incomplete at the end of this stage are unsuitable fot the competition and will be rejected (see Section [[#acceptance|Final acceptance of problems]]). The organizing committee takes the right to exclude benchmark problems whose provided instance family turns out to be blatantly too ''easy'' or too ''difficult'' in terms of expected evaluation time. |
| Line 135: | Line 89: |
| Line 136: | Line 91: |
| The final inclusion of problems in the competition is subject to to the conclusive approval of the competition organizing committee. The committee takes the right to discard problems which are not compliant with one or more of the following criteria: | |
| Line 137: | Line 93: |
| The final inclusion of problems in the competition is subject to to the conclusive approval of the competition organizing committee. Among the collected problems, the committee takes the right to discard problems which are not compliant with one or more of the following criteria: 1. The instance set (or the instances generator output) does not comply with the competition input/output format (see Section [[#input-output|Problem I/O and Instance Specification]]); 2. The set of provided instances counts less than 25 instances, or the provided generator is missing or failing; 3. The provided (or generated) instances are trivial or not provably solvable when run on the competition hardware; |
1. The instance set (or the instances generator output) does not comply with the competition input/output format (see [[ProblemIOSpecification|Problem I/O and Instance Specification]]); 1. The set of provided/generated instances counts less than 50 instances, or the provided generator is missing/failing; 1. The provided/generated instances are trivial or not provably solvable on the competition hardware. |
Call for Benchmark Problems
Final package submissions are handled by submitting the problem package via Easychair (here).
Contents
The 4th Open Answer Set Programming Competition is open to ASP systems and any other system based on a declarative specification paradigm.
Participants will compete on a selected collection of declarative specifications of benchmark problems, taken from a variety of domains as well as real world applications, and instances thereof.
These include but are not limited to:
- Deductive database tasks on large data-sets
- Sequential and Temporal Planning
- Classic and Applicative graph problems
- Puzzles and combinatorics
- Scheduling, timetabling and other resource allocation problems
- Combinatorial Optimization problems
- Ontology reasoning
- Automated Theorem Proving and model checking
- Reasoning tasks over large propositional instances
- Constraint Programming problems
- Other AI problems
We encourage to provide help by proposing and/or devising new challenging benchmark problems. The submission of problems arising from applications having practical impact are especially encouraged; problems used in the former ASP Competitions, or variants thereof, can be re-submitted (see Problem Submission).
Benchmark authors are asked to provide the committee with a problem specification and an instance set (or a generator thereof). Problems will be classified and selected in order to obtain a fair balance between several factors (including application domain, modeling difficulty, complexity category, expressiveness, etc).
Problem Submission
The problems submission procedure will consist of two stages:
- Proposal and discussion;
- Validation and final submission.
At the first stage, problem descriptions are submitted and made publicly available in the competition website. Everybody can discuss and improve the proposals on our discussion mailing list. At the second stage, the organizing committee will validate a selection of benchmarks; each selected problem will then be completed under the responsibility of the original contributor(s).
Submission, discussion and selection of benchmark problems is handled via email: details are reported below.
Schedule
Problem Proposal and Discussion
Aug 31th, 2012 - Problem submission deadline
Problem Validation and Final Submission
Sep. 30th, 2012 - Final submission deadline
How To Submit a Problem
Preliminary Submission (abstract):
To preliminarily submit a new problem just prepare a benchmark package (see Package Description) and send it by email to Benchmark Submission with subject:
. SUBMISSION: Benchmark Name
- Preliminary submissions are now over. Complete problem submissions are now reserved to selected problem authors.
Complete problem submission:
Final package submissions are handled by submitting the problem package via Easychair (here).
Package Description
A preliminary submission (see the Package Submission Specification) simply requires problem name and description/specification only, even if the submission of sample instances is encouraged (sample instances will not be used for the competition). In case you want to propose the problem for the System Competition, an ASP encoding is also strongly suggested.
A complete problem submission (see the Package Submission Specification), to be provided if the proposal has been accepted, has to be enclosed in a single compressed package (zip).
Problem Discussion and Validation
Benchmarks will be published in a specific section of the competition website.
Join the Discussion Group
All Researchers interested in discussing/improving problem proposals can join the Benchmark Discussion mailing list visiting the group page or sending a message to Benchmark Discussion Subscription Request. All communities are encouraged to participate to this crucial stage in which problems are submitted and evaluated.
Validation and final submission stage
Benchmark problems submitted and discussed in the first stage are evaluated by the competition organizing committee. As for paper submissions in a conference, authors will be notified regarding the final acceptance decision.
In this phase the contribution of benchmarks authors is fundamental; submission which results to be incomplete at the end of this stage are unsuitable fot the competition and will be rejected (see Section Final acceptance of problems). The organizing committee takes the right to exclude benchmark problems whose provided instance family turns out to be blatantly too easy or too difficult in terms of expected evaluation time.
Final acceptance of problems
The final inclusion of problems in the competition is subject to to the conclusive approval of the competition organizing committee. The committee takes the right to discard problems which are not compliant with one or more of the following criteria:
The instance set (or the instances generator output) does not comply with the competition input/output format (see Problem I/O and Instance Specification);
- The set of provided/generated instances counts less than 50 instances, or the provided generator is missing/failing;
- The provided/generated instances are trivial or not provably solvable on the competition hardware.