#acl EditorsGroup:read,write,delete,revert,admin All:read = System Track Final Results = <
> <> <
> == System Track Final Results: Overall == ||System ||Total ||Instance ||Time || ||[[http://goo.gl/4XXFz|claspd]] ||861 ||560 ||301 || ||[[http://goo.gl/4XXFz|claspfolio]] ||818 ||535 ||283 || ||[[http://goo.gl/4XXFz|clasp]] ||810 ||520 ||290 || ||[[http://goo.gl/4XXFz|idp]] ||781 ||500 ||281 || ||[[http://goo.gl/4XXFz|cmodels]] ||766 ||510 ||256 || ||[[http://goo.gl/4XXFz|lp2diffz3]] ||572 ||405 ||167 || ||[[http://goo.gl/4XXFz|sup]] ||541 ||380 ||161 || ||[[http://goo.gl/4XXFz|lp2sat2gminisat]] ||495 ||365 ||130 || ||[[http://goo.gl/4XXFz|lp2sat2minisat]] ||481 ||355 ||126 || ||[[http://goo.gl/4XXFz|lp2sat2lminisat]] ||472 ||350 ||122 || ||[[http://goo.gl/4XXFz|smodels]] ||449 ||295 ||154 || == Final results by categories == === Overview === ||||||||'''P''' || ||||||||'''NP''' || ||||||||'''Beyond NP''' || ||'''System''' ||'''Total Score''' ||'''Inst. Score''' ||'''Time Score''' ||'''System''' ||'''Total Score''' ||'''Inst. Score''' ||'''Time Score''' ||'''System''' ||'''Total Score''' ||'''Inst. Score''' ||'''Time Score''' || ||clasp ||213 ||150 ||63 ||claspfolio ||609 ||385 ||224 ||claspd ||103 ||60 ||43 || ||claspfolio ||209 ||150 ||59 ||clasp ||597 ||370 ||227 ||cmodels ||72 ||45 ||27 || ||claspd ||206 ||145 ||61 ||idp ||597 ||370 ||227 ||claspfolio ||- ||- ||- || ||sup ||195 ||140 ||55 ||claspd ||552 ||355 ||197 ||clasp ||- ||- ||- || ||lp2sat2gminisat ||185 ||140 ||45 ||cmodels ||510 ||335 ||175 ||idp ||- ||- ||- || ||cmodels ||184 ||130 ||54 ||lp2diffz3 ||394 ||270 ||124 ||lp2diffz3 ||- ||- ||- || ||idp ||184 ||130 ||54 ||sup ||346 ||240 ||106 ||sup ||- ||- ||- || ||smodels ||180 ||130 ||50 ||lp2sat2gminisat ||310 ||225 ||85 ||lp2sat2gminisat ||- ||- ||- || ||lp2sat2minisat ||179 ||135 ||44 ||lp2sat2minisat ||302 ||220 ||82 ||lp2sat2minisat ||- ||- ||- || ||lp2diffz3 ||178 ||135 ||43 ||lp2sat2lminisat ||301 ||220 ||81 ||lp2sat2lminisat ||- ||- ||- || ||lp2sat2lminisat ||171 ||130 ||41 ||smodels ||269 ||165 ||104 ||smodels ||- ||- ||- || === P Problems === ||||System ||Total ||Reachability ||Grammar-Based IE ||Hydraulic Leaking ||Hydraulic Planning ||Stable Marriage ||Partner Units Polynomial || ||clasp ||'''Total''' ||213 ||36 ||72 ||10 ||75 ||6 ||14 || ||Instance ||150 ||25 ||50 ||10 ||50 ||5 ||10 || ||Time ||63 ||11 ||22 ||0 ||25 ||1 ||4 || ||claspfolio ||'''Total''' ||209 ||35 ||72 ||10 ||74 ||5 ||13 || ||Instance ||150 ||25 ||50 ||10 ||50 ||5 ||10 || ||Time ||59 ||10 ||22 ||0 ||24 ||0 ||3 || ||claspd ||'''Total''' ||206 ||36 ||72 ||10 ||75 ||0 ||13 || ||Instance ||145 ||25 ||50 ||10 ||50 ||0 ||10 || ||Time ||61 ||11 ||22 ||0 ||25 ||0 ||3 || ||sup ||'''Total''' ||195 ||29 ||71 ||10 ||74 ||0 ||11 || ||Instance ||140 ||20 ||50 ||10 ||50 ||0 ||10 || ||Time ||55 ||9 ||21 ||0 ||24 ||0 ||1 || ||lp2sat2gminisat ||'''Total''' ||185 ||30 ||66 ||10 ||68 ||0 ||11 || ||Instance ||140 ||20 ||50 ||10 ||50 ||0 ||10 || ||Time ||45 ||10 ||16 ||0 ||18 ||0 ||1 || ||cmodels ||'''Total''' ||184 ||29 ||71 ||10 ||74 ||0 ||0 || ||Instance ||130 ||20 ||50 ||10 ||50 ||0 ||0 || ||Time ||54 ||9 ||21 ||0 ||24 ||0 ||0 || ||idp ||'''Total''' ||184 ||29 ||71 ||10 ||74 ||0 ||0 || ||Instance ||130 ||20 ||50 ||10 ||50 ||0 ||0 || ||Time ||54 ||9 ||21 ||0 ||24 ||0 ||0 || ||smodels ||'''Total''' ||180 ||28 ||70 ||10 ||72 ||0 ||0 || ||Instance ||130 ||20 ||50 ||10 ||50 ||0 ||0 || ||Time ||50 ||8 ||20 ||0 ||22 ||0 ||0 || ||lp2sat2minisat ||'''Total''' ||179 ||30 ||66 ||10 ||68 ||0 ||5 || ||Instance ||135 ||20 ||50 ||10 ||50 ||0 ||5 || ||Time ||44 ||10 ||16 ||0 ||18 ||0 ||0 || ||lp2diffz3 ||'''Total''' ||178 ||35 ||66 ||10 ||67 ||0 ||0 || ||Instance ||135 ||25 ||50 ||10 ||50 ||0 ||0 || ||Time ||43 ||10 ||16 ||0 ||17 ||0 ||0 || ||lp2sat2lminisat ||'''Total''' ||171 ||28 ||66 ||10 ||67 ||0 ||0 || ||Instance ||130 ||20 ||50 ||10 ||50 ||0 ||0 || ||Time ||41 ||8 ||16 ||0 ||17 ||0 ||0 || === NP Problems === ||||System ||Total ||Sokoban Decision ||Knight Tour ||Disjunctive Scheduling ||Packing Problem ||Labyrinth ||MCS Querying ||Numberlink ||Hanoi Tower ||Graph Colouring ||Solitaire ||Weight-Assignment Tree ||Maze Generation || ||claspfolio ||'''Total''' ||609 ||66 ||65 ||37 ||0 ||63 ||75 ||64 ||47 ||55 ||21 ||21 ||95 || ||Instance ||385 ||45 ||35 ||25 ||0 ||40 ||50 ||35 ||35 ||40 ||15 ||15 ||50 || ||Time ||224 ||21 ||30 ||12 ||0 ||23 ||25 ||29 ||12 ||15 ||6 ||6 ||45 || ||clasp ||'''Total''' ||597 ||78 ||63 ||38 ||0 ||78 ||75 ||65 ||39 ||23 ||21 ||21 ||96 || ||Instance ||370 ||50 ||35 ||25 ||0 ||50 ||50 ||35 ||30 ||15 ||15 ||15 ||50 || ||Time ||227 ||28 ||28 ||13 ||0 ||28 ||25 ||30 ||9 ||8 ||6 ||6 ||46 || ||idp ||'''Total''' ||597 ||64 ||74 ||38 ||0 ||52 ||75 ||70 ||65 ||18 ||38 ||8 ||95 || ||Instance ||370 ||45 ||45 ||25 ||0 ||30 ||50 ||40 ||45 ||10 ||25 ||5 ||50 || ||Time ||227 ||19 ||29 ||13 ||0 ||22 ||25 ||30 ||20 ||8 ||13 ||3 ||45 || ||claspd ||'''Total''' ||552 ||68 ||68 ||30 ||0 ||65 ||75 ||69 ||31 ||19 ||11 ||20 ||96 || ||Instance ||355 ||45 ||40 ||25 ||0 ||45 ||50 ||40 ||25 ||10 ||10 ||15 ||50 || ||Time ||197 ||23 ||28 ||5 ||0 ||20 ||25 ||29 ||6 ||9 ||1 ||5 ||46 || ||cmodels ||'''Total''' ||510 ||67 ||56 ||21 ||0 ||62 ||75 ||30 ||51 ||29 ||18 ||6 ||95 || ||Instance ||335 ||45 ||30 ||20 ||0 ||45 ||50 ||20 ||35 ||20 ||15 ||5 ||50 || ||Time ||175 ||22 ||26 ||1 ||0 ||17 ||25 ||10 ||16 ||9 ||3 ||1 ||45 || ||lp2diffz3 ||'''Total''' ||394 ||42 ||55 ||0 ||0 ||0 ||70 ||45 ||47 ||27 ||25 ||0 ||83 || ||Instance ||270 ||30 ||35 ||0 ||0 ||0 ||50 ||30 ||35 ||20 ||20 ||0 ||50 || ||Time ||124 ||12 ||20 ||0 ||0 ||0 ||20 ||15 ||12 ||7 ||5 ||0 ||33 || ||sup ||'''Total''' ||346 ||52 ||40 ||37 ||0 ||58 ||72 ||0 ||31 ||16 ||15 ||25 ||0 || ||Instance ||240 ||35 ||25 ||25 ||0 ||40 ||50 ||0 ||25 ||10 ||10 ||20 ||0 || ||Time ||106 ||17 ||15 ||12 ||0 ||18 ||22 ||0 ||6 ||6 ||5 ||5 ||0 || ||lp2sat2gminisat ||'''Total''' ||310 ||36 ||10 ||32 ||0 ||46 ||71 ||22 ||47 ||17 ||29 ||- ||0 || ||Instance ||225 ||30 ||5 ||25 ||0 ||35 ||50 ||15 ||35 ||10 ||20 ||- ||0 || ||Time ||85 ||6 ||5 ||7 ||0 ||11 ||21 ||7 ||12 ||7 ||9 ||- ||0 || ||lp2sat2minisat ||'''Total''' ||302 ||39 ||0 ||32 ||0 ||52 ||71 ||15 ||47 ||17 ||29 ||- ||0 || ||Instance ||220 ||30 ||0 ||25 ||0 ||40 ||50 ||10 ||35 ||10 ||20 ||- ||0 || ||Time ||82 ||9 ||0 ||7 ||0 ||12 ||21 ||5 ||12 ||7 ||9 ||- ||0 || ||lp2sat2lminisat ||'''Total''' ||301 ||35 ||0 ||32 ||0 ||53 ||71 ||17 ||47 ||17 ||29 ||- ||0 || ||Instance ||220 ||30 ||0 ||25 ||0 ||40 ||50 ||10 ||35 ||10 ||20 ||- ||0 || ||Time ||81 ||5 ||0 ||7 ||0 ||13 ||21 ||7 ||12 ||7 ||9 ||- ||0 || ||smodels ||'''Total''' ||269 ||0 ||55 ||36 ||0 ||9 ||53 ||27 ||0 ||0 ||0 ||0 ||89 || ||Instance ||165 ||0 ||30 ||25 ||0 ||5 ||35 ||20 ||0 ||0 ||0 ||0 ||50 || ||Time ||104 ||0 ||25 ||11 ||0 ||4 ||18 ||7 ||0 ||0 ||0 ||0 ||39 || === Beyond NP Problems === ||||System ||Total ||Strategic Companies ||Minimal Diagnosis || ||claspd ||'''Total''' ||103 ||12 ||91 || ||Instance ||60 ||10 ||50 || ||Time ||43 ||2 ||41 || ||cmodels ||'''Total''' ||72 ||0 ||72 || ||Instance ||45 ||0 ||45 || ||Time ||27 ||0 ||27 || == Detailed Results == === Details: raw file === Detailed results, as a ''raw'' text file, can be downloaded [[attachment:systemcompetition.final.results.raw.detailed.txt|HERE]]. <
> The file contains a line for each instance each system has been asked to solve. === Details: tables === Detailed results are also available as separate PDF files: * [[attachment:System-Results-P-Problems-detailed.pdf||System Track - Details - P problems]] * [[attachment:System-Results-NP-Problems-detailed.pdf||System Track - Details - NP problems]] * [[attachment:System-Results-BndNP-Problems-detailed.pdf||System Track - Details - Beyond P problems]] == Quick reference guide == Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the actual time spent. In case of optimization problems, "quality" of the solutions plays a role as well. Please have a look at the [[ParticipationRules|Scoring and Participation Rules]] and the [[http://www.mat.unical.it/aspcomp2011/files/LanguageSpecifications.pdf|file and language format specifications]] for more details. * System: the system name (quite easy to understand) * Benchmark: the benchmark problem * Instance: the specific instance of the problem * Out|Outcome: the answer provided by the system for the specific instance * Diagnostics: result of the evaluation of the answer (essentially, whether the system answered correctly or not) * Time: the total time spent by the system while solving the specific instance * Optimum Cost: the "cost" of the witness solution returned by the system, in case of istance of an optimization problem * Total Score: the total score rewarded for the specific instance * Inst\Opt Score - Time Score: the two components of the score, as introduced above. * WF (Cost) : Witness Found (Cost) * OF (Cost) : Optimum Found (Cost) * Inc : Inconsistent * TO : Time Out * MO/F : Mem Out Fail * N/A : Not Available / Not Applicable