welcome: please sign in
location: Diff for "SystemTrackFinalResults"
Differences between revisions 28 and 125 (spanning 97 versions)
⇤ ← Revision 28 as of 2011-05-11 11:53:47 →
Size: 22825
Editor: CarmenSantoro
Comment:
← Revision 125 as of 2011-05-22 09:53:57 → ⇥
Size: 23843
Editor: CarmenSantoro
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
#acl EditorsGroup:read,write,delete,revert,admin All: #acl EditorsGroup:read,write,delete,revert,admin All:read
Line 5: Line 5:
== P Problems == <<TableOfContents>>
Line 7: Line 8:
||||<style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">Reachability ||<style="font-weight: bold; text-align: center;">Grammar-Based IE ||<style="font-weight: bold; text-align: center;">Hydraulic Leaking ||<style="font-weight: bold; text-align: center;">Hydraulic Planning ||<style="font-weight: bold; text-align: center;">Stable Marriage ||<style="font-weight: bold; text-align: center;">Partner Units Polynomial ||   == System Track Final Results: Overall ==
||<tablewidth="1%"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">Instance ||<style="font-weight: bold; text-align: center;">Time ||
||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|claspd]] ||<style="font-weight: bold;">861 ||560 ||301 ||
||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|claspfolio]] ||<style="font-weight: bold;">818 ||535 ||283 ||
||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|clasp]] ||<style="font-weight: bold;">810 ||520 ||290 ||
||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|idp]] ||<style="font-weight: bold;">781 ||500 ||281 ||
||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|cmodels]] ||<style="font-weight: bold;">766 ||510 ||256 ||
||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|lp2diffz3]] ||<style="font-weight: bold;">572 ||405 ||167 ||
||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|sup]] ||<style="font-weight: bold;">541 ||380 ||161 ||
||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|lp2sat2gminisat]] ||<style="font-weight: bold;">495 ||365 ||130 ||
||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|lp2sat2minisat]] ||<style="font-weight: bold;">481 ||355 ||126 ||
||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|lp2sat2lminisat]] ||<style="font-weight: bold;">472 ||350 ||122 ||
||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|smodels]] ||<style="font-weight: bold;">449 ||295 ||154 ||

== Final results by categories ==
=== Overview ===
||||||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center;">'''P''' ||<style="text-align: center;" |13> ||||||||<style="text-align: center;">'''NP''' ||<style="text-align: center;" |13> ||||||||<style="text-align: center;">'''Beyond NP''' ||
||'''System''' ||'''Total Score''' ||'''Inst. Score''' ||'''Time Score''' ||<style="text-align: center;">'''System''' ||'''Total Score''' ||'''Inst. Score''' ||'''Time Score''' ||<style="text-align: center;">'''System''' ||'''Total Score''' ||'''Inst. Score''' ||'''Time Score''' ||
||<style="font-weight: bold;">clasp ||213 ||150 ||63 ||<style="text-align: center; font-weight: bold;">claspfolio ||609 ||385 ||224 ||<style="text-align: center; font-weight: bold;">claspd ||103 ||60 ||43 ||
||<style="font-weight: bold;">claspfolio ||209 ||150 ||59 ||<style="text-align: center; font-weight: bold;">clasp ||597 ||370 ||227 ||<style="text-align: center; font-weight: bold;">cmodels ||72 ||45 ||27 ||
||<style="font-weight: bold;">claspd ||206 ||145 ||61 ||<style="text-align: center; font-weight: bold;">idp ||597 ||370 ||227 ||<style="text-align: center; font-weight: bold;">claspfolio ||- ||- ||- ||
||<style="font-weight: bold;">sup ||195 ||140 ||55 ||<style="text-align: center; font-weight: bold;">claspd ||552 ||355 ||197 ||<style="text-align: center; font-weight: bold;">clasp ||- ||- ||- ||
||<style="font-weight: bold;">lp2sat2gminisat ||185 ||140 ||45 ||<style="text-align: center; font-weight: bold;">cmodels ||510 ||335 ||175 ||<style="text-align: center; font-weight: bold;">idp ||- ||- ||- ||
||<style="font-weight: bold;">cmodels ||184 ||130 ||54 ||<style="text-align: center; font-weight: bold;">lp2diffz3 ||394 ||270 ||124 ||<style="text-align: center; font-weight: bold;">lp2diffz3 ||- ||- ||- ||
||<style="font-weight: bold;">idp ||184 ||130 ||54 ||<style="text-align: center; font-weight: bold;">sup ||346 ||240 ||106 ||<style="text-align: center; font-weight: bold;">sup ||- ||- ||- ||
||<style="font-weight: bold;">smodels ||180 ||130 ||50 ||<style="text-align: center; font-weight: bold;">lp2sat2gminisat ||310 ||225 ||85 ||<style="text-align: center; font-weight: bold;">lp2sat2gminisat ||- ||- ||- ||
||<style="font-weight: bold;">lp2sat2minisat ||179 ||135 ||44 ||<style="text-align: center; font-weight: bold;">lp2sat2minisat ||302 ||220 ||82 ||<style="text-align: center; font-weight: bold;">lp2sat2minisat ||- ||- ||- ||
||<style="font-weight: bold;">lp2diffz3 ||178 ||135 ||43 ||<style="text-align: center; font-weight: bold;">lp2sat2lminisat ||301 ||220 ||81 ||<style="text-align: center; font-weight: bold;">lp2sat2lminisat ||- ||- ||- ||
||<style="font-weight: bold;">lp2sat2lminisat ||171 ||130 ||41 ||<style="text-align: center; font-weight: bold;">smodels ||269 ||165 ||104 ||<style="text-align: center; font-weight: bold;">smodels ||- ||- ||- ||

=== P Problems ===
||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">Reachability ||<style="font-weight: bold; text-align: center;">Grammar-Based IE ||<style="font-weight: bold; text-align: center;">Hydraulic Leaking ||<style="font-weight: bold; text-align: center;">Hydraulic Planning ||<style="font-weight: bold; text-align: center;">Stable Marriage ||<style="font-weight: bold; text-align: center;">Partner Units Polynomial ||
Line 42: Line 75:
=== NP Problems ===
||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">[[SystemSokobanDecisionResults|Sokoban Decision]] ||<style="font-weight: bold; text-align: center;">[[SystemKnightTourResults|Knight Tour]] ||<style="font-weight: bold; text-align: center;">[[SystemDisjunctiveSchedulingResults|Disjunctive Scheduling]] ||<style="font-weight: bold; text-align: center;">[[SystemPackingProblemResults|Packing Problem]] ||<style="font-weight: bold; text-align: center;">[[SystemLabyrinthResults|Labyrinth]] ||<style="font-weight: bold; text-align: center;">[[SystemMCSQueryingResults|MCS Querying]] ||<style="font-weight: bold; text-align: center;">[[SystemNumberlinkResults|Numberlink]] ||<style="font-weight: bold; text-align: center;">[[SystemHanoiTowerResults|Hanoi Tower]] ||<style="font-weight: bold; text-align: center;">[[SystemGraphColouringResults|Graph Colouring]] ||<style="font-weight: bold; text-align: center;">[[SystemSolitaireResults|Solitaire]] ||<style="font-weight: bold; text-align: center;">[[SystemWeightAssignmentTreeResults|Weight-Assignment Tree]] ||<style="font-weight: bold; text-align: center;">[[SystemMazeGenerationResults|Maze Generation]] ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>claspfolio ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">609 ||<style="font-weight: bold;">66 ||<style="font-weight: bold;">65 ||<style="font-weight: bold;">37 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">63 ||<style="font-weight: bold;">75 ||<style="font-weight: bold;">64 ||<style="font-weight: bold;">47 ||<style="font-weight: bold;">55 ||<style="font-weight: bold;">21 ||<style="font-weight: bold;">21 ||<style="font-weight: bold;">95 ||
||<style="text-align: right;">Instance ||385 ||45 ||35 ||25 ||0 ||40 ||50 ||35 ||35 ||40 ||15 ||15 ||50 ||
||<style="text-align: right;">Time ||224 ||21 ||30 ||12 ||0 ||23 ||25 ||29 ||12 ||15 ||6 ||6 ||45 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>clasp ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">597 ||<style="font-weight: bold;">78 ||<style="font-weight: bold;">63 ||<style="font-weight: bold;">38 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">78 ||<style="font-weight: bold;">75 ||<style="font-weight: bold;">65 ||<style="font-weight: bold;">39 ||<style="font-weight: bold;">23 ||<style="font-weight: bold;">21 ||<style="font-weight: bold;">21 ||<style="font-weight: bold;">96 ||
||<style="text-align: right;">Instance ||370 ||50 ||35 ||25 ||0 ||50 ||50 ||35 ||30 ||15 ||15 ||15 ||50 ||
||<style="text-align: right;">Time ||227 ||28 ||28 ||13 ||0 ||28 ||25 ||30 ||9 ||8 ||6 ||6 ||46 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>idp ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">597 ||<style="font-weight: bold;">64 ||<style="font-weight: bold;">74 ||<style="font-weight: bold;">38 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">52 ||<style="font-weight: bold;">75 ||<style="font-weight: bold;">70 ||<style="font-weight: bold;">65 ||<style="font-weight: bold;">18 ||<style="font-weight: bold;">38 ||<style="font-weight: bold;">8 ||<style="font-weight: bold;">95 ||
||<style="text-align: right;">Instance ||370 ||45 ||45 ||25 ||0 ||30 ||50 ||40 ||45 ||10 ||25 ||5 ||50 ||
||<style="text-align: right;">Time ||227 ||19 ||29 ||13 ||0 ||22 ||25 ||30 ||20 ||8 ||13 ||3 ||45 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>claspd ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">552 ||<style="font-weight: bold;">68 ||<style="font-weight: bold;">68 ||<style="font-weight: bold;">30 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">65 ||<style="font-weight: bold;">75 ||<style="font-weight: bold;">69 ||<style="font-weight: bold;">31 ||<style="font-weight: bold;">19 ||<style="font-weight: bold;">11 ||<style="font-weight: bold;">20 ||<style="font-weight: bold;">96 ||
||<style="text-align: right;">Instance ||355 ||45 ||40 ||25 ||0 ||45 ||50 ||40 ||25 ||10 ||10 ||15 ||50 ||
||<style="text-align: right;">Time ||197 ||23 ||28 ||5 ||0 ||20 ||25 ||29 ||6 ||9 ||1 ||5 ||46 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>cmodels ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">510 ||<style="font-weight: bold;">67 ||<style="font-weight: bold;">56 ||<style="font-weight: bold;">21 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">62 ||<style="font-weight: bold;">75 ||<style="font-weight: bold;">30 ||<style="font-weight: bold;">51 ||<style="font-weight: bold;">29 ||<style="font-weight: bold;">18 ||<style="font-weight: bold;">6 ||<style="font-weight: bold;">95 ||
||<style="text-align: right;">Instance ||335 ||45 ||30 ||20 ||0 ||45 ||50 ||20 ||35 ||20 ||15 ||5 ||50 ||
||<style="text-align: right;">Time ||175 ||22 ||26 ||1 ||0 ||17 ||25 ||10 ||16 ||9 ||3 ||1 ||45 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2diffz3 ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">394 ||<style="font-weight: bold;">42 ||<style="font-weight: bold;">55 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">70 ||<style="font-weight: bold;">45 ||<style="font-weight: bold;">47 ||<style="font-weight: bold;">27 ||<style="font-weight: bold;">25 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">83 ||
||<style="text-align: right;">Instance ||270 ||30 ||35 ||0 ||0 ||0 ||50 ||30 ||35 ||20 ||20 ||0 ||50 ||
||<style="text-align: right;">Time ||124 ||12 ||20 ||0 ||0 ||0 ||20 ||15 ||12 ||7 ||5 ||0 ||33 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>sup ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">346 ||<style="font-weight: bold;">52 ||<style="font-weight: bold;">40 ||<style="font-weight: bold;">37 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">58 ||<style="font-weight: bold;">72 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">31 ||<style="font-weight: bold;">16 ||<style="font-weight: bold;">15 ||<style="font-weight: bold;">25 ||<style="font-weight: bold;">0 ||
||<style="text-align: right;">Instance ||240 ||35 ||25 ||25 ||0 ||40 ||50 ||0 ||25 ||10 ||10 ||20 ||0 ||
||<style="text-align: right;">Time ||106 ||17 ||15 ||12 ||0 ||18 ||22 ||0 ||6 ||6 ||5 ||5 ||0 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2sat2gminisat ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">310 ||<style="font-weight: bold;">36 ||<style="font-weight: bold;">10 ||<style="font-weight: bold;">32 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">46 ||<style="font-weight: bold;">71 ||<style="font-weight: bold;">22 ||<style="font-weight: bold;">47 ||<style="font-weight: bold;">17 ||<style="font-weight: bold;">29 ||<style="font-weight: bold;">- ||<style="font-weight: bold;">0 ||
||<style="text-align: right;">Instance ||225 ||30 ||5 ||25 ||0 ||35 ||50 ||15 ||35 ||10 ||20 ||- ||0 ||
||<style="text-align: right;">Time ||85 ||6 ||5 ||7 ||0 ||11 ||21 ||7 ||12 ||7 ||9 ||- ||0 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2sat2minisat ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">302 ||<style="font-weight: bold;">39 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">32 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">52 ||<style="font-weight: bold;">71 ||<style="font-weight: bold;">15 ||<style="font-weight: bold;">47 ||<style="font-weight: bold;">17 ||<style="font-weight: bold;">29 ||<style="font-weight: bold;">- ||<style="font-weight: bold;">0 ||
||<style="text-align: right;">Instance ||220 ||30 ||0 ||25 ||0 ||40 ||50 ||10 ||35 ||10 ||20 ||- ||0 ||
||<style="text-align: right;">Time ||82 ||9 ||0 ||7 ||0 ||12 ||21 ||5 ||12 ||7 ||9 ||- ||0 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2sat2lminisat ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">301 ||<style="font-weight: bold;">35 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">32 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">53 ||<style="font-weight: bold;">71 ||<style="font-weight: bold;">17 ||<style="font-weight: bold;">47 ||<style="font-weight: bold;">17 ||<style="font-weight: bold;">29 ||<style="font-weight: bold;">- ||<style="font-weight: bold;">0 ||
||<style="text-align: right;">Instance ||220 ||30 ||0 ||25 ||0 ||40 ||50 ||10 ||35 ||10 ||20 ||- ||0 ||
||<style="text-align: right;">Time ||81 ||5 ||0 ||7 ||0 ||13 ||21 ||7 ||12 ||7 ||9 ||- ||0 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>smodels ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">269 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">55 ||<style="font-weight: bold;">36 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">9 ||<style="font-weight: bold;">53 ||<style="font-weight: bold;">27 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">89 ||
||<style="text-align: right;">Instance ||165 ||0 ||30 ||25 ||0 ||5 ||35 ||20 ||0 ||0 ||0 ||0 ||50 ||
||<style="text-align: right;">Time ||104 ||0 ||25 ||11 ||0 ||4 ||18 ||7 ||0 ||0 ||0 ||0 ||39 ||
Line 43: Line 111:
=== Beyond NP Problems ===
||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">[[SystemStrategicCompaniesResults|Strategic Companies]] ||<style="font-weight: bold; text-align: center;">[[SystemMinimalDiagnosisResults|Minimal Diagnosis]] ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>claspd ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">103 ||<style="font-weight: bold;">12 ||<style="font-weight: bold;">91 ||
||<style="text-align: right;">Instance ||60 ||10 ||50 ||
||<style="text-align: right;">Time ||43 ||2 ||41 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>cmodels ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">72 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">72 ||
||<style="text-align: right;">Instance ||45 ||0 ||45 ||
||<style="text-align: right;">Time ||27 ||0 ||27 ||
Line 44: Line 120:
== Detailed Results ==
=== Details: raw file ===
Detailed results, as a ''raw'' text file, can be downloaded [[attachment:systemcompetition.final.results.raw.detailed.txt|HERE]].
<<BR>>
The file contains a line for each instance each system has been asked to solve.
Line 45: Line 126:
== NP Problems ==
<<BR>>
||||<style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">Sokoban Decision ||<style="font-weight: bold; text-align: center;">Knight Tour ||<style="font-weight: bold; text-align: center;">Disjunctive Scheduling ||<style="font-weight: bold; text-align: center;">Packing Problem ||<style="font-weight: bold; text-align: center;">Labyrinth ||<style="font-weight: bold; text-align: center;">MCS Querying ||<style="font-weight: bold; text-align: center;">Numberlink ||<style="font-weight: bold; text-align: center;">Hanoi Tower ||<style="font-weight: bold; text-align: center;">Graph Colouring ||<style="font-weight: bold; text-align: center;">Solitaire ||<style="font-weight: bold; text-align: center;">Weight-Assignment Tree ||<style="font-weight: bold; text-align: center;">Maze Generation ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>claspfolio ||<style="text-align: right;">'''Total''' ||609 ||385 ||224 ||66 ||45 ||21 ||65 ||35 ||30 ||37 ||25 ||12 ||0 ||0 ||0 ||
||<style="text-align: right;">Instance ||63 ||40 ||23 ||75 ||50 ||25 ||64 ||35 ||29 ||47 ||35 ||12 ||
||<style="text-align: right;">Time ||55 ||40 ||15 ||21 ||15 ||6 ||21 ||15 ||6 ||95 ||50 ||45 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>clasp ||<style="text-align: right;">'''Total''' ||597 ||370 ||227 ||78 ||50 ||28 ||63 ||35 ||28 ||38 ||25 ||13 ||0 ||0 ||0 ||
||<style="text-align: right;">Instance ||78 ||50 ||28 ||75 ||50 ||25 ||65 ||35 ||30 ||39 ||30 ||9 ||
||<style="text-align: right;">Time ||23 ||15 ||8 ||21 ||15 ||6 ||21 ||15 ||6 ||96 ||50 ||46 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>idp ||<style="text-align: right;">'''Total''' ||597 ||370 ||227 ||64 ||45 ||19 ||74 ||45 ||29 ||38 ||25 ||13 ||0 ||0 ||0 ||
||<style="text-align: right;">Instance ||52 ||30 ||22 ||75 ||50 ||25 ||70 ||40 ||30 ||65 ||45 ||20 ||
||<style="text-align: right;">Time ||18 ||10 ||8 ||38 ||25 ||13 ||8 ||5 ||3 ||95 ||50 ||45 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>claspd ||<style="text-align: right;">'''Total''' ||552 ||355 ||197 ||68 ||45 ||23 ||68 ||40 ||28 ||30 ||25 ||5 ||0 ||0 ||0 ||
||<style="text-align: right;">Instance ||65 ||45 ||20 ||75 ||50 ||25 ||69 ||40 ||29 ||31 ||25 ||6 ||
||<style="text-align: right;">Time ||19 ||10 ||9 ||11 ||10 ||1 ||20 ||15 ||5 ||96 ||50 ||46 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>cmodels ||<style="text-align: right;">'''Total''' ||510 ||335 ||175 ||67 ||45 ||22 ||56 ||30 ||26 ||21 ||20 ||1 ||0 ||0 ||0 ||
||<style="text-align: right;">Instance ||62 ||45 ||17 ||75 ||50 ||25 ||30 ||20 ||10 ||51 ||35 ||16 ||
||<style="text-align: right;">Time ||29 ||20 ||9 ||18 ||15 ||3 ||6 ||5 ||1 ||95 ||50 ||45 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2diffz3 ||<style="text-align: right;">'''Total''' ||394 ||270 ||124 ||42 ||30 ||12 ||55 ||35 ||20 ||0 ||0 ||0 ||0 ||0 ||0 ||
||<style="text-align: right;">Instance ||0 ||0 ||0 ||70 ||50 ||20 ||45 ||30 ||15 ||47 ||35 ||12 ||
||<style="text-align: right;">Time ||27 ||20 ||7 ||25 ||20 ||5 ||0 ||0 ||0 ||83 ||50 ||33 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>sup ||<style="text-align: right;">'''Total''' ||346 ||240 ||106 ||52 ||35 ||17 ||40 ||25 ||15 ||37 ||25 ||12 ||0 ||0 ||0 ||
||<style="text-align: right;">Instance ||58 ||40 ||18 ||72 ||50 ||22 ||0 ||0 ||0 ||31 ||25 ||6 ||
||<style="text-align: right;">Time ||16 ||10 ||6 ||15 ||10 ||5 ||25 ||20 ||5 ||0 ||0 ||0 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2sat2gminisat ||<style="text-align: right;">'''Total''' ||310 ||225 ||85 ||36 ||30 ||6 ||10 ||5 ||5 ||32 ||25 ||7 ||0 ||0 ||0 ||
||<style="text-align: right;">Instance ||46 ||35 ||11 ||71 ||50 ||21 ||22 ||15 ||7 ||47 ||35 ||12 ||
||<style="text-align: right;">Time ||17 ||10 ||7 ||29 ||20 ||9 ||0 ||0 ||0 ||0 ||0 ||0 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2sat2minisat ||<style="text-align: right;">'''Total''' ||302 ||220 ||82 ||39 ||30 ||9 ||0 ||0 ||0 ||32 ||25 ||7 ||0 ||0 ||0 ||
||<style="text-align: right;">Instance ||52 ||40 ||12 ||71 ||50 ||21 ||15 ||10 ||5 ||47 ||35 ||12 ||
||<style="text-align: right;">Time ||17 ||10 ||7 ||29 ||20 ||9 ||0 ||0 ||0 ||0 ||0 ||0 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2sat2lminisat ||<style="text-align: right;">'''Total''' ||301 ||220 ||81 ||35 ||30 ||5 ||0 ||0 ||0 ||32 ||25 ||7 ||0 ||0 ||0 ||
||<style="text-align: right;">Instance ||53 ||40 ||13 ||71 ||50 ||21 ||17 ||10 ||7 ||47 ||35 ||12 ||
||<style="text-align: right;">Time ||17 ||10 ||7 ||29 ||20 ||9 ||0 ||0 ||0 ||0 ||0 ||0 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>smodels ||<style="text-align: right;">'''Total''' ||269 ||165 ||104 ||0 ||0 ||0 ||55 ||30 ||25 ||36 ||25 ||11 ||0 ||0 ||0 ||
||<style="text-align: right;">Instance ||9 ||5 ||4 ||53 ||35 ||18 ||27 ||20 ||7 ||0 ||0 ||0 ||
||<style="text-align: right;">Time ||0 ||0 ||0 ||0 ||0 ||0 ||0 ||0 ||0 ||89 ||50 ||39 ||		 === Details: tables ===
Detailed results are also available as separate PDF files:
 * [[attachment:System-Results-P-Problems-detailed.pdf||System Track - Details - P problems]]
 * [[attachment:System-Results-NP-Problems-detailed.pdf||System Track - Details - NP problems]]
 * [[attachment:System-Results-BndNP-Problems-detailed.pdf||System Track - Details - Beyond P problems]]
Line 82: Line 132:
== Quick reference guide ==
Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the actual time spent. In case of optimization problems, "quality" of the solutions plays a role as well. Please have a look at the [[ParticipationRules|Scoring and Participation Rules]] and the [[http://www.mat.unical.it/aspcomp2011/files/LanguageSpecifications.pdf|file and language format specifications]] for more details.
Line 83: Line 135:
 * System: the system name (quite easy to understand)
 * Benchmark: the benchmark problem
 * Instance: the specific instance of the problem
 * Out|Outcome: the answer provided by the system for the specific instance
 * Diagnostics: result of the evaluation of the answer (essentially, whether the system answered correctly or not)
 * Time: the total time spent by the system while solving the specific instance
 * Optimum Cost: the "cost" of the witness solution returned by the system, in case of istance of an optimization problem
 * Total Score: the total score rewarded for the specific instance
 * Inst\Opt Score - Time Score: the two components of the score, as introduced above.
Line 84: Line 145:

<<BR>>
||||<style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">P ||<style="font-weight: bold; text-align: center;">NP ||<style="font-weight: bold; text-align: center;">Beyond NP ||<style="font-weight: bold; text-align: center;">Reachability ||<style="font-weight: bold; text-align: center;">Grammar-Based IE ||<style="font-weight: bold; text-align: center;">Hydraulic Leaking ||<style="font-weight: bold; text-align: center;">Hydraulic Planning ||<style="font-weight: bold; text-align: center;">Stable Marriage ||<style="font-weight: bold; text-align: center;">Partner Units Polynomial ||<style="font-weight: bold; text-align: center;">Sokoban Decision ||<style="font-weight: bold; text-align: center;">Knight Tour ||<style="font-weight: bold; text-align: center;">Disjunctive Scheduling ||<style="font-weight: bold; text-align: center;">Packing Problem ||<style="font-weight: bold; text-align: center;">Labyrinth ||<style="font-weight: bold; text-align: center;">MCS Querying ||<style="font-weight: bold; text-align: center;">Numberlink ||<style="font-weight: bold; text-align: center;">Hanoi Tower ||<style="font-weight: bold; text-align: center;">Graph Colouring ||<style="font-weight: bold; text-align: center;">Solitaire ||<style="font-weight: bold; text-align: center;">Weight-Assignment Tree ||<style="font-weight: bold; text-align: center;">Maze Generation ||<style="font-weight: bold; text-align: center;">Strategic Companies ||<style="font-weight: bold; text-align: center;">Minimal Diagnosis ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>claspd ||<style="text-align: right;">'''Total''' ||'''861''' ||'''206''' ||'''552''' ||'''103''' ||'''36''' ||'''72''' ||'''10''' ||'''75''' ||'''0''' ||'''13''' ||'''68''' ||'''68''' ||'''30''' ||'''0''' ||'''65''' ||'''75''' ||'''69''' ||'''31''' ||'''19''' ||'''11''' ||'''20''' ||'''96''' ||'''12''' ||'''91''' ||
||<style="text-align: right;">Instance ||560 ||145 ||355 ||60 ||25 ||50 ||10 ||50 ||0 ||10 ||45 ||40 ||25 ||0 ||45 ||50 ||40 ||25 ||10 ||10 ||15 ||50 ||10 ||50 ||
||<style="text-align: right;">Time ||301 ||61 ||197 ||43 ||11 ||22 ||0 ||25 ||0 ||3 ||23 ||28 ||5 ||0 ||20 ||25 ||29 ||6 ||9 ||1 ||5 ||46 ||2 ||41 ||
||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>claspfolio ||<style="text-align: right;">'''Total''' ||'''818''' ||'''209''' ||'''609''' ||'''0''' ||'''35''' ||'''72''' ||'''10''' ||'''74''' ||'''5''' ||'''13''' ||'''66''' ||'''65''' ||'''37''' ||'''0''' ||'''63''' ||'''75''' ||'''64''' ||'''47''' ||'''55''' ||'''21''' ||'''21''' ||'''95''' ||'''0''' ||'''0''' ||
||<style="text-align: right;">Instance ||535 ||150 ||385 ||0 ||25 ||50 ||10 ||50 ||5 ||10 ||45 ||35 ||25 ||0 ||40 ||50 ||35 ||35 ||40 ||15 ||15 ||50 ||0 ||0 ||
||<style="text-align: right;">Time ||283 ||59 ||224 ||0 ||10 ||22 ||0 ||24 ||0 ||3 ||21 ||30 ||12 ||0 ||23 ||25 ||29 ||12 ||15 ||6 ||6 ||45 ||0 ||0 ||
||<style="text-align: center; vertical-align: top;" |3>'''clasp''' ||<style="text-align: right;">'''Total''' ||'''810''' ||'''213''' ||'''597''' ||'''0''' ||'''36''' ||'''72''' ||'''10''' ||'''75''' ||'''6''' ||'''14''' ||'''78''' ||'''63''' ||'''38''' ||'''0''' ||'''78''' ||'''75''' ||'''65''' ||'''39''' ||'''23''' ||'''21''' ||'''21''' ||'''96''' ||'''0''' ||'''0''' ||
||<style="text-align: right;">Instance ||520 ||150 ||370 ||0 ||25 ||50 ||10 ||50 ||5 ||10 ||50 ||35 ||25 ||0 ||50 ||50 ||35 ||30 ||15 ||15 ||15 ||50 ||0 ||0 ||
||<style="text-align: right;">Time ||290 ||63 ||227 ||0 ||11 ||22 ||0 ||25 ||1 ||4 ||28 ||28 ||13 ||0 ||28 ||25 ||30 ||9 ||8 ||6 ||6 ||46 ||0 ||0 ||
||<style="text-align: center; vertical-align: top;" |3>'''idp''' ||<style="text-align: right;">'''Total''' ||'''781''' ||'''184''' ||'''597''' ||'''0''' ||'''29''' ||'''71''' ||'''10''' ||'''74''' ||'''0''' ||'''0''' ||'''64''' ||'''74''' ||'''38''' ||'''0''' ||'''52''' ||'''75''' ||'''70''' ||'''65''' ||'''18''' ||'''38''' ||'''8''' ||'''95''' ||'''0''' ||'''0''' ||
||<style="text-align: right;">Instance ||500 ||130 ||370 ||0 ||20 ||50 ||10 ||50 ||0 ||0 ||45 ||45 ||25 ||0 ||30 ||50 ||40 ||45 ||10 ||25 ||5 ||50 ||0 ||0 ||
||<style="text-align: right;">Time ||281 ||54 ||227 ||0 ||9 ||21 ||0 ||24 ||0 ||0 ||19 ||29 ||13 ||0 ||22 ||25 ||30 ||20 ||8 ||13 ||3 ||45 ||0 ||0 ||
||<style="text-align: center; vertical-align: top;" |3>'''cmodels''' ||<style="text-align: right;">'''Total''' ||'''766''' ||'''184''' ||'''510''' ||'''72''' ||'''29''' ||'''71''' ||'''10''' ||'''74''' ||'''0''' ||'''0''' ||'''67''' ||'''56''' ||'''21''' ||'''0''' ||'''62''' ||'''75''' ||'''30''' ||'''51''' ||'''29''' ||'''18''' ||'''6''' ||'''95''' ||'''0''' ||'''72''' ||
||<style="text-align: right;">Instance ||510 ||130 ||335 ||45 ||20 ||50 ||10 ||50 ||0 ||0 ||45 ||30 ||20 ||0 ||45 ||50 ||20 ||35 ||20 ||15 ||5 ||50 ||0 ||45 ||
||<style="text-align: right;">Time ||256 ||54 ||175 ||27 ||9 ||21 ||0 ||24 ||0 ||0 ||22 ||26 ||1 ||0 ||17 ||25 ||10 ||16 ||9 ||3 ||1 ||45 ||0 ||27 ||
||<style="text-align: center; vertical-align: top;" |3>'''lp2diffz3''' ||<style="text-align: right;">'''Total''' ||'''572''' ||'''178''' ||'''394''' ||'''0''' ||'''35''' ||'''66''' ||'''10''' ||'''67''' ||'''0''' ||'''0''' ||'''42''' ||'''55''' ||'''0''' ||'''0''' ||'''0''' ||'''70''' ||'''45''' ||'''47''' ||'''27''' ||'''25''' ||'''0''' ||'''83''' ||'''0''' ||'''0''' ||
||<style="text-align: right;">Instance ||405 ||135 ||270 ||0 ||25 ||50 ||10 ||50 ||0 ||0 ||30 ||35 ||0 ||0 ||0 ||50 ||30 ||35 ||20 ||20 ||0 ||50 ||0 ||0 ||
||<style="text-align: right;">Time ||167 ||43 ||124 ||0 ||10 ||16 ||0 ||17 ||0 ||0 ||12 ||20 ||0 ||0 ||0 ||20 ||15 ||12 ||7 ||5 ||0 ||33 ||0 ||0 ||
||<style="text-align: center; vertical-align: top;" |3>'''sup''' ||<style="text-align: right;">'''Total''' ||'''541''' ||'''195''' ||'''346''' ||'''0''' ||'''29''' ||'''71''' ||'''10''' ||'''74''' ||'''0''' ||'''11''' ||'''52''' ||'''40''' ||'''37''' ||'''0''' ||'''58''' ||'''72''' ||'''0''' ||'''31''' ||'''16''' ||'''15''' ||'''25''' ||'''0''' ||'''0''' ||'''0''' ||
||<style="text-align: right;">Instance ||380 ||140 ||240 ||0 ||20 ||50 ||10 ||50 ||0 ||10 ||35 ||25 ||25 ||0 ||40 ||50 ||0 ||25 ||10 ||10 ||20 ||0 ||0 ||0 ||
||<style="text-align: right;">Time ||161 ||55 ||106 ||0 ||9 ||21 ||0 ||24 ||0 ||1 ||17 ||15 ||12 ||0 ||18 ||22 ||0 ||6 ||6 ||5 ||5 ||0 ||0 ||0 ||
||<style="text-align: center; vertical-align: top;" |3>'''lp2sat2gmsat''' ||<style="text-align: right;">'''Total''' ||'''495''' ||'''185''' ||'''310''' ||'''0''' ||'''30''' ||'''66''' ||'''10''' ||'''68''' ||'''0''' ||'''11''' ||'''36''' ||'''10''' ||'''32''' ||'''0''' ||'''46''' ||'''71''' ||'''22''' ||'''47''' ||'''17''' ||'''29''' ||'''-''' ||'''0''' ||'''0''' ||'''0''' ||
||<style="text-align: right;">Instance ||365 ||140 ||225 ||0 ||20 ||50 ||10 ||50 ||0 ||10 ||30 ||5 ||25 ||0 ||35 ||50 ||15 ||35 ||10 ||20 ||- ||0 ||0 ||0 ||
||<style="text-align: right;">Time ||130 ||45 ||85 ||0 ||10 ||16 ||0 ||18 ||0 ||1 ||6 ||5 ||7 ||0 ||11 ||21 ||7 ||12 ||7 ||9 ||- ||0 ||0 ||0 ||
||<style="text-align: center; vertical-align: top;" |3>'''lp2sat2msat''' ||<style="text-align: right;">'''Total''' ||'''481''' ||'''179''' ||'''302''' ||'''0''' ||'''30''' ||'''66''' ||'''10''' ||'''68''' ||'''0''' ||'''5''' ||'''39''' ||'''0''' ||'''32''' ||'''0''' ||'''52''' ||'''71''' ||'''15''' ||'''47''' ||'''17''' ||'''29''' ||'''-''' ||'''0''' ||'''0''' ||'''0''' ||
||<style="text-align: right;">Instance ||355 ||135 ||220 ||0 ||20 ||50 ||10 ||50 ||0 ||5 ||30 ||0 ||25 ||0 ||40 ||50 ||10 ||35 ||10 ||20 ||- ||0 ||0 ||0 ||
||<style="text-align: right;">Time ||126 ||44 ||82 ||0 ||10 ||16 ||0 ||18 ||0 ||0 ||9 ||0 ||7 ||0 ||12 ||21 ||5 ||12 ||7 ||9 ||- ||0 ||0 ||0 ||
||<style="text-align: center; vertical-align: top;" |3>'''lp2sat2lmsat''' ||<style="text-align: right;">'''Total''' ||'''472''' ||'''171''' ||'''301''' ||'''0''' ||'''28''' ||'''66''' ||'''10''' ||'''67''' ||'''0''' ||'''0''' ||'''35''' ||'''0''' ||'''32''' ||'''0''' ||'''53''' ||'''71''' ||'''17''' ||'''47''' ||'''17''' ||'''29''' ||'''-''' ||'''0''' ||'''0''' ||'''0''' ||
||<style="text-align: right;">Instance ||350 ||130 ||220 ||0 ||20 ||50 ||10 ||50 ||0 ||0 ||30 ||0 ||25 ||0 ||40 ||50 ||10 ||35 ||10 ||20 ||- ||0 ||0 ||0 ||
||<style="text-align: right;">Time ||122 ||41 ||81 ||0 ||8 ||16 ||0 ||17 ||0 ||0 ||5 ||0 ||7 ||0 ||13 ||21 ||7 ||12 ||7 ||9 ||- ||0 ||0 ||0 ||
||<style="text-align: center; vertical-align: top;" |3>'''smodels''' ||<style="text-align: right;">'''Total''' ||'''449''' ||'''180''' ||'''269''' ||'''0''' ||'''28''' ||'''70''' ||'''10''' ||'''72''' ||'''0''' ||'''0''' ||'''0''' ||'''55''' ||'''36''' ||'''0''' ||'''9''' ||'''53''' ||'''27''' ||'''0''' ||'''0''' ||'''0''' ||'''0''' ||'''89''' ||'''0''' ||'''0''' ||
||<style="text-align: right;">Instance ||295 ||130 ||165 ||0 ||20 ||50 ||10 ||50 ||0 ||0 ||0 ||30 ||25 ||0 ||5 ||35 ||20 ||0 ||0 ||0 ||0 ||50 ||0 ||0 ||
||<style="text-align: right;">Time ||154 ||50 ||104 ||0 ||8 ||20 ||0 ||22 ||0 ||0 ||0 ||25 ||11 ||0 ||4 ||18 ||7 ||0 ||0 ||0 ||0 ||39 ||0 ||0 ||




=== Final results by categories ===
<<BR>>
||||||||<style="text-align: center;">'''P''' || ||||||||<style="text-align: center;">'''NP''' || ||||||||<style="text-align: center;">'''Beyond NP''' ||
||'''System''' ||'''Total Score''' ||'''Inst. Score''' ||'''Time Score''' ||||<style="text-align: center;">'''System''' ||'''Total Score''' ||'''Inst. Score''' ||'''Time Score''' ||||<style="text-align: center;">'''System''' ||'''Total Score''' ||'''Inst. Score''' ||'''Time Score''' ||
||clasp ||213 ||150 ||63 ||||<style="text-align: center;">claspfolio ||609 ||385 ||224 ||||<style="text-align: center;">claspd ||103 ||60 ||43 ||
||claspfolio ||209 ||150 ||59 ||||<style="text-align: center;">clasp ||597 ||370 ||227 ||||<style="text-align: center;">cmodels ||72 ||45 ||27 ||
||claspd ||206 ||145 ||61 ||||<style="text-align: center;">idp ||597 ||370 ||227 ||||<style="text-align: center;">claspfolio ||0 ||0 ||0 ||
||sup ||195 ||140 ||55 ||||<style="text-align: center;">claspd ||552 ||355 ||197 ||||<style="text-align: center;">clasp ||0 ||0 ||0 ||
||lp2sat2gminisat ||185 ||140 ||45 ||||<style="text-align: center;">cmodels ||510 ||335 ||175 ||||<style="text-align: center;">idp ||0 ||0 ||0 ||
||cmodels ||184 ||130 ||54 ||||<style="text-align: center;">lp2diffz3 ||394 ||270 ||124 ||||<style="text-align: center;">lp2diffz3 ||0 ||0 ||0 ||
||idp ||184 ||130 ||54 ||||<style="text-align: center;">sup ||346 ||240 ||106 ||||<style="text-align: center;">sup ||0 ||0 ||0 ||
||smodels ||180 ||130 ||50 ||||<style="text-align: center;">lp2sat2gminisat ||310 ||225 ||85 ||||<style="text-align: center;">lp2sat2gminisat ||0 ||0 ||0 ||
||lp2sat2minisat ||179 ||135 ||44 ||||<style="text-align: center;">lp2sat2minisat ||302 ||220 ||82 ||||<style="text-align: center;">lp2sat2minisat ||0 ||0 ||0 ||
||lp2diffz3 ||178 ||135 ||43 ||||<style="text-align: center;">lp2sat2lminisat ||301 ||220 ||81 ||||<style="text-align: center;">lp2sat2lminisat ||0 ||0 ||0 ||
||lp2sat2lminisat ||171 ||130 ||41 ||||<style="text-align: center;">smodels ||269 ||165 ||104 ||||<style="text-align: center;">smodels ||0 ||0 ||0 ||		  * WF (Cost) : Witness Found (Cost)
 * OF (Cost) : Optimum Found (Cost)
 * Inc : Inconsistent
 * TO : Time Out
 * MO/F : Mem Out Fail
 * N/A : Not Available / Not Applicable

System Track Final Results


Contents

  1. System Track Final Results
    1. System Track Final Results: Overall
    2. Final results by categories
      1. Overview
      2. P Problems
      3. NP Problems
      4. Beyond NP Problems
    3. Detailed Results
      1. Details: raw file
      2. Details: tables
    4. Quick reference guide


System Track Final Results: Overall

System

Total

Instance

Time

claspd

861

560

301

claspfolio

818

535

283

clasp

810

520

290

idp

781

500

281

cmodels

766

510

256

lp2diffz3

572

405

167

sup

541

380

161

lp2sat2gminisat

495

365

130

lp2sat2minisat

481

355

126

lp2sat2lminisat

472

350

122

smodels

449

295

154

Final results by categories

Overview

P

NP

Beyond NP

System

Total Score

Inst. Score

Time Score

System

Total Score

Inst. Score

Time Score

System

Total Score

Inst. Score

Time Score

clasp

213

150

63

claspfolio

609

385

224

claspd

103

60

43

claspfolio

209

150

59

clasp

597

370

227

cmodels

72

45

27

claspd

206

145

61

idp

597

370

227

claspfolio

-

-

-

sup

195

140

55

claspd

552

355

197

clasp

-

-

-

lp2sat2gminisat

185

140

45

cmodels

510

335

175

idp

-

-

-

cmodels

184

130

54

lp2diffz3

394

270

124

lp2diffz3

-

-

-

idp

184

130

54

sup

346

240

106

sup

-

-

-

smodels

180

130

50

lp2sat2gminisat

310

225

85

lp2sat2gminisat

-

-

-

lp2sat2minisat

179

135

44

lp2sat2minisat

302

220

82

lp2sat2minisat

-

-

-

lp2diffz3

178

135

43

lp2sat2lminisat

301

220

81

lp2sat2lminisat

-

-

-

lp2sat2lminisat

171

130

41

smodels

269

165

104

smodels

-

-

-

P Problems

System

Total

Reachability

Grammar-Based IE

Hydraulic Leaking

Hydraulic Planning

Stable Marriage

Partner Units Polynomial

clasp

Total

213

36

72

10

75

6

14

Instance

150

25

50

10

50

5

10

Time

63

11

22

0

25

1

4

claspfolio

Total

209

35

72

10

74

5

13

Instance

150

25

50

10

50

5

10

Time

59

10

22

0

24

0

3

claspd

Total

206

36

72

10

75

0

13

Instance

145

25

50

10

50

0

10

Time

61

11

22

0

25

0

3

sup

Total

195

29

71

10

74

0

11

Instance

140

20

50

10

50

0

10

Time

55

9

21

0

24

0

1

lp2sat2gminisat

Total

185

30

66

10

68

0

11

Instance

140

20

50

10

50

0

10

Time

45

10

16

0

18

0

1

cmodels

Total

184

29

71

10

74

0

0

Instance

130

20

50

10

50

0

0

Time

54

9

21

0

24

0

0

idp

Total

184

29

71

10

74

0

0

Instance

130

20

50

10

50

0

0

Time

54

9

21

0

24

0

0

smodels

Total

180

28

70

10

72

0

0

Instance

130

20

50

10

50

0

0

Time

50

8

20

0

22

0

0

lp2sat2minisat

Total

179

30

66

10

68

0

5

Instance

135

20

50

10

50

0

5

Time

44

10

16

0

18

0

0

lp2diffz3

Total

178

35

66

10

67

0

0

Instance

135

25

50

10

50

0

0

Time

43

10

16

0

17

0

0

lp2sat2lminisat

Total

171

28

66

10

67

0

0

Instance

130

20

50

10

50

0

0

Time

41

8

16

0

17

0

0

NP Problems

System

Total

Sokoban Decision

Knight Tour

Disjunctive Scheduling

Packing Problem

Labyrinth

MCS Querying

Numberlink

Hanoi Tower

Graph Colouring

Solitaire

Weight-Assignment Tree

Maze Generation

claspfolio

Total

609

66

65

37

0

63

75

64

47

55

21

21

95

Instance

385

45

35

25

0

40

50

35

35

40

15

15

50

Time

224

21

30

12

0

23

25

29

12

15

6

6

45

clasp

Total

597

78

63

38

0

78

75

65

39

23

21

21

96

Instance

370

50

35

25

0

50

50

35

30

15

15

15

50

Time

227

28

28

13

0

28

25

30

9

8

6

6

46

idp

Total

597

64

74

38

0

52

75

70

65

18

38

8

95

Instance

370

45

45

25

0

30

50

40

45

10

25

5

50

Time

227

19

29

13

0

22

25

30

20

8

13

3

45

claspd

Total

552

68

68

30

0

65

75

69

31

19

11

20

96

Instance

355

45

40

25

0

45

50

40

25

10

10

15

50

Time

197

23

28

5

0

20

25

29

6

9

1

5

46

cmodels

Total

510

67

56

21

0

62

75

30

51

29

18

6

95

Instance

335

45

30

20

0

45

50

20

35

20

15

5

50

Time

175

22

26

1

0

17

25

10

16

9

3

1

45

lp2diffz3

Total

394

42

55

0

0

0

70

45

47

27

25

0

83

Instance

270

30

35

0

0

0

50

30

35

20

20

0

50

Time

124

12

20

0

0

0

20

15

12

7

5

0

33

sup

Total

346

52

40

37

0

58

72

0

31

16

15

25

0

Instance

240

35

25

25

0

40

50

0

25

10

10

20

0

Time

106

17

15

12

0

18

22

0

6

6

5

5

0

lp2sat2gminisat

Total

310

36

10

32

0

46

71

22

47

17

29

-

0

Instance

225

30

5

25

0

35

50

15

35

10

20

-

0

Time

85

6

5

7

0

11

21

7

12

7

9

-

0

lp2sat2minisat

Total

302

39

0

32

0

52

71

15

47

17

29

-

0

Instance

220

30

0

25

0

40

50

10

35

10

20

-

0

Time

82

9

0

7

0

12

21

5

12

7

9

-

0

lp2sat2lminisat

Total

301

35

0

32

0

53

71

17

47

17

29

-

0

Instance

220

30

0

25

0

40

50

10

35

10

20

-

0

Time

81

5

0

7

0

13

21

7

12

7

9

-

0

smodels

Total

269

0

55

36

0

9

53

27

0

0

0

0

89

Instance

165

0

30

25

0

5

35

20

0

0

0

0

50

Time

104

0

25

11

0

4

18

7

0

0

0

0

39

Beyond NP Problems

System

Total

Strategic Companies

Minimal Diagnosis

claspd

Total

103

12

91

Instance

60

10

50

Time

43

2

41

cmodels

Total

72

0

72

Instance

45

0

45

Time

27

0

27

Detailed Results

Details: raw file

Detailed results, as a raw text file, can be downloaded HERE.
The file contains a line for each instance each system has been asked to solve.

Details: tables

Detailed results are also available as separate PDF files:

Quick reference guide

Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the actual time spent. In case of optimization problems, "quality" of the solutions plays a role as well. Please have a look at the Scoring and Participation Rules and the file and language format specifications for more details.

  • System: the system name (quite easy to understand)
  • Benchmark: the benchmark problem
  • Instance: the specific instance of the problem
  • Out|Outcome: the answer provided by the system for the specific instance
  • Diagnostics: result of the evaluation of the answer (essentially, whether the system answered correctly or not)
  • Time: the total time spent by the system while solving the specific instance
  • Optimum Cost: the "cost" of the witness solution returned by the system, in case of istance of an optimization problem
  • Total Score: the total score rewarded for the specific instance
  • Inst\Opt Score - Time Score: the two components of the score, as introduced above.
  • WF (Cost) : Witness Found (Cost)
  • OF (Cost) : Optimum Found (Cost)
  • Inc : Inconsistent
  • TO : Time Out
  • MO/F : Mem Out Fail
  • N/A : Not Available / Not Applicable

ASP Competition 2011: SystemTrackFinalResults (last edited 2011-05-24 09:11:12 by FrancescoCalimeri)