|
Size: 15900
Comment:
|
Size: 23701
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| #acl EditorsGroup:read,write,delete,revert,admin All: | #acl EditorsGroup:read,write,delete,revert,admin All:read |
| Line 3: | Line 3: |
| <<BR>> <<TableOfContents>> |
|
| Line 4: | Line 6: |
| <<TableOfContents>> | == Model&Solve Track Final Results: Overall == ||<tablewidth="1%"style="text-align:center;font-weight:bold;">System ||<style="text-align: center; font-weight: bold;">Total ||<style="font-weight:bold;text-align:center;">Instance ||<style="font-weight:bold;text-align:center;">Time || ||<style="font-weight:bold;text-align:center;">[[http://goo.gl/4XXFz|clasp]] ||<style="font-weight: bold;">2431 ||1432 ||999 || ||<style="font-weight:bold;text-align:center;">[[http://goo.gl/4XXFz|aclasp]] ||<style="font-weight: bold;">1953 ||1140 ||813 || ||<style="font-weight:bold;text-align:center;">[[http://goo.gl/4XXFz|bpsolver]] ||<style="font-weight: bold;">1878 ||1064 ||814 || ||<style="font-weight:bold;text-align:center;">[[http://goo.gl/4XXFz|ezcsp]] ||<style="font-weight: bold;">1773 ||993 ||780 || ||<style="font-weight:bold;text-align:center;">[[http://goo.gl/4XXFz|idp]] ||<style="font-weight: bold;">1442 ||918 ||524 || ||<style="font-weight:bold;text-align:center;">[[http://goo.gl/4XXFz|fastdownward]] ||<style="font-weight: bold;">367 ||217 ||150 || == Final results by categories == === Overview === ||||||||<tablewidth="1%"style="text-align:center;font-weight:bold;">'''P''' ||<style="font-weight:bold;text-align:center;" |7> ||||||||<style="font-weight:bold;text-align:center;">'''NP''' ||<style="font-weight:bold;text-align:center;" |7> ||||||||<style="font-weight:bold;text-align:center;">'''Beyond NP''' ||<style="font-weight:bold;text-align:center;" |7> ||||||||<style="font-weight:bold;text-align:center;">'''Optimization''' || ||<style="font-weight:bold;text-align:center; ;text-align:center">'''System''' ||<style="text-align:center">'''Total Score''' ||<style="text-align:center">'''Inst. Score''' ||<style="text-align:center">'''Time Score''' ||<style="text-align:center">'''System''' ||<style="text-align:center">'''Total Score''' ||<style="text-align:center">'''Inst. Score''' ||<style="text-align:center">'''Time Score''' ||<style="text-align:center">'''System''' ||<style="text-align:center">'''Total Score''' ||<style="text-align:center">'''Inst. Score''' ||<style="text-align:center">'''Time Score''' ||<style="text-align:center">'''System''' ||<style="text-align:center">'''Total Score''' ||<style="text-align:center">'''Inst. Score''' ||<style="text-align:center">'''Time Score''' || ||<style="font-weight:bold;text-align:center;">clasp ||497 ||290 ||290 ||<style="font-weight:bold; ;text-align:center">clasp ||1481 ||848 ||633 ||<style="font-weight:bold; ;text-align:center">clasp ||110 ||60 ||50 ||<style="font-weight:bold; ;text-align:center">clasp ||343 ||234 ||109 || ||<style="font-weight:bold;text-align:center;">bpsolver ||459 ||253 ||206 ||<style="font-weight:bold; ;text-align:center">ezcsp ||1419 ||786 ||633 ||<style="font-weight:bold; ;text-align:center">bpsolver ||86 ||43 ||43 ||<style="font-weight:bold; ;text-align:center">aclasp ||325 ||220 ||105 || ||<style="font-weight:bold;text-align:center;">aclasp ||404 ||240 ||164 ||<style="font-weight:bold; ;text-align:center">aclasp ||1224 ||680 ||544 ||<style="font-weight:bold; ;text-align:center">idp ||0 ||0 ||0 ||<style="font-weight:bold; ;text-align:center">bpsolver ||207 ||161 ||46 || ||<style="font-weight:bold;text-align:center;">ezcsp ||320 ||173 ||147 ||<style="font-weight:bold; ;text-align:center">idp ||1127 ||680 ||447 ||<style="font-weight:bold; ;text-align:center">fastdownward ||0 ||0 ||0 ||<style="font-weight:bold; ;text-align:center">idp ||140 ||121 ||19 || ||<style="font-weight:bold;text-align:center;">fastdownward ||181 ||100 ||81 ||<style="font-weight:bold; ;text-align:center">bpsolver ||1126 ||607 ||519 ||<style="font-weight:bold; ;text-align:center">ezcsp ||0 ||0 ||0 ||<style="font-weight:bold; ;text-align:center">fastdownward ||81 ||47 ||34 || ||<style="font-weight:bold;text-align:center;">idp ||175 ||117 ||58 || ||<style="font-weight:bold; ;text-align:center">fastdownward ||105 ||70 ||35 || ||<style="font-weight:bold; ;text-align:center">aclasp ||0 ||0 ||0 || ||<style="font-weight:bold; ;text-align:center">ezcsp ||34 ||34 ||0 || === P Problems === ||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">Reachability||<style="font-weight: bold; text-align: center;">Company Controls||<style="font-weight: bold; text-align: center;">Grammar-Based IE||<style="font-weight: bold; text-align: center;">Hydraulic Leaking||<style="font-weight: bold; text-align: center;">Hydraulic Planning||<style="font-weight: bold; text-align: center;">Stable Marriage||<style="font-weight: bold; text-align: center;">Partner Units Polynomial|| ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>clasp ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">497 ||<style="font-weight: bold;">87 ||<style="font-weight: bold;">89 ||<style="font-weight: bold;">80 ||<style="font-weight: bold;">78 ||<style="font-weight: bold;">100 ||<style="font-weight: bold;">25 ||<style="font-weight: bold;">38 || ||<style="text-align:right;">Instance ||290 ||50 ||50 ||50 ||50 ||50 ||17 ||23 || ||<style="text-align:right;">Time ||207 ||37 ||39 ||30 ||28 ||50 ||8 ||15 || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>bpsolver ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">459 ||<style="font-weight:bold;">90 ||<style="font-weight:bold;">68 ||<style="font-weight:bold;">96 ||<style="font-weight:bold;">100 ||<style="font-weight:bold;">100 ||<style="font-weight:bold;">5 ||<style="font-weight:bold;">- || ||<style="text-align:right;">Instance ||253 ||50 ||50 ||50 ||50 ||50 ||3 ||- || ||<style="text-align:right;">Time ||206 ||40 ||18 ||46 ||50 ||50 ||2 ||- || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>aclasp ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">404 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">84 ||<style="font-weight:bold;">80 ||<style="font-weight:bold;">76 ||<style="font-weight:bold;">100 ||<style="font-weight:bold;">25 ||<style="font-weight:bold;">39 || ||<style="text-align:right;">Instance ||240 ||- ||50 ||50 ||50 ||50 ||17 ||23 || ||<style="text-align:right;">Time ||164 ||- ||34 ||30 ||26 ||50 ||8 ||16 || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>ezcsp ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">320 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">99 ||<style="font-weight:bold;">99 ||<style="font-weight:bold;">87 ||<style="font-weight:bold;">35 || ||<style="text-align:right;">Instance ||173 ||- ||- ||- ||50 ||50 ||50 ||23 || ||<style="text-align:right;">Time ||147 ||- ||- ||- ||49 ||49 ||37 ||12 || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>fastdownward ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">181 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">87 ||<style="font-weight:bold;">94 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- || ||<style="text-align:right;">Instance ||100 ||- ||- ||- ||50 ||50 ||- ||- || ||<style="text-align:right;">Time ||81 ||- ||- ||- ||37 ||44 ||- ||- || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>idp ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">175 ||<style="font-weight:bold;">76 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">72 ||<style="font-weight:bold;">27 || ||<style="text-align:right;">Instance ||117 ||50 ||- ||- ||- ||- ||50 ||17 || ||<style="text-align:right;">Time ||58 ||26 ||- ||- ||- ||- ||22 ||10 || === NP Problems === ||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">Generalized Slitherlink||<style="font-weight: bold; text-align: center;">Fastfood Optimality Check||<style="font-weight: bold; text-align: center;">Sokoban Decision||<style="font-weight: bold; text-align: center;">Knight Tour||<style="font-weight: bold; text-align: center;">Disjunctive Scheduling||<style="font-weight: bold; text-align: center;">Packing Problem||<style="font-weight: bold; text-align: center;">Labyrinth||<style="font-weight: bold; text-align: center;">Numberlink||<style="font-weight: bold; text-align: center;">Reverse Folding||<style="font-weight: bold; text-align: center;">Hanoi Tower||<style="font-weight: bold; text-align: center;">Magic Square Sets||<style="font-weight: bold; text-align: center;">Airport Pickup||<style="font-weight: bold; text-align: center;">Graph Colouring||<style="font-weight: bold; text-align: center;">Solitaire||<style="font-weight: bold; text-align: center;">Partner Units||<style="font-weight: bold; text-align: center;">Weight-Assignment Tree||<style="font-weight: bold; text-align: center;">Maze Generation||<style="font-weight: bold; text-align: center;">Incremental Scheduling||<style="font-weight: bold; text-align: center;">Tangram|| ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>clasp ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">1481 ||<style="font-weight:bold;">97 ||<style="font-weight:bold;">97 ||<style="font-weight:bold;">85 ||<style="font-weight:bold;">97 ||<style="font-weight:bold;">67 ||<style="font-weight:bold;">81 ||<style="font-weight:bold;">71 ||<style="font-weight:bold;">72 ||<style="font-weight:bold;">94 ||<style="font-weight:bold;">81 ||<style="font-weight:bold;">98 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">51 ||<style="font-weight:bold;">73 ||<style="font-weight:bold;">79 ||<style="font-weight:bold;">56 ||<style="font-weight:bold;">97 ||<style="font-weight:bold;">85 ||<style="font-weight:bold;">100 || ||<style="text-align:right;">Instance ||848 ||50 ||50 ||50 ||50 ||50 ||50 ||47 ||37 ||47 ||50 ||50 ||- ||37 ||40 ||50 ||43 ||50 ||47 ||50 || ||<style="text-align:right;">Time ||633 ||47 ||47 ||35 ||47 ||17 ||31 ||24 ||35 ||47 ||31 ||48 ||- ||14 ||33 ||29 ||13 ||47 ||38 ||50 || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>ezcsp ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">1419 ||<style="font-weight:bold;">94 ||<style="font-weight:bold;">97 ||<style="font-weight:bold;">81 ||<style="font-weight:bold;">75 ||<style="font-weight:bold;">26 ||<style="font-weight:bold;">99 ||<style="font-weight:bold;">69 ||<style="font-weight:bold;">70 ||<style="font-weight:bold;">94 ||<style="font-weight:bold;">27 ||<style="font-weight:bold;">99 ||<style="font-weight:bold;">81 ||<style="font-weight:bold;">23 ||<style="font-weight:bold;">50 ||<style="font-weight:bold;">65 ||<style="font-weight:bold;">96 ||<style="font-weight:bold;">96 ||<style="font-weight:bold;">77 ||<style="font-weight:bold;">100 || ||<style="text-align:right;">Instance ||786 ||50 ||50 ||50 ||40 ||13 ||50 ||43 ||37 ||47 ||23 ||50 ||50 ||13 ||30 ||50 ||50 ||50 ||40 ||50 || ||<style="text-align:right;">Time ||633 ||44 ||47 ||31 ||35 ||13 ||49 ||26 ||33 ||47 ||4 ||49 ||31 ||10 ||20 ||15 ||46 ||46 ||37 ||50 || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>aclasp ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">1224 ||<style="font-weight:bold;">96 ||<style="font-weight:bold;">97 ||<style="font-weight:bold;">83 ||<style="font-weight:bold;">97 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">83 ||<style="font-weight:bold;">68 ||<style="font-weight:bold;">72 ||<style="font-weight:bold;">80 ||<style="font-weight:bold;">77 ||<style="font-weight:bold;">98 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">28 ||<style="font-weight:bold;">73 ||<style="font-weight:bold;">77 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">95 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">100 || ||<style="text-align:right;">Instance ||680 ||50 ||50 ||50 ||50 ||- ||50 ||43 ||37 ||40 ||50 ||50 ||- ||20 ||40 ||50 ||- ||50 ||- ||50 || ||<style="text-align:right;">Time ||544 ||46 ||47 ||33 ||47 ||- ||33 ||25 ||35 ||40 ||27 ||48 ||- ||8 ||33 ||27 ||- ||45 ||- ||50 || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>idp ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">1127 ||<style="font-weight:bold;">91 ||<style="font-weight:bold;">76 ||<style="font-weight:bold;">69 ||<style="font-weight:bold;">95 ||<style="font-weight:bold;">15 ||<style="font-weight:bold;">17 ||<style="font-weight:bold;">42 ||<style="font-weight:bold;">99 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">74 ||<style="font-weight:bold;">93 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">66 ||<style="font-weight:bold;">71 ||<style="font-weight:bold;">33 ||<style="font-weight:bold;">83 ||<style="font-weight:bold;">73 ||<style="font-weight:bold;">30 ||<style="font-weight:bold;">100 || ||<style="text-align:right;">Instance ||680 ||50 ||50 ||43 ||50 ||13 ||13 ||27 ||50 ||- ||50 ||47 ||- ||47 ||40 ||30 ||50 ||50 ||20 ||50 || ||<style="text-align:right;">Time ||447 ||41 ||26 ||26 ||45 ||2 ||4 ||15 ||49 ||- ||24 ||46 ||- ||19 ||31 ||3 ||33 ||23 ||10 ||50 || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>bpsolver ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">1126 ||<style="font-weight:bold;">* ||<style="font-weight:bold;">92 ||<style="font-weight:bold;">60 ||<style="font-weight:bold;">97 ||<style="font-weight:bold;">92 ||<style="font-weight:bold;">82 ||<style="font-weight:bold;">72 ||<style="font-weight:bold;">73 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">100 ||<style="font-weight:bold;">93 ||<style="font-weight:bold;">15 ||<style="font-weight:bold;">28 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">97 ||<style="font-weight:bold;">49 ||<style="font-weight:bold;">76 ||<style="font-weight:bold;">100 || ||<style="text-align:right;">Instance ||607 ||* ||50 ||30 ||50 ||50 ||43 ||40 ||37 ||- ||- ||50 ||50 ||10 ||20 ||- ||50 ||37 ||40 ||50 || ||<style="text-align:right;">Time ||519 ||* ||42 ||30 ||47 ||42 ||39 ||32 ||36 ||- ||- ||50 ||43 ||5 ||8 ||- ||47 ||12 ||36 ||50 || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>fastdownward ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">105 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">* ||<style="font-weight:bold;">5 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">27 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">10 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">63 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- || ||<style="text-align:right;">Instance ||70 ||- ||- ||* ||5 ||- ||- ||- ||- ||- ||23 ||- ||7 ||- ||35 ||- ||- ||- ||- ||- || ||<style="text-align:right;">Time ||35 ||- ||- ||* ||0 ||- ||- ||- ||- ||- ||4 ||- ||3 ||- ||28 ||- ||- ||- ||- ||- || === Beyond NP Problems === ||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">Strategic Companies||<style="font-weight: bold; text-align: center;">Minimal Diagnosis|| ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>clasp ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">110 ||<style="font-weight:bold;">11 ||<style="font-weight:bold;">99 || ||<style="text-align:right;">Instance ||60 ||10 ||50 || ||<style="text-align:right;">Time ||50 ||1 ||49 || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>bpsolver ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">86 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">86 || ||<style="text-align:right;">Instance ||43 ||- ||43 || ||<style="text-align:right;">Time ||43 ||- ||43 || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>idp ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">0 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- || ||<style="text-align:right;">Instance ||0 ||- ||- || ||<style="text-align:right;">Time ||0 ||- ||- || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>fastdownward ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">0 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- || ||<style="text-align:right;">Instance ||0 ||- ||- || ||<style="text-align:right;">Time ||0 ||- ||- || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>ezcsp ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">0 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- || ||<style="text-align:right;">Instance ||0 ||- ||- || ||<style="text-align:right;">Time ||0 ||- ||- || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>aclasp ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">0 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- || ||<style="text-align:right;">Instance ||0 ||- ||- || ||<style="text-align:right;">Time ||0 ||- ||- || === Optimization Problems === ||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">Crossing Minimization||<style="font-weight: bold; text-align: center;">Company Controls Optimize||<style="font-weight: bold; text-align: center;">Fastfood Optimization||<style="font-weight: bold; text-align: center;">Sokoban Optimization||<style="font-weight: bold; text-align: center;">Tomography||<style="font-weight: bold; text-align: center;">Maximal Clique|| ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>clasp ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">343 ||<style="font-weight:bold;">37 ||<style="font-weight:bold;">87 ||<style="font-weight:bold;">83 ||<style="font-weight:bold;">87 ||<style="font-weight:bold;">13 ||<style="font-weight:bold;">36 || ||<style="text-align:right;">Instance ||234 ||35 ||50 ||50 ||50 ||13 ||36 || ||<style="text-align:right;">Time ||109 ||2 ||37 ||33 ||37 ||0 ||0 || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>aclasp ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">325 ||<style="font-weight:bold;">16 ||<style="font-weight:bold;">87 ||<style="font-weight:bold;">83 ||<style="font-weight:bold;">85 ||<style="font-weight:bold;">18 ||<style="font-weight:bold;">36 || ||<style="text-align:right;">Instance ||220 ||16 ||50 ||50 ||50 ||18 ||36 || ||<style="text-align:right;">Time ||105 ||0 ||37 ||33 ||35 ||0 ||0 || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>bpsolver ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">207 ||<style="font-weight:bold;">13 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">48 ||<style="font-weight:bold;">70 ||<style="font-weight:bold;">39 ||<style="font-weight:bold;">37 || ||<style="text-align:right;">Instance ||161 ||13 ||- ||35 ||37 ||39 ||37 || ||<style="text-align:right;">Time ||46 ||0 ||- ||13 ||33 ||0 ||0 || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>idp ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">140 ||<style="font-weight:bold;">13 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">64 ||<style="font-weight:bold;">21 ||<style="font-weight:bold;">13 ||<style="font-weight:bold;">29 || ||<style="text-align:right;">Instance ||121 ||13 ||- ||48 ||18 ||13 ||29 || ||<style="text-align:right;">Time ||19 ||0 ||- ||16 ||3 ||0 ||0 || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>fastdownward ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">81 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">81 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- || ||<style="text-align:right;">Instance ||47 ||- ||- ||- ||47 ||- ||- || ||<style="text-align:right;">Time ||34 ||- ||- ||- ||34 ||- ||- || ||<style="font-weight:bold;text-align:center;vertical-align:top;" |3>ezcsp ||<style="text-align:right;">'''Total''' ||<style="font-weight:bold;">34 ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">- ||<style="font-weight:bold;">34 ||<style="font-weight:bold;">- || ||<style="text-align:right;">Instance ||34 ||- ||- ||- ||- ||34 ||- || ||<style="text-align:right;">Time ||0 ||- ||- ||- ||- ||0 ||- || == Detailed Results == === Details: raw file === Detailed results, as a ''raw'' text file, can be downloaded [[attachment:teamcompetition.final.results.raw.detailed.txt|HERE]]. |
| Line 6: | Line 115: |
| == Overall == <<BR>> ||<tablewidth="1%"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">Instance ||<style="font-weight: bold; text-align: center;">Time || ||<style="font-weight: bold; text-align: center;">clasp ||2431 ||1432 ||999 || ||<style="font-weight: bold; text-align: center;">aclasp ||1953 ||1140 ||813 || ||<style="font-weight: bold; text-align: center;">bpsolver ||1878 ||1064 ||814 || ||<style="font-weight: bold; text-align: center;">ezcsp ||1773 ||993 ||780 || ||<style="font-weight: bold; text-align: center;">idp ||1442 ||918 ||524 || ||<style="font-weight: bold; text-align: center;">fastdownward ||367 ||217 ||150 || <<BR>> == Final results by categories == <<BR>> ||||||||<tablewidth="1%"style="text-align: center; font-weight: bold;">'''P''' ||<style="font-weight: bold; text-align: center;" |7> ||||||||<style="font-weight: bold; text-align: center;">'''NP''' ||<style="font-weight: bold; text-align: center;" |7> ||||||||<style="font-weight: bold; text-align: center;">'''Beyond NP''' ||<style="font-weight: bold; text-align: center;" |7> ||||||||<style="font-weight: bold; text-align: center;">'''Optimization''' || ||<: style="font-weight: bold; text-align: center;">'''System''' ||<:>'''Total Score''' ||<:>'''Inst. Score''' ||<:>'''Time Score''' ||<:>'''System''' ||<:>'''Total Score''' ||<:>'''Inst. Score''' ||<:>'''Time Score''' ||<:>'''System''' ||<:>'''Total Score''' ||<:>'''Inst. Score''' ||<:>'''Time Score''' ||<:>'''System''' ||<:>'''Total Score''' ||<:>'''Inst. Score''' ||<:>'''Time Score''' || ||<style="font-weight: bold; text-align: center;">clasp ||497 ||290 ||290 ||<: style="font-weight: bold;">clasp ||1481 ||848 ||633 ||<: style="font-weight: bold;">clasp ||110 ||60 ||50 ||<: style="font-weight: bold;">clasp ||343 ||234 ||109 || ||<style="font-weight: bold; text-align: center;">bpsolver ||459 ||253 ||206 ||<: style="font-weight: bold;">ezcsp ||1419 ||786 ||633 ||<: style="font-weight: bold;">bpsolver ||86 ||43 ||43 ||<: style="font-weight: bold;">aclasp ||325 ||220 ||105 || ||<style="font-weight: bold; text-align: center;">aclasp ||404 ||240 ||164 ||<: style="font-weight: bold;">aclasp ||1224 ||680 ||544 ||<: style="font-weight: bold;">idp ||0 ||0 ||0 ||<: style="font-weight: bold;">bpsolver ||207 ||161 ||46 || ||<style="font-weight: bold; text-align: center;">ezcsp ||320 ||173 ||147 ||<: style="font-weight: bold;">idp ||1127 ||680 ||447 ||<: style="font-weight: bold;">fastdownward ||0 ||0 ||0 ||<: style="font-weight: bold;">idp ||140 ||121 ||19 || ||<style="font-weight: bold; text-align: center;">fastdownward ||181 ||100 ||81 ||<: style="font-weight: bold;">bpsolver ||1126 ||607 ||519 ||<: style="font-weight: bold;">ezcsp ||0 ||0 ||0 ||<: style="font-weight: bold;">fastdownward ||81 ||47 ||34 || ||<style="font-weight: bold; text-align: center;">idp ||175 ||117 ||58 || ||<: style="font-weight: bold;">fastdownward ||105 ||70 ||35 || ||<: style="font-weight: bold;">aclasp ||0 ||0 ||0 || ||<: style="font-weight: bold;">ezcsp ||34 ||34 ||0 || <<BR>> == P Problems == <<BR>> ||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">Reachability ||<style="font-weight: bold; text-align: center;">Company Controls ||<style="font-weight: bold; text-align: center;">Grammar-Based IE ||<style="font-weight: bold; text-align: center;">Hydraulic Leaking ||<style="font-weight: bold; text-align: center;">Hydraulic Planning ||<style="font-weight: bold; text-align: center;">Stable Marriage ||<style="font-weight: bold; text-align: center;">Partner Units Polynomial || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>clasp ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">497 ||<style="font-weight: bold;">87 ||<style="font-weight: bold;">89 ||<style="font-weight: bold;">80 ||<style="font-weight: bold;">78 ||<style="font-weight: bold;">100 ||<style="font-weight: bold;">25 ||<style="font-weight: bold;">38 || ||<style="text-align: right;">Instance ||290 ||50 ||50 ||50 ||50 ||50 ||17 ||23 || ||<style="text-align: right;">Time ||207 ||37 ||39 ||30 ||28 ||50 ||8 ||15 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>bpsolver ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">459 ||<style="font-weight: bold;">90 ||<style="font-weight: bold;">68 ||<style="font-weight: bold;">96 ||<style="font-weight: bold;">100 ||<style="font-weight: bold;">100 ||<style="font-weight: bold;">5 ||<style="font-weight: bold;">- || ||<style="text-align: right;">Instance ||253 ||50 ||50 ||50 ||50 ||50 ||3 ||- || ||<style="text-align: right;">Time ||206 ||40 ||18 ||46 ||50 ||50 ||2 ||- || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>aclasp ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">404 ||<style="font-weight: bold;">- ||<style="font-weight: bold;">84 ||<style="font-weight: bold;">80 ||<style="font-weight: bold;">76 ||<style="font-weight: bold;">100 ||<style="font-weight: bold;">25 ||<style="font-weight: bold;">39 || ||<style="text-align: right;">Instance ||240 ||- ||50 ||50 ||50 ||50 ||17 ||23 || ||<style="text-align: right;">Time ||164 ||- ||34 ||30 ||26 ||50 ||8 ||16 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>ezcsp ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">320 ||<style="font-weight: bold;">- ||<style="font-weight: bold;">- ||<style="font-weight: bold;">- ||<style="font-weight: bold;">99 ||<style="font-weight: bold;">99 ||<style="font-weight: bold;">87 ||<style="font-weight: bold;">35 || ||<style="text-align: right;">Instance ||173 ||- ||- ||- ||50 ||50 ||50 ||23 || ||<style="text-align: right;">Time ||147 ||- ||- ||- ||49 ||49 ||37 ||12 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>fastdownward ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">181 ||<style="font-weight: bold;">- ||<style="font-weight: bold;">- ||<style="font-weight: bold;">- ||<style="font-weight: bold;">87 ||<style="font-weight: bold;">94 ||<style="font-weight: bold;">- ||<style="font-weight: bold;">- || ||<style="text-align: right;">Instance ||100 ||- ||- ||- ||50 ||50 ||- ||- || ||<style="text-align: right;">Time ||81 ||- ||- ||- ||37 ||44 ||- ||- || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>idp ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">175 ||<style="font-weight: bold;">76 ||<style="font-weight: bold;">- ||<style="font-weight: bold;">- ||<style="font-weight: bold;">- ||<style="font-weight: bold;">- ||<style="font-weight: bold;">72 ||<style="font-weight: bold;">27 || ||<style="text-align: right;">Instance ||117 ||50 ||- ||- ||- ||- ||50 ||17 || ||<style="text-align: right;">Time ||58 ||26 ||- ||- ||- ||- ||22 ||10 || <<BR>> == NP Problems == <<BR>> ||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">Generalized Slitherlink ||<style="font-weight: bold; text-align: center;">Fastfood Optimality Check ||<style="font-weight: bold; text-align: center;">Sokoban Decision ||<style="font-weight: bold; text-align: center;">Knight Tour ||<style="font-weight: bold; text-align: center;">Disjunctive Scheduling ||<style="font-weight: bold; text-align: center;">Packing Problem ||<style="font-weight: bold; text-align: center;">Labyrinth ||<style="font-weight: bold; text-align: center;">Numberlink ||<style="font-weight: bold; text-align: center;">Reverse Folding ||<style="font-weight: bold; text-align: center;">Hanoi Tower ||<style="font-weight: bold; text-align: center;">Magic Square Sets ||<style="font-weight: bold; text-align: center;">Airport Pickup ||<style="font-weight: bold; text-align: center;">Graph Colouring ||<style="font-weight: bold; text-align: center;">Solitaire ||<style="font-weight: bold; text-align: center;">Partner Units ||<style="font-weight: bold; text-align: center;">Weight-Assignment Tree ||<style="font-weight: bold; text-align: center;">Maze Generation ||<style="font-weight: bold; text-align: center;">Incremental Scheduling ||<style="font-weight: bold; text-align: center;">Tangram || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>clasp ||<style="text-align: right;">'''Total''' ||1481 ||97 ||97 ||85 ||97 ||67 ||81 ||71 ||72 ||94 ||81 ||98 ||- ||51 ||73 ||79 ||56 ||97 ||85 ||100 || ||<style="text-align: right;">Instance ||848 ||50 ||50 ||50 ||50 ||50 ||50 ||47 ||37 ||47 ||50 ||50 ||- ||37 ||40 ||50 ||43 ||50 ||47 ||50 || ||<style="text-align: right;">Time ||633 ||47 ||47 ||35 ||47 ||17 ||31 ||24 ||35 ||47 ||31 ||48 ||- ||14 ||33 ||29 ||13 ||47 ||38 ||50 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>ezcsp ||<style="text-align: right;">'''Total''' ||1419 ||94 ||97 ||81 ||75 ||26 ||99 ||69 ||70 ||94 ||27 ||99 ||81 ||23 ||50 ||65 ||96 ||96 ||77 ||100 || ||<style="text-align: right;">Instance ||786 ||50 ||50 ||50 ||40 ||13 ||50 ||43 ||37 ||47 ||23 ||50 ||50 ||13 ||30 ||50 ||50 ||50 ||40 ||50 || ||<style="text-align: right;">Time ||633 ||44 ||47 ||31 ||35 ||13 ||49 ||26 ||33 ||47 ||4 ||49 ||31 ||10 ||20 ||15 ||46 ||46 ||37 ||50 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>aclasp ||<style="text-align: right;">'''Total''' ||1224 ||96 ||97 ||83 ||97 ||- ||83 ||68 ||72 ||80 ||77 ||98 ||- ||28 ||73 ||77 ||- ||95 ||- ||100 || ||<style="text-align: right;">Instance ||680 ||50 ||50 ||50 ||50 ||- ||50 ||43 ||37 ||40 ||50 ||50 ||- ||20 ||40 ||50 ||- ||50 ||- ||50 || ||<style="text-align: right;">Time ||544 ||46 ||47 ||33 ||47 ||- ||33 ||25 ||35 ||40 ||27 ||48 ||- ||8 ||33 ||27 ||- ||45 ||- ||50 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>idp ||<style="text-align: right;">'''Total''' ||1127 ||91 ||76 ||69 ||95 ||15 ||17 ||42 ||99 ||- ||74 ||93 ||- ||66 ||71 ||33 ||83 ||73 ||30 ||100 || ||<style="text-align: right;">Instance ||680 ||50 ||50 ||43 ||50 ||13 ||13 ||27 ||50 ||- ||50 ||47 ||- ||47 ||40 ||30 ||50 ||50 ||20 ||50 || ||<style="text-align: right;">Time ||447 ||41 ||26 ||26 ||45 ||2 ||4 ||15 ||49 ||- ||24 ||46 ||- ||19 ||31 ||3 ||33 ||23 ||10 ||50 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>bpsolver ||<style="text-align: right;">'''Total''' ||1126 ||* ||92 ||60 ||97 ||92 ||82 ||72 ||73 ||- ||- ||100 ||93 ||15 ||28 ||- ||97 ||49 ||76 ||100 || ||<style="text-align: right;">Instance ||607 ||* ||50 ||30 ||50 ||50 ||43 ||40 ||37 ||- ||- ||50 ||50 ||10 ||20 ||- ||50 ||37 ||40 ||50 || ||<style="text-align: right;">Time ||519 ||* ||42 ||30 ||47 ||42 ||39 ||32 ||36 ||- ||- ||50 ||43 ||5 ||8 ||- ||47 ||12 ||36 ||50 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>fastdownward ||<style="text-align: right;">'''Total''' ||105 ||- ||- ||* ||5 ||- ||- ||- ||- ||- ||27 ||- ||10 ||- ||63 ||- ||- ||- ||- ||- || ||<style="text-align: right;">Instance ||70 ||- ||- ||* ||5 ||- ||- ||- ||- ||- ||23 ||- ||7 ||- ||35 ||- ||- ||- ||- ||- || ||<style="text-align: right;">Time ||35 ||- ||- ||* ||0 ||- ||- ||- ||- ||- ||4 ||- ||3 ||- ||28 ||- ||- ||- ||- ||- || <<BR>> == Beyond NP Problems == <<BR>> ||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">Strategic Companies ||<style="font-weight: bold; text-align: center;">Minimal Diagnosis || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>clasp ||<style="text-align: right;">'''Total''' ||110 ||11 ||99 || ||<style="text-align: right;">Instance ||60 ||10 ||50 || ||<style="text-align: right;">Time ||50 ||1 ||49 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>bpsolver ||<style="text-align: right;">'''Total''' ||86 ||- ||86 || ||<style="text-align: right;">Instance ||43 ||- ||43 || ||<style="text-align: right;">Time ||43 ||- ||43 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>idp ||<style="text-align: right;">'''Total''' ||0 ||- ||- || ||<style="text-align: right;">Instance ||0 ||- ||- || ||<style="text-align: right;">Time ||0 ||- ||- || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>fastdownward ||<style="text-align: right;">'''Total''' ||0 ||- ||- || ||<style="text-align: right;">Instance ||0 ||- ||- || ||<style="text-align: right;">Time ||0 ||- ||- || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>ezcsp ||<style="text-align: right;">'''Total''' ||0 ||- ||- || ||<style="text-align: right;">Instance ||0 ||- ||- || ||<style="text-align: right;">Time ||0 ||- ||- || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>aclasp ||<style="text-align: right;">'''Total''' ||0 ||- ||- || ||<style="text-align: right;">Instance ||0 ||- ||- || ||<style="text-align: right;">Time ||0 ||- ||- || <<BR>> == Optimization Problems == <<BR>> ||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">Crossing Minimization ||<style="font-weight: bold; text-align: center;">Company Controls Optimize ||<style="font-weight: bold; text-align: center;">Fastfood Optimization ||<style="font-weight: bold; text-align: center;">Sokoban Optimization ||<style="font-weight: bold; text-align: center;">Tomography ||<style="font-weight: bold; text-align: center;">Maximal Clique || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>clasp ||<style="text-align: right;">'''Total''' ||343||37||87||83||87||13||36|| ||<style="text-align: right;">Instance ||234||35||50||50||50||13||36|| ||<style="text-align: right;">Time ||109||2||37||33||37||0||0|| ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>aclasp ||<style="text-align: right;">'''Total''' ||325||16||87||83||85||18||36|| ||<style="text-align: right;">Instance ||220||16||50||50||50||18||36|| ||<style="text-align: right;">Time ||105||0||37||33||35||0||0|| ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>bpsolver ||<style="text-align: right;">'''Total''' ||207||13||-||48||70||39||37|| ||<style="text-align: right;">Instance ||161||13||-||35||37||39||37|| ||<style="text-align: right;">Time ||46||0||-||13||33||0||0|| ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>idp ||<style="text-align: right;">'''Total''' ||140||13||-||64||21||13||29|| ||<style="text-align: right;">Instance ||121||13||-||48||18||13||29|| ||<style="text-align: right;">Time ||19||0||-||16||3||0||0|| ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>fastdownward ||<style="text-align: right;">'''Total''' ||81||-||-||-||81||-||-|| ||<style="text-align: right;">Instance ||47||-||-||-||47||-||-|| ||<style="text-align: right;">Time ||34||-||-||-||34||-||-|| ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>ezcsp ||<style="text-align: right;">'''Total''' ||34||-||-||-||-||34||-|| ||<style="text-align: right;">Instance ||34||-||-||-||-||34||-|| ||<style="text-align: right;">Time ||0||-||-||-||-||0||-|| |
The file contains a line for each instance each system has been asked to solve. === Details: tables === Detailed results are also available as separate PDF files: * [[attachment:Model&Solve-Results-P-Problems-detailed.pdf||System Track - Details - P problems]] * [[attachment:Model&Solve-Results-NP-Problems-detailed.pdf||System Track - Details - NP problems]] * [[attachment:Model&Solve-Results-BndNP-Problems-detailed.pdf||System Track - Details - Beyond P problems]] * [[attachment:Model&Solve-Results-Opt-Problems-detailed.pdf||System Track - Details - Beyond P problems]] == Quick reference guide == Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the actual time spent. In case of optimization problems, "quality" of the solutions plays a role as well. Please have a look at the [[ParticipationRules|Scoring and Participation Rules]] and the [[http://www.mat.unical.it/aspcomp2011/files/LanguageSpecifications.pdf|file and language format specifications]] for more details. * System: the system name (quite easy to understand) * Benchmark: the benchmark problem * Instance: the specific instance of the problem * Out|Outcome: the answer provided by the system for the specific instance * Diagnostics: result of the evaluation of the answer (essentially, whether the system answered correctly or not) * Time: the total time spent by the system while solving the specific instance * Optimum Cost: the "cost" of the witness solution returned by the system, in case of istance of an optimization problem * Total Score: the total score rewarded for the specific instance * Inst\Opt Score - Time Score: the two components of the score, as introduced above. * WF (Cost) : Witness Found (Cost) * OF (Cost) : Optimum Found (Cost) * Inc : Inconsistent * TO : Time Out * MO/F : Mem Out Fail * N/A : Not Available / Not Applicable |
Model & Solve Track Final Results
Contents
Model&Solve Track Final Results: Overall
System |
Total |
Instance |
Time |
2431 |
1432 |
999 |
|
1953 |
1140 |
813 |
|
1878 |
1064 |
814 |
|
1773 |
993 |
780 |
|
1442 |
918 |
524 |
|
367 |
217 |
150 |
Final results by categories
Overview
P |
|
NP |
|
Beyond NP |
|
Optimization |
||||||||||||
System |
Total Score |
Inst. Score |
Time Score |
System |
Total Score |
Inst. Score |
Time Score |
System |
Total Score |
Inst. Score |
Time Score |
System |
Total Score |
Inst. Score |
Time Score |
|||
clasp |
497 |
290 |
290 |
clasp |
1481 |
848 |
633 |
clasp |
110 |
60 |
50 |
clasp |
343 |
234 |
109 |
|||
bpsolver |
459 |
253 |
206 |
ezcsp |
1419 |
786 |
633 |
bpsolver |
86 |
43 |
43 |
aclasp |
325 |
220 |
105 |
|||
aclasp |
404 |
240 |
164 |
aclasp |
1224 |
680 |
544 |
idp |
0 |
0 |
0 |
bpsolver |
207 |
161 |
46 |
|||
ezcsp |
320 |
173 |
147 |
idp |
1127 |
680 |
447 |
fastdownward |
0 |
0 |
0 |
idp |
140 |
121 |
19 |
|||
fastdownward |
181 |
100 |
81 |
bpsolver |
1126 |
607 |
519 |
ezcsp |
0 |
0 |
0 |
fastdownward |
81 |
47 |
34 |
|||
idp |
175 |
117 |
58 |
|
fastdownward |
105 |
70 |
35 |
|
aclasp |
0 |
0 |
0 |
|
ezcsp |
34 |
34 |
0 |
P Problems
System |
Total |
Reachability |
Company Controls |
Grammar-Based IE |
Hydraulic Leaking |
Hydraulic Planning |
Stable Marriage |
Partner Units Polynomial |
|
clasp |
Total |
497 |
87 |
89 |
80 |
78 |
100 |
25 |
38 |
Instance |
290 |
50 |
50 |
50 |
50 |
50 |
17 |
23 |
|
Time |
207 |
37 |
39 |
30 |
28 |
50 |
8 |
15 |
|
bpsolver |
Total |
459 |
90 |
68 |
96 |
100 |
100 |
5 |
- |
Instance |
253 |
50 |
50 |
50 |
50 |
50 |
3 |
- |
|
Time |
206 |
40 |
18 |
46 |
50 |
50 |
2 |
- |
|
aclasp |
Total |
404 |
- |
84 |
80 |
76 |
100 |
25 |
39 |
Instance |
240 |
- |
50 |
50 |
50 |
50 |
17 |
23 |
|
Time |
164 |
- |
34 |
30 |
26 |
50 |
8 |
16 |
|
ezcsp |
Total |
320 |
- |
- |
- |
99 |
99 |
87 |
35 |
Instance |
173 |
- |
- |
- |
50 |
50 |
50 |
23 |
|
Time |
147 |
- |
- |
- |
49 |
49 |
37 |
12 |
|
fastdownward |
Total |
181 |
- |
- |
- |
87 |
94 |
- |
- |
Instance |
100 |
- |
- |
- |
50 |
50 |
- |
- |
|
Time |
81 |
- |
- |
- |
37 |
44 |
- |
- |
|
idp |
Total |
175 |
76 |
- |
- |
- |
- |
72 |
27 |
Instance |
117 |
50 |
- |
- |
- |
- |
50 |
17 |
|
Time |
58 |
26 |
- |
- |
- |
- |
22 |
10 |
|
NP Problems
System |
Total |
Generalized Slitherlink |
Fastfood Optimality Check |
Sokoban Decision |
Knight Tour |
Disjunctive Scheduling |
Packing Problem |
Labyrinth |
Numberlink |
Reverse Folding |
Hanoi Tower |
Magic Square Sets |
Airport Pickup |
Graph Colouring |
Solitaire |
Partner Units |
Weight-Assignment Tree |
Maze Generation |
Incremental Scheduling |
Tangram |
|
clasp |
Total |
1481 |
97 |
97 |
85 |
97 |
67 |
81 |
71 |
72 |
94 |
81 |
98 |
- |
51 |
73 |
79 |
56 |
97 |
85 |
100 |
Instance |
848 |
50 |
50 |
50 |
50 |
50 |
50 |
47 |
37 |
47 |
50 |
50 |
- |
37 |
40 |
50 |
43 |
50 |
47 |
50 |
|
Time |
633 |
47 |
47 |
35 |
47 |
17 |
31 |
24 |
35 |
47 |
31 |
48 |
- |
14 |
33 |
29 |
13 |
47 |
38 |
50 |
|
ezcsp |
Total |
1419 |
94 |
97 |
81 |
75 |
26 |
99 |
69 |
70 |
94 |
27 |
99 |
81 |
23 |
50 |
65 |
96 |
96 |
77 |
100 |
Instance |
786 |
50 |
50 |
50 |
40 |
13 |
50 |
43 |
37 |
47 |
23 |
50 |
50 |
13 |
30 |
50 |
50 |
50 |
40 |
50 |
|
Time |
633 |
44 |
47 |
31 |
35 |
13 |
49 |
26 |
33 |
47 |
4 |
49 |
31 |
10 |
20 |
15 |
46 |
46 |
37 |
50 |
|
aclasp |
Total |
1224 |
96 |
97 |
83 |
97 |
- |
83 |
68 |
72 |
80 |
77 |
98 |
- |
28 |
73 |
77 |
- |
95 |
- |
100 |
Instance |
680 |
50 |
50 |
50 |
50 |
- |
50 |
43 |
37 |
40 |
50 |
50 |
- |
20 |
40 |
50 |
- |
50 |
- |
50 |
|
Time |
544 |
46 |
47 |
33 |
47 |
- |
33 |
25 |
35 |
40 |
27 |
48 |
- |
8 |
33 |
27 |
- |
45 |
- |
50 |
|
idp |
Total |
1127 |
91 |
76 |
69 |
95 |
15 |
17 |
42 |
99 |
- |
74 |
93 |
- |
66 |
71 |
33 |
83 |
73 |
30 |
100 |
Instance |
680 |
50 |
50 |
43 |
50 |
13 |
13 |
27 |
50 |
- |
50 |
47 |
- |
47 |
40 |
30 |
50 |
50 |
20 |
50 |
|
Time |
447 |
41 |
26 |
26 |
45 |
2 |
4 |
15 |
49 |
- |
24 |
46 |
- |
19 |
31 |
3 |
33 |
23 |
10 |
50 |
|
bpsolver |
Total |
1126 |
* |
92 |
60 |
97 |
92 |
82 |
72 |
73 |
- |
- |
100 |
93 |
15 |
28 |
- |
97 |
49 |
76 |
100 |
Instance |
607 |
* |
50 |
30 |
50 |
50 |
43 |
40 |
37 |
- |
- |
50 |
50 |
10 |
20 |
- |
50 |
37 |
40 |
50 |
|
Time |
519 |
* |
42 |
30 |
47 |
42 |
39 |
32 |
36 |
- |
- |
50 |
43 |
5 |
8 |
- |
47 |
12 |
36 |
50 |
|
fastdownward |
Total |
105 |
- |
- |
* |
5 |
- |
- |
- |
- |
- |
27 |
- |
10 |
- |
63 |
- |
- |
- |
- |
- |
Instance |
70 |
- |
- |
* |
5 |
- |
- |
- |
- |
- |
23 |
- |
7 |
- |
35 |
- |
- |
- |
- |
- |
|
Time |
35 |
- |
- |
* |
0 |
- |
- |
- |
- |
- |
4 |
- |
3 |
- |
28 |
- |
- |
- |
- |
- |
|
Beyond NP Problems
System |
Total |
Strategic Companies |
Minimal Diagnosis |
|
clasp |
Total |
110 |
11 |
99 |
Instance |
60 |
10 |
50 |
|
Time |
50 |
1 |
49 |
|
bpsolver |
Total |
86 |
- |
86 |
Instance |
43 |
- |
43 |
|
Time |
43 |
- |
43 |
|
idp |
Total |
0 |
- |
- |
Instance |
0 |
- |
- |
|
Time |
0 |
- |
- |
|
fastdownward |
Total |
0 |
- |
- |
Instance |
0 |
- |
- |
|
Time |
0 |
- |
- |
|
ezcsp |
Total |
0 |
- |
- |
Instance |
0 |
- |
- |
|
Time |
0 |
- |
- |
|
aclasp |
Total |
0 |
- |
- |
Instance |
0 |
- |
- |
|
Time |
0 |
- |
- |
|
Optimization Problems
System |
Total |
Crossing Minimization |
Company Controls Optimize |
Fastfood Optimization |
Sokoban Optimization |
Tomography |
Maximal Clique |
|
clasp |
Total |
343 |
37 |
87 |
83 |
87 |
13 |
36 |
Instance |
234 |
35 |
50 |
50 |
50 |
13 |
36 |
|
Time |
109 |
2 |
37 |
33 |
37 |
0 |
0 |
|
aclasp |
Total |
325 |
16 |
87 |
83 |
85 |
18 |
36 |
Instance |
220 |
16 |
50 |
50 |
50 |
18 |
36 |
|
Time |
105 |
0 |
37 |
33 |
35 |
0 |
0 |
|
bpsolver |
Total |
207 |
13 |
- |
48 |
70 |
39 |
37 |
Instance |
161 |
13 |
- |
35 |
37 |
39 |
37 |
|
Time |
46 |
0 |
- |
13 |
33 |
0 |
0 |
|
idp |
Total |
140 |
13 |
- |
64 |
21 |
13 |
29 |
Instance |
121 |
13 |
- |
48 |
18 |
13 |
29 |
|
Time |
19 |
0 |
- |
16 |
3 |
0 |
0 |
|
fastdownward |
Total |
81 |
- |
- |
- |
81 |
- |
- |
Instance |
47 |
- |
- |
- |
47 |
- |
- |
|
Time |
34 |
- |
- |
- |
34 |
- |
- |
|
ezcsp |
Total |
34 |
- |
- |
- |
- |
34 |
- |
Instance |
34 |
- |
- |
- |
- |
34 |
- |
|
Time |
0 |
- |
- |
- |
- |
0 |
- |
|
Detailed Results
Details: raw file
Detailed results, as a raw text file, can be downloaded HERE.
The file contains a line for each instance each system has been asked to solve.
Details: tables
Detailed results are also available as separate PDF files:
Quick reference guide
Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the actual time spent. In case of optimization problems, "quality" of the solutions plays a role as well. Please have a look at the Scoring and Participation Rules and the file and language format specifications for more details.
- System: the system name (quite easy to understand)
- Benchmark: the benchmark problem
- Instance: the specific instance of the problem
- Out|Outcome: the answer provided by the system for the specific instance
- Diagnostics: result of the evaluation of the answer (essentially, whether the system answered correctly or not)
- Time: the total time spent by the system while solving the specific instance
- Optimum Cost: the "cost" of the witness solution returned by the system, in case of istance of an optimization problem
- Total Score: the total score rewarded for the specific instance
- Inst\Opt Score - Time Score: the two components of the score, as introduced above.
- WF (Cost) : Witness Found (Cost)
- OF (Cost) : Optimum Found (Cost)
- Inc : Inconsistent
- TO : Time Out
- MO/F : Mem Out Fail
- N/A : Not Available / Not Applicable