|
Size: 19427
Comment:
|
Size: 24186
Comment: add detailed results as PDF tables
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| #acl EditorsGroup:read,write,delete,revert,admin All: | #acl EditorsGroup:read,write,delete,revert,admin All:read |
| Line 5: | Line 5: |
| == P Problems == | <<TableOfContents>> |
| Line 7: | Line 8: |
| ||||<style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">P ||<style="font-weight: bold; text-align: center;">Reachability ||<style="font-weight: bold; text-align: center;">Grammar-Based IE ||<style="font-weight: bold; text-align: center;">Hydraulic Leaking ||<style="font-weight: bold; text-align: center;">Hydraulic Planning ||<style="font-weight: bold; text-align: center;">Stable Marriage ||<style="font-weight: bold; text-align: center;">Partner Units Polynomial || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>clasp||<style="text-align: right;">'''Total''' ||213||150||63||36||25||11||72|| ||<style="text-align: right;">Instance ||50||22||10||10||0||75||50|| ||<style="text-align: right;">Time ||25||6||5||1||14||10||4|| ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>claspfolio||<style="text-align: right;">'''Total''' ||209||150||59||35||25||10||72|| ||<style="text-align: right;">Instance ||50||22||10||10||0||74||50|| ||<style="text-align: right;">Time ||24||5||5||0||13||10||3|| ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>claspd||<style="text-align: right;">'''Total''' ||206||145||61||36||25||11||72|| ||<style="text-align: right;">Instance ||50||22||10||10||0||75||50|| ||<style="text-align: right;">Time ||25||0||0||0||13||10||3|| ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>sup||<style="text-align: right;">'''Total''' ||195||140||55||29||20||9||71|| ||<style="text-align: right;">Instance ||50||21||10||10||0||74||50|| ||<style="text-align: right;">Time ||24||0||0||0||11||10||1|| ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2sat2gminisat||<style="text-align: right;">'''Total''' ||185||140||45||30||20||10||66|| ||<style="text-align: right;">Instance ||50||16||10||10||0||68||50|| ||<style="text-align: right;">Time ||18||0||0||0||11||10||1|| ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>cmodels||<style="text-align: right;">'''Total''' ||184||130||54||29||20||9||71|| ||<style="text-align: right;">Instance ||50||21||10||10||0||74||50|| ||<style="text-align: right;">Time ||24||0||0||0||0||0||0|| ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>idp||<style="text-align: right;">'''Total''' ||184||130||54||29||20||9||71|| ||<style="text-align: right;">Instance ||50||21||10||10||0||74||50|| ||<style="text-align: right;">Time ||24||0||0||0||0||0||0|| ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>smodels||<style="text-align: right;">'''Total''' ||180||130||50||28||20||8||70|| ||<style="text-align: right;">Instance ||50||20||10||10||0||72||50|| ||<style="text-align: right;">Time ||22||0||0||0||0||0||0|| ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2sat2minisat||<style="text-align: right;">'''Total''' ||179||135||44||30||20||10||66|| ||<style="text-align: right;">Instance ||50||16||10||10||0||68||50|| ||<style="text-align: right;">Time ||18||0||0||0||5||5||0|| ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2diffz3||<style="text-align: right;">'''Total''' ||178||135||43||35||25||10||66|| ||<style="text-align: right;">Instance ||50||16||10||10||0||67||50|| ||<style="text-align: right;">Time ||17||0||0||0||0||0||0|| ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2sat2lminisat||<style="text-align: right;">'''Total''' ||171||130||41||28||20||8||66|| ||<style="text-align: right;">Instance ||50||16||10||10||0||67||50|| ||<style="text-align: right;">Time ||17||0||0||0||0||0||0|| |
== Legenda == Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the actual time spent. In case of optimization problems, "quality" of the solutions plays a role as well. Please have a look at the [[ParticipationRules|Scoring and Participation Rules]] for all details. == Detailed Results == === Details: raw file === Detailed results, as a ''raw'' text file, can be downloaded [[attachment:systemcompetition.final.results.raw.detailed.txt|HERE]]. <<BR>> The file contains a line for each instance each system has been asked to solve. === Details: tables === Detailed results are also available as separate PDF files: * [[attachment:System-Results-P-Problems-detailed.pdf||System Track - Details - P problems]] * [[attachment:System-Results-NP-Problems-detailed.pdf||System Track - Details - NP problems]] * [[attachment:System-Results-BndP-Problems-detailed.pdf||System Track - Details - Beyond P problems]] === Quick reference guide === Please have a look at the [[http://www.mat.unical.it/aspcomp2011/files/LanguageSpecifications.pdf|file and language format specifications]] for more details. * System: the system name (quite easy to understand) * Benchmark: the benchmark problem * Instance: the specific instance of the problem * Out: the answer provided by the system for the specific instance * Diagnostics: result of the evaluation of the answer (essentially, whether the system answered correctly or not) * Time: the total time spent by the system while solving the specific instance * Optimum Cost: the "cost" of the witness solution returned by the system, in case of istance of an optimization problem * Total Score: the total score rewarded for the specific instance * Inst\Opt Score - Time Score: the two components of the score, as introduced above. * WF (Cost) : Witness Found (Cost) * OF (Cost) : Optimum Found (Cost) * Inc : Inconsistent * TO : Time Out * MO/F : Mem Out Fail * N/A : Not Available / Not Applicable == Overall == <<BR>> ||<tablewidth="1%"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">Instance ||<style="font-weight: bold; text-align: center;">Time || ||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|claspd]] ||<style="font-weight: bold;">861 ||560 ||301 || ||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|claspfolio]] ||<style="font-weight: bold;">818 ||535 ||283 || ||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|clasp]] ||<style="font-weight: bold;">810 ||520 ||290 || ||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|idp]] ||<style="font-weight: bold;">781 ||500 ||281 || ||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|cmodels]] ||<style="font-weight: bold;">766 ||510 ||256 || ||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|lp2diffz3]] ||<style="font-weight: bold;">572 ||405 ||167 || ||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|sup]] ||<style="font-weight: bold;">541 ||380 ||161 || ||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|lp2sat2gminisat]] ||<style="font-weight: bold;">495 ||365 ||130 || ||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|lp2sat2minisat]] ||<style="font-weight: bold;">481 ||355 ||126 || ||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|lp2sat2lminisat]] ||<style="font-weight: bold;">472 ||350 ||122 || ||<style="font-weight: bold; text-align: center;">[[http://goo.gl/4XXFz|smodels]] ||<style="font-weight: bold;">449 ||295 ||154 || |
| Line 46: | Line 67: |
| ||||<style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">P ||<style="font-weight: bold; text-align: center;">Reachability ||<style="font-weight: bold; text-align: center;">Grammar-Based IE ||<style="font-weight: bold; text-align: center;">Hydraulic Leaking ||<style="font-weight: bold; text-align: center;">Hydraulic Planning ||<style="font-weight: bold; text-align: center;">Stable Marriage ||<style="font-weight: bold; text-align: center;">Partner Units Polynomial || | |
| Line 48: | Line 68: |
| ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>claspd ||<style="text-align: right;">'''Total''' ||'''206''' ||'''36''' ||'''72''' ||'''10''' ||'''75''' ||'''0''' ||'''13''' || ||<style="text-align: right;">Instance ||145 ||25 ||50 ||10 ||50 ||0 ||10 || ||<style="text-align: right;">Time ||61 ||11 ||22 ||0 ||25 ||0 ||3 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>claspfolio ||<style="text-align: right;">'''Total''' ||'''209''' ||'''35''' ||'''72''' ||'''10''' ||'''74''' ||'''5''' ||'''13''' || ||<style="text-align: right;">Instance ||150 ||25 ||50 ||10 ||50 ||5 ||10 || ||<style="text-align: right;">Time ||59 ||10 ||22 ||0 ||24 ||0 ||3 || ||<style="text-align: center; vertical-align: top;" |3>'''clasp''' ||<style="text-align: right;">'''Total''' ||'''213''' ||'''36''' ||'''72''' ||'''10''' ||'''75''' ||'''6''' ||'''14''' || ||<style="text-align: right;">Instance ||150 ||25 ||50 ||10 ||50 ||5 ||10 || ||<style="text-align: right;">Time ||63 ||11 ||22 ||0 ||25 ||1 ||4 || ||<style="text-align: center; vertical-align: top;" |3>'''idp''' ||<style="text-align: right;">'''Total''' ||'''184''' ||'''29''' ||'''71''' ||'''10''' ||'''74''' ||'''0''' ||'''0''' || ||<style="text-align: right;">Instance ||130 ||20 ||50 ||10 ||50 ||0 ||0 || ||<style="text-align: right;">Time ||54 ||9 ||21 ||0 ||24 ||0 ||0 || ||<style="text-align: center; vertical-align: top;" |3>'''cmodels''' ||<style="text-align: right;">'''Total''' ||'''184''' ||'''29''' ||'''71''' ||'''10''' ||'''74''' ||'''0''' ||'''0''' || ||<style="text-align: right;">Instance ||130 ||20 ||50 ||10 ||50 ||0 ||0 || ||<style="text-align: right;">Time ||54 ||9 ||21 ||0 ||24 ||0 ||0 || ||<style="text-align: center; vertical-align: top;" |3>'''lp2diffz3''' ||<style="text-align: right;">'''Total''' ||'''178''' ||'''35''' ||'''66''' ||'''10''' ||'''67''' ||'''0''' ||'''0''' || ||<style="text-align: right;">Instance ||135 ||25 ||50 ||10 ||50 ||0 ||0 || ||<style="text-align: right;">Time ||43 ||10 ||16 ||0 ||17 ||0 ||0 || ||<style="text-align: center; vertical-align: top;" |3>'''sup''' ||<style="text-align: right;">'''Total''' ||'''195''' ||'''29''' ||'''71''' ||'''10''' ||'''74''' ||'''0''' ||'''11''' || ||<style="text-align: right;">Instance ||140 ||20 ||50 ||10 ||50 ||0 ||10 || ||<style="text-align: right;">Time ||55 ||9 ||21 ||0 ||24 ||0 ||1 || ||<style="text-align: center; vertical-align: top;" |3>'''lp2sat2gmsat''' ||<style="text-align: right;">'''Total''' ||'''185''' ||'''30''' ||'''66''' ||'''10''' ||'''68''' ||'''0''' ||'''11'''|| ||<style="text-align: right;">Instance ||140 ||20 ||50 ||10 ||50 ||0 ||10 || ||<style="text-align: right;">Time ||45 ||10 ||16 ||0 ||18 ||0 ||1 || ||<style="text-align: center; vertical-align: top;" |3>'''lp2sat2msat''' ||<style="text-align: right;">'''Total''' ||'''179''' ||'''30''' ||'''66''' ||'''10''' ||'''68''' ||'''0''' ||'''5''' || ||<style="text-align: right;">Instance ||135 ||20 ||50 ||10 ||50 ||0 ||5 || ||<style="text-align: right;">Time ||44 ||10 ||16 ||0 ||18 ||0 ||0 || ||<style="text-align: center; vertical-align: top;" |3>'''lp2sat2lmsat''' ||<style="text-align: right;">'''Total''' ||'''171''' ||'''28''' ||'''66''' ||'''10''' ||'''67''' ||'''0''' ||'''0''' || ||<style="text-align: right;">Instance ||130 ||20 ||50 ||10 ||50 ||0 ||0 || ||<style="text-align: right;">Time ||41 ||8 ||16 ||0 ||17 ||0 ||0 || ||<style="text-align: center; vertical-align: top;" |3>'''smodels''' ||<style="text-align: right;">'''Total''' ||'''180''' ||'''28''' ||'''70''' ||'''10''' ||'''72''' ||'''0''' ||'''0''' || ||<style="text-align: right;">Instance ||130 ||20 ||50 ||10 ||50 ||0 ||0 || ||<style="text-align: right;">Time ||50 ||8 ||20 ||0 ||22 ||0 ||0 || |
== Final results by categories == |
| Line 83: | Line 70: |
||||<style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">P ||<style="font-weight: bold; text-align: center;">NP ||<style="font-weight: bold; text-align: center;">Beyond NP ||<style="font-weight: bold; text-align: center;">Reachability ||<style="font-weight: bold; text-align: center;">Grammar-Based IE ||<style="font-weight: bold; text-align: center;">Hydraulic Leaking ||<style="font-weight: bold; text-align: center;">Hydraulic Planning ||<style="font-weight: bold; text-align: center;">Stable Marriage ||<style="font-weight: bold; text-align: center;">Partner Units Polynomial ||<style="font-weight: bold; text-align: center;">Sokoban Decision ||<style="font-weight: bold; text-align: center;">Knight Tour ||<style="font-weight: bold; text-align: center;">Disjunctive Scheduling ||<style="font-weight: bold; text-align: center;">Packing Problem ||<style="font-weight: bold; text-align: center;">Labyrinth ||<style="font-weight: bold; text-align: center;">MCS Querying ||<style="font-weight: bold; text-align: center;">Numberlink ||<style="font-weight: bold; text-align: center;">Hanoi Tower ||<style="font-weight: bold; text-align: center;">Graph Colouring ||<style="font-weight: bold; text-align: center;">Solitaire ||<style="font-weight: bold; text-align: center;">Weight-Assignment Tree ||<style="font-weight: bold; text-align: center;">Maze Generation ||<style="font-weight: bold; text-align: center;">Strategic Companies ||<style="font-weight: bold; text-align: center;">Minimal Diagnosis || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>claspd ||<style="text-align: right;">'''Total''' ||'''861''' ||'''206''' ||'''552''' ||'''103''' ||'''36''' ||'''72''' ||'''10''' ||'''75''' ||'''0''' ||'''13''' ||'''68''' ||'''68''' ||'''30''' ||'''0''' ||'''65''' ||'''75''' ||'''69''' ||'''31''' ||'''19''' ||'''11''' ||'''20''' ||'''96''' ||'''12''' ||'''91''' || ||<style="text-align: right;">Instance ||560 ||145 ||355 ||60 ||25 ||50 ||10 ||50 ||0 ||10 ||45 ||40 ||25 ||0 ||45 ||50 ||40 ||25 ||10 ||10 ||15 ||50 ||10 ||50 || ||<style="text-align: right;">Time ||301 ||61 ||197 ||43 ||11 ||22 ||0 ||25 ||0 ||3 ||23 ||28 ||5 ||0 ||20 ||25 ||29 ||6 ||9 ||1 ||5 ||46 ||2 ||41 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>claspfolio ||<style="text-align: right;">'''Total''' ||'''818''' ||'''209''' ||'''609''' ||'''0''' ||'''35''' ||'''72''' ||'''10''' ||'''74''' ||'''5''' ||'''13''' ||'''66''' ||'''65''' ||'''37''' ||'''0''' ||'''63''' ||'''75''' ||'''64''' ||'''47''' ||'''55''' ||'''21''' ||'''21''' ||'''95''' ||'''0''' ||'''0''' || ||<style="text-align: right;">Instance ||535 ||150 ||385 ||0 ||25 ||50 ||10 ||50 ||5 ||10 ||45 ||35 ||25 ||0 ||40 ||50 ||35 ||35 ||40 ||15 ||15 ||50 ||0 ||0 || ||<style="text-align: right;">Time ||283 ||59 ||224 ||0 ||10 ||22 ||0 ||24 ||0 ||3 ||21 ||30 ||12 ||0 ||23 ||25 ||29 ||12 ||15 ||6 ||6 ||45 ||0 ||0 || ||<style="text-align: center; vertical-align: top;" |3>'''clasp''' ||<style="text-align: right;">'''Total''' ||'''810''' ||'''213''' ||'''597''' ||'''0''' ||'''36''' ||'''72''' ||'''10''' ||'''75''' ||'''6''' ||'''14''' ||'''78''' ||'''63''' ||'''38''' ||'''0''' ||'''78''' ||'''75''' ||'''65''' ||'''39''' ||'''23''' ||'''21''' ||'''21''' ||'''96''' ||'''0''' ||'''0''' || ||<style="text-align: right;">Instance ||520 ||150 ||370 ||0 ||25 ||50 ||10 ||50 ||5 ||10 ||50 ||35 ||25 ||0 ||50 ||50 ||35 ||30 ||15 ||15 ||15 ||50 ||0 ||0 || ||<style="text-align: right;">Time ||290 ||63 ||227 ||0 ||11 ||22 ||0 ||25 ||1 ||4 ||28 ||28 ||13 ||0 ||28 ||25 ||30 ||9 ||8 ||6 ||6 ||46 ||0 ||0 || ||<style="text-align: center; vertical-align: top;" |3>'''idp''' ||<style="text-align: right;">'''Total''' ||'''781''' ||'''184''' ||'''597''' ||'''0''' ||'''29''' ||'''71''' ||'''10''' ||'''74''' ||'''0''' ||'''0''' ||'''64''' ||'''74''' ||'''38''' ||'''0''' ||'''52''' ||'''75''' ||'''70''' ||'''65''' ||'''18''' ||'''38''' ||'''8''' ||'''95''' ||'''0''' ||'''0''' || ||<style="text-align: right;">Instance ||500 ||130 ||370 ||0 ||20 ||50 ||10 ||50 ||0 ||0 ||45 ||45 ||25 ||0 ||30 ||50 ||40 ||45 ||10 ||25 ||5 ||50 ||0 ||0 || ||<style="text-align: right;">Time ||281 ||54 ||227 ||0 ||9 ||21 ||0 ||24 ||0 ||0 ||19 ||29 ||13 ||0 ||22 ||25 ||30 ||20 ||8 ||13 ||3 ||45 ||0 ||0 || ||<style="text-align: center; vertical-align: top;" |3>'''cmodels''' ||<style="text-align: right;">'''Total''' ||'''766''' ||'''184''' ||'''510''' ||'''72''' ||'''29''' ||'''71''' ||'''10''' ||'''74''' ||'''0''' ||'''0''' ||'''67''' ||'''56''' ||'''21''' ||'''0''' ||'''62''' ||'''75''' ||'''30''' ||'''51''' ||'''29''' ||'''18''' ||'''6''' ||'''95''' ||'''0''' ||'''72''' || ||<style="text-align: right;">Instance ||510 ||130 ||335 ||45 ||20 ||50 ||10 ||50 ||0 ||0 ||45 ||30 ||20 ||0 ||45 ||50 ||20 ||35 ||20 ||15 ||5 ||50 ||0 ||45 || ||<style="text-align: right;">Time ||256 ||54 ||175 ||27 ||9 ||21 ||0 ||24 ||0 ||0 ||22 ||26 ||1 ||0 ||17 ||25 ||10 ||16 ||9 ||3 ||1 ||45 ||0 ||27 || ||<style="text-align: center; vertical-align: top;" |3>'''lp2diffz3''' ||<style="text-align: right;">'''Total''' ||'''572''' ||'''178''' ||'''394''' ||'''0''' ||'''35''' ||'''66''' ||'''10''' ||'''67''' ||'''0''' ||'''0''' ||'''42''' ||'''55''' ||'''0''' ||'''0''' ||'''0''' ||'''70''' ||'''45''' ||'''47''' ||'''27''' ||'''25''' ||'''0''' ||'''83''' ||'''0''' ||'''0''' || ||<style="text-align: right;">Instance ||405 ||135 ||270 ||0 ||25 ||50 ||10 ||50 ||0 ||0 ||30 ||35 ||0 ||0 ||0 ||50 ||30 ||35 ||20 ||20 ||0 ||50 ||0 ||0 || ||<style="text-align: right;">Time ||167 ||43 ||124 ||0 ||10 ||16 ||0 ||17 ||0 ||0 ||12 ||20 ||0 ||0 ||0 ||20 ||15 ||12 ||7 ||5 ||0 ||33 ||0 ||0 || ||<style="text-align: center; vertical-align: top;" |3>'''sup''' ||<style="text-align: right;">'''Total''' ||'''541''' ||'''195''' ||'''346''' ||'''0''' ||'''29''' ||'''71''' ||'''10''' ||'''74''' ||'''0''' ||'''11''' ||'''52''' ||'''40''' ||'''37''' ||'''0''' ||'''58''' ||'''72''' ||'''0''' ||'''31''' ||'''16''' ||'''15''' ||'''25''' ||'''0''' ||'''0''' ||'''0''' || ||<style="text-align: right;">Instance ||380 ||140 ||240 ||0 ||20 ||50 ||10 ||50 ||0 ||10 ||35 ||25 ||25 ||0 ||40 ||50 ||0 ||25 ||10 ||10 ||20 ||0 ||0 ||0 || ||<style="text-align: right;">Time ||161 ||55 ||106 ||0 ||9 ||21 ||0 ||24 ||0 ||1 ||17 ||15 ||12 ||0 ||18 ||22 ||0 ||6 ||6 ||5 ||5 ||0 ||0 ||0 || ||<style="text-align: center; vertical-align: top;" |3>'''lp2sat2gmsat''' ||<style="text-align: right;">'''Total''' ||'''495''' ||'''185''' ||'''310''' ||'''0''' ||'''30''' ||'''66''' ||'''10''' ||'''68''' ||'''0''' ||'''11''' ||'''36''' ||'''10''' ||'''32''' ||'''0''' ||'''46''' ||'''71''' ||'''22''' ||'''47''' ||'''17''' ||'''29''' ||'''-''' ||'''0''' ||'''0''' ||'''0''' || ||<style="text-align: right;">Instance ||365 ||140 ||225 ||0 ||20 ||50 ||10 ||50 ||0 ||10 ||30 ||5 ||25 ||0 ||35 ||50 ||15 ||35 ||10 ||20 ||- ||0 ||0 ||0 || ||<style="text-align: right;">Time ||130 ||45 ||85 ||0 ||10 ||16 ||0 ||18 ||0 ||1 ||6 ||5 ||7 ||0 ||11 ||21 ||7 ||12 ||7 ||9 ||- ||0 ||0 ||0 || ||<style="text-align: center; vertical-align: top;" |3>'''lp2sat2msat''' ||<style="text-align: right;">'''Total''' ||'''481''' ||'''179''' ||'''302''' ||'''0''' ||'''30''' ||'''66''' ||'''10''' ||'''68''' ||'''0''' ||'''5''' ||'''39''' ||'''0''' ||'''32''' ||'''0''' ||'''52''' ||'''71''' ||'''15''' ||'''47''' ||'''17''' ||'''29''' ||'''-''' ||'''0''' ||'''0''' ||'''0''' || ||<style="text-align: right;">Instance ||355 ||135 ||220 ||0 ||20 ||50 ||10 ||50 ||0 ||5 ||30 ||0 ||25 ||0 ||40 ||50 ||10 ||35 ||10 ||20 ||- ||0 ||0 ||0 || ||<style="text-align: right;">Time ||126 ||44 ||82 ||0 ||10 ||16 ||0 ||18 ||0 ||0 ||9 ||0 ||7 ||0 ||12 ||21 ||5 ||12 ||7 ||9 ||- ||0 ||0 ||0 || ||<style="text-align: center; vertical-align: top;" |3>'''lp2sat2lmsat''' ||<style="text-align: right;">'''Total''' ||'''472''' ||'''171''' ||'''301''' ||'''0''' ||'''28''' ||'''66''' ||'''10''' ||'''67''' ||'''0''' ||'''0''' ||'''35''' ||'''0''' ||'''32''' ||'''0''' ||'''53''' ||'''71''' ||'''17''' ||'''47''' ||'''17''' ||'''29''' ||'''-''' ||'''0''' ||'''0''' ||'''0''' || ||<style="text-align: right;">Instance ||350 ||130 ||220 ||0 ||20 ||50 ||10 ||50 ||0 ||0 ||30 ||0 ||25 ||0 ||40 ||50 ||10 ||35 ||10 ||20 ||- ||0 ||0 ||0 || ||<style="text-align: right;">Time ||122 ||41 ||81 ||0 ||8 ||16 ||0 ||17 ||0 ||0 ||5 ||0 ||7 ||0 ||13 ||21 ||7 ||12 ||7 ||9 ||- ||0 ||0 ||0 || ||<style="text-align: center; vertical-align: top;" |3>'''smodels''' ||<style="text-align: right;">'''Total''' ||'''449''' ||'''180''' ||'''269''' ||'''0''' ||'''28''' ||'''70''' ||'''10''' ||'''72''' ||'''0''' ||'''0''' ||'''0''' ||'''55''' ||'''36''' ||'''0''' ||'''9''' ||'''53''' ||'''27''' ||'''0''' ||'''0''' ||'''0''' ||'''0''' ||'''89''' ||'''0''' ||'''0''' || ||<style="text-align: right;">Instance ||295 ||130 ||165 ||0 ||20 ||50 ||10 ||50 ||0 ||0 ||0 ||30 ||25 ||0 ||5 ||35 ||20 ||0 ||0 ||0 ||0 ||50 ||0 ||0 || ||<style="text-align: right;">Time ||154 ||50 ||104 ||0 ||8 ||20 ||0 ||22 ||0 ||0 ||0 ||25 ||11 ||0 ||4 ||18 ||7 ||0 ||0 ||0 ||0 ||39 ||0 ||0 || |
||||||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center;">'''P''' ||<style="text-align: center;" |13> ||||||||<style="text-align: center;">'''NP''' ||<style="text-align: center;" |13> ||||||||<style="text-align: center;">'''Beyond NP''' || ||'''System''' ||'''Total Score''' ||'''Inst. Score''' ||'''Time Score''' ||<style="text-align: center;">'''System''' ||'''Total Score''' ||'''Inst. Score''' ||'''Time Score''' ||<style="text-align: center;">'''System''' ||'''Total Score''' ||'''Inst. Score''' ||'''Time Score''' || ||<style="font-weight: bold;">clasp ||213 ||150 ||63 ||<style="text-align: center; font-weight: bold;">claspfolio ||609 ||385 ||224 ||<style="text-align: center; font-weight: bold;">claspd ||103 ||60 ||43 || ||<style="font-weight: bold;">claspfolio ||209 ||150 ||59 ||<style="text-align: center; font-weight: bold;">clasp ||597 ||370 ||227 ||<style="text-align: center; font-weight: bold;">cmodels ||72 ||45 ||27 || ||<style="font-weight: bold;">claspd ||206 ||145 ||61 ||<style="text-align: center; font-weight: bold;">idp ||597 ||370 ||227 ||<style="text-align: center; font-weight: bold;">claspfolio ||- ||- ||- || ||<style="font-weight: bold;">sup ||195 ||140 ||55 ||<style="text-align: center; font-weight: bold;">claspd ||552 ||355 ||197 ||<style="text-align: center; font-weight: bold;">clasp ||- ||- ||- || ||<style="font-weight: bold;">lp2sat2gminisat ||185 ||140 ||45 ||<style="text-align: center; font-weight: bold;">cmodels ||510 ||335 ||175 ||<style="text-align: center; font-weight: bold;">idp ||- ||- ||- || ||<style="font-weight: bold;">cmodels ||184 ||130 ||54 ||<style="text-align: center; font-weight: bold;">lp2diffz3 ||394 ||270 ||124 ||<style="text-align: center; font-weight: bold;">lp2diffz3 ||- ||- ||- || ||<style="font-weight: bold;">idp ||184 ||130 ||54 ||<style="text-align: center; font-weight: bold;">sup ||346 ||240 ||106 ||<style="text-align: center; font-weight: bold;">sup ||- ||- ||- || ||<style="font-weight: bold;">smodels ||180 ||130 ||50 ||<style="text-align: center; font-weight: bold;">lp2sat2gminisat ||310 ||225 ||85 ||<style="text-align: center; font-weight: bold;">lp2sat2gminisat ||- ||- ||- || ||<style="font-weight: bold;">lp2sat2minisat ||179 ||135 ||44 ||<style="text-align: center; font-weight: bold;">lp2sat2minisat ||302 ||220 ||82 ||<style="text-align: center; font-weight: bold;">lp2sat2minisat ||- ||- ||- || ||<style="font-weight: bold;">lp2diffz3 ||178 ||135 ||43 ||<style="text-align: center; font-weight: bold;">lp2sat2lminisat ||301 ||220 ||81 ||<style="text-align: center; font-weight: bold;">lp2sat2lminisat ||- ||- ||- || ||<style="font-weight: bold;">lp2sat2lminisat ||171 ||130 ||41 ||<style="text-align: center; font-weight: bold;">smodels ||269 ||165 ||104 ||<style="text-align: center; font-weight: bold;">smodels ||- ||- ||- || |
| Line 122: | Line 87: |
| === Final results by categories === | |
| Line 124: | Line 88: |
| ||||||||<style="text-align: center;">'''P''' || ||||||||<style="text-align: center;">'''NP''' || ||||||||<style="text-align: center;">'''Beyond NP''' || ||'''System''' ||'''Total Score''' ||'''Inst. Score''' ||'''Time Score''' ||||<style="text-align: center;">'''System''' ||'''Total Score''' ||'''Inst. Score''' ||'''Time Score''' ||||<style="text-align: center;">'''System''' ||'''Total Score''' ||'''Inst. Score''' ||'''Time Score''' || ||clasp ||213 ||150 ||63 ||||<style="text-align: center;">claspfolio ||609 ||385 ||224 ||||<style="text-align: center;">claspd ||103 ||60 ||43 || ||claspfolio ||209 ||150 ||59 ||||<style="text-align: center;">clasp ||597 ||370 ||227 ||||<style="text-align: center;">cmodels ||72 ||45 ||27 || ||claspd ||206 ||145 ||61 ||||<style="text-align: center;">idp ||597 ||370 ||227 ||||<style="text-align: center;">claspfolio ||0 ||0 ||0 || ||sup ||195 ||140 ||55 ||||<style="text-align: center;">claspd ||552 ||355 ||197 ||||<style="text-align: center;">clasp ||0 ||0 ||0 || ||lp2sat2gminisat ||185 ||140 ||45 ||||<style="text-align: center;">cmodels ||510 ||335 ||175 ||||<style="text-align: center;">idp ||0 ||0 ||0 || ||cmodels ||184 ||130 ||54 ||||<style="text-align: center;">lp2diffz3 ||394 ||270 ||124 ||||<style="text-align: center;">lp2diffz3 ||0 ||0 ||0 || ||idp ||184 ||130 ||54 ||||<style="text-align: center;">sup ||346 ||240 ||106 ||||<style="text-align: center;">sup ||0 ||0 ||0 || ||smodels ||180 ||130 ||50 ||||<style="text-align: center;">lp2sat2gminisat ||310 ||225 ||85 ||||<style="text-align: center;">lp2sat2gminisat ||0 ||0 ||0 || ||lp2sat2minisat ||179 ||135 ||44 ||||<style="text-align: center;">lp2sat2minisat ||302 ||220 ||82 ||||<style="text-align: center;">lp2sat2minisat ||0 ||0 ||0 || ||lp2diffz3 ||178 ||135 ||43 ||||<style="text-align: center;">lp2sat2lminisat ||301 ||220 ||81 ||||<style="text-align: center;">lp2sat2lminisat ||0 ||0 ||0 || ||lp2sat2lminisat ||171 ||130 ||41 ||||<style="text-align: center;">smodels ||269 ||165 ||104 ||||<style="text-align: center;">smodels ||0 ||0 ||0 || |
<<BR>> === P Problems === <<BR>> ||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">[[SystemReachabilityResults|Reachability]] ||<style="font-weight: bold; text-align: center;">[[SystemGrammarBasedIEResults|Grammar-Based IE]] ||<style="font-weight: bold; text-align: center;">[[SystemHydraulicLeakingResults|Hydraulic Leaking]] ||<style="font-weight: bold; text-align: center;">[[SystemHydraulicPlanningResults|Hydraulic Planning]] ||<style="font-weight: bold; text-align: center;">[[SystemStableMarriageResults|Stable Marriage]] ||<style="font-weight: bold; text-align: center;">[[SystemPartnerUnitsPolynomialResults|Partner Units Polynomial]] || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>clasp ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">213 ||<style="font-weight: bold;">36 ||<style="font-weight: bold;">72 ||<style="font-weight: bold;">10 ||<style="font-weight: bold;">75 ||<style="font-weight: bold;">6 ||<style="font-weight: bold;">14 || ||<style="text-align: right;">Instance ||150 ||25 ||50 ||10 ||50 ||5 ||10 || ||<style="text-align: right;">Time ||63 ||11 ||22 ||0 ||25 ||1 ||4 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>claspfolio ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">209 ||<style="font-weight: bold;">35 ||<style="font-weight: bold;">72 ||<style="font-weight: bold;">10 ||<style="font-weight: bold;">74 ||<style="font-weight: bold;">5 ||<style="font-weight: bold;">13 || ||<style="text-align: right;">Instance ||150 ||25 ||50 ||10 ||50 ||5 ||10 || ||<style="text-align: right;">Time ||59 ||10 ||22 ||0 ||24 ||0 ||3 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>claspd ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">206 ||<style="font-weight: bold;">36 ||<style="font-weight: bold;">72 ||<style="font-weight: bold;">10 ||<style="font-weight: bold;">75 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">13 || ||<style="text-align: right;">Instance ||145 ||25 ||50 ||10 ||50 ||0 ||10 || ||<style="text-align: right;">Time ||61 ||11 ||22 ||0 ||25 ||0 ||3 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>sup ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">195 ||<style="font-weight: bold;">29 ||<style="font-weight: bold;">71 ||<style="font-weight: bold;">10 ||<style="font-weight: bold;">74 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">11 || ||<style="text-align: right;">Instance ||140 ||20 ||50 ||10 ||50 ||0 ||10 || ||<style="text-align: right;">Time ||55 ||9 ||21 ||0 ||24 ||0 ||1 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2sat2gminisat ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">185 ||<style="font-weight: bold;">30 ||<style="font-weight: bold;">66 ||<style="font-weight: bold;">10 ||<style="font-weight: bold;">68 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">11 || ||<style="text-align: right;">Instance ||140 ||20 ||50 ||10 ||50 ||0 ||10 || ||<style="text-align: right;">Time ||45 ||10 ||16 ||0 ||18 ||0 ||1 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>cmodels ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">184 ||<style="font-weight: bold;">29 ||<style="font-weight: bold;">71 ||<style="font-weight: bold;">10 ||<style="font-weight: bold;">74 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">0 || ||<style="text-align: right;">Instance ||130 ||20 ||50 ||10 ||50 ||0 ||0 || ||<style="text-align: right;">Time ||54 ||9 ||21 ||0 ||24 ||0 ||0 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>idp ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">184 ||<style="font-weight: bold;">29 ||<style="font-weight: bold;">71 ||<style="font-weight: bold;">10 ||<style="font-weight: bold;">74 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">0 || ||<style="text-align: right;">Instance ||130 ||20 ||50 ||10 ||50 ||0 ||0 || ||<style="text-align: right;">Time ||54 ||9 ||21 ||0 ||24 ||0 ||0 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>smodels ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">180 ||<style="font-weight: bold;">28 ||<style="font-weight: bold;">70 ||<style="font-weight: bold;">10 ||<style="font-weight: bold;">72 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">0 || ||<style="text-align: right;">Instance ||130 ||20 ||50 ||10 ||50 ||0 ||0 || ||<style="text-align: right;">Time ||50 ||8 ||20 ||0 ||22 ||0 ||0 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2sat2minisat ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">179 ||<style="font-weight: bold;">30 ||<style="font-weight: bold;">66 ||<style="font-weight: bold;">10 ||<style="font-weight: bold;">68 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">5 || ||<style="text-align: right;">Instance ||135 ||20 ||50 ||10 ||50 ||0 ||5 || ||<style="text-align: right;">Time ||44 ||10 ||16 ||0 ||18 ||0 ||0 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2diffz3 ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">178 ||<style="font-weight: bold;">35 ||<style="font-weight: bold;">66 ||<style="font-weight: bold;">10 ||<style="font-weight: bold;">67 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">0 || ||<style="text-align: right;">Instance ||135 ||25 ||50 ||10 ||50 ||0 ||0 || ||<style="text-align: right;">Time ||43 ||10 ||16 ||0 ||17 ||0 ||0 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2sat2lminisat ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">171 ||<style="font-weight: bold;">28 ||<style="font-weight: bold;">66 ||<style="font-weight: bold;">10 ||<style="font-weight: bold;">67 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">0 || ||<style="text-align: right;">Instance ||130 ||20 ||50 ||10 ||50 ||0 ||0 || ||<style="text-align: right;">Time ||41 ||8 ||16 ||0 ||17 ||0 ||0 || <<BR>> <<BR>> === NP Problems === <<BR>> ||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">[[SystemSokobanDecisionResults|Sokoban Decision]] ||<style="font-weight: bold; text-align: center;">[[SystemKnightTourResults|Knight Tour]] ||<style="font-weight: bold; text-align: center;">[[SystemDisjunctiveSchedulingResults|Disjunctive Scheduling]] ||<style="font-weight: bold; text-align: center;">[[SystemPackingProblemResults|Packing Problem]] ||<style="font-weight: bold; text-align: center;">[[SystemLabyrinthResults|Labyrinth]] ||<style="font-weight: bold; text-align: center;">[[SystemMCSQueryingResults|MCS Querying]] ||<style="font-weight: bold; text-align: center;">[[SystemNumberlinkResults|Numberlink]] ||<style="font-weight: bold; text-align: center;">[[SystemHanoiTowerResults|Hanoi Tower]] ||<style="font-weight: bold; text-align: center;">[[SystemGraphColouringResults|Graph Colouring]] ||<style="font-weight: bold; text-align: center;">[[SystemSolitaireResults|Solitaire]] ||<style="font-weight: bold; text-align: center;">[[SystemWeightAssignmentTreeResults|Weight-Assignment Tree]] ||<style="font-weight: bold; text-align: center;">[[SystemMazeGenerationResults|Maze Generation]] || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>claspfolio ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">609 ||<style="font-weight: bold;">66 ||<style="font-weight: bold;">65 ||<style="font-weight: bold;">37 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">63 ||<style="font-weight: bold;">75 ||<style="font-weight: bold;">64 ||<style="font-weight: bold;">47 ||<style="font-weight: bold;">55 ||<style="font-weight: bold;">21 ||<style="font-weight: bold;">21 ||<style="font-weight: bold;">95 || ||<style="text-align: right;">Instance ||385 ||45 ||35 ||25 ||0 ||40 ||50 ||35 ||35 ||40 ||15 ||15 ||50 || ||<style="text-align: right;">Time ||224 ||21 ||30 ||12 ||0 ||23 ||25 ||29 ||12 ||15 ||6 ||6 ||45 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>clasp ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">597 ||<style="font-weight: bold;">78 ||<style="font-weight: bold;">63 ||<style="font-weight: bold;">38 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">78 ||<style="font-weight: bold;">75 ||<style="font-weight: bold;">65 ||<style="font-weight: bold;">39 ||<style="font-weight: bold;">23 ||<style="font-weight: bold;">21 ||<style="font-weight: bold;">21 ||<style="font-weight: bold;">96 || ||<style="text-align: right;">Instance ||370 ||50 ||35 ||25 ||0 ||50 ||50 ||35 ||30 ||15 ||15 ||15 ||50 || ||<style="text-align: right;">Time ||227 ||28 ||28 ||13 ||0 ||28 ||25 ||30 ||9 ||8 ||6 ||6 ||46 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>idp ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">597 ||<style="font-weight: bold;">64 ||<style="font-weight: bold;">74 ||<style="font-weight: bold;">38 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">52 ||<style="font-weight: bold;">75 ||<style="font-weight: bold;">70 ||<style="font-weight: bold;">65 ||<style="font-weight: bold;">18 ||<style="font-weight: bold;">38 ||<style="font-weight: bold;">8 ||<style="font-weight: bold;">95 || ||<style="text-align: right;">Instance ||370 ||45 ||45 ||25 ||0 ||30 ||50 ||40 ||45 ||10 ||25 ||5 ||50 || ||<style="text-align: right;">Time ||227 ||19 ||29 ||13 ||0 ||22 ||25 ||30 ||20 ||8 ||13 ||3 ||45 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>claspd ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">552 ||<style="font-weight: bold;">68 ||<style="font-weight: bold;">68 ||<style="font-weight: bold;">30 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">65 ||<style="font-weight: bold;">75 ||<style="font-weight: bold;">69 ||<style="font-weight: bold;">31 ||<style="font-weight: bold;">19 ||<style="font-weight: bold;">11 ||<style="font-weight: bold;">20 ||<style="font-weight: bold;">96 || ||<style="text-align: right;">Instance ||355 ||45 ||40 ||25 ||0 ||45 ||50 ||40 ||25 ||10 ||10 ||15 ||50 || ||<style="text-align: right;">Time ||197 ||23 ||28 ||5 ||0 ||20 ||25 ||29 ||6 ||9 ||1 ||5 ||46 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>cmodels ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">510 ||<style="font-weight: bold;">67 ||<style="font-weight: bold;">56 ||<style="font-weight: bold;">21 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">62 ||<style="font-weight: bold;">75 ||<style="font-weight: bold;">30 ||<style="font-weight: bold;">51 ||<style="font-weight: bold;">29 ||<style="font-weight: bold;">18 ||<style="font-weight: bold;">6 ||<style="font-weight: bold;">95 || ||<style="text-align: right;">Instance ||335 ||45 ||30 ||20 ||0 ||45 ||50 ||20 ||35 ||20 ||15 ||5 ||50 || ||<style="text-align: right;">Time ||175 ||22 ||26 ||1 ||0 ||17 ||25 ||10 ||16 ||9 ||3 ||1 ||45 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2diffz3 ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">394 ||<style="font-weight: bold;">42 ||<style="font-weight: bold;">55 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">70 ||<style="font-weight: bold;">45 ||<style="font-weight: bold;">47 ||<style="font-weight: bold;">27 ||<style="font-weight: bold;">25 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">83 || ||<style="text-align: right;">Instance ||270 ||30 ||35 ||0 ||0 ||0 ||50 ||30 ||35 ||20 ||20 ||0 ||50 || ||<style="text-align: right;">Time ||124 ||12 ||20 ||0 ||0 ||0 ||20 ||15 ||12 ||7 ||5 ||0 ||33 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>sup ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">346 ||<style="font-weight: bold;">52 ||<style="font-weight: bold;">40 ||<style="font-weight: bold;">37 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">58 ||<style="font-weight: bold;">72 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">31 ||<style="font-weight: bold;">16 ||<style="font-weight: bold;">15 ||<style="font-weight: bold;">25 ||<style="font-weight: bold;">0 || ||<style="text-align: right;">Instance ||240 ||35 ||25 ||25 ||0 ||40 ||50 ||0 ||25 ||10 ||10 ||20 ||0 || ||<style="text-align: right;">Time ||106 ||17 ||15 ||12 ||0 ||18 ||22 ||0 ||6 ||6 ||5 ||5 ||0 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2sat2gminisat ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">310 ||<style="font-weight: bold;">36 ||<style="font-weight: bold;">10 ||<style="font-weight: bold;">32 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">46 ||<style="font-weight: bold;">71 ||<style="font-weight: bold;">22 ||<style="font-weight: bold;">47 ||<style="font-weight: bold;">17 ||<style="font-weight: bold;">29 ||<style="font-weight: bold;">- ||<style="font-weight: bold;">0 || ||<style="text-align: right;">Instance ||225 ||30 ||5 ||25 ||0 ||35 ||50 ||15 ||35 ||10 ||20 ||- ||0 || ||<style="text-align: right;">Time ||85 ||6 ||5 ||7 ||0 ||11 ||21 ||7 ||12 ||7 ||9 ||- ||0 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2sat2minisat ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">302 ||<style="font-weight: bold;">39 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">32 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">52 ||<style="font-weight: bold;">71 ||<style="font-weight: bold;">15 ||<style="font-weight: bold;">47 ||<style="font-weight: bold;">17 ||<style="font-weight: bold;">29 ||<style="font-weight: bold;">- ||<style="font-weight: bold;">0 || ||<style="text-align: right;">Instance ||220 ||30 ||0 ||25 ||0 ||40 ||50 ||10 ||35 ||10 ||20 ||- ||0 || ||<style="text-align: right;">Time ||82 ||9 ||0 ||7 ||0 ||12 ||21 ||5 ||12 ||7 ||9 ||- ||0 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>lp2sat2lminisat ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">301 ||<style="font-weight: bold;">35 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">32 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">53 ||<style="font-weight: bold;">71 ||<style="font-weight: bold;">17 ||<style="font-weight: bold;">47 ||<style="font-weight: bold;">17 ||<style="font-weight: bold;">29 ||<style="font-weight: bold;">- ||<style="font-weight: bold;">0 || ||<style="text-align: right;">Instance ||220 ||30 ||0 ||25 ||0 ||40 ||50 ||10 ||35 ||10 ||20 ||- ||0 || ||<style="text-align: right;">Time ||81 ||5 ||0 ||7 ||0 ||13 ||21 ||7 ||12 ||7 ||9 ||- ||0 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>smodels ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">269 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">55 ||<style="font-weight: bold;">36 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">9 ||<style="font-weight: bold;">53 ||<style="font-weight: bold;">27 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">89 || ||<style="text-align: right;">Instance ||165 ||0 ||30 ||25 ||0 ||5 ||35 ||20 ||0 ||0 ||0 ||0 ||50 || ||<style="text-align: right;">Time ||104 ||0 ||25 ||11 ||0 ||4 ||18 ||7 ||0 ||0 ||0 ||0 ||39 || <<BR>> <<BR>> === Beyond NP Problems === <<BR>> ||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">[[SystemStrategicCompaniesResults|Strategic Companies]] ||<style="font-weight: bold; text-align: center;">[[SystemMinimalDiagnosisResults|Minimal Diagnosis]] || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>claspd ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">103 ||<style="font-weight: bold;">12 ||<style="font-weight: bold;">91 || ||<style="text-align: right;">Instance ||60 ||10 ||50 || ||<style="text-align: right;">Time ||43 ||2 ||41 || ||<style="font-weight: bold; text-align: center; vertical-align: top;" |3>cmodels ||<style="text-align: right;">'''Total''' ||<style="font-weight: bold;">72 ||<style="font-weight: bold;">0 ||<style="font-weight: bold;">72 || ||<style="text-align: right;">Instance ||45 ||0 ||45 || ||<style="text-align: right;">Time ||27 ||0 ||27 || <<BR>> |
System Track Final Results
Contents
Legenda
Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the actual time spent. In case of optimization problems, "quality" of the solutions plays a role as well. Please have a look at the Scoring and Participation Rules for all details.
Detailed Results
Details: raw file
Detailed results, as a raw text file, can be downloaded HERE.
The file contains a line for each instance each system has been asked to solve.
Details: tables
Detailed results are also available as separate PDF files:
Quick reference guide
Please have a look at the file and language format specifications for more details.
- System: the system name (quite easy to understand)
- Benchmark: the benchmark problem
- Instance: the specific instance of the problem
- Out: the answer provided by the system for the specific instance
- Diagnostics: result of the evaluation of the answer (essentially, whether the system answered correctly or not)
- Time: the total time spent by the system while solving the specific instance
- Optimum Cost: the "cost" of the witness solution returned by the system, in case of istance of an optimization problem
- Total Score: the total score rewarded for the specific instance
- Inst\Opt Score - Time Score: the two components of the score, as introduced above.
- WF (Cost) : Witness Found (Cost)
- OF (Cost) : Optimum Found (Cost)
- Inc : Inconsistent
- TO : Time Out
- MO/F : Mem Out Fail
- N/A : Not Available / Not Applicable
Overall
System |
Total |
Instance |
Time |
861 |
560 |
301 |
|
818 |
535 |
283 |
|
810 |
520 |
290 |
|
781 |
500 |
281 |
|
766 |
510 |
256 |
|
572 |
405 |
167 |
|
541 |
380 |
161 |
|
495 |
365 |
130 |
|
481 |
355 |
126 |
|
472 |
350 |
122 |
|
449 |
295 |
154 |
Final results by categories
P |
|
NP |
|
Beyond NP |
|||||||||
System |
Total Score |
Inst. Score |
Time Score |
System |
Total Score |
Inst. Score |
Time Score |
System |
Total Score |
Inst. Score |
Time Score |
||
clasp |
213 |
150 |
63 |
claspfolio |
609 |
385 |
224 |
claspd |
103 |
60 |
43 |
||
claspfolio |
209 |
150 |
59 |
clasp |
597 |
370 |
227 |
cmodels |
72 |
45 |
27 |
||
claspd |
206 |
145 |
61 |
idp |
597 |
370 |
227 |
claspfolio |
- |
- |
- |
||
sup |
195 |
140 |
55 |
claspd |
552 |
355 |
197 |
clasp |
- |
- |
- |
||
lp2sat2gminisat |
185 |
140 |
45 |
cmodels |
510 |
335 |
175 |
idp |
- |
- |
- |
||
cmodels |
184 |
130 |
54 |
lp2diffz3 |
394 |
270 |
124 |
lp2diffz3 |
- |
- |
- |
||
idp |
184 |
130 |
54 |
sup |
346 |
240 |
106 |
sup |
- |
- |
- |
||
smodels |
180 |
130 |
50 |
lp2sat2gminisat |
310 |
225 |
85 |
lp2sat2gminisat |
- |
- |
- |
||
lp2sat2minisat |
179 |
135 |
44 |
lp2sat2minisat |
302 |
220 |
82 |
lp2sat2minisat |
- |
- |
- |
||
lp2diffz3 |
178 |
135 |
43 |
lp2sat2lminisat |
301 |
220 |
81 |
lp2sat2lminisat |
- |
- |
- |
||
lp2sat2lminisat |
171 |
130 |
41 |
smodels |
269 |
165 |
104 |
smodels |
- |
- |
- |
||
P Problems
System |
Total |
|||||||
clasp |
Total |
213 |
36 |
72 |
10 |
75 |
6 |
14 |
Instance |
150 |
25 |
50 |
10 |
50 |
5 |
10 |
|
Time |
63 |
11 |
22 |
0 |
25 |
1 |
4 |
|
claspfolio |
Total |
209 |
35 |
72 |
10 |
74 |
5 |
13 |
Instance |
150 |
25 |
50 |
10 |
50 |
5 |
10 |
|
Time |
59 |
10 |
22 |
0 |
24 |
0 |
3 |
|
claspd |
Total |
206 |
36 |
72 |
10 |
75 |
0 |
13 |
Instance |
145 |
25 |
50 |
10 |
50 |
0 |
10 |
|
Time |
61 |
11 |
22 |
0 |
25 |
0 |
3 |
|
sup |
Total |
195 |
29 |
71 |
10 |
74 |
0 |
11 |
Instance |
140 |
20 |
50 |
10 |
50 |
0 |
10 |
|
Time |
55 |
9 |
21 |
0 |
24 |
0 |
1 |
|
lp2sat2gminisat |
Total |
185 |
30 |
66 |
10 |
68 |
0 |
11 |
Instance |
140 |
20 |
50 |
10 |
50 |
0 |
10 |
|
Time |
45 |
10 |
16 |
0 |
18 |
0 |
1 |
|
cmodels |
Total |
184 |
29 |
71 |
10 |
74 |
0 |
0 |
Instance |
130 |
20 |
50 |
10 |
50 |
0 |
0 |
|
Time |
54 |
9 |
21 |
0 |
24 |
0 |
0 |
|
idp |
Total |
184 |
29 |
71 |
10 |
74 |
0 |
0 |
Instance |
130 |
20 |
50 |
10 |
50 |
0 |
0 |
|
Time |
54 |
9 |
21 |
0 |
24 |
0 |
0 |
|
smodels |
Total |
180 |
28 |
70 |
10 |
72 |
0 |
0 |
Instance |
130 |
20 |
50 |
10 |
50 |
0 |
0 |
|
Time |
50 |
8 |
20 |
0 |
22 |
0 |
0 |
|
lp2sat2minisat |
Total |
179 |
30 |
66 |
10 |
68 |
0 |
5 |
Instance |
135 |
20 |
50 |
10 |
50 |
0 |
5 |
|
Time |
44 |
10 |
16 |
0 |
18 |
0 |
0 |
|
lp2diffz3 |
Total |
178 |
35 |
66 |
10 |
67 |
0 |
0 |
Instance |
135 |
25 |
50 |
10 |
50 |
0 |
0 |
|
Time |
43 |
10 |
16 |
0 |
17 |
0 |
0 |
|
lp2sat2lminisat |
Total |
171 |
28 |
66 |
10 |
67 |
0 |
0 |
Instance |
130 |
20 |
50 |
10 |
50 |
0 |
0 |
|
Time |
41 |
8 |
16 |
0 |
17 |
0 |
0 |
|
NP Problems
System |
Total |
|||||||||||||
claspfolio |
Total |
609 |
66 |
65 |
37 |
0 |
63 |
75 |
64 |
47 |
55 |
21 |
21 |
95 |
Instance |
385 |
45 |
35 |
25 |
0 |
40 |
50 |
35 |
35 |
40 |
15 |
15 |
50 |
|
Time |
224 |
21 |
30 |
12 |
0 |
23 |
25 |
29 |
12 |
15 |
6 |
6 |
45 |
|
clasp |
Total |
597 |
78 |
63 |
38 |
0 |
78 |
75 |
65 |
39 |
23 |
21 |
21 |
96 |
Instance |
370 |
50 |
35 |
25 |
0 |
50 |
50 |
35 |
30 |
15 |
15 |
15 |
50 |
|
Time |
227 |
28 |
28 |
13 |
0 |
28 |
25 |
30 |
9 |
8 |
6 |
6 |
46 |
|
idp |
Total |
597 |
64 |
74 |
38 |
0 |
52 |
75 |
70 |
65 |
18 |
38 |
8 |
95 |
Instance |
370 |
45 |
45 |
25 |
0 |
30 |
50 |
40 |
45 |
10 |
25 |
5 |
50 |
|
Time |
227 |
19 |
29 |
13 |
0 |
22 |
25 |
30 |
20 |
8 |
13 |
3 |
45 |
|
claspd |
Total |
552 |
68 |
68 |
30 |
0 |
65 |
75 |
69 |
31 |
19 |
11 |
20 |
96 |
Instance |
355 |
45 |
40 |
25 |
0 |
45 |
50 |
40 |
25 |
10 |
10 |
15 |
50 |
|
Time |
197 |
23 |
28 |
5 |
0 |
20 |
25 |
29 |
6 |
9 |
1 |
5 |
46 |
|
cmodels |
Total |
510 |
67 |
56 |
21 |
0 |
62 |
75 |
30 |
51 |
29 |
18 |
6 |
95 |
Instance |
335 |
45 |
30 |
20 |
0 |
45 |
50 |
20 |
35 |
20 |
15 |
5 |
50 |
|
Time |
175 |
22 |
26 |
1 |
0 |
17 |
25 |
10 |
16 |
9 |
3 |
1 |
45 |
|
lp2diffz3 |
Total |
394 |
42 |
55 |
0 |
0 |
0 |
70 |
45 |
47 |
27 |
25 |
0 |
83 |
Instance |
270 |
30 |
35 |
0 |
0 |
0 |
50 |
30 |
35 |
20 |
20 |
0 |
50 |
|
Time |
124 |
12 |
20 |
0 |
0 |
0 |
20 |
15 |
12 |
7 |
5 |
0 |
33 |
|
sup |
Total |
346 |
52 |
40 |
37 |
0 |
58 |
72 |
0 |
31 |
16 |
15 |
25 |
0 |
Instance |
240 |
35 |
25 |
25 |
0 |
40 |
50 |
0 |
25 |
10 |
10 |
20 |
0 |
|
Time |
106 |
17 |
15 |
12 |
0 |
18 |
22 |
0 |
6 |
6 |
5 |
5 |
0 |
|
lp2sat2gminisat |
Total |
310 |
36 |
10 |
32 |
0 |
46 |
71 |
22 |
47 |
17 |
29 |
- |
0 |
Instance |
225 |
30 |
5 |
25 |
0 |
35 |
50 |
15 |
35 |
10 |
20 |
- |
0 |
|
Time |
85 |
6 |
5 |
7 |
0 |
11 |
21 |
7 |
12 |
7 |
9 |
- |
0 |
|
lp2sat2minisat |
Total |
302 |
39 |
0 |
32 |
0 |
52 |
71 |
15 |
47 |
17 |
29 |
- |
0 |
Instance |
220 |
30 |
0 |
25 |
0 |
40 |
50 |
10 |
35 |
10 |
20 |
- |
0 |
|
Time |
82 |
9 |
0 |
7 |
0 |
12 |
21 |
5 |
12 |
7 |
9 |
- |
0 |
|
lp2sat2lminisat |
Total |
301 |
35 |
0 |
32 |
0 |
53 |
71 |
17 |
47 |
17 |
29 |
- |
0 |
Instance |
220 |
30 |
0 |
25 |
0 |
40 |
50 |
10 |
35 |
10 |
20 |
- |
0 |
|
Time |
81 |
5 |
0 |
7 |
0 |
13 |
21 |
7 |
12 |
7 |
9 |
- |
0 |
|
smodels |
Total |
269 |
0 |
55 |
36 |
0 |
9 |
53 |
27 |
0 |
0 |
0 |
0 |
89 |
Instance |
165 |
0 |
30 |
25 |
0 |
5 |
35 |
20 |
0 |
0 |
0 |
0 |
50 |
|
Time |
104 |
0 |
25 |
11 |
0 |
4 |
18 |
7 |
0 |
0 |
0 |
0 |
39 |
|
Beyond NP Problems
System |
Total |
|||
claspd |
Total |
103 |
12 |
91 |
Instance |
60 |
10 |
50 |
|
Time |
43 |
2 |
41 |
|
cmodels |
Total |
72 |
0 |
72 |
Instance |
45 |
0 |
45 |
|
Time |
27 |
0 |
27 |
|