23425
Comment: add some comments to scores
|
25008
|
Deletions are marked like this. | Additions are marked like this. |
Line 7: | Line 7: |
== NOTE == <<newicon>> Links to '''detailed results''' for all problems will be available soon. == Legenda == Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the time spent. Please have a look at the [[ParticipationRules|Scoring and Participation Rules]] for all details. == Detailed Results == Detailed results can be downloaded [[attachment:teamcompetition.final.results.raw.detailed.txt|HERE]] as a ''raw'' text file. == Overall == <<BR>> |
== Model&Solve Track Final Results: Overall == |
Line 29: | Line 16: |
<<BR>> |
|
Line 35: | Line 17: |
<<BR>> | === Overview === |
Line 45: | Line 27: |
<<BR>> |
|
Line 51: | Line 28: |
<<BR>> |
|
Line 73: | Line 48: |
<<BR>> |
|
Line 77: | Line 49: |
<<BR>> |
|
Line 99: | Line 69: |
<<BR>> |
|
Line 103: | Line 70: |
<<BR>> |
|
Line 125: | Line 90: |
<<BR>> |
|
Line 129: | Line 91: |
<<BR>> |
|
Line 150: | Line 110: |
== Detailed Results == === Details: raw file === Detailed results, as a ''raw'' text file, can be downloaded [[attachment:teamcompetition.final.results.raw.detailed.txt|HERE]]. <<BR>> The file contains a line for each instance each system has been asked to solve. === Details: tables === Detailed results are also available as separate PDF files: * [[attachment:Model&Solve-Results-P-Problems-detailed.pdf||System Track - Details - P problems]] * [[attachment:Model&Solve-Results-NP-Problems-detailed.pdf||System Track - Details - NP problems]] * [[attachment:Model&Solve-Results-BndNP-Problems-detailed.pdf||System Track - Details - Beyond P problems]] * [[attachment:Model&Solve-Results-Opt-Problems-detailed.pdf||System Track - Details - Beyond P problems]] == Quick reference guide == Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the actual time spent. In case of optimization problems, "quality" of the solutions plays a role as well. Please have a look at the [[ParticipationRules|Scoring and Participation Rules]] and the [[http://www.mat.unical.it/aspcomp2011/files/LanguageSpecifications.pdf|file and language format specifications]] for more details. * System: the system name (quite easy to understand) * Benchmark: the benchmark problem * Instance: the specific instance of the problem * Out|Outcome: the answer provided by the system for the specific instance * Diagnostics: result of the evaluation of the answer (essentially, whether the system answered correctly or not) * Time: the total time spent by the system while solving the specific instance * Optimum Cost: the "cost" of the witness solution returned by the system, in case of istance of an optimization problem * Total Score: the total score rewarded for the specific instance * Inst\Opt Score - Time Score: the two components of the score, as introduced above. * WF (Cost) : Witness Found (Cost) * OF (Cost) : Optimum Found (Cost) * Inc : Inconsistent * TO : Time Out * MO/F : Mem Out Fail * N/A : Not Available / Not Applicable |
Model & Solve Track Final Results
Contents
Model&Solve Track Final Results: Overall
System |
Total |
Instance |
Time |
2431 |
1432 |
999 |
|
1953 |
1140 |
813 |
|
1878 |
1064 |
814 |
|
1773 |
993 |
780 |
|
1442 |
918 |
524 |
|
367 |
217 |
150 |
Final results by categories
Overview
P |
|
NP |
|
Beyond NP |
|
Optimization |
||||||||||||
System |
Total Score |
Inst. Score |
Time Score |
System |
Total Score |
Inst. Score |
Time Score |
System |
Total Score |
Inst. Score |
Time Score |
System |
Total Score |
Inst. Score |
Time Score |
|||
clasp |
497 |
290 |
290 |
clasp |
1481 |
848 |
633 |
clasp |
110 |
60 |
50 |
clasp |
343 |
234 |
109 |
|||
bpsolver |
459 |
253 |
206 |
ezcsp |
1419 |
786 |
633 |
bpsolver |
86 |
43 |
43 |
aclasp |
325 |
220 |
105 |
|||
aclasp |
404 |
240 |
164 |
aclasp |
1224 |
680 |
544 |
idp |
0 |
0 |
0 |
bpsolver |
207 |
161 |
46 |
|||
ezcsp |
320 |
173 |
147 |
idp |
1127 |
680 |
447 |
fastdownward |
0 |
0 |
0 |
idp |
140 |
121 |
19 |
|||
fastdownward |
181 |
100 |
81 |
bpsolver |
1126 |
607 |
519 |
ezcsp |
0 |
0 |
0 |
fastdownward |
81 |
47 |
34 |
|||
idp |
175 |
117 |
58 |
|
fastdownward |
105 |
70 |
35 |
|
aclasp |
0 |
0 |
0 |
|
ezcsp |
34 |
34 |
0 |
P Problems
System |
Total |
||||||||
clasp |
Total |
497 |
87 |
89 |
80 |
78 |
100 |
25 |
38 |
Instance |
290 |
50 |
50 |
50 |
50 |
50 |
17 |
23 |
|
Time |
207 |
37 |
39 |
30 |
28 |
50 |
8 |
15 |
|
bpsolver |
Total |
459 |
90 |
68 |
96 |
100 |
100 |
5 |
- |
Instance |
253 |
50 |
50 |
50 |
50 |
50 |
3 |
- |
|
Time |
206 |
40 |
18 |
46 |
50 |
50 |
2 |
- |
|
aclasp |
Total |
404 |
- |
84 |
80 |
76 |
100 |
25 |
39 |
Instance |
240 |
- |
50 |
50 |
50 |
50 |
17 |
23 |
|
Time |
164 |
- |
34 |
30 |
26 |
50 |
8 |
16 |
|
ezcsp |
Total |
320 |
- |
- |
- |
99 |
99 |
87 |
35 |
Instance |
173 |
- |
- |
- |
50 |
50 |
50 |
23 |
|
Time |
147 |
- |
- |
- |
49 |
49 |
37 |
12 |
|
fastdownward |
Total |
181 |
- |
- |
- |
87 |
94 |
- |
- |
Instance |
100 |
- |
- |
- |
50 |
50 |
- |
- |
|
Time |
81 |
- |
- |
- |
37 |
44 |
- |
- |
|
idp |
Total |
175 |
76 |
- |
- |
- |
- |
72 |
27 |
Instance |
117 |
50 |
- |
- |
- |
- |
50 |
17 |
|
Time |
58 |
26 |
- |
- |
- |
- |
22 |
10 |
NP Problems
System |
Total |
||||||||||||||||||||
clasp |
Total |
1481 |
97 |
97 |
85 |
97 |
67 |
81 |
71 |
72 |
94 |
81 |
98 |
- |
51 |
73 |
79 |
56 |
97 |
85 |
100 |
Instance |
848 |
50 |
50 |
50 |
50 |
50 |
50 |
47 |
37 |
47 |
50 |
50 |
- |
37 |
40 |
50 |
43 |
50 |
47 |
50 |
|
Time |
633 |
47 |
47 |
35 |
47 |
17 |
31 |
24 |
35 |
47 |
31 |
48 |
- |
14 |
33 |
29 |
13 |
47 |
38 |
50 |
|
ezcsp |
Total |
1419 |
94 |
97 |
81 |
75 |
26 |
99 |
69 |
70 |
94 |
27 |
99 |
81 |
23 |
50 |
65 |
96 |
96 |
77 |
100 |
Instance |
786 |
50 |
50 |
50 |
40 |
13 |
50 |
43 |
37 |
47 |
23 |
50 |
50 |
13 |
30 |
50 |
50 |
50 |
40 |
50 |
|
Time |
633 |
44 |
47 |
31 |
35 |
13 |
49 |
26 |
33 |
47 |
4 |
49 |
31 |
10 |
20 |
15 |
46 |
46 |
37 |
50 |
|
aclasp |
Total |
1224 |
96 |
97 |
83 |
97 |
- |
83 |
68 |
72 |
80 |
77 |
98 |
- |
28 |
73 |
77 |
- |
95 |
- |
100 |
Instance |
680 |
50 |
50 |
50 |
50 |
- |
50 |
43 |
37 |
40 |
50 |
50 |
- |
20 |
40 |
50 |
- |
50 |
- |
50 |
|
Time |
544 |
46 |
47 |
33 |
47 |
- |
33 |
25 |
35 |
40 |
27 |
48 |
- |
8 |
33 |
27 |
- |
45 |
- |
50 |
|
idp |
Total |
1127 |
91 |
76 |
69 |
95 |
15 |
17 |
42 |
99 |
- |
74 |
93 |
- |
66 |
71 |
33 |
83 |
73 |
30 |
100 |
Instance |
680 |
50 |
50 |
43 |
50 |
13 |
13 |
27 |
50 |
- |
50 |
47 |
- |
47 |
40 |
30 |
50 |
50 |
20 |
50 |
|
Time |
447 |
41 |
26 |
26 |
45 |
2 |
4 |
15 |
49 |
- |
24 |
46 |
- |
19 |
31 |
3 |
33 |
23 |
10 |
50 |
|
bpsolver |
Total |
1126 |
* |
92 |
60 |
97 |
92 |
82 |
72 |
73 |
- |
- |
100 |
93 |
15 |
28 |
- |
97 |
49 |
76 |
100 |
Instance |
607 |
* |
50 |
30 |
50 |
50 |
43 |
40 |
37 |
- |
- |
50 |
50 |
10 |
20 |
- |
50 |
37 |
40 |
50 |
|
Time |
519 |
* |
42 |
30 |
47 |
42 |
39 |
32 |
36 |
- |
- |
50 |
43 |
5 |
8 |
- |
47 |
12 |
36 |
50 |
|
fastdownward |
Total |
105 |
- |
- |
* |
5 |
- |
- |
- |
- |
- |
27 |
- |
10 |
- |
63 |
- |
- |
- |
- |
- |
Instance |
70 |
- |
- |
* |
5 |
- |
- |
- |
- |
- |
23 |
- |
7 |
- |
35 |
- |
- |
- |
- |
- |
|
Time |
35 |
- |
- |
* |
0 |
- |
- |
- |
- |
- |
4 |
- |
3 |
- |
28 |
- |
- |
- |
- |
- |
Beyond NP Problems
System |
Total |
|||
clasp |
Total |
110 |
11 |
99 |
Instance |
60 |
10 |
50 |
|
Time |
50 |
1 |
49 |
|
bpsolver |
Total |
86 |
- |
86 |
Instance |
43 |
- |
43 |
|
Time |
43 |
- |
43 |
|
idp |
Total |
0 |
- |
- |
Instance |
0 |
- |
- |
|
Time |
0 |
- |
- |
|
fastdownward |
Total |
0 |
- |
- |
Instance |
0 |
- |
- |
|
Time |
0 |
- |
- |
|
ezcsp |
Total |
0 |
- |
- |
Instance |
0 |
- |
- |
|
Time |
0 |
- |
- |
|
aclasp |
Total |
0 |
- |
- |
Instance |
0 |
- |
- |
|
Time |
0 |
- |
- |
Optimization Problems
System |
Total |
|||||||
clasp |
Total |
343 |
37 |
87 |
83 |
87 |
13 |
36 |
Instance |
234 |
35 |
50 |
50 |
50 |
13 |
36 |
|
Time |
109 |
2 |
37 |
33 |
37 |
0 |
0 |
|
aclasp |
Total |
325 |
16 |
87 |
83 |
85 |
18 |
36 |
Instance |
220 |
16 |
50 |
50 |
50 |
18 |
36 |
|
Time |
105 |
0 |
37 |
33 |
35 |
0 |
0 |
|
bpsolver |
Total |
207 |
13 |
- |
48 |
70 |
39 |
37 |
Instance |
161 |
13 |
- |
35 |
37 |
39 |
37 |
|
Time |
46 |
0 |
- |
13 |
33 |
0 |
0 |
|
idp |
Total |
140 |
13 |
- |
64 |
21 |
13 |
29 |
Instance |
121 |
13 |
- |
48 |
18 |
13 |
29 |
|
Time |
19 |
0 |
- |
16 |
3 |
0 |
0 |
|
fastdownward |
Total |
81 |
- |
- |
- |
81 |
- |
- |
Instance |
47 |
- |
- |
- |
47 |
- |
- |
|
Time |
34 |
- |
- |
- |
34 |
- |
- |
|
ezcsp |
Total |
34 |
- |
- |
- |
- |
34 |
- |
Instance |
34 |
- |
- |
- |
- |
34 |
- |
|
Time |
0 |
- |
- |
- |
- |
0 |
- |
Detailed Results
Details: raw file
Detailed results, as a raw text file, can be downloaded HERE.
The file contains a line for each instance each system has been asked to solve.
Details: tables
Detailed results are also available as separate PDF files:
Quick reference guide
Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the actual time spent. In case of optimization problems, "quality" of the solutions plays a role as well. Please have a look at the Scoring and Participation Rules and the file and language format specifications for more details.
- System: the system name (quite easy to understand)
- Benchmark: the benchmark problem
- Instance: the specific instance of the problem
- Out|Outcome: the answer provided by the system for the specific instance
- Diagnostics: result of the evaluation of the answer (essentially, whether the system answered correctly or not)
- Time: the total time spent by the system while solving the specific instance
- Optimum Cost: the "cost" of the witness solution returned by the system, in case of istance of an optimization problem
- Total Score: the total score rewarded for the specific instance
- Inst\Opt Score - Time Score: the two components of the score, as introduced above.
- WF (Cost) : Witness Found (Cost)
- OF (Cost) : Optimum Found (Cost)
- Inc : Inconsistent
- TO : Time Out
- MO/F : Mem Out Fail
- N/A : Not Available / Not Applicable