welcome: please sign in
location: Diff for "Model&SolveTrackFinalResults"
Differences between revisions 56 and 60 (spanning 4 versions)
Revision 56 as of 2011-05-20 00:49:36
Size: 23425
Comment: add some comments to scores
Revision 60 as of 2011-05-20 14:59:00
Size: 25008
Comment:
Deletions are marked like this. Additions are marked like this.
Line 7: Line 7:
== NOTE ==
<<newicon>> Links to '''detailed results''' for all problems will be available soon.


== Legenda ==
Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the time spent. Please have a look at the [[ParticipationRules|Scoring and Participation Rules]] for all details.


== Detailed Results ==
Detailed results can be downloaded [[attachment:teamcompetition.final.results.raw.detailed.txt|HERE]] as a ''raw'' text file.


== Overall ==
<<BR>>
== Model&Solve Track Final Results: Overall ==
Line 29: Line 16:



<<BR>>
Line 35: Line 17:
<<BR>> === Overview ===
Line 45: Line 27:



<<BR>>
Line 51: Line 28:
<<BR>>
Line 73: Line 48:

<<BR>>
Line 77: Line 49:
<<BR>>
Line 99: Line 69:

<<BR>>
Line 103: Line 70:
<<BR>>
Line 125: Line 90:

<<BR>>
Line 129: Line 91:
<<BR>>
Line 150: Line 110:

== Detailed Results ==
=== Details: raw file ===
Detailed results, as a ''raw'' text file, can be downloaded [[attachment:teamcompetition.final.results.raw.detailed.txt|HERE]].
<<BR>>
The file contains a line for each instance each system has been asked to solve.

=== Details: tables ===
Detailed results are also available as separate PDF files:
 * [[attachment:Model&Solve-Results-P-Problems-detailed.pdf||System Track - Details - P problems]]
 * [[attachment:Model&Solve-Results-NP-Problems-detailed.pdf||System Track - Details - NP problems]]
 * [[attachment:Model&Solve-Results-BndNP-Problems-detailed.pdf||System Track - Details - Beyond P problems]]
 * [[attachment:Model&Solve-Results-Opt-Problems-detailed.pdf||System Track - Details - Beyond P problems]]

== Quick reference guide ==
Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the actual time spent. In case of optimization problems, "quality" of the solutions plays a role as well. Please have a look at the [[ParticipationRules|Scoring and Participation Rules]] and the [[http://www.mat.unical.it/aspcomp2011/files/LanguageSpecifications.pdf|file and language format specifications]] for more details.

 * System: the system name (quite easy to understand)
 * Benchmark: the benchmark problem
 * Instance: the specific instance of the problem
 * Out|Outcome: the answer provided by the system for the specific instance
 * Diagnostics: result of the evaluation of the answer (essentially, whether the system answered correctly or not)
 * Time: the total time spent by the system while solving the specific instance
 * Optimum Cost: the "cost" of the witness solution returned by the system, in case of istance of an optimization problem
 * Total Score: the total score rewarded for the specific instance
 * Inst\Opt Score - Time Score: the two components of the score, as introduced above.

 * WF (Cost) : Witness Found (Cost)
 * OF (Cost) : Optimum Found (Cost)
 * Inc : Inconsistent
 * TO : Time Out
 * MO/F : Mem Out Fail
 * N/A : Not Available / Not Applicable

Model & Solve Track Final Results



Model&Solve Track Final Results: Overall

System

Total

Instance

Time

clasp

2431

1432

999

aclasp

1953

1140

813

bpsolver

1878

1064

814

ezcsp

1773

993

780

idp

1442

918

524

fastdownward

367

217

150

Final results by categories

Overview

P

NP

Beyond NP

Optimization

System

Total Score

Inst. Score

Time Score

System

Total Score

Inst. Score

Time Score

System

Total Score

Inst. Score

Time Score

System

Total Score

Inst. Score

Time Score

clasp

497

290

290

clasp

1481

848

633

clasp

110

60

50

clasp

343

234

109

bpsolver

459

253

206

ezcsp

1419

786

633

bpsolver

86

43

43

aclasp

325

220

105

aclasp

404

240

164

aclasp

1224

680

544

idp

0

0

0

bpsolver

207

161

46

ezcsp

320

173

147

idp

1127

680

447

fastdownward

0

0

0

idp

140

121

19

fastdownward

181

100

81

bpsolver

1126

607

519

ezcsp

0

0

0

fastdownward

81

47

34

idp

175

117

58

fastdownward

105

70

35

aclasp

0

0

0

ezcsp

34

34

0

P Problems

System

Total

Reachability

Company Controls

Grammar-Based IE

Hydraulic Leaking

Hydraulic Planning

Stable Marriage

Partner Units Polynomial

clasp

Total

497

87

89

80

78

100

25

38

Instance

290

50

50

50

50

50

17

23

Time

207

37

39

30

28

50

8

15

bpsolver

Total

459

90

68

96

100

100

5

-

Instance

253

50

50

50

50

50

3

-

Time

206

40

18

46

50

50

2

-

aclasp

Total

404

-

84

80

76

100

25

39

Instance

240

-

50

50

50

50

17

23

Time

164

-

34

30

26

50

8

16

ezcsp

Total

320

-

-

-

99

99

87

35

Instance

173

-

-

-

50

50

50

23

Time

147

-

-

-

49

49

37

12

fastdownward

Total

181

-

-

-

87

94

-

-

Instance

100

-

-

-

50

50

-

-

Time

81

-

-

-

37

44

-

-

idp

Total

175

76

-

-

-

-

72

27

Instance

117

50

-

-

-

-

50

17

Time

58

26

-

-

-

-

22

10

NP Problems

System

Total

Generalized Slitherlink

Fastfood Optimality Check

Sokoban Decision

Knight Tour

Disjunctive Scheduling

Packing Problem

Labyrinth

Numberlink

Reverse Folding

Hanoi Tower

Magic Square Sets

Airport Pickup

Graph Colouring

Solitaire

Partner Units

Weight-Assignment Tree

Maze Generation

Incremental Scheduling

Tangram

clasp

Total

1481

97

97

85

97

67

81

71

72

94

81

98

-

51

73

79

56

97

85

100

Instance

848

50

50

50

50

50

50

47

37

47

50

50

-

37

40

50

43

50

47

50

Time

633

47

47

35

47

17

31

24

35

47

31

48

-

14

33

29

13

47

38

50

ezcsp

Total

1419

94

97

81

75

26

99

69

70

94

27

99

81

23

50

65

96

96

77

100

Instance

786

50

50

50

40

13

50

43

37

47

23

50

50

13

30

50

50

50

40

50

Time

633

44

47

31

35

13

49

26

33

47

4

49

31

10

20

15

46

46

37

50

aclasp

Total

1224

96

97

83

97

-

83

68

72

80

77

98

-

28

73

77

-

95

-

100

Instance

680

50

50

50

50

-

50

43

37

40

50

50

-

20

40

50

-

50

-

50

Time

544

46

47

33

47

-

33

25

35

40

27

48

-

8

33

27

-

45

-

50

idp

Total

1127

91

76

69

95

15

17

42

99

-

74

93

-

66

71

33

83

73

30

100

Instance

680

50

50

43

50

13

13

27

50

-

50

47

-

47

40

30

50

50

20

50

Time

447

41

26

26

45

2

4

15

49

-

24

46

-

19

31

3

33

23

10

50

bpsolver

Total

1126

*

92

60

97

92

82

72

73

-

-

100

93

15

28

-

97

49

76

100

Instance

607

*

50

30

50

50

43

40

37

-

-

50

50

10

20

-

50

37

40

50

Time

519

*

42

30

47

42

39

32

36

-

-

50

43

5

8

-

47

12

36

50

fastdownward

Total

105

-

-

*

5

-

-

-

-

-

27

-

10

-

63

-

-

-

-

-

Instance

70

-

-

*

5

-

-

-

-

-

23

-

7

-

35

-

-

-

-

-

Time

35

-

-

*

0

-

-

-

-

-

4

-

3

-

28

-

-

-

-

-

Beyond NP Problems

System

Total

Strategic Companies

Minimal Diagnosis

clasp

Total

110

11

99

Instance

60

10

50

Time

50

1

49

bpsolver

Total

86

-

86

Instance

43

-

43

Time

43

-

43

idp

Total

0

-

-

Instance

0

-

-

Time

0

-

-

fastdownward

Total

0

-

-

Instance

0

-

-

Time

0

-

-

ezcsp

Total

0

-

-

Instance

0

-

-

Time

0

-

-

aclasp

Total

0

-

-

Instance

0

-

-

Time

0

-

-

Optimization Problems

System

Total

Crossing Minimization

Company Controls Optimize

Fastfood Optimization

Sokoban Optimization

Tomography

Maximal Clique

clasp

Total

343

37

87

83

87

13

36

Instance

234

35

50

50

50

13

36

Time

109

2

37

33

37

0

0

aclasp

Total

325

16

87

83

85

18

36

Instance

220

16

50

50

50

18

36

Time

105

0

37

33

35

0

0

bpsolver

Total

207

13

-

48

70

39

37

Instance

161

13

-

35

37

39

37

Time

46

0

-

13

33

0

0

idp

Total

140

13

-

64

21

13

29

Instance

121

13

-

48

18

13

29

Time

19

0

-

16

3

0

0

fastdownward

Total

81

-

-

-

81

-

-

Instance

47

-

-

-

47

-

-

Time

34

-

-

-

34

-

-

ezcsp

Total

34

-

-

-

-

34

-

Instance

34

-

-

-

-

34

-

Time

0

-

-

-

-

0

-

Detailed Results

Details: raw file

Detailed results, as a raw text file, can be downloaded HERE.
The file contains a line for each instance each system has been asked to solve.

Details: tables

Detailed results are also available as separate PDF files:

Quick reference guide

Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the actual time spent. In case of optimization problems, "quality" of the solutions plays a role as well. Please have a look at the Scoring and Participation Rules and the file and language format specifications for more details.

  • System: the system name (quite easy to understand)
  • Benchmark: the benchmark problem
  • Instance: the specific instance of the problem
  • Out|Outcome: the answer provided by the system for the specific instance
  • Diagnostics: result of the evaluation of the answer (essentially, whether the system answered correctly or not)
  • Time: the total time spent by the system while solving the specific instance
  • Optimum Cost: the "cost" of the witness solution returned by the system, in case of istance of an optimization problem
  • Total Score: the total score rewarded for the specific instance
  • Inst\Opt Score - Time Score: the two components of the score, as introduced above.
  • WF (Cost) : Witness Found (Cost)
  • OF (Cost) : Optimum Found (Cost)
  • Inc : Inconsistent
  • TO : Time Out
  • MO/F : Mem Out Fail
  • N/A : Not Available / Not Applicable

ASP Competition 2011: Model&SolveTrackFinalResults (last edited 2011-06-08 14:40:28 by CarmenSantoro)