welcome: please sign in
location: Diff for "SystemTrackFinalResults"
Differences between revisions 118 and 119
Revision 118 as of 2011-05-20 00:48:22
Size: 22597
Comment:
Revision 119 as of 2011-05-20 01:25:14
Size: 23670
Comment: add notes to detailed RAW results
Deletions are marked like this. Additions are marked like this.
Line 13: Line 13:
Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the time spent. Please have a look at the [[ParticipationRules|Scoring and Participation Rules]] for all details. Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the actual time spent. In case of optimization problems, "quality" of the solutions plays a role as well. Please have a look at the [[ParticipationRules|Scoring and Participation Rules]] for all details.
Line 17: Line 17:
Detailed results can be downloaded [[attachment:systemcompetition.final.results.raw.detailed.txt|HERE]] as a ''raw'' text file. Detailed results, as a ''raw'' text file, can be downloaded [[attachment:systemcompetition.final.results.raw.detailed.txt|HERE]].
<<BR>>
The file contains a line for each instance each system has been asked to solve. Some quick info about what's in the file (please have a look at the [[http://www.mat.unical.it/aspcomp2011/files/LanguageSpecifications.pdf|file and language format specifications]] for more details):
 * System: the system name (quite easy to understand)
 * Benchmark: the benchmark problem
 * Instance: the specific instance of the problem
 * Out: the answer provided by the system for the specific instance
 * Diagnostics: result of the evaluation of the answer (essentially, whether the system answered correctly or not)
 * Time: the total time spent by the system while solving the specific instance
 * Optimum Cost: the "cost" of the witness solution returned by the system, in case of istance of an optimization problem
 * Total Score: the total score rewarded for the specific instance
 * Inst\Opt Score - Time Score: the two components of the score, as introduced above.

System Track Final Results



NOTES

new_2.gif Links to detailed results for all problems will be available soon.

Legenda

Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the actual time spent. In case of optimization problems, "quality" of the solutions plays a role as well. Please have a look at the Scoring and Participation Rules for all details.

Detailed Results

Detailed results, as a raw text file, can be downloaded HERE.
The file contains a line for each instance each system has been asked to solve. Some quick info about what's in the file (please have a look at the file and language format specifications for more details):

  • System: the system name (quite easy to understand)
  • Benchmark: the benchmark problem
  • Instance: the specific instance of the problem
  • Out: the answer provided by the system for the specific instance
  • Diagnostics: result of the evaluation of the answer (essentially, whether the system answered correctly or not)
  • Time: the total time spent by the system while solving the specific instance
  • Optimum Cost: the "cost" of the witness solution returned by the system, in case of istance of an optimization problem
  • Total Score: the total score rewarded for the specific instance
  • Inst\Opt Score - Time Score: the two components of the score, as introduced above.

Overall


System

Total

Instance

Time

claspd

861

560

301

claspfolio

818

535

283

clasp

810

520

290

idp

781

500

281

cmodels

766

510

256

lp2diffz3

572

405

167

sup

541

380

161

lp2sat2gminisat

495

365

130

lp2sat2minisat

481

355

126

lp2sat2lminisat

472

350

122

smodels

449

295

154


Final results by categories


P

NP

Beyond NP

System

Total Score

Inst. Score

Time Score

System

Total Score

Inst. Score

Time Score

System

Total Score

Inst. Score

Time Score

clasp

213

150

63

claspfolio

609

385

224

claspd

103

60

43

claspfolio

209

150

59

clasp

597

370

227

cmodels

72

45

27

claspd

206

145

61

idp

597

370

227

claspfolio

-

-

-

sup

195

140

55

claspd

552

355

197

clasp

-

-

-

lp2sat2gminisat

185

140

45

cmodels

510

335

175

idp

-

-

-

cmodels

184

130

54

lp2diffz3

394

270

124

lp2diffz3

-

-

-

idp

184

130

54

sup

346

240

106

sup

-

-

-

smodels

180

130

50

lp2sat2gminisat

310

225

85

lp2sat2gminisat

-

-

-

lp2sat2minisat

179

135

44

lp2sat2minisat

302

220

82

lp2sat2minisat

-

-

-

lp2diffz3

178

135

43

lp2sat2lminisat

301

220

81

lp2sat2lminisat

-

-

-

lp2sat2lminisat

171

130

41

smodels

269

165

104

smodels

-

-

-



P Problems


System

Total

Reachability

Grammar-Based IE

Hydraulic Leaking

Hydraulic Planning

Stable Marriage

Partner Units Polynomial

clasp

Total

213

36

72

10

75

6

14

Instance

150

25

50

10

50

5

10

Time

63

11

22

0

25

1

4

claspfolio

Total

209

35

72

10

74

5

13

Instance

150

25

50

10

50

5

10

Time

59

10

22

0

24

0

3

claspd

Total

206

36

72

10

75

0

13

Instance

145

25

50

10

50

0

10

Time

61

11

22

0

25

0

3

sup

Total

195

29

71

10

74

0

11

Instance

140

20

50

10

50

0

10

Time

55

9

21

0

24

0

1

lp2sat2gminisat

Total

185

30

66

10

68

0

11

Instance

140

20

50

10

50

0

10

Time

45

10

16

0

18

0

1

cmodels

Total

184

29

71

10

74

0

0

Instance

130

20

50

10

50

0

0

Time

54

9

21

0

24

0

0

idp

Total

184

29

71

10

74

0

0

Instance

130

20

50

10

50

0

0

Time

54

9

21

0

24

0

0

smodels

Total

180

28

70

10

72

0

0

Instance

130

20

50

10

50

0

0

Time

50

8

20

0

22

0

0

lp2sat2minisat

Total

179

30

66

10

68

0

5

Instance

135

20

50

10

50

0

5

Time

44

10

16

0

18

0

0

lp2diffz3

Total

178

35

66

10

67

0

0

Instance

135

25

50

10

50

0

0

Time

43

10

16

0

17

0

0

lp2sat2lminisat

Total

171

28

66

10

67

0

0

Instance

130

20

50

10

50

0

0

Time

41

8

16

0

17

0

0



NP Problems


System

Total

Sokoban Decision

Knight Tour

Disjunctive Scheduling

Packing Problem

Labyrinth

MCS Querying

Numberlink

Hanoi Tower

Graph Colouring

Solitaire

Weight-Assignment Tree

Maze Generation

claspfolio

Total

609

66

65

37

0

63

75

64

47

55

21

21

95

Instance

385

45

35

25

0

40

50

35

35

40

15

15

50

Time

224

21

30

12

0

23

25

29

12

15

6

6

45

clasp

Total

597

78

63

38

0

78

75

65

39

23

21

21

96

Instance

370

50

35

25

0

50

50

35

30

15

15

15

50

Time

227

28

28

13

0

28

25

30

9

8

6

6

46

idp

Total

597

64

74

38

0

52

75

70

65

18

38

8

95

Instance

370

45

45

25

0

30

50

40

45

10

25

5

50

Time

227

19

29

13

0

22

25

30

20

8

13

3

45

claspd

Total

552

68

68

30

0

65

75

69

31

19

11

20

96

Instance

355

45

40

25

0

45

50

40

25

10

10

15

50

Time

197

23

28

5

0

20

25

29

6

9

1

5

46

cmodels

Total

510

67

56

21

0

62

75

30

51

29

18

6

95

Instance

335

45

30

20

0

45

50

20

35

20

15

5

50

Time

175

22

26

1

0

17

25

10

16

9

3

1

45

lp2diffz3

Total

394

42

55

0

0

0

70

45

47

27

25

0

83

Instance

270

30

35

0

0

0

50

30

35

20

20

0

50

Time

124

12

20

0

0

0

20

15

12

7

5

0

33

sup

Total

346

52

40

37

0

58

72

0

31

16

15

25

0

Instance

240

35

25

25

0

40

50

0

25

10

10

20

0

Time

106

17

15

12

0

18

22

0

6

6

5

5

0

lp2sat2gminisat

Total

310

36

10

32

0

46

71

22

47

17

29

-

0

Instance

225

30

5

25

0

35

50

15

35

10

20

-

0

Time

85

6

5

7

0

11

21

7

12

7

9

-

0

lp2sat2minisat

Total

302

39

0

32

0

52

71

15

47

17

29

-

0

Instance

220

30

0

25

0

40

50

10

35

10

20

-

0

Time

82

9

0

7

0

12

21

5

12

7

9

-

0

lp2sat2lminisat

Total

301

35

0

32

0

53

71

17

47

17

29

-

0

Instance

220

30

0

25

0

40

50

10

35

10

20

-

0

Time

81

5

0

7

0

13

21

7

12

7

9

-

0

smodels

Total

269

0

55

36

0

9

53

27

0

0

0

0

89

Instance

165

0

30

25

0

5

35

20

0

0

0

0

50

Time

104

0

25

11

0

4

18

7

0

0

0

0

39



Beyond NP Problems


System

Total

Strategic Companies

Minimal Diagnosis

claspd

Total

103

12

91

Instance

60

10

50

Time

43

2

41

cmodels

Total

72

0

72

Instance

45

0

45

Time

27

0

27


ASP Competition 2011: SystemTrackFinalResults (last edited 2011-05-24 09:11:12 by FrancescoCalimeri)