Model & Solve Track Final Results

Contents

Model & Solve Track Final Results

Model&Solve Track Final Results: Overall

System	Total	Instance	Time
clasp	2431	1432	999
bpsolver	1970	1111	859
aclasp	1953	1140	813
ezcsp	1773	993	780
idp	1442	918	524
fastdownward	367	217	150

Final results by categories

Overview

P				NP				Beyond NP				Optimization
System	Total Score	Inst. Score	Time Score	System	Total Score	Inst. Score	Time Score	System	Total Score	Inst. Score	Time Score	System	Total Score	Inst. Score	Time Score
clasp	497	290	290	clasp	1481	848	633	clasp	110	60	50	clasp	343	234	109
bpsolver	459	253	206	ezcsp	1419	786	633	bpsolver	86	43	43	aclasp	325	220	105
aclasp	404	240	164	bpsolver	1218	654	564	idp	0	0	0	bpsolver	207	161	46
ezcsp	320	173	147	aclasp	1224	680	544	fastdownward	0	0	0	idp	140	121	19
fastdownward	181	100	81	idp	1127	680	447	ezcsp	0	0	0	fastdownward	81	47	34
idp	175	117	58	fastdownward	105	70	35	aclasp	0	0	0	ezcsp	34	34	0

P Problems

System		Total	Reachability	Company Controls	Grammar-Based IE	Hydraulic Leaking	Hydraulic Planning	Stable Marriage	Partner Units Polynomial
clasp	Total	497	87	89	80	78	100	25	38
	Instance	290	50	50	50	50	50	17	23
	Time	207	37	39	30	28	50	8	15
bpsolver	Total	459	90	68	96	100	100	5	-
	Instance	253	50	50	50	50	50	3	-
	Time	206	40	18	46	50	50	2	-
aclasp	Total	404	-	84	80	76	100	25	39
	Instance	240	-	50	50	50	50	17	23
	Time	164	-	34	30	26	50	8	16
ezcsp	Total	320	-	-	-	99	99	87	35
	Instance	173	-	-	-	50	50	50	23
	Time	147	-	-	-	49	49	37	12
fastdownward	Total	181	-	-	-	87	94	-	-
	Instance	100	-	-	-	50	50	-	-
	Time	81	-	-	-	37	44	-	-
idp	Total	175	76	-	-	-	-	72	27
	Instance	117	50	-	-	-	-	50	17
	Time	58	26	-	-	-	-	22	10

NP Problems

System		Total	Generalized Slitherlink	Fastfood Optimality Check	Sokoban Decision	Knight Tour	Disjunctive Scheduling	Packing Problem	Labyrinth	Numberlink	Reverse Folding	Hanoi Tower	Magic Square Sets	Airport Pickup	Graph Colouring	Solitaire	Partner Units	Weight-Assignment Tree	Maze Generation	Incremental Scheduling	Tangram
clasp	Total	1481	97	97	85	97	67	81	71	72	94	81	98	-	51	73	79	56	97	85	100
	Instance	848	50	50	50	50	50	50	47	37	47	50	50	-	37	40	50	43	50	47	50
	Time	633	47	47	35	47	17	31	24	35	47	31	48	-	14	33	29	13	47	38	50
ezcsp	Total	1419	94	97	81	75	26	99	69	70	94	27	99	81	23	50	65	96	96	77	100
	Instance	786	50	50	50	40	13	50	43	37	47	23	50	50	13	30	50	50	50	40	50
	Time	633	44	47	31	35	13	49	26	33	47	4	49	31	10	20	15	46	46	37	50
aclasp	Total	1224	96	97	83	97	-	83	68	72	80	77	98	-	28	73	77	-	95	-	100
	Instance	680	50	50	50	50	-	50	43	37	40	50	50	-	20	40	50	-	50	-	50
	Time	544	46	47	33	47	-	33	25	35	40	27	48	-	8	33	27	-	45	-	50
idp	Total	1127	91	76	69	95	15	17	42	99	-	74	93	-	66	71	33	83	73	30	100
	Instance	680	50	50	43	50	13	13	27	50	-	50	47	-	47	40	30	50	50	20	50
	Time	447	41	26	26	45	2	4	15	49	-	24	46	-	19	31	3	33	23	10	50
bpsolver	Total	1126	*	92	60	97	92	82	72	73	-	94*	100	93	15	28	-	97	49	76	100
	Instance	607	*	50	30	50	50	43	40	37	-	47*	50	50	10	20	-	50	37	40	50
	Time	519	*	42	30	47	42	39	32	36	-	47*	50	43	5	8	-	47	12	36	50
fastdownward	Total	105	-	-	*	5	-	-	-	-	-	27	-	10	-	63	-	-	-	-	-
	Instance	70	-	-	*	5	-	-	-	-	-	23	-	7	-	35	-	-	-	-	-
	Time	35	-	-	*	0	-	-	-	-	-	4	-	3	-	28	-	-	-	-	-

Beyond NP Problems

System		Total	Strategic Companies	Minimal Diagnosis
clasp	Total	110	11	99
	Instance	60	10	50
	Time	50	1	49
bpsolver	Total	86	-	86
	Instance	43	-	43
	Time	43	-	43
idp	Total	0	-	-
	Instance	0	-	-
	Time	0	-	-
fastdownward	Total	0	-	-
	Instance	0	-	-
	Time	0	-	-
ezcsp	Total	0	-	-
	Instance	0	-	-
	Time	0	-	-
aclasp	Total	0	-	-
	Instance	0	-	-
	Time	0	-	-

Optimization Problems

System		Total	Crossing Minimization	Company Controls Optimize	Fastfood Optimization	Sokoban Optimization	Tomography	Maximal Clique
clasp	Total	343	37	87	83	87	13	36
	Instance	234	35	50	50	50	13	36
	Time	109	2	37	33	37	0	0
aclasp	Total	325	16	87	83	85	18	36
	Instance	220	16	50	50	50	18	36
	Time	105	0	37	33	35	0	0
bpsolver	Total	207	13	-	48	70	39	37
	Instance	161	13	-	35	37	39	37
	Time	46	0	-	13	33	0	0
idp	Total	140	13	-	64	21	13	29
	Instance	121	13	-	48	18	13	29
	Time	19	0	-	16	3	0	0
fastdownward	Total	81	-	-	-	81	-	-
	Instance	47	-	-	-	47	-	-
	Time	34	-	-	-	34	-	-
ezcsp	Total	34	-	-	-	-	34	-
	Instance	34	-	-	-	-	34	-
	Time	0	-	-	-	-	0	-

Detailed Results

Details: raw file

Detailed results, as a raw text file, can be downloaded HERE.
The file contains a line for each instance each system has been asked to solve.

Details: tables

Detailed results are also available as separate PDF files:

Quick reference guide

Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the actual time spent. In case of optimization problems, "quality" of the solutions plays a role as well. Please have a look at the Scoring and Participation Rules and the file and language format specifications for more details.

System: the system name (quite easy to understand)
Benchmark: the benchmark problem
Instance: the specific instance of the problem
Out|Outcome: the answer provided by the system for the specific instance
Diagnostics: result of the evaluation of the answer (essentially, whether the system answered correctly or not)
Time: the total time spent by the system while solving the specific instance
Optimum Cost: the "cost" of the witness solution returned by the system, in case of istance of an optimization problem
Total Score: the total score rewarded for the specific instance
Inst\Opt Score - Time Score: the two components of the score, as introduced above.
WF (Cost) : Witness Found (Cost)
OF (Cost) : Optimum Found (Cost)
Inc : Inconsistent
TO : Time Out
MO/F : Mem Out Fail
N/A : Not Available / Not Applicable
* : Score obtained on a late submission of the problem's solution

ASP Competition 2011: Model&SolveTrackFinalResults (last edited 2011-06-08 14:40:28 by CarmenSantoro)