Differences between revisions 108 and 128 (spanning 20 versions)

System Track Final Results

Contents

System Track Final Results

System Track Final Results: Overall

System	Total	Instance	Time
claspd	861	560	301
claspfolio	818	535	283
clasp	810	520	290
idp	781	500	281
cmodels	766	510	256
lp2diffz3	572	405	167
sup	541	380	161
lp2sat2gminisat	495	365	130
lp2sat2minisat	481	355	126
lp2sat2lminisat	472	350	122
smodels	449	295	154

Final results by categories

Overview

P				NP				Beyond NP
System	Total Score	Inst. Score	Time Score	System	Total Score	Inst. Score	Time Score	System	Total Score	Inst. Score	Time Score
clasp	213	150	63	claspfolio	609	385	224	claspd	103	60	43
claspfolio	209	150	59	clasp	597	370	227	cmodels	72	45	27
claspd	206	145	61	idp	597	370	227	claspfolio	-	-	-
sup	195	140	55	claspd	552	355	197	clasp	-	-	-
lp2sat2gminisat	185	140	45	cmodels	510	335	175	idp	-	-	-
cmodels	184	130	54	lp2diffz3	394	270	124	lp2diffz3	-	-	-
idp	184	130	54	sup	346	240	106	sup	-	-	-
smodels	180	130	50	lp2sat2gminisat	310	225	85	lp2sat2gminisat	-	-	-
lp2sat2minisat	179	135	44	lp2sat2minisat	302	220	82	lp2sat2minisat	-	-	-
lp2diffz3	178	135	43	lp2sat2lminisat	301	220	81	lp2sat2lminisat	-	-	-
lp2sat2lminisat	171	130	41	smodels	269	165	104	smodels	-	-	-

P Problems

System		Total	Reachability	Grammar-Based IE	Hydraulic Leaking	Hydraulic Planning	Stable Marriage	Partner Units Polynomial
clasp	Total	213	36	72	10	75	6	14
	Instance	150	25	50	10	50	5	10
	Time	63	11	22	0	25	1	4
claspfolio	Total	209	35	72	10	74	5	13
	Instance	150	25	50	10	50	5	10
	Time	59	10	22	0	24	0	3
claspd	Total	206	36	72	10	75	0	13
	Instance	145	25	50	10	50	0	10
	Time	61	11	22	0	25	0	3
sup	Total	195	29	71	10	74	0	11
	Instance	140	20	50	10	50	0	10
	Time	55	9	21	0	24	0	1
lp2sat2gminisat	Total	185	30	66	10	68	0	11
	Instance	140	20	50	10	50	0	10
	Time	45	10	16	0	18	0	1
cmodels	Total	184	29	71	10	74	0	0
	Instance	130	20	50	10	50	0	0
	Time	54	9	21	0	24	0	0
idp	Total	184	29	71	10	74	0	0
	Instance	130	20	50	10	50	0	0
	Time	54	9	21	0	24	0	0
smodels	Total	180	28	70	10	72	0	0
	Instance	130	20	50	10	50	0	0
	Time	50	8	20	0	22	0	0
lp2sat2minisat	Total	179	30	66	10	68	0	5
	Instance	135	20	50	10	50	0	5
	Time	44	10	16	0	18	0	0
lp2diffz3	Total	178	35	66	10	67	0	0
	Instance	135	25	50	10	50	0	0
	Time	43	10	16	0	17	0	0
lp2sat2lminisat	Total	171	28	66	10	67	0	0
	Instance	130	20	50	10	50	0	0
	Time	41	8	16	0	17	0	0

NP Problems

System		Total	Sokoban Decision	Knight Tour	Disjunctive Scheduling	Packing Problem	Labyrinth	MCS Querying	Numberlink	Hanoi Tower	Graph Colouring	Solitaire	Weight-Assignment Tree	Maze Generation
claspfolio	Total	609	66	65	37	0	63	75	64	47	55	21	21	95
	Instance	385	45	35	25	0	40	50	35	35	40	15	15	50
	Time	224	21	30	12	0	23	25	29	12	15	6	6	45
clasp	Total	597	78	63	38	0	78	75	65	39	23	21	21	96
	Instance	370	50	35	25	0	50	50	35	30	15	15	15	50
	Time	227	28	28	13	0	28	25	30	9	8	6	6	46
idp	Total	597	64	74	38	0	52	75	70	65	18	38	8	95
	Instance	370	45	45	25	0	30	50	40	45	10	25	5	50
	Time	227	19	29	13	0	22	25	30	20	8	13	3	45
claspd	Total	552	68	68	30	0	65	75	69	31	19	11	20	96
	Instance	355	45	40	25	0	45	50	40	25	10	10	15	50
	Time	197	23	28	5	0	20	25	29	6	9	1	5	46
cmodels	Total	510	67	56	21	0	62	75	30	51	29	18	6	95
	Instance	335	45	30	20	0	45	50	20	35	20	15	5	50
	Time	175	22	26	1	0	17	25	10	16	9	3	1	45
lp2diffz3	Total	394	42	55	0	0	0	70	45	47	27	25	0	83
	Instance	270	30	35	0	0	0	50	30	35	20	20	0	50
	Time	124	12	20	0	0	0	20	15	12	7	5	0	33
sup	Total	346	52	40	37	0	58	72	0	31	16	15	25	0
	Instance	240	35	25	25	0	40	50	0	25	10	10	20	0
	Time	106	17	15	12	0	18	22	0	6	6	5	5	0
lp2sat2gminisat	Total	310	36	10	32	0	46	71	22	47	17	29	-	0
	Instance	225	30	5	25	0	35	50	15	35	10	20	-	0
	Time	85	6	5	7	0	11	21	7	12	7	9	-	0
lp2sat2minisat	Total	302	39	0	32	0	52	71	15	47	17	29	-	0
	Instance	220	30	0	25	0	40	50	10	35	10	20	-	0
	Time	82	9	0	7	0	12	21	5	12	7	9	-	0
lp2sat2lminisat	Total	301	35	0	32	0	53	71	17	47	17	29	-	0
	Instance	220	30	0	25	0	40	50	10	35	10	20	-	0
	Time	81	5	0	7	0	13	21	7	12	7	9	-	0
smodels	Total	269	0	55	36	0	9	53	27	0	0	0	0	89
	Instance	165	0	30	25	0	5	35	20	0	0	0	0	50
	Time	104	0	25	11	0	4	18	7	0	0	0	0	39

Beyond NP Problems

System		Total	Strategic Companies	Minimal Diagnosis
claspd	Total	103	12	91
	Instance	60	10	50
	Time	43	2	41
cmodels	Total	72	0	72
	Instance	45	0	45
	Time	27	0	27

Detailed Results

Details: raw file

Detailed results, as a raw text file, can be downloaded HERE.
The file contains a line for each instance each system has been asked to solve.

Details: tables

Detailed results are also available as separate PDF files:

Quick reference guide

Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the actual time spent. In case of optimization problems, "quality" of the solutions plays a role as well. Please have a look at the Scoring and Participation Rules and the file and language format specifications for more details.

System: the system name (quite easy to understand)
Benchmark: the benchmark problem
Instance: the specific instance of the problem
Out|Outcome: the answer provided by the system for the specific instance
Diagnostics: result of the evaluation of the answer (essentially, whether the system answered correctly or not)
Time: the total time spent by the system while solving the specific instance
Optimum Cost: the "cost" of the witness solution returned by the system, in case of istance of an optimization problem
Total Score: the total score rewarded for the specific instance
Inst\Opt Score - Time Score: the two components of the score, as introduced above.
WF (Cost) : Witness Found (Cost)
OF (Cost) : Optimum Found (Cost)
Inc : Inconsistent
TO : Time Out
MO/F : Mem Out Fail
N/A : Not Available / Not Applicable

ASP Competition 2011: SystemTrackFinalResults (last edited 2011-05-24 09:11:12 by FrancescoCalimeri)

-  ⇤ ← Revision 108 as of 2011-05-19 07:36:09 → 
  Size: 24124
  Editor: FrancescoCalimeri
  Comment: add detailed results as a raw text file
+   ← Revision 128 as of 2011-05-24 09:11:12 → ⇥
  Size: 23403
  Editor: FrancescoCalimeri
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 8:
-== Detailed Results ==
Detailed results can be downloaded [[attachment:systemcompetition.final.results.raw.detailed|HERE]] as a ''raw'' text file.

== Overall ==
<<BR>>
+  == System Track Final Results: Overall ==
-Line 27:
+Line 23:
-<<BR>>
-Line 33:
+Line 24:
-<<BR>>
+=== Overview ===
-Line 48:
+Line 39:
-<<BR>>
<<BR>>
-Line 55:
+Line 40:
-<<BR>>
{{{#!wiki comment
||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">[[SystemReachabilityResults|Reachability]] ||<style="font-weight: bold; text-align: center;">[[SystemGrammarBasedIEResults|Grammar-Based IE]] ||<style="font-weight: bold; text-align: center;">[[SystemHydraulicLeakingResults|Hydraulic Leaking]] ||<style="font-weight: bold; text-align: center;">[[SystemHydraulicPlanningResults|Hydraulic Planning]] ||<style="font-weight: bold; text-align: center;">[[SystemStableMarriageResults|Stable Marriage]] ||<style="font-weight: bold; text-align: center;">[[SystemPartnerUnitsPolynomialResults|Partner Units Polynomial]] ||
}}}
-Line 94:
+Line 75:
-<<BR>>
<<BR>>
-Line 98:
+Line 76:
-<<BR>>
{{{#!wiki comment
||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">[[SystemSokobanDecisionResults|Sokoban Decision]] ||<style="font-weight: bold; text-align: center;">[[SystemKnightTourResults|Knight Tour]] ||<style="font-weight: bold; text-align: center;">[[SystemDisjunctiveSchedulingResults|Disjunctive Scheduling]] ||<style="font-weight: bold; text-align: center;">[[SystemPackingProblemResults|Packing Problem]] ||<style="font-weight: bold; text-align: center;">[[SystemLabyrinthResults|Labyrinth]] ||<style="font-weight: bold; text-align: center;">[[SystemMCSQueryingResults|MCS Querying]] ||<style="font-weight: bold; text-align: center;">[[SystemNumberlinkResults|Numberlink]] ||<style="font-weight: bold; text-align: center;">[[SystemHanoiTowerResults|Hanoi Tower]] ||<style="font-weight: bold; text-align: center;">[[SystemGraphColouringResults|Graph Colouring]] ||<style="font-weight: bold; text-align: center;">[[SystemSolitaireResults|Solitaire]] ||<style="font-weight: bold; text-align: center;">[[SystemWeightAssignmentTreeResults|Weight-Assignment Tree]] ||<style="font-weight: bold; text-align: center;">[[SystemMazeGenerationResults|Maze Generation]] ||
}}}
-Line 137:
+Line 111:
-<<BR>>
<<BR>>
-Line 141:
+Line 112:
-<<BR>>
{{{#!wiki comment
||||<tablewidth="1%" tablestyle="text-align: center;"style="text-align: center; font-weight: bold;">System ||<style="font-weight: bold; text-align: center;">Total ||<style="font-weight: bold; text-align: center;">[[SystemStrategicCompaniesResults|Strategic Companies]] ||<style="font-weight: bold; text-align: center;">[[SystemMinimalDiagnosisResults|Minimal Diagnosis]] ||
}}}
-Line 153:
+Line 120:
+== Detailed Results ==
=== Details: raw file ===
Detailed results, as a ''raw'' text file, can be downloaded [[attachment:systemcompetition.final.results.raw.detailed.txt|HERE]].
<<BR>>
The file contains a line for each instance each system has been asked to solve.
-Line 154:
+Line 126:
+=== Details: tables ===
Detailed results are also available as separate PDF files: 
 * [[attachment:System-Results-P-Problems-detailed.pdf||System Track - Details - P problems]]
 * [[attachment:System-Results-NP-Problems-detailed.pdf||System Track - Details - NP problems]]
 * [[attachment:System-Results-BndNP-Problems-detailed.pdf||System Track - Details - Beyond P problems]]
-Line 155:
+Line 132:
+== Quick reference guide ==
Scores result from two components: the first depending on the number of instances solved within the given time-bound, the second on the actual time spent. In case of optimization problems, "quality" of the solutions plays a role as well. Please have a look at the [[ParticipationRules|Scoring and Participation Rules]] and the [[http://www.mat.unical.it/aspcomp2011/files/LanguageSpecifications.pdf|file and language format specifications]] for more details.
-Line 156:
+Line 135:
-<<BR>>
+ * System: the system name (quite easy to understand) 
 * Benchmark: the benchmark problem
 * Instance: the specific instance of the problem
 * Out|Outcome: the answer provided by the system for the specific instance
 * Diagnostics: result of the evaluation of the answer (essentially, whether the system answered correctly or not)
 * Time: the total time spent by the system while solving the specific instance
 * Optimum Cost: the "cost" of the witness solution returned by the system, in case of istance of an optimization problem
 * Total Score: the total score rewarded for the specific instance
 * Inst\Opt Score -  Time Score: the two components of the score, as introduced above.

 * WF (Cost) : Witness Found (Cost)
 * OF (Cost) : Optimum Found (Cost)
 * Inc       : Inconsistent
 * TO        : Time Out
 * MO/F      : Mem Out Fail
 * N/A       : Not Available / Not Applicable

ASP Competition 2011

Quick Links

Search Wiki

Page Tools