Results

Method details

Download PDF file of article
ObserverC

Abstract:
This is one of the three manual annotation from which the reference standard is created. In the paper we describe how the segmentation of each of the three observers are turned into a signed distance map, and how the observer results are combined into the reference standard. Observers are ranked with respect to the reference standard in the same way the other methods are ranked. Observer A is the closest to the reference standard, followed by Observer B and Observer C This document is also available online: http://cls2009.bigr.nl/2009/download/evaluationframework2009.pdf

Summary lumen

Measure% / mmrank
 min.max.avg.min.max.avg.
dice 87.4% 96.9% 94.4% 2 4 3.20
msd 0.06mm 0.24mm 0.12mm 2 4 3.12
Hausdorff 0.27mm 2.37mm 0.97mm 2 4 3.39
Total (lumen)       2 4 3.24

Summary stenosis

Measure%rank
 min.max.avg.min.max.avg.
diam 0.0% 23.0% 5.7% 1 4 2.61
area 0.0% 21.0% 5.0% 1 4 2.76
diam* 0.0% 100.0% 11.2% -- -- --
area* 0.0% 40.3% 10.3% -- -- --
Total (stenosis)       1 4 2.68



Averages lumen

TeamTotaldicemsdHausdorffTotal
namesuccess%rankmmrankmmrankrank
ObserverA 41 95.1 2.6 0.10 2.7 0.65 2.7 2.7
ObserverB 41 94.6 3.2 0.11 3.2 0.83 2.9 3.1
ObserverC 41 94.4 3.2 0.12 3.1 0.97 3.4 3.2
Reference 41 100.0 1.0 0.01 1.0 0.06 1.0 1.0
ObserverC 41 94.4 3.2 0.12 3.1 0.97 3.4 3.2

Averages stenosis

TeamTotaldiamareaBaselineTotal
namesuccess%rank%rankdiam*area*lumensrank
ObserverA 41 3.4 2.2 2.9 2.0 6.7 8.2 41 2.1
ObserverB 41 5.4 2.4 4.3 2.3 9.0 8.7 41 2.3
ObserverC 41 5.7 2.6 5.0 2.8 11.2 10.3 41 2.7
Reference 41 0.0 1.0 0.0 1.0 6.6 8.0 41 1.0
ObserverC 41 5.7 2.6 5.0 2.8 11.2 10.3 41 2.7





* The measures indicated with an asterix (*) or in italics are not determined by the participants, but are the result of applying a standardized algorithm by the framework. These measures are not used in the calculation of the ranks.

Your results compared to the observers

DatasetTeamdicemsdHausdorff
nr.name%rankmmrankmmrank
009 ObserverA 95.1 4 0.09 4 0.58 2
009 ObserverB 95.2 3 0.09 3 0.61 3
009 ObserverC 95.5 2 0.08 2 1.11 4
009 Reference 100.0 1 0.01 1 0.05 1
010 ObserverA 96.2 2 0.09 2 0.42 2
010 ObserverB 95.7 3 0.10 3 0.53 3
010 ObserverC 95.4 4 0.11 4 1.68 4
010 Reference 100.0 1 0.01 1 0.04 1
011 ObserverA 96.3 2 0.06 2 0.42 3
011 ObserverB 95.4 4 0.08 3 0.36 2
011 ObserverC 95.4 3 0.09 4 0.43 4
011 Reference 100.0 1 0.01 1 0.07 1
012 ObserverA 95.4 3 0.15 3 0.76 3
012 ObserverB 95.6 2 0.14 2 0.77 4
012 ObserverC 93.6 4 0.21 4 0.69 2
012 Reference 100.0 1 0.01 1 0.07 1
013 ObserverA 95.6 2 0.08 2 0.64 3
013 ObserverB 95.2 3 0.09 3 0.55 2
013 ObserverC 95.1 4 0.09 4 0.78 4
013 Reference 100.0 1 0.01 1 0.05 1
014 ObserverA 97.1 2 0.08 2 0.62 4
014 ObserverB 96.5 4 0.10 4 0.46 3
014 ObserverC 96.9 3 0.09 3 0.38 2
014 Reference 100.0 1 0.01 1 0.06 1
015 ObserverA 95.3 4 0.09 4 0.50 2
015 ObserverB 95.4 3 0.08 3 0.61 3
015 ObserverC 95.5 2 0.08 2 0.72 4
015 Reference 100.0 1 0.01 1 0.06 1
016 ObserverA 95.3 2 0.09 2 0.94 4
016 ObserverB 94.8 4 0.10 4 0.75 3
016 ObserverC 95.1 3 0.10 3 0.70 2
016 Reference 100.0 1 0.01 1 0.06 1
017 ObserverA 95.2 2 0.09 2 0.77 2
017 ObserverB 94.6 4 0.10 4 0.97 3
017 ObserverC 94.7 3 0.10 3 1.30 4
017 Reference 100.0 1 0.01 1 0.06 1
018 ObserverA 96.4 2 0.09 2 0.78 2
018 ObserverB 95.9 3 0.10 3 0.82 3
018 ObserverC 95.6 4 0.11 4 0.99 4
018 Reference 100.0 1 0.01 1 0.05 1
019 ObserverA 96.2 4 0.08 4 0.42 3
019 ObserverB 96.3 3 0.08 3 0.46 4
019 ObserverC 96.9 2 0.06 2 0.38 2
019 Reference 100.0 1 0.00 1 0.08 1
020 ObserverA 96.0 4 0.09 4 0.51 4
020 ObserverB 96.3 3 0.08 3 0.41 2
020 ObserverC 96.9 2 0.06 2 0.44 3
020 Reference 100.0 1 0.01 1 0.06 1
021 ObserverA 96.9 2 0.08 2 0.78 4
021 ObserverB 96.2 4 0.10 4 0.52 2
021 ObserverC 96.6 3 0.08 3 0.73 3
021 Reference 100.0 1 0.01 1 0.04 1
022 ObserverA 95.9 3 0.08 3 1.18 4
022 ObserverB 95.1 4 0.10 4 0.89 3
022 ObserverC 96.4 2 0.07 2 0.63 2
022 Reference 100.0 1 0.01 1 0.05 1
023 ObserverA 96.4 4 0.08 4 0.42 4
023 ObserverB 96.6 2 0.07 2 0.32 2
023 ObserverC 96.6 3 0.08 3 0.41 3
023 Reference 100.0 1 0.01 1 0.06 1
024 ObserverA 95.4 2 0.09 2 0.76 3
024 ObserverB 94.2 4 0.12 4 0.63 2
024 ObserverC 94.5 3 0.11 3 1.36 4
024 Reference 100.0 1 0.01 1 0.08 1
025 ObserverA 96.3 2 0.06 2 0.29 2
025 ObserverB 95.2 4 0.08 4 0.35 3
025 ObserverC 95.5 3 0.08 3 1.43 4
025 Reference 100.0 1 0.01 1 0.07 1
026 ObserverA 96.0 3 0.08 3 0.48 3
026 ObserverB 96.3 2 0.07 2 0.46 2
026 ObserverC 95.7 4 0.09 4 1.17 4
026 Reference 100.0 1 0.01 1 0.05 1
027 ObserverA 96.5 2 0.05 2 0.33 3
027 ObserverB 96.1 4 0.06 4 0.40 4
027 ObserverC 96.4 3 0.06 3 0.27 2
027 Reference 100.0 1 0.01 1 0.05 1
028 ObserverA 96.2 2 0.08 2 0.67 4
028 ObserverB 95.8 4 0.09 4 0.49 2
028 ObserverC 96.1 3 0.08 3 0.61 3
028 Reference 100.0 1 0.01 1 0.08 1
029 ObserverA 96.1 2 0.08 2 0.91 2
029 ObserverB 95.2 4 0.10 4 0.92 3
029 ObserverC 95.6 3 0.09 3 1.11 4
029 Reference 100.0 1 0.01 1 0.05 1
030 ObserverA 95.2 4 0.08 4 0.42 3
030 ObserverB 95.5 2 0.08 2 0.35 2
030 ObserverC 95.5 3 0.08 3 0.55 4
030 Reference 100.0 1 0.01 1 0.04 1
031 ObserverA 96.2 4 0.08 4 0.59 3
031 ObserverB 96.4 3 0.08 3 0.49 2
031 ObserverC 96.5 2 0.07 2 1.15 4
031 Reference 100.0 1 0.00 1 0.04 1
032 ObserverA 95.0 4 0.09 4 0.74 3
032 ObserverB 95.4 2 0.08 2 0.49 2
032 ObserverC 95.1 3 0.09 3 1.79 4
032 Reference 100.0 1 0.01 1 0.05 1
033 ObserverA 96.0 4 0.06 4 0.28 2
033 ObserverB 96.3 2 0.05 2 0.29 3
033 ObserverC 96.1 3 0.06 3 0.37 4
033 Reference 100.0 1 0.01 1 0.05 1
034 ObserverA 96.3 2 0.06 3 0.26 2
034 ObserverB 96.1 4 0.07 4 0.33 3
034 ObserverC 96.3 3 0.06 2 0.45 4
034 Reference 100.0 1 0.01 1 0.05 1
035 ObserverA 95.6 2 0.08 2 0.44 4
035 ObserverB 95.5 3 0.08 3 0.35 2
035 ObserverC 95.3 4 0.08 4 0.44 3
035 Reference 100.0 1 0.01 1 0.05 1
103 ObserverA 93.5 3 0.14 3 0.64 3
103 ObserverB 93.6 2 0.13 2 0.62 2
103 ObserverC 90.6 4 0.21 4 1.70 4
103 Reference 100.0 1 0.01 1 0.08 1
104 ObserverA 94.1 2 0.13 2 0.85 2
104 ObserverB 92.4 4 0.17 4 0.90 3
104 ObserverC 92.5 3 0.15 3 0.93 4
104 Reference 100.0 1 0.01 1 0.08 1
105 ObserverA 94.3 2 0.12 2 0.77 3
105 ObserverB 92.1 4 0.18 4 0.76 2
105 ObserverC 92.4 3 0.17 3 0.81 4
105 Reference 100.0 1 0.01 1 0.06 1
106 ObserverA 93.3 2 0.13 2 0.76 2
106 ObserverB 91.7 3 0.17 3 0.89 3
106 ObserverC 89.9 4 0.21 4 2.37 4
106 Reference 100.0 1 0.01 1 0.11 1
107 ObserverA 93.0 2 0.16 2 0.86 2
107 ObserverB 90.7 3 0.21 3 1.79 4
107 ObserverC 89.5 4 0.24 4 1.05 3
107 Reference 100.0 1 0.01 1 0.06 1
108 ObserverA 91.6 2 0.13 2 0.76 2
108 ObserverB 89.5 3 0.16 3 0.88 3
108 ObserverC 87.4 4 0.20 4 2.20 4
108 Reference 100.0 1 0.02 1 0.10 1
109 ObserverA 93.6 2 0.16 2 0.72 2
109 ObserverB 91.8 3 0.22 3 1.21 3
109 ObserverC 91.5 4 0.22 4 1.27 4
109 Reference 100.0 1 0.01 1 0.05 1
203 ObserverA 95.4 2 0.08 3 0.40 2
203 ObserverB 95.3 3 0.08 2 0.42 3
203 ObserverC 95.0 4 0.09 4 1.73 4
203 Reference 100.0 1 0.01 1 0.05 1
204 ObserverA 95.3 2 0.12 2 0.62 2
204 ObserverB 94.3 3 0.15 4 0.79 4
204 ObserverC 94.3 4 0.15 3 0.69 3
204 Reference 100.0 1 0.01 1 0.07 1
205 ObserverA 95.7 2 0.11 2 0.68 2
205 ObserverB 95.1 4 0.13 4 1.23 4
205 ObserverC 95.2 3 0.13 3 0.76 3
205 Reference 100.0 1 0.01 1 0.05 1
206 ObserverA 95.5 2 0.11 3 0.71 3
206 ObserverB 95.0 4 0.13 4 2.17 4
206 ObserverC 95.4 3 0.11 2 0.53 2
206 Reference 100.0 1 0.01 1 0.08 1
207 ObserverA 92.0 2 0.17 2 1.16 2
207 ObserverB 91.1 3 0.20 4 1.83 4
207 ObserverC 91.0 4 0.19 3 1.35 3
207 Reference 100.0 1 0.01 1 0.05 1
208 ObserverA 90.3 3 0.20 3 0.77 2
208 ObserverB 90.1 4 0.20 4 2.57 4
208 ObserverC 91.0 2 0.17 2 1.25 3
208 Reference 100.0 1 0.01 1 0.08 1
209 ObserverA 92.5 3 0.19 3 0.86 2
209 ObserverB 93.3 2 0.17 2 3.25 4
209 ObserverC 91.8 4 0.21 4 1.02 3
209 Reference 100.0 1 0.01 1 0.09 1