Frequently Asked Questions
I have lost the password of my registered team.
What should I do?
Send us an . A password reminder
will be mailed to the e-mail address provided with the
I have downloaded the data. How do I read it?
Read the data format section on the
Can I submit my results to this evaluation framework?
Yes, you can, if you have registerd your team and send in the data confidentiality form
What's the submission format
Read the submssion format section on the
How are the methods evaluated?
The evaluation framework (data, reference standard, measures, scores,
and ranking) is described in
Why can't I download the reference segmentation for
the test data and perform the evaluation myself?
From our previous experiences with making data sets publicly
available we have learned that if we would release the 'truth'
for the test data, groups would perform slightly or vastly
different types of evaluations. This may lead to incomparable
results between papers that have used the same data.
To avoid this, we decided on the current procedure, which makes sure
that each system is evaluated in exactly the same way.
If my segmentation extents beyond the specified region, will this result in large errors?
No, your submitted segmentation is only evaluated within the region as specified in Section 2.1 of
this document. However you need to make sure that you
cut off the external carotid artery within the region sepcified in the same Section.
Will my results be shown on the website?
Your results will only be shown on the website after you have
- sucessfully submitted the results
- confirmed that submission through email
- you have submitted a paper with a detailed description of your method or provided us with
a link to an online journal publication (for commercial companies: please the organizers)
- the paper has been accepted by the framework organisation
All my average evaluation measures show a better average value than those of another submission, but I still have a rank that's worse.
This due to the way the ranks are calculated. If for a certain dataset many submissions have a measure that
is closes to the others, your may reveive a rank that is much worse than the other submission, although your evaluation measures are close. This
higher rank will have a relatively large impact on the average rank.
I improved my method. Can I submit again?
Yes, you can. But to show the new results on the website you will have to provide us with an updated description of your method.
What to do if I have further questions?
Send an e-mail to one of the