Frequently Asked Questions


I have lost the password of my registered team. What should I do?
Send us an email. A password reminder will be mailed to the e-mail address provided with the registered team.

I have downloaded the data. How do I read it?
Read the data format section on the about page.

Can I submit my results to this evaluation framework?
Yes, you can, if you have registerd your team and send in the data confidentiality form

What's the submission format
Read the submssion format section on the about page.

How are the methods evaluated?
The evaluation framework (data, reference standard, measures, scores, and ranking) is described in this document.

Why can't I download the reference segmentation for the test data and perform the evaluation myself?
From our previous experiences with making data sets publicly available we have learned that if we would release the 'truth' for the test data, groups would perform slightly or vastly different types of evaluations. This may lead to incomparable results between papers that have used the same data. To avoid this, we decided on the current procedure, which makes sure that each system is evaluated in exactly the same way.

If my segmentation extents beyond the specified region, will this result in large errors?
No, your submitted segmentation is only evaluated within the region as specified in Section 2.1 of this document. However you need to make sure that you cut off the external carotid artery within the region sepcified in the same Section.

Will my results be shown on the website?
Your results will only be shown on the website after you have

  • sucessfully submitted the results
  • confirmed that submission through email
  • you have submitted a paper with a detailed description of your method or provided us with a link to an online journal publication (for commercial companies: please contact the organizers)
  • the paper has been accepted by the framework organisation

All my average evaluation measures show a better average value than those of another submission, but I still have a rank that's worse.
This due to the way the ranks are calculated. If for a certain dataset many submissions have a measure that is closes to the others, your may reveive a rank that is much worse than the other submission, although your evaluation measures are close. This higher rank will have a relatively large impact on the average rank.

I improved my method. Can I submit again?
Yes, you can. But to show the new results on the website you will have to provide us with an updated description of your method.

What to do if I have further questions?
Send an e-mail to one of the organizers.