. (2018). “On the Importance of Drill-Down Analysis for Assessing Gold Standards and Named Entity Linking Performance”. Proceedings of the 14th International Conference on Semantic Systems (SEMANTICS 2018), Vienna, Austria
Rigorous evaluations and analyses of evaluation results are key towards improving Named Entity Linking systems. Nevertheless,most current evaluation tools are focused on benchmarking and comparative evaluations. Therefore, they only provide aggregatedstatistics such as precision, recall and F1-measure to assess system performance and no means for conducting detailed analyses upto the level of individual annotations.This paper addresses the need for transparent benchmarking and fine-grained error analysis by introducing Orbis, an extensibleframework that supports drill-down analysis, multiple annotation tasks and resource versioning. Orbis complements approacheslike those deployed through the GERBIL and TAC KBP tools and helps developers to better understand and address shortcomingsin their Named Entity Linking tools.We present three uses cases in order to demonstrate the usefulness of Orbis for both research and production systems: (i)improving Named Entity Linking tools; (ii) detecting gold standard errors; and (iii) performing Named Entity Linking evaluationswith multiple versions of the included resources.