Imperial College London
Dr Felipe Orihuela-Espina

DrFelipeOrihuela-Espina

Faculty of Medicine, Department of Surgery & Cancer

Honorary Lecturer
 
 
 
//

Contact

 

f.orihuela-espina

 
 
//

Location

 

//

Summary

 

Publications

Citation

BibTex format

@article{Garcia-Mendoza:2022:10.26342/2022-68-5,
author = {Garcia-Mendoza, J-L and Villasenor-Pineda, L and Orihuela-Espina, F},
doi = {10.26342/2022-68-5},
journal = {PROCESAMIENTO DEL LENGUAJE NATURAL},
pages = {71--83},
title = {Risks of misinterpretation in the evaluation of Distant Supervision for Relation Extraction},
url = {http://dx.doi.org/10.26342/2022-68-5},
volume = {68},
year = {2022}
}
Download

RIS format (EndNote, RefMan)

TY  - JOUR
AB  - Distant Supervision is frequently used for addressing Relation Extraction. The evaluation of Distant Supervision in Relation Extraction has been attempted through Precision-Recall curves and/or calculation of Precision at N elements. However, such evaluation is challenging because the labeling of the instances results from an automatic process that can introduce noise into the labels. Consequently, the labels are not necessarily correct, affecting the learning process and the interpretation of the evaluation results. Therefore, this research aims to show that the performance of the methods measured with the mentioned evaluation strategies varies significantly if the correct labels are used during the evaluation. Besides, based on the preceding, the current interpretation of the results of these measures is questioned. To this end, we manually labeled a subset of a well-known data set and evaluated the performance of 6 traditional Distant Supervision approaches. We demonstrate quantitative differences in the evaluation scores when considering manually versus automatically labeled subsets. Consequently, the ranking of performance among distant supervision methods is different with both labeled. Keywords: Relation Extraction. Distant Supervision evaluation. Precision-Recall curves. Precision at N.
AU  - Garcia-Mendoza,J-L
AU  - Villasenor-Pineda,L
AU  - Orihuela-Espina,F
DO  - 10.26342/2022-68-5
EP  - 83
PY  - 2022///
SN  - 1135-5948
SP  - 71
TI  - Risks of misinterpretation in the evaluation of Distant Supervision for Relation Extraction
T2  - PROCESAMIENTO DEL LENGUAJE NATURAL
UR  - http://dx.doi.org/10.26342/2022-68-5
UR  - https://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=PARTNER_APP&SrcAuth=LinksAMR&KeyUT=WOS:000782644600005&DestLinkType=FullRecord&DestApp=ALL_WOS&UsrCustomerID=a2bf6146997ec60c407a63945d4e92bb
UR  - http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6407
VL  - 68
ER  -
Download