Uncertainty estimation in BERT-based Named Entity Recognition | ML in PL 22

Опубликовано: 09 Октябрь 2024
на канале: ML in PL

234

Uncertainty estimation in BERT-based Named Entity Recognition by Alicja Rączkowska (Allegro Machine Learning Research)

Named Entity Recognition (NER) is one of the standard tasks in natural language processing and models optimized for it are routinely deployed for users. With the advent of deep learning in general, and Transformer-based architectures in particular, the SOTA for NER moved towards large parameter spaces. However, such models are known to be poorly calibrated black boxes, which do not provide any uncertainty estimates. These estimates are crucial for misclassification detection and out-of-distribution detection in deployed machine learning models. Despite the common application of NER, the issue of uncertainty estimation in that task is not well studied as of yet. This talk will cover a novel exploration of NER uncertainty estimation in the context of e-commerce product descriptions.

We pre-trained a BERT-based language model on our internal corpus of around 35 million product descriptions. We then used that pre-trained backbone to fine-tune NER classifiers in several product categories and utilized the variational dropout technique to measure the predictive entropy for each observation, which served as a general-purpose uncertainty metric.

With variational dropout, we were able to properly calibrate our NER models and the entropy uncertainty measure proved to be a good indicator of both misclassified and out-of-distribution observations. Moreover, thanks to the effective model averaging provided by variational dropout, we improved our internal NER evaluation metrics. The solution discussed in this talk is a relatively simple method of uncertainty estimation and can be effectively utilized in production-grade NER models.

The talk was delivered during ML in PL Conference 2022 as a part of Contributed Talks. The conference was organized by a non-profit NGO called ML in PL Association.

ML in PL Association website: https://mlinpl.org/
ML in PL Conference 2022 website: https://conference2022.mlinpl.org/
ML In PL Conference 2023 website: https://conference2023.mlinpl.org/

---

ML in PL Association was founded based on the experiences in organizing of the ML in PL Conference (formerly PL in ML), the ML in PL Association is a non-profit organization devoted to fostering the machine learning community in Poland and Europe and promoting a deep understanding of ML methods. Even though ML in PL is based in Poland, it seeks to provide opportunities for international cooperation.