Adaptable Focal Loss for Imbalanced Text Classification

Lu Cao; Xinyue Liu; Hong Shen

doi:10.1007/978-3-030-96772-7_43

Adaptable Focal Loss for Imbalanced Text Classification

Lu Cao, Xinyue Liu, Hong Shen

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Citations (Scopus)

Abstract

In this paper, we study the problem of imbalanced text classification based on the pre-trained language models. We propose the Adaptable Focal Loss (AFL) method to solve this problem. Firstly, we use the word embeddings from the pre-trained models to construct the sentence level prior by the sum of the word embeddings in the sentence. Then, we extend the Focal Loss, which is widely used in the field of object detection, by replacing the task-special parameters with the scaled-softmax of the distance between the fine-tuned embeddings and the prior embeddings from the pre-trained models. By removing the task-special parameters in Focal Loss, not only can the parameters of arbitrary imbalanced proportion distribution be adjusted automatically according to the task, but also the sentences that are difficult to classify can be given a higher weight. Experimental results show that our methods can easily combine with the common classifier models and significantly improve their performances.

Original language	English
Title of host publication	Parallel and Distributed Computing, Applications and Technologies - 22nd International Conference, PDCAT 2021, Proceedings
Editors	Hong Shen, Yingpeng Sang, Yong Zhang, Nong Xiao, Hamid R. Arabnia, Geoffrey Fox, Ajay Gupta, Manu Malek
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	466-475
Number of pages	10
ISBN (Print)	9783030967710
DOIs	https://doi.org/10.1007/978-3-030-96772-7_43
Publication status	Published - 2022
Externally published	Yes
Event	22nd International Conference on Parallel and Distributed Computing, Applications and Technologies, PDCAT 2021 - Guangzhou, China Duration: 17 Dec 2021 → 19 Dec 2021

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13148 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	22nd International Conference on Parallel and Distributed Computing, Applications and Technologies, PDCAT 2021
Country/Territory	China
City	Guangzhou
Period	17/12/21 → 19/12/21

Keywords

Adaptive training
Focal loss
Imbalanced text classification
Pre-trained models

Access to Document

10.1007/978-3-030-96772-7_43

Cite this

Cao, L., Liu, X., & Shen, H. (2022). Adaptable Focal Loss for Imbalanced Text Classification. In H. Shen, Y. Sang, Y. Zhang, N. Xiao, H. R. Arabnia, G. Fox, A. Gupta, & M. Malek (Eds.), Parallel and Distributed Computing, Applications and Technologies - 22nd International Conference, PDCAT 2021, Proceedings (pp. 466-475). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13148 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-96772-7_43

Cao, Lu ; Liu, Xinyue ; Shen, Hong. / Adaptable Focal Loss for Imbalanced Text Classification. Parallel and Distributed Computing, Applications and Technologies - 22nd International Conference, PDCAT 2021, Proceedings. editor / Hong Shen ; Yingpeng Sang ; Yong Zhang ; Nong Xiao ; Hamid R. Arabnia ; Geoffrey Fox ; Ajay Gupta ; Manu Malek. Springer Science and Business Media Deutschland GmbH, 2022. pp. 466-475 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{e322da2f7df544ec8b97e466c46c60ce,

title = "Adaptable Focal Loss for Imbalanced Text Classification",

abstract = "In this paper, we study the problem of imbalanced text classification based on the pre-trained language models. We propose the Adaptable Focal Loss (AFL) method to solve this problem. Firstly, we use the word embeddings from the pre-trained models to construct the sentence level prior by the sum of the word embeddings in the sentence. Then, we extend the Focal Loss, which is widely used in the field of object detection, by replacing the task-special parameters with the scaled-softmax of the distance between the fine-tuned embeddings and the prior embeddings from the pre-trained models. By removing the task-special parameters in Focal Loss, not only can the parameters of arbitrary imbalanced proportion distribution be adjusted automatically according to the task, but also the sentences that are difficult to classify can be given a higher weight. Experimental results show that our methods can easily combine with the common classifier models and significantly improve their performances.",

keywords = "Adaptive training, Focal loss, Imbalanced text classification, Pre-trained models",

author = "Lu Cao and Xinyue Liu and Hong Shen",

note = "Publisher Copyright: {\textcopyright} 2022, Springer Nature Switzerland AG.; 22nd International Conference on Parallel and Distributed Computing, Applications and Technologies, PDCAT 2021 ; Conference date: 17-12-2021 Through 19-12-2021",

year = "2022",

doi = "10.1007/978-3-030-96772-7_43",

language = "English",

isbn = "9783030967710",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "466--475",

editor = "Hong Shen and Yingpeng Sang and Yong Zhang and Nong Xiao and Arabnia, {Hamid R.} and Geoffrey Fox and Ajay Gupta and Manu Malek",

booktitle = "Parallel and Distributed Computing, Applications and Technologies - 22nd International Conference, PDCAT 2021, Proceedings",

address = "Germany",

}

Cao, L, Liu, X & Shen, H 2022, Adaptable Focal Loss for Imbalanced Text Classification. in H Shen, Y Sang, Y Zhang, N Xiao, HR Arabnia, G Fox, A Gupta & M Malek (eds), Parallel and Distributed Computing, Applications and Technologies - 22nd International Conference, PDCAT 2021, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13148 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 466-475, 22nd International Conference on Parallel and Distributed Computing, Applications and Technologies, PDCAT 2021, Guangzhou, China, 17/12/21. https://doi.org/10.1007/978-3-030-96772-7_43

Adaptable Focal Loss for Imbalanced Text Classification. / Cao, Lu; Liu, Xinyue; Shen, Hong.
Parallel and Distributed Computing, Applications and Technologies - 22nd International Conference, PDCAT 2021, Proceedings. ed. / Hong Shen; Yingpeng Sang; Yong Zhang; Nong Xiao; Hamid R. Arabnia; Geoffrey Fox; Ajay Gupta; Manu Malek. Springer Science and Business Media Deutschland GmbH, 2022. p. 466-475 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13148 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Adaptable Focal Loss for Imbalanced Text Classification

AU - Cao, Lu

AU - Liu, Xinyue

AU - Shen, Hong

PY - 2022

Y1 - 2022

N2 - In this paper, we study the problem of imbalanced text classification based on the pre-trained language models. We propose the Adaptable Focal Loss (AFL) method to solve this problem. Firstly, we use the word embeddings from the pre-trained models to construct the sentence level prior by the sum of the word embeddings in the sentence. Then, we extend the Focal Loss, which is widely used in the field of object detection, by replacing the task-special parameters with the scaled-softmax of the distance between the fine-tuned embeddings and the prior embeddings from the pre-trained models. By removing the task-special parameters in Focal Loss, not only can the parameters of arbitrary imbalanced proportion distribution be adjusted automatically according to the task, but also the sentences that are difficult to classify can be given a higher weight. Experimental results show that our methods can easily combine with the common classifier models and significantly improve their performances.

AB - In this paper, we study the problem of imbalanced text classification based on the pre-trained language models. We propose the Adaptable Focal Loss (AFL) method to solve this problem. Firstly, we use the word embeddings from the pre-trained models to construct the sentence level prior by the sum of the word embeddings in the sentence. Then, we extend the Focal Loss, which is widely used in the field of object detection, by replacing the task-special parameters with the scaled-softmax of the distance between the fine-tuned embeddings and the prior embeddings from the pre-trained models. By removing the task-special parameters in Focal Loss, not only can the parameters of arbitrary imbalanced proportion distribution be adjusted automatically according to the task, but also the sentences that are difficult to classify can be given a higher weight. Experimental results show that our methods can easily combine with the common classifier models and significantly improve their performances.

KW - Adaptive training

KW - Focal loss

KW - Imbalanced text classification

KW - Pre-trained models

UR - http://www.scopus.com/inward/record.url?scp=85127708109&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-96772-7_43

DO - 10.1007/978-3-030-96772-7_43

M3 - Conference contribution

AN - SCOPUS:85127708109

SN - 9783030967710

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 466

EP - 475

BT - Parallel and Distributed Computing, Applications and Technologies - 22nd International Conference, PDCAT 2021, Proceedings

A2 - Shen, Hong

A2 - Sang, Yingpeng

A2 - Zhang, Yong

A2 - Xiao, Nong

A2 - Arabnia, Hamid R.

A2 - Fox, Geoffrey

A2 - Gupta, Ajay

A2 - Malek, Manu

PB - Springer Science and Business Media Deutschland GmbH

T2 - 22nd International Conference on Parallel and Distributed Computing, Applications and Technologies, PDCAT 2021

Y2 - 17 December 2021 through 19 December 2021

ER -

Cao L, Liu X, Shen H. Adaptable Focal Loss for Imbalanced Text Classification. In Shen H, Sang Y, Zhang Y, Xiao N, Arabnia HR, Fox G, Gupta A, Malek M, editors, Parallel and Distributed Computing, Applications and Technologies - 22nd International Conference, PDCAT 2021, Proceedings. Springer Science and Business Media Deutschland GmbH. 2022. p. 466-475. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-96772-7_43

Adaptable Focal Loss for Imbalanced Text Classification

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this