PK-BERT: Knowledge Enhanced Pre-trained Models with Prompt for Few-Shot Learning

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

2 Citations (Scopus)


The amount of data in some fields are scarce because they are difficult or expensive to obtain. The general practice is to pre-train a model on similar data sets and fine-tune the models in downstream tasks by transfer learning. The pre-trained models could learn the general language representation from large-scale corpora but their downstream task may be different from the pre-trained tasks in form and type. It also lacks related semantic knowledge. Therefore, we propose PK-BERT—Knowledge Enhanced Pre-trained Models with Prompt for Few-shot Learning. It (1) achieves few-shot learning by using small samples with pre-trained models; (2) constructs the prefix that contains the masked label to shorten the gap between downstream task and pre-trained task; (3) uses the explicit representation to inject knowledge graph triples into the text to enhance the sentence information; and (4) uses masked language modelling (MLM) head to convert the classification task into generation task. The experiments show that our proposed model PK-BERT achieves better results.

Original languageEnglish
Title of host publicationComputer and Information Science
EditorsRoger Lee
PublisherSpringer Science and Business Media Deutschland GmbH
Number of pages14
ISBN (Print)9783031121265
Publication statusPublished - 2023
Event22nd IEEE/ACIS International Conference on Computer and Information Science, ICIS 2022 - Zhuhai, China
Duration: 26 Jun 202228 Jun 2022

Publication series

NameStudies in Computational Intelligence
ISSN (Print)1860-949X
ISSN (Electronic)1860-9503


Conference22nd IEEE/ACIS International Conference on Computer and Information Science, ICIS 2022


  • Few-shot learning
  • Knowledge graph
  • Masked language modelling
  • Pre-trained models
  • Prompt


Dive into the research topics of 'PK-BERT: Knowledge Enhanced Pre-trained Models with Prompt for Few-Shot Learning'. Together they form a unique fingerprint.

Cite this