Distributed Hierarchical Sentence Embeddings for Unsupervised Extractive Text Summarization

Guanjie Huang, Hong Shen

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


Unsupervised text summarization is a promising approach that avoids human efforts in generating reference summaries, which is particularly important for large-scale datasets. To improve its performance, we propose a hierarchical BERT [1] model that contains both word-level and sentence-level training processes to achieve semantic-rich sentence embeddings. We use the vanilla BERT as the word-level training, and redesign it for the sentence-level training with the new "Sentence Token Prediction"and "Local Shuffle Recovery"training tasks and suitable input format. We first train word-level model to get preliminary sentence embeddings, then we input them into the sentence-level model to further extract higher level and inter-sentence semantic information. After that, we obtain the context sensitive sentence embeddings and utilize them for the KMeans cluster algorithm to finally generate summaries by extracting sentences from the document. To accelerate the training of the BERT model, we adopt the PipeDream [2] model parallelism that distributes the model layers among multiple machines to conduct the training process in parallel. Finally, we show through experimental results that our proposed model outperforms most popular models and achieves a speedup of 2.7 in training time on 4 machines.

Original languageEnglish
Title of host publicationICBDC 2021 - 2021 6th International Conference on Big Data and Computing
PublisherAssociation for Computing Machinery
Number of pages7
ISBN (Electronic)9781450389808
Publication statusPublished - 22 May 2021
Externally publishedYes
Event6th International Conference on Big Data and Computing, ICBDC 2021 - Virtual, Online, China
Duration: 22 May 202124 May 2021

Publication series

NameACM International Conference Proceeding Series


Conference6th International Conference on Big Data and Computing, ICBDC 2021
CityVirtual, Online


  • model parallelism
  • sentence embeddings
  • unsupervised extractive text summarization


Dive into the research topics of 'Distributed Hierarchical Sentence Embeddings for Unsupervised Extractive Text Summarization'. Together they form a unique fingerprint.

Cite this