Using Multiple Heads to Subsize Meta-memorization Problem

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Citation (Scopus)


The memorization problem is a meta-level overfitting phenomenon in meta-learning. The trained model prefers to remember learned tasks instead of adapting to new tasks. This issue limits many meta-learning approaches to generalize. In this paper, we mitigate this limitation issue by proposing multiple supervisions through a multi-objective optimization process. The design leads to a Multi-Input Multi-Output (MIMO) configuration for meta-learning. The model has multiple outputs through different heads. Each head is supervised by a different order of labels for the same task. This leads to different memories, resulting in meta-level conflicts as regularization to avoid meta-overfitting. The resulting MIMO configuration is applicable to all MAML-like algorithms with minor increments in training computation, the inference calculation can be reduced through early-exit policy or better performance can be achieved through low cost ensemble. In experiments, identical model and training settings are used in all test cases, our proposed design is able to suppress the meta-overfitting issue, achieve smoother loss landscapes, and improve generalisation.

Original languageEnglish
Title of host publicationArtificial Neural Networks and Machine Learning - ICANN 2022 - 31st International Conference on Artificial Neural Networks, Proceedings
EditorsElias Pimenidis, Mehmet Aydin, Plamen Angelov, Chrisina Jayne, Antonios Papaleonidas
PublisherSpringer Science and Business Media Deutschland GmbH
Number of pages12
ISBN (Print)9783031159367
Publication statusPublished - 2022
Event31st International Conference on Artificial Neural Networks, ICANN 2022 - Bristol, United Kingdom
Duration: 6 Sept 20229 Sept 2022

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13532 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference31st International Conference on Artificial Neural Networks, ICANN 2022
Country/TerritoryUnited Kingdom


  • Meta-learning
  • Meta-overfitting
  • Multi-head


Dive into the research topics of 'Using Multiple Heads to Subsize Meta-memorization Problem'. Together they form a unique fingerprint.

Cite this