double PT: Enhancing Meta-Learning Performance with Pre-Train and Meta-Pre-Train Processes

With meta-learning, models are trained on multiple tasks, and resulting trained models are expected to be capable of ' learning' new tasks effectively. MAML (Model Agnostic Meta-Learning) was one such early design which allowed models to reuse learned features, but with limited performance. Pre-training is another known method to improve the performance of a final trained model. Pre-train starts by assisting models to seek better initialization points, thus offering better feature representations. In this paper, we propose doublePT (double-pretrain) which is a two-stage pre-training method with goals to 1) reduce the number of parameters, 2) acquire better feature representations, and 3) achieve competitive overall performances in different benchmark measurements. For the first stage operation, we use a universal pre-training model to capture general features from a large dataset. In the second stage, we propose to use MAML to fine-tune the pre-training model to enhance the feature adaptability. Since the first-stage pre-training model has already learned general feature representations, it reduces the training activities for the second-stage fine-tuning operations, and enables better feature extractions in new tasks. Validated through our experiments, we find that our proposed doublePT approach fine-tunes across different tasks, and performs better than that of one-stage pre-training approach. Upon combining doublePT and DINOv2, and comparing to the latest PMF meta-learning design, the number of parameters required by the PMF pipelining model needs 304.8% more parameters than in our proposed DINOv2+doublePT model design. Performance-wise, the DINOv2+doublePT also has the best accuracies across different benchmark measurements.