Researchers from Tsinghua University have developed a groundbreaking machine learning algorithm called Meta-Semi, which falls under the meta-learning paradigm. The algorithm aims to enhance model performance in scenarios where only a small fraction of the training data is labeled. By leveraging both labeled and unlabeled data, this novel approach achieves outstanding results with the adjustment of just one additional hyper-parameter.
Supervised tasks in deep learning have witnessed significant advancements due to the availability of large amounts of labeled training data. However, collecting accurate labels requires substantial effort and financial resources. In many practical contexts, only a small portion of the training data is labeled. This is where semi-supervised learning (SSL) comes into play, seeking to enhance model performance by utilizing both labeled and unlabeled input.
Effective SSL approaches often employ unsupervised consistency regularization to leverage the unlabeled data. However, these state-of-the-art consistency-based algorithms introduce multiple configurable hyper-parameters, which need to be fine-tuned for optimal performance. Unfortunately, hyper-parameter searching poses challenges in real-world SSL scenarios where annotated data are scarce, resulting in high variance during cross-validation. Moreover, the computational cost becomes overwhelming as the search space expands exponentially with the number of hyper-parameters.
In light of these challenges, the researchers at Tsinghua University have introduced Meta-Semi, a meta-learning-based SSL algorithm that addresses these issues by adjusting just one additional hyper-parameter. The team was inspired by the idea that a network can be successfully trained using appropriately pseudo-labeled unlabeled examples. During the online training phase, pseudo-soft labels are generated for the unlabeled data based on the network predictions. Unreliable or incorrect pseudo-labeled samples are then discarded, and the remaining data is used to train the model. The key is to ensure that the distribution of correctly pseudo-labeled data is comparable to that of the labeled data so that minimizing the loss on the former also minimizes it on the latter.
To accomplish this, the researchers introduced the meta-reweighting objective, which aims to minimize the final loss on the labeled data by selecting the most suitable weights. However, they encountered computational difficulties when using optimization algorithms to solve this problem. Therefore, they proposed an approximation formulation that allows for a closed-form solution. The theoretical analysis revealed that each training iteration only requires a single meta gradient step to achieve approximate solutions.
In summary, the researchers suggest a dynamic weighting approach to reweight previously pseudo-labeled samples using 0-1 weights. Results demonstrate that this approach eventually reaches the stationary point of the supervised loss function. In various image classification benchmarks, Meta-Semi outperformed state-of-the-art deep networks, such as ICT and MixMatch, in challenging SSL tasks like CIFAR-100 and STL-10 while achieving somewhat better performance on CIFAR-10. Additionally, incorporating consistency regularization further enhanced the algorithm’s performance.
However, Meta-Semi does require slightly more time to train, which the researchers plan to address in future work. The algorithm shows promise in domains such as medical image processing, hyper-spectral image classification, network traffic recognition, and document recognition. Meta-Semi offers a significant advancement in SSL algorithms, effectively leveraging labeled and unlabeled data while minimizing the reliance on hyper-parameter tuning. Its outstanding performance and potential applications make it a valuable addition to the field of deep learning research.
