rss-training-iclr2024
Out-Of-Domain Unlabeled Data Improves Generalization - ICLR 2024 (Spotlight)
📜 Click for Abstract
We propose a novel framework for incorporating unlabeled data into semi-supervised classification problems, where scenarios involving the minimization of either: - *i)* adversarially robust, or - *ii)* non-robust loss functions have been considered. Notably, we allow the unlabeled samples to deviate slightly (in the total variation sense) from the in-domain distribution. The core idea behind our framework is to combine **Distributionally Robust Optimization (DRO)** with **self-supervised training**. As a result, we also leverage **efficient polynomial-time algorithms** for the training stage. From a theoretical standpoint, we apply our framework to the classification problem of a mixture of two Gaussians in $\mathbb{R}^d$, where, in addition to the $m$ independent and labeled samples from the true distribution, a set of $n$ (usually with $n \gg m$) out-of-domain and unlabeled samples are also provided. Using only the labeled data, it is known that the generalization error can be bounded by: $$\propto \left(\frac{d}{m}\right)^{1/2}.$$ However, using our method on both isotropic and non-isotropic Gaussian mixture models, one can derive a new set of analytically explicit and non-asymptotic bounds which show substantial improvement in the generalization error compared to ERM. Our results underscore two significant insights: 1. Out-of-domain samples, even when unlabeled, can be harnessed to narrow the generalization gap, provided that the true data distribution adheres to a form of the *"cluster assumption"*. 2. The semi-supervised learning paradigm can be regarded as a special case of our framework when there are no distributional shifts. We validate our claims through experiments conducted on a variety of synthetic and real-world datasets.💡 Methodology: Robust Self-Supervised (RSS) Training
The Robust Self-Supervised (RSS) Training framework enhances the ERM loss function by incorporating a robust regularization term. This term leverages out-of-domain unlabeled data, guiding the classifier away from crowded, dense regions to improve overall performance and robustness.
Overview of the RSS Training Framework
✨ Key Contributions
Our work introduces a polynomial-time framework that effectively integrates both labeled and slightly out-of-domain unlabeled data to improve generalization. Key contributions include:
- Non-asymptotic bounds for both robust and non-robust learning scenarios.
- Enhanced generalization over traditional ERM techniques, particularly when $n \geq \Omega(m^2/d)$.
- Dimension-independent sample complexity under specific conditions.
- Improved sample complexity from $O(d/\epsilon^2)$ to $O(d/\epsilon)$ when $n = O(d/\epsilon^6)$.
📊 Conference Poster
ICLR 2024 Conference Poster
You can access the LaTeX code for this poster here.
🚩 Updates
We are in the process of preparing the code for public release. Stay tuned for updates!
📚 Citation
If you find our work useful, please consider citing our paper:
@inproceedings{
saberi2024outofdomain,
title={Out-Of-Domain Unlabeled Data Improves Generalization},
author={Seyed Amir Hossein Saberi and Amir Najafi and Alireza Heidari and Mohammad Hosein Movasaghinia and Abolfazl Motahari and Babak Khalaj},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Bo6GpQ3B9a}
}
📝 License
This project is licensed under the Apache 2.0 License. For more details, please see the LICENSE file.