A Minimax Approach for Robustifying Statistical Learning Algorithms ; Une approche minimax pour la robustificationdes algorithmes d’apprentissage statistique

In this paper, we address the challenge of learning models that stay reliable under distribution shifts when only finite training data is available. A novel training-dependent minimax problem is proposed to design learning algorithms that are robust to the worst-case datagenerating distribution. For the ambiguity sets, we construct Kullback-Leibler (KL) divergence neighborhoods on both model and data distributions. We obtain an analytical solution to this minimax problem, referred to as the robust learner. We show that the robust learner follows a Gibbs distribution, in which the prior can be chosen as any baseline learning algorithm to be robustified. Such a robust learner minimizes the worst-case generalization gap within a KL-divergence neighbourhood of unseen new data and it provides a smaller generalization error compared with the baseline learning algorithm Q. Under certain conditions, the robust learner also guarantees a smaller expected testing error than Q. We also provide a training-dependent PAC-Bayes bound on the robust learner's performance on unseen data. Closed-form expressions for generalization error and expected loss of the robust learner are given in terms of mutual information, lautum information, and KL-divergence. As a by-product, we show that the proposed minimax problem admits a two-player zero-sum game formulation, for which a unique Nash equilibrium exists. This enhances our understanding of learning algorithm robustification. Numerical experiments validate the applicability of the results and the benefits of robustification.