Towards self-certified learning: Probabilistic neural networks trained by PAC-Bayes with Backprop
Towards self-certified learning: Probabilistic neural networks trained by PAC-Bayes with Backprop by Omar Rivasplata (University College London)
This talk is based on the content of my paper "Tighter Risk Certificates for Neural Networks" (JMLR, 2021) which reports the results of empirical studies regarding training probabilistic (aka stochastic) neural networks using training objectives derived from PAC-Bayes bounds. In this context, the output of training is a probability distribution over network weights, rather than a fixed setting of the weights. We present two training objectives, used for the first time in connection with training neural networks, which were derived from tight PAC-Bayes bounds. We also re-implement a previously used training objective based on a classical PAC-Bayes bound, to compare the properties of the predictors learned using the different training objectives. We compute risk certificates for the learnt predictors, which are valid on unseen data from the same distribution that generated the training data. The risk certificates are computed using part of the data used to learn the predictors. We further experiment with different types of priors on the weights (both data-free and data-dependent priors) and neural network architectures. Our experiments on MNIST and CIFAR-10 show that our training methods produce stochastic neural network classifiers with competitive test set errors and non-vacuous risk bounds with much tighter values than previous results in the literature. Thus, our results show promise not only to guide the learning algorithm through bounding the risk but also for model selection. These observations suggest that the methods studied in our work might be good candidates for self-certified learning, in the sense of using the whole data set for learning a predictor and certifying its performance with a risk certificate that is valid on unseen data (from the same distribution as the training data) and reasonably tight, so that the certificate value is informative of the value of the true error on unseen data.
The talk was delivered during ML in PL Conference 2022 as a part of Contributed Talks. The conference was organized by a non-profit NGO called ML in PL Association.
ML in PL Association website: https://mlinpl.org/
ML in PL Conference 2022 website: https://conference2022.mlinpl.org/
ML In PL Conference 2023 website: https://conference2023.mlinpl.org/
---
ML in PL Association was founded based on the experiences in organizing of the ML in PL Conference (formerly PL in ML), the ML in PL Association is a non-profit organization devoted to fostering the machine learning community in Poland and Europe and promoting a deep understanding of ML methods. Even though ML in PL is based in Poland, it seeks to provide opportunities for international cooperation.
