MicroProbe: Efficient Reliability Assessment for Foundation Models with Minimal Data

This paper introduces MicroProbe, a novel method for efficiently assessing the reliability of foundation models. It addresses the challenge of computationally expensive and time-consuming reliability evaluations by using only 100 strategically selected probe examples. The method combines prompt diversity, uncertainty quantification, and adaptive weighting to detect failure modes effectively. Empirical results demonstrate significant improvements in reliability scores compared to random sampling, validated by expert AI safety researchers. MicroProbe offers a promising solution for reducing assessment costs while maintaining high statistical power and coverage, contributing to responsible AI deployment by enabling efficient model evaluation. The approach seems particularly valuable for resource-constrained environments or rapid model iteration cycles.