Holder： Jean Feng(University of California)
Time：2025-12-16 10:00-12:00
Location：Tencent Meeting:625-476-374
Abstract:
Given the challenge of automatically evaluating free-form outputs from large language models (LLMs), an increasingly common solution is to use LLMs themselves as the judging mechanism, without any gold-standard scores. lmplicitly, this practice accounts for only sampling variability(aleatoric uncertainty) and ignores uncertainty about judge quality (epistemic uncertainty). While this is justified if judges are perfect, it is unclear when such an approach is (i)theoretically valid and (ii)practically robust. We study these questions for the task of ranking LLM candidates from a novel geometric perspective:for M-level scoring systems, both LLM judges and candidates can be represented as points on a (M-1)dimensional probability simplex, where geometric concepts (e.g.,triangle areas)correspond to key ranking concepts. This perspective yields intuitive theoretical conditions and visual proofs for when rankings are identifiable, for instance, we provide a formal basis for the "folk wisdom" that LLM judges are more effective with binary scoring (M=2) than with multi-level scoring (M>2). Leveraging the simplex, we design geometric Bayesian priors that encode epistemic uncertainty about judge quality and vary the priors to conduct sensitivity analyses. Experiments on LLM benchmarks show that rankings based solely on LLM judges are robust in many but not all datasets, underscoring both their widespread success and the need for caution. Our Bayesian method achieves substantially higher coverage rates than existing procedures,highlighting the importance of modeling epistemic uncertainty.
About the Speaker:
Jean Feng is an Associate Professor in the Department of Epidemiology and Biostatistics at the University of California, San Francisco and the UCSF-UC Berkeley Joint Program in Computational Precision Health and a principal investigator at the UCSF-Stanford Center of Excellence in Regulatory Science and Innovation(CERSl). She is also the data science lead on the PROSPECT team,the digital innovation taskforce for the Zuckerberg San Francisco General Hospital . Her research interests span the interpretability,reliability, and regulation of machine learning (ML) algorithms in healthcare.
Your participation is warmly welcomed!
欢迎扫码关注北大统计科学中心公众号，了解更多讲座信息!
