As (large) language models find applications across an increasingly broad spectrum of tasks, the necessity for reliable confidence estimates—or uncertainty quantification (UQ)—on their predictions is critical. However, the selection of appropriate and efficient UQ methods presents a considerable challenge, particularly in scenarios where access to the model’s parameters and its training process is restricted. In this presentation, we will delve into conformal prediction, a method that allows us to compute confidence intervals in the form of prediction sets, with established guarantees over coverage of the ground truth. I will discuss both regression and generation tasks within Natural Language Processing (NLP), using machine translation evaluation and machine translation (MT) paradigms respectively. With a focus on MT evaluation, we will explore how conformal prediction can guide the selection of fitting UQ methods, yielding meaningful confidence intervals that assist in identifying and addressing biases inherent in these approaches. Turning to generation, we face an additional challenge due to the sequential nature of these tasks, where the dependence on preceding tokens must be considered. This talk will address how we adapt our approach to accommodate such dependencies, and show how our method ‘Non-Exchangeable Conformal Language Generation with Nearest Neighbors’ can be used post-hoc for an arbitrary model without extra training to supply token-level, calibrated prediction sets. Finally, we will see how such prediction sets can be used for sampling in machine translation and language modelling, showing encouraging results in generation quality.