My last post outlined the kinds of evaluation I need and want to understand how well my finetuned LLM is performing in the task of structured data extraction from press releases. Let’s start with the core metric I’m interested in, accuracy, and then later we can dive into some of the other evaluation metrics as well. TL;DR The headline for this post could well have been: finetuned models beat OpenAI, but evals were a bit painful to implement. There’s a lot of hidden code here in this post and it...