I enjoyed this article by Ken about production LLM use cases with OpenAI models.
When it comes to prompts, less is more
This resonated with me. I’ve found that too much instruction can lead a model to perform worse on a task.
GPT is really bad at producing the null hypothesis
This also seems to confirm what I’ve seen empirically, but I never ask for it. I ask for something like, “return an empty JSON array if you can’t find anything”.
GPT really cannot give back more than 10 items. Trying to have it give you back 15 items? Maybe it does it 15% of the time.
I didn’t notice this and haven’t tried it explicitly. I did a bit of recipe ingredients extraction where there were more than 10 ingredients and that seems to work okay. I also recently did another use case where I’m extracting a variable number of items and extracting 10 items seems to work consistently, but I am just waving my hands. I need to collect data to confirm.
vector databases, and RAG/embeddings are mostly useless for us mere mortals
This surprised me. I’ve found embeddings to be useful, though I can’t say I’ve gone to production with a RAG use case. I agree that putting your embeddings in a specialized database doesn’t make sense. I’ve done most of my exploration storing embeddings in SQLite.
Ken’s article also linked to The Bitter Lesson by Rich. He lays out what he believes to be the biggest lesson learned in the history of AI
general methods that leverage computation are ultimately the most effective, and by a large margin.
One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning.
A general note on my experience with Nix: most things have taken multiple attempts, but persistence has allowed me to eventually find a workable solution for the problems I’ve faced.
I love the combo of direnv and flakes to build a directory-specific environment.
It seems a number of dev tools are popping up that make a very friendly interface to this functionality and use Nix underneath to make it happen.
I used one called devbox to play around with postgres and recently learned of another called devenv.