One of the greatest misconceptions concerning LLMs is the idea that they are easy to use. They really aren’t: getting great results out of them requires a great deal of experience and hard-fought intuition, combined with deep domain knowledge of the problem you are applying them to.

The whole “LLMs are useful” section hits for me. I have an experience similar to Simon’s and I also wouldn’t claim LLMs are without issue or controversy. For me they are unquestionably useful. They help me more quickly and effectively get ideas out of my head and into the real world. They help me learn more quickly and solve problems and answer questions as I’m learning. They increase my capabilities as an individual in the same sort of way getting access to Google for the first time did. These days, not having access to a language model makes me feel like I’ve had an essential tool taken away from me, like not having a calculator or documentation.

Did a bit more work on a LLM evaluator for connections. I’m mostly trying it with gpt-4 and claude-3-opus. On today’s puzzle, the best either did was 2/4 correct. I’m unsure how much more improvement is possible with prompting or even fine tuning, but it’s an interesting challenge.

Darwin, who kept a notebook where he wrote down facts that contradicted him, observed that frustrating, cognitively dissonant things were the first to slip his memory.

Read More…

Setup a Temporal worker in Ruby and got familiar with its ergonomics.

Tried out this gpt-4v demo repo Experimented with OCR capabilities of open source multi-modal language models. Tried llava:32b (1.6) and bakllava but neither seemed to touch gpt-4-vison’s performance. It was cool to see the former run on a macbook though.

I use the hyper+u keyboard shortcut to open a language model playground for convenience. I might use this 10-20 times a day. For the last year or so that I’ve been doing this, it has always pointed to https://platform.openai.com/playground. As of today, I’ve switched it to point to https://console.anthropic.com/workbench?new=1. Lately, I’ve preferred claude-3-opus to gpt-4. For a while, I had completely stopped looking for other models as gpt-4 seemed to be unchallenged, but it’s exciting to see new options available.

I tried setting up sqlite-vss with Deno following these instructions but got stuck on this error

❯ deno task dev
Task dev deno run --allow-env --allow-read --allow-write --allow-net --unstable-ffi --allow-ffi --watch main.ts
Watcher Process started.
error: Uncaught (in promise) TypeError: readCstr is not a function
export const SQLITE_VERSION = readCstr(sqlite3_libversion());
                              ^
    at https://deno.land/x/[email protected]/src/database.ts:101:31
    at eventLoopTick (ext:core/01_core.js:169:7)

so I pivoted to Python. That effort eventually turned into this post.

I’ve spend almost a week, on and off, trying to install ollama using Nix in such a way that ollama serve will be run and managed automatically in the background. Initially, I had tried to install ollama via home-manager. This was straightforward, but finding a way to have ollama serve run automatically so that I didn’t need to do it myself every time I wanted to interact with or pull a model. This need send me down that macOS rabbithole of launch agents/daemons and plists. The macOS-specific nature of this research could have tipped me off that this was a job for nix-darwin rather than home-manager. Eventually, I tried installing ollama via nix-darwin and it seemed to Just Work. I still haven’t been able to figure out where it’s running ollama serve from.

I spent some time exploring Deepgram’s Next.js starter app. I was hoping I could use it to generate a transcription in realtime but it was more like real-time captions. The responses from the server were sometimes corrections of previous transcriptions. Maybe there is a way to make this a transcription but I wasn’t sure.

I also tried out vocode’s Python library for building voice-based LLM applications. By far the hardest part of this was getting an Azure account, which I believe is used to synthesize the LLM response as speech. I had the demo working end to end but it was a bit sensitive to background noise as they note. I haven’t had a chance to play around with any configurations they provide.

I played around with AgentGPT using Reworkd’s cloud hosted instance. I tried a few different goals. The first was travel related. I was pleasantly surprised (unnerved) to see the agent return links for flights to my proposed location from my actual current origin (I guess they used my IP to determine this). One of my first impressions was the likeness to Perplexity, particularly in how the links were displayed. It turns out, Reworkd open sourced code for this exact thing. AgentGPT also uses inline citations like Perplexity does, though those aren’t shown in the perplexity-style-streaming demo video.

I tried to get claude-3-opus to solve today’s Connections. It did well enough that I am going to try doing this more consistently.

I grew tried of Nix and decided to setup Homebrew as well. I finally gave in after trying to install llm and not being able to figure out how to install plugins despite the message

If you want to install extra llm plugins, use llm.withPlugins([]) expression.

In tracing this error, I also learning nixpkgs can apply patches to libraries in its registry.

I like Nix a lot. I still don’t understand a lot about it. I cannot lose an hour to installing a dependencies when that is a solved problem using Homebrew.

Ok fine, I will manage Homebrew with nix.