I tried stacking multiple pages of a pdf vertically as a single image to a model, then doing data extraction from this. It didn’t work. I imagine this is because models aren’t trained on much data like this. The inference seemed to output made up data.
An interesting pitch written by Hillel for preferring reStructuredText to Markdown.
Multiple studies have shown that hallucinations can be significantly reduced by giving the model the right context via retrieval or tools that the model can use to gather context (e.g., web search).
While hallucinations are dealbreakers for many applications, they can be sufficiently curtailed to make GenAI usable for many more.
A search with Perplexity seemed to corroborate this assertion. It led me to this paper which used RAG to reduce hallucinations.
Jon wrote about his computer setup and tools. I always take inspiration from these kinds of write-ups.
I’d like to get something like open-webui set up to run remote or local using the same UI, that I can customize. Still thinking about what this could look like.
A great video by Mark Russinovich about AI security. He discusses model fingerprinting, backdoors, jailbreaking, model tool abuse, crescendo attacks and more. Definitely a lot to keep in mind when building with models.