I can't believe we're back to discussing LLMs' ability to reason. Where have you been these past two years? In a bunker? If you'd actually worked with LLMs during this time, you'd know by now that they're obviously pattern-matching machines. Try asking one to write incorrect… pic.twitter.com/KPcDCI2cjD
— Andriy Burkov (@burkov) August 18, 2024
I haven’t viewed the LLMs-can, LLM-can’t discourse through this lens explicitly.
they’re obviously pattern-matching machines
I’m not sure if I understand at what point these are different things. Maybe it’s a consequence of how I learn, but I generally develop skills on the foundations of seeing and understanding how someone more skilled than myself solves a problem.
My foundational knowledge of Python, Protobuf and prompting language models allowed me to combine these skills and develop impulse.
Could a language model have (I say this loosely) invented this solution given a high-level description of the problem?
Probably not without a high degree of specificity.
How would I even describe the problem at a high enough level to not prescribe the solution but in enough detail that I didn’t already do most of the work designing the solution myself?
Returning to the original point
they’re obviously pattern-matching machines
A huge amount of “work” in the world is pattern matching. That includes jobs done on computers and more generally the application of widely understood expertise. A model can translate data from form A to B or write a script to do that. A model can also analyze a picture of an error light on a thermostat and serve as a productive information source to help you identify and potentially solve the problem yourself (though I can’t say I have first-hand confirmation for this use case but I did read a relatively well-documented account of the process). With information accessibility and pattern matching readily available to scale, the reason, insight or vision is the remaining part of work for which there is a premium and deep need.
Folks are skeptical that models can generate anything original
Asking an LLM for an “original thought” is almost oxymoronic, if not just moronic. It was built with the express purpose of not doing that.
A really good predictability machine is not very helpful in artistic expression
LLMs were funniest at their earliest stages. Image generation was funniest here as well. Remember those “trail cam” images we got from Dall-e mini? As our systems got better, the humor was lost.
A really good predictability machine is not very helpful in artistic expression.
In contrast, other folks find models notably creative
I’ve long been enamored by DALL-E 2’s specific flavor of visual creativity. Especially given the text-to-image AI system’s age, it seems to have an incredible command over color, light and dark, the abstract and the concrete, and the emotional resonance that their careful combination can conjure.
In a way, it’s almost like they’re talking about two entirely different things. Models are elusive and much of an individual’s positive or negative experience using one or seeing what it can do seems to be contingent on how they prompt the model.