Yeah this is exactly my view. We've had several years of work on the tech, and LLMs are just as prone to randomly spitting out garbage as they were the first day. They are not a tool which is fit for any serious work, because you need to be able to rely on your tools. A tool which is sometimes good and sometimes bad is worse than having no tool at all.
Probably. According to the paper, 83.82% of automated commits were already made by algorithmic tools (non-LLM). For the remainder, a three-phase LLM approach was tried, and achieved a success rate of 30%. Based on these numbers, it probably would have been faster, cheaper, and more efficient to just enhance their current strategy rather than screwing around with text generators.
If you're not seeing the hallucinations, I'd assert you're either not using it enough, or (more likely) you don't have enough knowledge in the subject matter to notice when it's hallucinating.
I'm not interested in getting into some argument about who has "more knowledge in the subject matter". I'm genuinely curious: do you think Opus 4.6 hallucinates just as much as GPT-3.5?
Hmm no way. Ive used to see hallucinations like 50% of the time prompting gpt3.5 for simple functions.
I don't remember the last time ive seen a made up library/methods these days and Im definitely using way more for more complex stuff. The tool calling changed the game.
Even for work I do almost 100% of my coding telling claude what to do. I mean I break down the tasks and tell it more or less exactly what I want but I find "rename this thing across these two repos" easier than doing it myself
I ran into the non existent methods and functions far more a year ago than I do today. I hadn’t even considered it as I don’t write a lot of code,
Most of my job is talking with people to understand the problems and to drive strategy.
What a condescending post. You either haven't used any recent models if you make that statement.
Anyone who used GPT3.5 and any other newer model know that hallucinations have gone down tremendously.
Of course it's not perfect, here and there are inaccuracies or plain hallucinations, but it's impossible to state that it's still the same garbage it was 3 years ago.
