Why does no one seem to care that AI gives wrong answers?

My graduate research was in this area. My lab group developed swarm robots for various terrestrial and space exploration tasks. I spent a lot of time probing why our swarm robots developed pathological behavioral breakdowns - running away from construction projects, burying each other, etc... The issue was so fundamental to our machine learning methods that we never found a way to reliably address it—by the time I left, anyway. No matter how we reconfigured the neural networks, trained, punished, deprived, or implemented forced forgetting or fine-tuned, nothing seemed to eliminate the catastrophic behavioral edge cases—nothing except for dramatically simplifying the neural networks.

Once I started seeing these behaviors in our robots, their appearance became much more pronounced every time I dug deeply into proposed ML systems: autonomous vehicles, robotic assistants, chatbots, and LLMs.

As I've had time to reflect on our challenges, I think that neural networks very quickly tend to overfit, and deep neural networks are incomparably overfitted. That condition makes them sensitive to hidden attractors that cause the system to break down when it is near these areas - catastrophically.

How do we define "near"? That would have to be determined using some topological method. But these systems are so complicated that we can't analyze their networks' topology or even brute-force probe their activations. Further, the larger, deeper, and more highly connected the network, the more challenging these hidden attractors are to find.

I was bothered by this topic a decade ago, and nothing I have seen today has alleviated my concern. We are building larger, deeper, and more connected networks on the premise that we'll eventually get to a state so unimaginably overfitted that it becomes stable again. I am unnerved by this idea and by the amount of money flowing in that direction with reckless abandon.

Personally, I and people I've spoken with use LLMs less and less because of how often they're wrong. The other day I asked ChatGPT about a specific built-in method in Java and it told me that it couldn't do one specific thing. I was already using it in that context so I pushed back and it said "Oh yeah you're right, sorry"

I feel like I can't trust anything it says. Mostly I use it to parse things I don't understand and then do my own verification that it's correct.

All that to say, from my perspective, they're losing some small amount of ground. The other side is that the big corps that run them don't want their golden gooses to be cooked. So they keep pushing them and shoving them into everything unnecessarily and we just have to eat it.

So I think it's a perception thing. The corps want us to think it's super useful so it continues to give them record profits. While the rest of us are slowly waking up to how useless they are if they will confidently tell us incorrect answers and are moving away from it.

So you may just be seeing sleezy marketing at work here.

LLMs would be better nomenclature than AI in this context.

LLMs are not factual databases. They are not trained to retrieve or produce factual statements.

LLMs give you the most likely word after some prior words. They are incredibly accurate at estimating the probabilities of the next word.

It is a weird accident that you can use auto-regressive next word prediction to make a chat bot. It's even weirder that you can ask the chatbot questions and give it requests and it appears to produce coherent answers and responses.

LLMs are best thought of as language generators (or "writers") not as repositories of knowledge and facts.

LLM chatbots were a happy and fascinating (and for some, very helpful) accident. But they were not designed to be "factually correct" they were designed to predict words.

People don't care about (or are willing to accept) the "wrong answers" because there are enough use cases for "writing" that don't require factual accuracy. (see for instance, the entire genre of fiction writing)

I would argue that it is precisely LLMs ability to escape the strict accuracy requirements of the rest of CS and just write/hallucinate some fiction that is actually what makes this tech fascinating and uniquely novel.

So, if you dealt with a person who knew all the vocabulary related to a field, and could make well-constructed sentences about that field, and sounded confident, it would almost always mean they had spent a lot of time studying that field. That tends to mean that, although they may occasionally make a mistake, they will usually be correct. People apply the same intuition to LLMs, and because it's not a person (and it's not intelligent), this intuition is way off.

There is, additionally, the fact that there is no easy (or even medium difficult) way to fix this aspect of LLM's, and it means that the choices are either: 1) ship it now anyway and hope people pay for it regardless 2) admit that this is a niche product, useful in certain situations but not for most

Option 1 means you get a lot of money (at least for a little while). Option 2 doesn't.

I haven't found a human that answers every single question correctly, either. You know whom to ask a question based off that person's domain of expertise. Well, AI's domain of expertise is everything (supposedly).

What gets difficult is evaluating the response, but let's not pretend that's any easier to do when interacting with a human. Experts give wrong answers all the time. It's generally other experts who point out wrong answers provided by one of their peers.

My solution? Query multiple LLMs. I'd like to have three so I can establish a quorum on an answer, but I only have two. If they agree then I'm reasonably confident the answer is correct. If they don't agree - well, that's where some digging is required.

To your point, nobody is expecting these systems to be infallible because I think we intuitively understand that nothing knows everything. Wouldn't be surprised if someone wrote a paper on this very topic.

> investors/founders/PMs don’t really care

Garry Tan from YC is a great example of this.

It's not that he doesn't care. It's just that he believes that the next model will be the one that fixes it. And companies that jump on board now can simply update their model and be in prime position. Similar to how Tesla FSD is always 2 weeks away from perfection and when it happens they will dominate the market.

And because companies are experimenting with how to apply AI these startups are making money. So investors jump in on the optimism.

The problem is that for many use cases e.g. AI agents, assistance, search, process automation etc. they very much do care about accuracy. And they are starting to run out of patience for the empty promises. So there is a reckoning coming for AI in the coming year or two and it will be brutal. Especially in this fundraising environment.

When I was a kid there was this new thing that came out called Wikipedia. I couldn't convince anyone it was useful though because they pointed out it was wrong sometimes. Eventually they came around though.

AI is like that right now. It's only right sometimes. You need to use judgement. Still useful though.

Your point is valid if you believe LLM/Generative AI is deterministic; it is not. It is inference-based, and thus it provides different answers even given the same input at times.

The question then becomes, "How wrong can it be and still be useful?" This depends on the use case. It is much harder for applications that require high deterministic output but less important for those that do not. So yes, it does provide wrong outputs, but it depends on what the output is and the tolerance for variation. In the context of Question and Answer, where there is only one right answer, it may seem wrong, but it could also provide the right answer in three different ways. Therefore, understanding your tolerance for variation is most important, in my humble opinion.

Lots of people care.

From a coding perspective, proper technical systems already have checks and balances (e.g. test cases) to catch bad code, and is something that's important to have regardless of generative AI usage.

From a creative/informational perspective, there are stories every day of hallucinations and the tech companies are correctly dunked on because of it. That's more product management error than AI error.

AI hallucination isn't a showstopper issue, it just has to be worked around.

I’m also curious about this. This morning I needed to generate summaries of news articles when I noticed Bing AI was inserting facts that weren’t present in the source article. (Not correct at all) It really hurts the potential of what AI could do if I have to double check everything it generates. We wouldn’t accept a spreadsheet program that required double checking with a calculator, why do LLMs get a pass?

The reason they don't care is the typical user doesn't notice. He asks the bot questions to which he does not know the answer, leaving him unable to detect when the bot answer is wrong.

My completely baseless theory is that there is an unbelievable amount of astroturfing happening in defense of this technology at a rate never seen before because of how badly people with capital want this to work and how "close" the tech is to achieving that.

If this is correct, then it's less of "people don't care" and more "the hype is louder than them."

That said: I, too, am completely perplexed by people within the tech community using LLMs heavily in making software while, unironically, saying that they have to keep an eye on it since it might produce incorrect work.

I work in AI product eng for a larger company. The honest answer is that with good RAG and few-shot prompting, we can consider actual incorrect output to be a serious and reproducible bug. This means that when we call LLMs in production, we get about the same wrong-answer rate as we do any other kind of product engineering bug.

Think of it as a system returning complex results after a search. Think of it as a synthetic search result. Think of it as a result where you still have to evaluate the source for reliability. Think of it as a junior engineer making mistakes. Think of it as a reason why you will have job security for complex tasks because the easy tasks can be done by dumber project managers. Think of it like you are now a senior engineer with a junior engineer doing the mundane stuff.

Do the mundane stuff in school/college/boot camp. Do the cool stuff at work.

In 1942, atom bombs didn't work at all. Does that mean nobody on the Manahattan Project cared? In my mind, when I hear that nobody on earth has achieved something, but a massive bubble of people are feverishly working on it, my conclusion is that a lot of people care a lot. I assume by how your post was phrased, you're using the term AI to mean LLM. You noticed that 100% of LLMs, every last one of them, in research labs and commercial businesses, hallucinate and give wrong answers when prompted for something very different from their training data. Thousands are this way. None at all exist that are the other way. A very peculiar property of an entire technology. But your conclusion wasn't that it's inherent property of LLMs (a statistics machine), or that we need to move beyond LLMs to achieve AGI. It wasn't that LLMs have a lot of powerful uses when kept inside the narrow scope of their training. Your conclusion was that, across the entire earth, without exception, "investors/founders/PMs don’t really care". I'm sorry, I am not following — perhaps if you elaborate more on each logical steps you took to get to that conclusion, we can shed more light on what you're missing.

So far this year the following companies have asked me for money for their new "AI" features:

  - slack   
  - github   
  - microsoft   
  - google    
  - atlassian   
  - notion   
  - clickup   
  - hubspot

So ask yourself: Who benefits from the hype? And who would benefit from a better general understanding of the flaws?

Hype. Every so many years some sort of semi-novel software gets invented/improved and some dude puts 100mg of 2c-b in his Huel shake and realizes that he’s invented the godhead. This dude invariably has buddies that do VC.

It’s the same reason why we heard about blockchain for years despite it having near zero practical uses

Isn't this just about unrealistic expectations? Lots of people derive lots of value from AI, but it isn't good for everything, and can't be trusted on its own for many things.

Because everyone incorrectly assumes there is intelligence at work and because people don't want to critically evaluate answers because that takes a lot of time to do.

I found LLMs to be quite useful, so the time saved is worth the effort to double check answers.

Granted it might have to do with how I use LLMs. If you just blindly ask a question you increase the chance of hallucinations. If you give a lengthy input, and the output is highly dependent on the input than you will get better results. Think email re-writing, summarizing, translation.

Think about this: when was the last time SV or the broader tech industry brought a revolutionary innovation to the consumer that improved their lives? The smartphone? Video streaming? I can't think of a single thing since. And those were both over a decade ago.

The tech industry is an environment composed almost entirely of companies running a loss to prove viability (and don't see that as ironic) to raise more funding from investors. AI is just the latest in a long series of empty hype to keep the gravy train running, last year it was VR, and it looks like at this point that the whole thing is teetering on a cliff. It's a bunch of MBAs scrambling for a sales pitch.

LLMs are useful. But "extremely lossy compression of documents with natural language lookup built in" doesn't sell endless subscriptions as well as "we created a mind." So they sell hype, which of course they cannot live up to because LLMs aren't minds.

Because 99% of people which have this issue ask a question to just a LLM. What you do is you add RAG to it, or you make it an agent which can retrieve information, and suddenly it's very accurate.

LLMs are language models, not magical information models with all information in the world somehow fit into several gigabytes. Use them right.

If people are paying for their product, why should they care?

As for why people are paying for a product that returns incorrect results, could be any number of reasons:

- People buy into the hype/marketing and actually think AI-thing is going to replace some part of their workflow

- People want to experiment and see how well it does at replacing part of their workflow

- Whatever the AI-thing does for a customer is not reliant on it being correct, therefore producing incorrect output simply doesn't matter

A good example would be my company's corporate IT AI bot that is effectively a very poor search engine for internal corporate wiki self-help articles on IT and HR related stuff. The actual IT/HR portal has a traditional search that, if you know the terms to search for, does a much better job. So most people ignore the AI, but I'm pretty sure we bought the engine from someone.

The LLMs are basically useless for anything requiring reasoning or to help solve real problems, but the big use-case is that LLMs are competing with google search. Google search is so polluted with low quality ads-first SEO garbage that the may-be-hallucinating-LLM is a more effective way to find some sorts of information than manually slogging through the garbage and possibly getting the wrong answer anyway.

I suppose that there is also some hope that the hallucination problem will erode as more effort/compute is poured into the training. There may need to be a paradigm shift though, the current structure around generating tokens based on probabilities seems like it will forever be a 'regurgitator'.

It can give wrong answers sometimes and still be useful. Also there are many tasks where it almost always gives correct answers, text to speech with function calling is 100x better now than it was 2 years ago. And in some spaces, correctness is a fuzzy concept anyway (creative spaces).

Of course most everyone cares, but the value proposition is high enough that people aren't going to hold off on using it until it is perfect.

Nonetheless, as with autopilot, you don't want to substitute paying attention with it. "Trust, but verify" as Reagan said.

Probably for the same reason that tech hasn't improved anything in a long time.

Tolerable Pizza delivery is ruined. The Internet is a walled wasteland now. Far too much "content" that doesn't need to exist. Everything is an ad.

None of our lives have been improved by software.

We built a correctness checker for LLM-generated SQL code for the military before LLMs were commercially available, it is going live soon on http://sql.ai . Some people do care about this problem, but it is hard to solve; even for SQL alone, this requires significant computer algebra, automated theorem proving, having to define what 'correct' even means, and much else etc.

To paraphrase Upton Sinclair, it is difficult to get a man to care about something, when his salary depends on his not caring about it.

A lot of money has poured into AI, money potentially well in excess of the return on investment over the next several years. The field, from investors to CEOs and downwards to developers, is in a state of collective suspension of disbelief. There is going to be a lot of people out of work when reality reasserts itself.

It's not apparent if you're not already an expert in the domain you're querying, so users trust its answers, especially because it's delivered with an air of confidence (until you challenge it).

Unfortunately that's good enough for a lot of people, especially when you don't actually care and just need an output to give to someone else (office jobs etc).

There are use cases where it doesn't matter, i.e. creative writing. Additionally, I don't think AI engineers have even figured out the path for LLMs to be hallucination free and extremely accurate. It's better to ship something that is not perfect (or even not great) now, and that way the industry gains experience and the tools slowly but surely get better.

It's because AI is useful enough despite its current limitations.

Developers work with what we have on the table, not what we may have years later.

It depends on how you look at it. The creative process isn't just a piece of code. It usually involves trying, tweaking, testing, and tuning before an optimal solution is reached. In both the real world and with software development, achieving a perfect result in a single shot is more the exception than the rule.

"Show me the incentive, and I'll show you the outcome."

There is a belief, cynical or otherwise, that AI will make (a very small number of) people extraordinarily wealthy. The drive to stuff it into every facet of the digital experience reflects this belief.

I’ll be honest, every day people tell me the magical things all these new AI tools can do and I try them and usually find the results useless.

Every AI chatbot I’ve ever interacted with has been unable to help me. The things I’ve had them write do usually pass the Turing Test, but are rarely even close to as good as what I could write myself. (I admit, being self-employed for a long time, I can just avoid a lot of busy work that many people cannot, so I may be missing lots of great use cases there. I never find myself having to write something that isn’t great and wanting to just get it over with. AI might be great if you do. )

I’ve been trying to use image/video creation to do lots of other things and I’ve not even come close to getting anything usable.

I appreciate certain things (ability to summarize, great voice to text transcription, etc.) but find a lot of it to be not very useful and overhyped in its current form.

(1) Some problems are probabilistic either in theory or practice. For instance there could be a sentiment analysis problem where the state of the art was 67% accuracy 5 years ago and with an LLM it is easy to get 85% accuracy. 100% accuracy is going to be impossible anyway because sometimes you really can't figure how somebody feels.

(2) It's a big topic that could be addressed in different ways but I'll boil it down to "people are sloppy" and that many people become uncomfortable with complex problems that have high stakes answers and will trade correctness for good vibes.

(3) LLMs are good at seducing people. To take an example, I know that I was born the same day as a famous baseball player who was also born exactly a year before an even more famous cricket player. I tried to get Microsoft's Copilot to recognize this situation but it struggled, thinking they were born on the same day or a day apart rather than a whole year. Once I laid it out explicitly and my own personal connection it had effusive praise and said I must be really happy to be connected to some sports legends like that, which I am. That kind of praise works on people.

(4) A lot of people think that fixing LLMs is going to be easy. For instance I'll point out that Copilot is completely unable to put items in orders that aren't excessively easy (like US states in reverse alphabetical order) and others will point out that Copilot could just write a Python program that does the sorting.

That's right and it is part of the answer, but it just puts off the problem. What's really irksome about Copilot's inability to sort is that it doesn't know that it can't sort, if you ask it what the probability is that it will sort a list in the right order it will tell you that it is very high. It's not so easy to know what is possible in terms of algorithms either, see

https://en.wikipedia.org/wiki/Collatz_conjecture

as evidence that it's (practically) impossible to completely understand very simple programs. See the book

https://en.wikipedia.org/wiki/G%C3%B6del,_Escher,_Bach

for interesting meditations on what a chatbot can and can't do. My take is that LLMs as we know them will reach an asymptote and not improve explosively with more investment, but who knows?

You'd consider it a bug because you can fix a bug. Doesn't seem like the tech's there yet, but it's still good enough to be useful.

Honestly I find LLMs to be a great tool, when using them right, and with the sufficient skills to know when they’re wrong. And for some problems, you don’t need a 100% right answer.

Earlier today I asked ChatGPT to give me a Go script to parse a Go codebase (making heavy use of Go AST libraries which I never use normally) and it gave me a 90% good solution which saved me a lot of time. To be clear the solution was non functional on its own, but it still saved me from doing exploration work and gave me a quick overview of the APIs I would need.

A few days ago it helped me generate code for some obscure AWS API using aws-sdk-go-v2. It was again almost fully working, and better than the examples I could find online.

I have examples like this every week. It’s not as amazing as some people say, but still pretty useful. I rejected AI stuff at first but don’t regret adding LLMs to my toolbelt.

In my usage it's competing with Google searches often those bring up a lot of nonsense. You have to filter it.

Another question, why does nobody care about the enormous and absurd energy cost to train and run models?

Does anyone know why this post no longer appears on HN.

It hasn't been flagged.

Because point often is not in providing a working solution. Instead it is selling just a solution. Just look at so many software projects in history. Did they provide correctly working solution? Or did they generate lot of billable work?

I didn't care too much at first because it seemed that the rate of improvement was sufficient to cover a multitude of sins. But I'm starting to, because it is becoming clear that progress hit an absolute brick wall with GPT4. If anything it has has gone backward since then.

Just today, ChatGPT4o screwed up a rudimentary arithmetic problem ( https://i.imgur.com/2jNXPBF.png ) that I'd swear the previous GPT4 model would have gotten right.

And then there's this shitshow: https://news.ycombinator.com/item?id=40894167 Which is still happening as of this morning, only now all my previous history is gone. Nothing left but links to other peoples' chats. If someone at OpenAI still cares what they are doing, it's not obvious.

I think this is the community that's an aberration. More often I see that "AI" has become synonymous with hallucinations and slop.

Blind optimism

Are you saying the emperor has no clothes!?

Everyone I know that doesn't work in tech is disappointed with AI assistants blurting out wrong answers. Investors will catch on soon.

"It is difficult to get a man to understand something when his NVIDIA shares depend upon his not understanding it"

IMHO because over the last two decades we have become so accustomed to lies and brokenness being the standard that it just doesn't matter to most people.

Move fast and break things, and don't pay anyone, but when you do that long enough and burn billions in VC money, you end up rich. Why does that work?

Why can someone like Trump lie and lie and lie and be convicted for felonies and turn up on the worst people list and nobody seems to care?

There are no more consequences. You break software, people don't care if it's the only thing available in the walled garden. You fuck up games, people don't care if you shove a TB worth of updates down their pipes later. You rugpull millions of dollars and walk out unscathed, as long as someone made a profit they will keep praising you.

You used to be actually shunned and driven out of the village for shit behavior. Not anymore. We find all kinds of ways to justify being terrible at stuff.

So along comes tech that costs us barely anything to use and produces meh results most of the time. That's amazing. It used to take thousands of talentless hacks to come up with all that mediocre wrong shit and they all wanted a paycheck. It's progress in a world where nothing means anything anymore.

It’s the grift. Doesn’t matter. Just slap AI on ANYTHING as fast as you can and hopefully no one will notice and hopefully someone else will have fixed it. Oh did I mention AI!