So, how hard was it to get your AI to work?

I've been playing around with data extraction using OpenAI's structured outputs feature last week. Perhaps this is due to me still being fairly new to this world, but man, has it been an exercise in frustration. My general takeaway so far is that LLMs are fantastic for creative writing, and are a much harder beast to tame when you're looking for exact, reproducible answers. Even when those answers should be the most obvious ones. My engineering brain does not appreciate all the prompt engineering hacks, the hopeful JSON schema tweaking, the "let's try this other random thing just to see if it works" approach to making progress. The worst has been getting to an OK point with one of the fields, only to find out later that giving directions for another field blew away the earlier progress. I'm sure I must be missing something since there are products that are doing this, but so far I have little to show for hours spent other than growing knowledge of LLM quirks and subtle behaviors.

Wasn't too hard for me. It took about a month for a full prototype, another month to turn it into a full product, and another to refactor it into the existing codebase. But the team had trouble with it and there was a lot of stuff they thought was impossible that turned out to be easy. It's the kind of thing I've been doing since 2021, so it could be a matter of experience as well?

I find most people have trouble because they assume it's like programming - they talk to it like a robot and assume it uses robot speak. But it's trained on human language and works better when talked to like a person. It's more like talking to a gifted child. Some,like Claude, have been trained to use tags and such. There's also a lot of core stuff that people don't understand, things like when to fine tune, when to RAG, when to prompt engineer and use large context windows, the difference between models and how they're trained. It's useful to read the documentations; GPT and Claude function similarly at first glance, but the documentation tells you the differences.

Many mistakes are solved in the same way you would do it with a child. If they're getting the answer wrong, there might not be enough context or enough hints. People say that LLMs only get sarcasm when they know the source of the material, well, that's exactly how humans understand sarcasm as well. Instead of asking, "Solve this math question," a prompt like, "What's the best way to solve this math question?" might lead to better results. Things like Cursor works so much better vs Copilot despite the same models, because it's trained to think on the solution.

I would recommend doing hackathons to learn. Set a small goal, one that can be solved in a day or two. Solve it using AI. If you can't, you'll at least learn why not.

A few months ago I made some toy AI apps that had amazing results - when it worked. When it failed, it would get me so annoyed that I'm feeling burnt out and don't want to keep "fixing" issues that keep appearing.