What is your most disturbing moment with generative AI?

How much I've come to trust the answers, responses, and information it feeds me for my increasingly frequent queries.

Doing a project to migrate from one LMS to another, I put ChatGPT in the middle to fix various mistakes in the content, add alt text for images, transcribe audio, etc.

When importing the content back into Moodle, I come to find that one of the transcripts is 30k+ characters, and errored out on import.

For whatever reason, it got stuck in a loop that started like this:

"And since the dawn of time, wow time, its so important, time is so important. What is time, time is so important, theres not enough time, time is so important time"... repeat "time is so important" until token limit.

This really gave me a bit of existential dread.

Nothing is disturbing.

I find myself occasionally wondering if 8.11 is in fact greater than 8.9

Deep fakes have always been horrible. The idea that someone - anyone - can take your image and represent you in ways that can ruin your reputation, is appalling. For example, revenge porn.

Having your likeness used to express an opinion that is the opposite of your own is nasty too. You can produce the kind of thing that has no courtesy, no grace, no kindness or care for the people around you.

The mass extraction and substitution of art has also caused a lot of unnecessary grief. Instead of AI enabling us to pursue creative work… it’s producing slop and making it harder for newbies to develop their craft. And making a lot of people anxious, fearful, and angry.

And finally of course astroturfing, phishing, that kind of thing has in principle become a lot more sophisticated.

It unnerves me that people can pull this capital lever against each other in ways that don’t obviously advance the common good.