me.dm is one of the many independent Mastodon servers you can use to participate in the fediverse.
Ideas and information to deepen your understanding of the world. Run by the folks at Medium.

Administered by:

Server stats:

1.2K
active users

#fact

16 posts7 participants0 posts today

On "AI"...

I finally had an opportunity to personally evaluate the output of an #LLM that was asked to do a technical evaluation / summary of the state of an industry. A friend asked a modern model about a particular #technical field and asked it to summarize various aspects of it. It happened to involve supercomputing, which I have some knowledge of, and he asked me to look at the answer it gave.

It was a detailed writeup, a little over 1300 words. It had the structure of the kind of document he'd asked it to create, with appropriate sections, headers, with the information divided up sensibly, etc. It very much sounded like the type of thing you would expect a computer science #expert to respond with if asked the same question.

But some of the specific points in the answers were obviously wrong to me. Some I suspected were wrong. Some I didn't know. So I checked. I didn't check every fact and figure, but a decent number of them - and every single one I checked was flat-out wrong. It gave specific numbers or attributes for many aspects of the state of the #tech, and they were all wrong. Sometimes badly #wrong, and some stupendously wrong.

There were only a couple of things in the text that were obviously inconsistent, so unless you happened to already know something about the subject, there was almost no hint of how incorrectly it stated the facts. And of course it sounded supremely confident.

1/2

#AI#fact#incorrect