This Is How to Tell if Writing Was Made by AI

SPEAKER_2 (0:02)

Bloomberg Audio Studios, Podcasts, Radio, News.

Joe Weisenthal (0:18)

Hello, and welcome to another episode of the Odd Lots Podcast. I'm Joe Weisenthal.

Tracy Alloway (0:23)

And I'm Tracy Alloway.

Joe Weisenthal (0:24)

So Tracy, you know, you ever come across some writing and you can't articulate exactly why, but you're like, I'm pretty sure AI wrote this. Does this happen too much?

Tracy Alloway (0:34)

So full disclosure, I haven't really thought about it that much. Yeah, because the thing is, I probably should think about it more, but there's a lot of bad writing out there, and I've become sort of a nerd to it. And I also think that, I don't know, trying to figure out whether or not something was generated by AI nowadays, if you actually dedicate a lot of your own time to doing that, that is a huge mental burden to be attempting. Especially, you and I are in the journalism industry. How many of the pitches do you think that we get from PRs right now are being generated by AI? Imagine if you're reading each one of those and trying to figure it out on a daily basis.

Joe Weisenthal (1:13)

You know what I suppose I think about it the most is someone will respond to a tweet. I'd be like, well, if this is a real person, then maybe this person deserves some engagement and ask a question or I want to respond.

If there's a person in the bot, then obviously I don't. That's where I'm like, you know what, I want to figure it out. I would like to know the answer. You know, I have a controversial view about AI writing, by the way, which is that it's pretty good. I mean, by and large, and I said this I think maybe in a recent episode, when you consider the fact that, I don't know, the majority of the population doesn't know where to put a comma. Within a sentence.

Tracy Alloway (1:47)

Well, this is my point, right?

Joe Weisenthal (1:48)

It's actually pretty good. I mean, one thing I'll say about AI is it never gets the placement of a comma wrong. On some level, it's perfect.

Tracy Alloway (1:54)

Did you do that? I think it was in the New York Times, the test.

Joe Weisenthal (1:57)

Yeah, I kind of hated that.

Tracy Alloway (1:58)

Okay, why?

Joe Weisenthal (1:59)

Well, because, I'll tell you why. First of all, there's only five examples. There's not very many. Two, it asked the reader, which do you prefer?

Tracy Alloway (2:05)

And they were different subjects as well.

Joe Weisenthal (2:07)

Yeah, and also I think most people probably treated that as, can you guess which one is a human? Because everyone wants to say they prefer the human. I didn't think it was like a great test. Nonetheless, look, not only is it often indistinguishable, not often, is it often fine writing. Sometimes AI could come up with a really remarkable turn of phrase, but I still by and large don't like it. You read like a thing, especially a long text that's AI, and it's like, even if you can't articulate it, it's like, this feels AI. It has a certain sickliness, sweetness to it that is often annoying, it's so annoying.

Tracy Alloway (2:38)

What I notice about it is it doesn't do style very well, right? So if you ask it to write something in the style of a writer, if you choose anything other than something really obvious, like Shakespeare, it really, it suffers. But the text that it actually outputs is pretty clear, right? Like for basic understanding, it's probably better than a lot of what's on the Internet.

Joe Weisenthal (3:01)

The real people who are going to have to worry about this are like teachers, obviously, the universities and-

Tracy Alloway (3:07)

Lawyers.

Joe Weisenthal (3:07)

Student lawyers and maybe law, it's fine. But there are some times it's like, okay, did someone write this or not? There has to be, it'd be nice if we could know the answer.

Tracy Alloway (3:16)

Well, the other thing that's starting to happen is, have you seen any books out there that actually come with a disclosure or a disclaimer that say this book has been written only by humans, no AI used at all? I saw that for the first time on a book that we actually read for an Odd Lots episode. I don't think it's come out yet, but that threw me.

Joe Weisenthal (3:34)

Yeah, no, it's more and more. Anyway, as we enter a world at which the vast majority, if not already, of words written are written by AI, there's going to be interest in this question of whether we know. Anyway, there's this company called Pangram Labs, and they have a little thing and you can pay for it, but also a free service where you can drop like a text in, and it'll say the odds that is written by human or AI. I'm pretty impressed by it. I did some samples of my own writing and then AI outputs, it got them all right, but then I did some like further, like I tried to stump it to see if like, so what I did was I took a piece of AI writing and then I had it translated into Chinese, and then I had it translate that into high Chinese, so it's like, okay, imagine this is being written by a more formal register, and then I had that translated into Hebrew, and then I had that translated into English, so the original thing through this series of AI telephone, through various translations, and then I put that output back into Pangram, and it got that right. It said it was AI, so even after a series of sort of transformations designed to obfuscate the original style of the piece, to see if eventually it would emerge in something else, so I was pretty impressed. It seems to work, and I think that's interesting for a couple of reasons, which is maybe there is something that you can just tell, but two, it sort of worries me because there have been articles, and they'll say like, this is written by AI, and I think one of my big fears would be that I write something.