What does that have to do with copyright infringement though? And how would it be illegal?

I could totally start a website, maybe call it "New York Stories", read every news article about New York (I'd get a lot of them from NYT) and then working off my own memory, not copy/pasting the text write/publish the same story. That would not be copyright infringement. In fact the NYT themselves do it all the time, publishing things that were originally reported elsewhere. You're allowed to do that as long as you don't produce exact copies.

LLMs generally don't do exact copies of anything - they're just not exact at all. If you ask the AI exactly the same question a thousand times, you won't get precisely the same exact response twice.

For example asking "What should I eat in New York?" gave me:

New York City offers a vast array of culinary experiences, reflecting its diverse culture. Here's a mix of iconic eats and modern must-tries:

1. Pizza: New York-style pizza is famous worldwide. Visit classic spots like Di Fara, Lombardi's, or newer favorites like Lucali for a slice of this iconic dish.
2. Bagels and Lox: New York bagels are [... several more paragraphs ...]

Then the same question again:

New York City is a melting pot of cultures, making it one of the best places in the world to explore a wide variety of cuisines. Here are some iconic foods and places to consider when deciding what to eat in New York:

1. Pizza: New York-style pizza is famous worldwide. Look for places with a long history and great reviews, such as Lombardi's (America's first pizzeria), Di Fara Pizza, or Joe's Pizza for a classic slice.

2. Bagels: Another iconic New York [...]

It's approximately the same response but not exactly the same and even recommends different restaurants.

Being exact matters when it comes to copyright infringement. Like OpenAI I'm genuinely curious how they got it to output a verbatim copy of anything. That's highly unusual behaviour and if they had reported it to the company I'm sure it would have been fixed. Just like if someone posted an exact copy of an NYT article in this community it would be removed and nobody would be taken to court.