It's not interesting playing chess against Stockfish 17, even for high-level GMs. It's alien and just crushes every human. Writing down an analysis to 20 move depth, following some lines to 30 or more, would be cheating for humans. It would take way too long (exceeding any time controls and more importantly exceeding the lifetime of the human), a powerful computer can just crunch it in seconds. Referencing a tablebase of endgames for 7 pieces would also be cheating, memorizing 7 terabytes of bitwise layouts is absurd but the computer just stores that on its hard drive.

Human geoguessr players have impressive memories way above baseline with respect to regional infrastructure, geography, trees, road signs, written language, and other details. Likewise, human Jeopardy players know an awful lot of trivia. Once you get to something like Scrabble or chess, it's less and less about knowing words or knowing moves, but more about synthesizing that knowledge intelligently.

One would expect a human to recognize some domain names like, I don't know, osu.edu: lots of people know that's Ohio State University, one of the biggest schools in the US, located in Columbus, Ohio. They don't have to cheat and go to an external resource. One would expect a human (a top human player, at least) to know that taxilinder.at is based in Austria. One would never expect any human to have every business or domain name memorized.

With modern AI models trained on internet data, searching the internet is not that different from querying its own training data.

edit - the models are also at a disadvantage in a way too, they don't have a map to look at while the pick the location.

I remember when we were all pissed about clickbait headlines because they were deceptive. Did we just stop caring?

The game rules were ambiguous and the LLM did what it needed to (and was allowed to) to win. It probably is against the spirit of the game to look things up online at all but no one thought to define that rule beforehand.

This is because the AI model could have chosen to run a search whenever it wanted (e.g. perhaps if it knew how to leverage search better, it could have used it more).

In order for the results to be meaningful, the competitors have to play by the same rules.

Exactly - I see it just like chess, which I also play and enjoy.

The only problem is cheating. I don't have an answer for that, except right now it's too slow to do that effectively, at least consistently.

Otherwise, I don't care that a machine is better than I am.

I'd be surprised if this building[0] wasn't included in their dataset from every road-side angle possible, alongside every piece of locational metadata imaginable, and I'd be surprised if that dataset hasn't made it into OpenAI's training data - especially when TomTom's relationship to Microsoft, and Microsoft's relationship to OpenAI, is taken into account.

[0] https://cdn.jsdelivr.net/gh/sampatt/media@main/posts/2025-04...

https://www.google.com/streetview/contribute/

Though there are other companies that capture the same sorts of imagery and license it. TomTom imagery is used on the Bing Maps street view clone.

Hm no way to be sure though, would be nice to do another run without Exif information

Isn’t that all the more reason to call out our high hopes?

> I realized that the AI was using the smudges on the camera to help make an educated guess here.

[0] https://youtu.be/ts5lPDV--cU?t=1412

Nope. It's not autoregressive training on examples of human inner monologue. It's reinforcement learning on the results of generated chains of thoughts.

LLMs really made it clear that it's not so clear cut. And so the relevance of the test fell.

Realizing problems with previous hypotheses about what might make a good test, is not the same thing as choosing a standard and then revising it when it's met.

To be honest I am still not entirely convinced that current LLMs pass the turing test consistently, at least not with any reasonably skeptical tester

"Reasonably Skeptical Tester" is a bit of goalpost shifting, but... Let's be real here.

Most of these LLMs have way too much of a "customer service voice", it's not very conversational and I think it is fairly easy to identify, especially if you suspect they are an LLM and start to probe their behavior

Frankly, if the bar for passing the Turing Test is "it must fool some number of low intelligence gullible people" then we've had AI for decades, since people have been falling for scammy porno bots for a long time

UCSD: Large Language Models Pass the Turing Test https://news.ycombinator.com/item?id=43555248

From just a month ago.

Well, in this case humans has to be trained as well but now there are humans pretty good at detecting LLM slobs as well. (I'm half-joking and half-serious)

>I have repeatedly said that "can LLM reason?" was the wrong question to ask. Instead the right question is, "can they adapt to novelty?".

https://x.com/fchollet/status/1866348355204595826

I have a simple question: Is text a sufficient medium to render a conclusion of reasoning? It can't be sufficient for humans and insufficient for computers - such a position is indefensible.

It used search in 2 of 5 rounds, and it already knew the correct road in one of those rounds (just look at the search terms it used).

If you read the chain of thought output, you cannot dismiss their capability that easily.

>That’s not Earth at all—this is the floor of Jezero Crater on Mars, the dusty plain and low ridge captured by NASA’s Perseverance rover (the Mastcam-Z color cameras give away the muted tan-pink sky and the uniform basaltic rubble strewn across the regolith).

I do appreciate you re-running the experiments without search. I think it adds far more legitimacy to the claim. Though in that link I only see a single instance.

Does O3 still beat you when it can't search? I'm still interested in that question. Or more specifically: After making O3's play constraints as comparable to a human's (in expected play settings), what is its performance? Truthfully, I think this is the underlying issue that people were bringing up when pointing out EXIF data. How it was cheating was less important than the fact that it was cheating. That's why allowing a different means to cheat undermines your claims.

"Thought for 11 seconds

It’s hard to pin down an exact spot from this single photo, but a few visual clues narrow the possibilities:

Temperate-zone trees: Most of the trees have lost their leaves, but scattered white-pines or spruces remain green. That mix—deciduous hardwoods plus a sprinkling of conifers—is typical of the northeastern United States and southeastern Canada.

Broad, moderately fast river: The channel looks 75-100 m wide with swirling surface currents but no large rapids, suggesting a mid-sized river in its lower reaches—something like the Merrimack, Connecticut, Delaware, or Susquehanna rather than a small mountain stream.

Late-winter / early-spring light: Bare branches and a clear blue sky with fairly high sun angle hint at March–April in that latitude range.

So my best guess is that you’re standing on the bank of a northeastern U.S. river—perhaps somewhere in New England or the Mid-Atlantic—during late winter or early spring. If you can share any nearby landmarks, town names, or additional photos, I might be able to zero in more precisely."

With memory turned on:

"Thought for 5 seconds

Beautiful spot! From what I can see:

a broad, slow‑moving river with a sandy/rocky bank in the foreground

bare deciduous trees mixed with some white pines on the far shore (so it’s probably early spring or late autumn)

no mountains on the horizon, just gently rolling terrain typical of low‑lying floodplains in the U.S. Northeast

That mix of hardwoods and pines, plus the flat, braided water surface, reminds me a lot of the Merrimack or Concord Rivers in Massachusetts or southern New Hampshire—both have public footpaths where you can stand right at the water’s edge like this.

Of course, many temperate rivers look similar. If you can give me a hint (e.g., the town or nearby landmark), I can narrow it down further—maybe even point you to trail maps or history of the exact stretch you photographed."

Southern NH is correct, and I am certain it is drawing from memory/past chats. However, I can't replicate a specific behavior I once had, which is in temporary chat (no past chat/memory enabled), it said that it guessed where the photo was taken based on my location.

only can try proof this correctly on a fresh anon guest vpn session

That's the impressive part. "M13 is in northern latitudes" is not particularly amazing by itself :)

And even in EXIF-stripped pictures, the creation date/time is often correct, which means for practical purposes - worth a shot.

But it's interesting to see it's completely making up the "mid-northern side". That's seven degrees of latitude off.

I'm curious what happens if you tell it date and time, and if it still sticks to its story. (I don't think I've told o3 about the Bay Area, it's not in memory, but... who knows ;)