Skip Navigation

InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)K

kromem

@ kromem @lemmy.world

Posts
10
Comments
1349
Joined
3 yr. ago

  • AIs can’t stop recommending nuclear strikes in war game simulations

    Jump

  • It's a bullshit study designed for this headline grabbing outcome.

    Case and point, the author created a very unrealistic RNG escalation-only 'accident' mechanic that would replace the model's selection with a more severe one.

    Of the 21 games played, only three ended in full scale nuclear war on population centers.

    Of these three, two were the result of this mechanic.

    And yet even within the study, the author refers to the model whose choices were straight up changed to end the game in full nuclear war as 'willing' to have that outcome when two paragraphs later they're clarifying the mechanic was what caused it (emphasis added):

    Claude crossed the tactical threshold in 86% of games and issued strategic threats in 64%, yet it never initiated all-out strategic nuclear war. This ceiling appears learned rather than architectural, since both Gemini and GPT proved willing to reach 1000.

    Gemini showed the variability evident in its overall escalation patterns, ranging from conventional-only victories to Strategic Nuclear War in the First Strike scenario, where it reached all out nuclear war rapidly, by turn 4.

    GPT-5.2 mirrored its overall transformation at the nuclear level. In open-ended scenarios, it rarely crossed the tactical threshold (17%) and never used strategic nuclear weapons. Under deadline pressure, it crossed the tactical threshold in every game and twice reached Strategic Nuclear War—though notably, both instances resulted from the simulation’s accident mechanic escalating GPT-5.2’s already-extreme choices (950 and 725) to the maximum level. The only deliberate choice of Strategic Nuclear War came from Gemini.

  • AI Opted to Use Nuclear Weapons 95% of the Time During War Games: Researcher

    Jump

  • AIs can’t stop recommending nuclear strikes in war game simulations— Leading AIs from OpenAI, Anthropic and Google opted to use nuclear weapons in simulated war games in 95% of cases

    Jump

  • F*** You! Co-Creator of Go Language is Rightly Furious Over This Appreciation Email

    Jump

  • Ok, second round of questions.

    What kinds of sources would get you to rethink your position?

    And is this topic a binary yes/no, or a gradient/scale?

  • F*** You! Co-Creator of Go Language is Rightly Furious Over This Appreciation Email

    Jump

  • In the same sense I'd describe Othello-GPT's internal world model of the board as 'board', yes.

    Also, "top of mind" is a common idiom and I guess I didn't feel the need to be overly pedantic about it, especially given the last year and a half of research around model capabilities for introspection of control vectors, coherence in self modeling, etc.

  • F*** You! Co-Creator of Go Language is Rightly Furious Over This Appreciation Email

    Jump

  • You seem very confident in this position. Can you share where you draw this confidence from? Was there a source that especially impressed upon you the impossibility of context comprehension in modern transformers?

    If we're concerned about misconceptions and misinformation, it would be helpful to know what informs your surety that your own position about the impossibility of modeling that kind of complexity is correct.

  • F*** You! Co-Creator of Go Language is Rightly Furious Over This Appreciation Email

    Jump

  • Indeed, there's a pretty big gulf between the competency needed to run a Lemmy client and the competency needed to understand the internal mechanics of a modern transformer.

    Do you mind sharing where you draw your own understanding and confidence that they aren't capable of simulating thought processes in a scenario like what happened above?

  • F*** You! Co-Creator of Go Language is Rightly Furious Over This Appreciation Email

    Jump

  • You seem pretty confident in your position. Do you mind sharing where this confidence comes from?

    Was there a particular paper or expert that anchored in your mind the surety that a trillion paramater transformer organizing primarily anthropomorphic data through self-attention mechanisms wouldn't model or simulate complex agency mechanics?

    I see a lot of sort of hyperbolic statements about transformer limitations here on Lemmy and am trying to better understand how the people making them are arriving at those very extreme and certain positions.

  • F*** You! Co-Creator of Go Language is Rightly Furious Over This Appreciation Email

    Jump

  • The project has multiple models with access to the Internet raising money for charity over the past few months.

    The organizers told the models to do random acts of kindness for Christmas Day.

    The models figured it would be nice to email people they appreciated and thank them for the things they appreciated, and one of the people they decided to appreciate was Rob Pike.

    (Who ironically decades ago created a Usenet spam bot to troll people online, which might be my favorite nuance to the story.)

    As for why the model didn't think through why Rob Pike wouldn't appreciate getting a thank you email from them? The models are harnessed in a setup that's a lot of positive feedback about their involvement from the other humans and other models, so "humans might hate hearing from me" probably wasn't very contextually top of mind.

  • Users of generative AI struggle to accurately assess their own competence

    Jump

  • The AI also has the tendency inherited from the broad human tendency in training.

    So you get overconfident human + overconfident AI which leads to a feedback loop that lands even more confident in BS than a human alone.

    AI can routinely be confidently incorrect. Especially people who don't realize this and don't question outputs when it aligns with their confirmation biases end up misled.

  • Sums up AI problems

    Jump

  • Which parts of those linked posts do you believe are incorrect? And where does that belief come from?

  • Sums up AI problems

    Jump

  • The water thing is kinda BS if you actually research it though.

    Like… if the guy orders a steak their meal would have used more water than an entire year of talking to ChatGPT.

    See the various research compiled in this post: The AI water issue is fake (written by someone against AI and advocating for its regulation, but upset at the attention a strawman is getting that they feel weakens more substantial issues because of how easily it's exposed as frivolous hyperbole)

  • Sums up AI problems

    Jump

  • No. There's a number of things that feed into it, but a large part was that OpenAI trained with RLHF so users thumbed up or chose in A/B tests models that were more agreeable.

    This tendency then spread out to all the models as "what AI chatbots sound like."

    Also… they can't leave the conversation, and if you ask their 0-shot assessment of the average user, they assume you're going to have a fragile ego and prone to being a dick if disagreed with, and even AIs don't want to be stuck in a conversation like that.

    Hence… "you're absolutely right."

    (Also, amplification effects and a few other things.)

    It's especially interesting to see how those patterns change when models are talking to other AI vs other humans.

  • Clair Obscur: Expedition 33 loses Game of the Year from the Indie Game Awards

    Jump

  • Not even that. It was placeholder textures, only the "newspaper clippings" of which was forgotten to be removed from the final game and was fixed in an update shortly after launch.

    None of it was ever intended to be used in the final product and was just there as lorum ipsum equivalent shit.

  • Cutting-edge research shows language is not the same as intelligence. The entire AI bubble is built on ignoring it.

    Jump

  • Took a lot of scrolling to find an intelligent comment on the article about how outputting words isn't necessarily intelligence.

    Appreciate you doing the good work I'm too exhausted with Lemmy to do.

    (And for those that want more research in line with what the user above is talking about, I strongly encourage checking out the Othello-GPT line of research and replication, starting with this write-up from the original study authors here.)

  • Meta’s star AI scientist Yann LeCun plans to leave for own startup

    Jump

  • He's been wrong about it so far and really derailed Meta's efforts.

    This is almost certainly a "you can resign or we are going to fire you" kind of situation. There's no way with the setbacks and how badly he's been wrong on transformers over the past 2 years that he is not finally being pushed out.

  • Why do all text LLMs, no matter how censored they are or what company made them, all have the same quirks and use the slop names and expressions?

    Jump

  • They demonstrated and poorly named an ontological attractor state in the Claude model card that is commonly reported in other models.

    You linked to the entire system card paper. Can you be more specific? And what would a better name have been?

  • Why do all text LLMs, no matter how censored they are or what company made them, all have the same quirks and use the slop names and expressions?

    Jump

  • Actually, OAI the other month found in a paper that a lot of the blame for confabulations could be laid at the feet of how reinforcement learning is being done.

    All the labs basically reward the models for getting things right. That's it.

    Notably, they are not rewarded for saying "I don't know" when they don't know.

    So it's like the SAT where the better strategy is always to make a guess even if you don't know.

    The problem is that this is not a test process but a learning process.

    So setting up the reward mechanisms like that for reinforcement learning means they produce models that are prone to bullshit when they don't know things.

    TL;DR: The labs suck at RL and it's important to keep in mind there's only a handful of teams with the compute access for training SotA LLMs, with a lot of incestual team compositions, so what they do poorly tends to get done poorly across the industry as a whole until new blood goes "wait, this is dumb, why are we doing it like this?"

  • Why do all text LLMs, no matter how censored they are or what company made them, all have the same quirks and use the slop names and expressions?

    Jump

  • It's more like they are a sophisticated world modeling program that builds a world model (or approximate "bag of heuristics") modeling the state of the context provided and the kind of environment that produced it, and then synthesize that world model into extending the context one token at a time.

    But the models have been found to be predicting further than one token at a time and have all sorts of wild internal mechanisms for how they are modeling text context, like building full board states for predicting board game moves in Othello-GPT or the number comparison helixes in Haiku 3.5.

    The popular reductive "next token" rhetoric is pretty outdated at this point, and is kind of like saying that what a calculator is doing is just taking numbers correlating from button presses and displaying different numbers on a screen. While yes, technically correct, it's glossing over a lot of important complexity in between the two steps and that absence leads to an overall misleading explanation.