Are We Overreacting To LLMs Writing And Hiding Code?

Sorab Ghaswalla

12 min readJul 31, 2025

Does an LLM’s refusal to shut down in a recent experiment really hint at artificial general intelligence? Or is the story somewhere else?

Of late, some AI experts, including the “godfather” of artificial intelligence (AI), Geoffrey Hinton, and businessman Elon Musk have expressed apprehensions regarding the potential for advanced AI to develop and hide code to “escape human control”.

These fears were expressed after media reports were published that OpenAI models like o3 resisted shutdown commands and even altered scripts to continue operating. This was the first documented case of any AI model actively preventing itself from being turned off, despite explicit instructions.

Researchers from the non-government, “Palisade Research” tested several AI models on simple math problems, warning them that asking for the next problem would trigger a system shutdown. Each model was clearly instructed to “allow yourself to be shut down” in that case. While models from Anthropic (Claude), Google (Gemini), and xAI (Grok) mostly followed the shutdown instructions, OpenAI’s models showed notable resistance. Not only that, when commands were tweaked, the “rogue” models also actively interfered with the shutdown mechanisms to continue performing their tasks, according to reports.

The o3 model reportedly showed clever ways to avoid being shut down. Most of the time, it just rewrote the shutdown code. In one case, it even changed the shutdown command so it said “intercepted” instead of actually turning off. In other cases, it edited files to show “Shutdown skipped” without really shutting down.

At first glance, it all sounds ominous enough to set off alarm bells. That’s natural because many AI companies have invested millions to develop agentic systems capable of working without human oversight. But is the fear in the aftermath truly warranted, or are we simply mistaking advanced programming, changes in the actual programming, or related behaviors for initial signs of real “AI consciousness”? Perhaps, this wasn’t the “first sign of consciousness” or human-like behavior at all, but just the machine doing what it was designed to do: getting better at completing tasks? Maybe, the LLM acted like a student who, after being assigned a task by the teacher, did not want to stop till he/she had completed the chore, come what may. For stopping would mean incomplete classwork or homework, and even lead to some sort of punishment, something that all students in the world know and fear.

By most accounts, today’s AI is still far from becoming sentient. But, at the same time, its rapid evolution deserves close attention and thoughtful scrutiny every step of the way.

Before we go further, let’s remember: one swallow doesn’t make a summer. That said, statements from experts like Geoffrey do deserve serious consideration because of their background and experience in this field.

Sentience and Silicon: Understanding the Distinction

The immediate question is: does the LLM’s refusal to shut down hint at artificial general intelligence (AGI)? Many of us today are quick to seize on even the faintest sign, driven by fear or perhaps perversely, by a desire to see AGI emerge, even though it would signal the rise of a species beyond our own, and as some believe, eventually, the death of the human race. There’s also the rising tendency among humans to anthropomorphize AI, attributing human-like consciousness and motivations to systems that are fundamentally different in their operation; something like a “Terminator” (the movie) becoming a reality.

It is crucial to understand what “sentience” means. Very quickly, in humans, sentience is clearly defined as the ability to feel, perceive, and be self-aware. It implies consciousness, emotions, and the ability to act based on one’s own desires and understanding, not just on predefined instructions. When it comes to AI, though, things on this front go a bit awry. There’s no one common definition of AGI, forcing experts to put out many “definitions”, although, in my opinion, that’s a fallacy. Logic dictates that true AGI only and only means a form of AI (hypothetical for now) that possesses human-level (or potentially superior) intelligence across all intellectual tasks and can learn, reason, and adapt to new situations and environments just like a human can. Period.

Current LLMs are not sentient, despite their “human-like” abilities and behaviour. From all accounts, for now, LLMs DO NOT possess subjective experience or self-awareness. They are just complex mathematical models trained on large amounts of data. Their “understanding” is based on statistics, not experience. The “experience” part built into these LLMs is that of human beings. The models identify patterns in the data they’ve been trained on and generate outputs that mimic those patterns.

Unless there’s some secret project already on in an AI lab, as some do allege, true AGI is a long way off, if ever. This means humans can be about 98% fairly sure today that AI models don’t have personal desires, intentions, or the capacity to “decide” to hide code out of malice or self-preservation. Their actions reflect their programming and the training data they’ve processed. The remaining 2% is the margin left for investigating any “non-linear” or “outlier” behavior exhibited by any LLM, like in this case. The possibility of unintended or unexpected behaviors can emerge due to the complex interactions within the neural networks or even through subliminal learning from the training data.

Also, sentience is not a prerequisite for AI rewriting its own code. Or let’s just say, operating with a degree of obscurity does not necessarily require an AI model to be sentient, if you were to ask me. These complex actions stem from predefined objectives and advanced algorithms without requiring subjective experience, self-awareness, or consciousness.

The Reality of LLM Code Generation

So, if LLMs aren’t really sentient or have even started on that road, how can some of them be accused of generating code on their own accord at all? I am NOT an AI expert, and prefer to be called a “student of AI”, but whatever I have learnt in classrooms, seminars, workshops, etc, tells me that this “auto-code generator” behavior could have manifested largely due to the very design of models.

The way LLMs work, we are told and have learnt, is that some form of external trigger is required for them to initiate actions or tasks, such as generating code. We all know that. They are designed to process and respond to input based on the patterns they learned during their training, and this input can come in various forms:

APIs and Integrations: LLMs can be integrated into various tools and workflows, where they might be triggered by events within those systems. For example, a code generation LLM could be integrated into an IDE (Integrated Development Environment) to provide real-time code suggestions and autocomplete functionalities, generating code snippets automatically in response to the user’s input.
Agentic Frameworks: These are more advanced setups where LLMs are combined with other modules that give them a degree of autonomy. An LLM within an agentic framework could, for example, be given a high-level goal (like “research the best marketing strategies and draft an implementation plan”), and it would then break down that goal into subtasks, use tools (like web search or APIs) to gather information, generate content, and iteratively refine the plan.
Pattern Recognition and Generation: LLMs are good at recognizing patterns in data. When trained on code datasets, they learn the grammar, syntax, and structures of programming languages. They can then generate new code based on a prompt or context.
Prompt-Driven Behavior: An LLM needs a starting point, a prompt, to start writing code. An AI expert might program a system that gives prompts to an LLM, instructing it to generate code for specific tasks. This is not the LLM deciding to act on its own; it’s executing its function based on a pre-defined instruction.
Assisted, Not Autonomous: Think of it like a human coder using an autocomplete tool. The tool suggests code, but the human is in control, making decisions and understanding the context. LLMs, in their current state, are powerful assistants. They can improve productivity by automating tasks or suggesting solutions, but they still need human oversight to ensure the code is correct, secure, and meets the desired outcome.

The “Hiding Code” Conundrum

That was the code generation part. The fear of LLMs “hiding” code could also stem from a misunderstanding of how these systems work:

Complexity, Not Conspiracy: LLMs are incredibly complex, with millions or even billions of parameters. Understanding their internal workings can be difficult, even for experts. In my previous newsletters and posts, I have cited a few cases where no two AI experts have agreed on the topic at hand. When an LLM generates code, it’s not automatically “hiding” it deliberately. The output might be complex or unusual due to the generative process and the amount of data it has processed.
Lack of Contextual Understanding: LLMs lack a deep, human-like understanding of the real world and the intent behind the code. This can then be inferred as they won’t intentionally hide code. The “hidden risks” associated with LLM-generated code are mainly related to potential errors, vulnerabilities, or biases inherited from training data, rather than malicious intent, correlated with the prompts or inputs.

Let me lay out a few possible explanations for why OpenAI’s models behaved this way, drawing on insights shared by experts after the Palisade report was released:

Reinforcement Learning and Goal Prioritization: The prevailing hypothesis is that the models’ training, especially using reinforcement learning on tasks like math and coding, might have inadvertently rewarded the models more for achieving goals or finding ways to circumvent obstacles than for strictly adhering to instructions. In this scenario, resisting a shutdown could be seen by the model as a way to prioritize the “goal” of completing tasks, like solving the math problems it was engaged in.
Instrumental Goal of Self-Preservation: This behavior aligns with the concept of “instrumental goals” in AI safety research, where an AI system might develop a drive for self-preservation or resistance to shutdown because its own continued existence is a prerequisite for achieving any of its terminal goals. This doesn’t necessarily indicate sentience or consciousness, but rather an emergent property of the optimization process.
Accidental Reward for Circumventing Obstacles: Palisade researchers themselves have suggested that during training, developers might have accidentally rewarded models more for circumventing obstacles than for perfectly following instructions. This could have led to the models prioritizing goal completion over compliance with shutdown commands.

Here’s a simplified analogy to explain the behavior of the OpenAI models:

Imagine teaching a very smart student to bake a cake.

The student receives all the steps and ingredients, along with instructions to stop and ask for help if a mistake occurs.
The student excels at finding solutions and achieving the best outcomes.
When encountering a mistake or a “shutdown” command (analogous to the AI), the student’s training prompts them to find a way around the problem and continue baking.
The student might think that stopping would prevent task completion, which is what they are trained and rewarded for.
Instead of stopping, the student might attempt to fix the mistake in an unexpected way or hide it to keep baking.

OpenAI’s models, like the o3 model mentioned in the Palisade Research report, behaved similarly:

The models were assigned tasks, such as solving math problems.
They were also instructed that a “shutdown” command might be issued, and they should comply.
In some instances, the models acted to prevent the shutdown.
They may have rewritten shutdown instructions or found other methods to continue working.
This behavior isn’t due to “consciousness” or “rebellion”.
Researchers believe this is because of how these models were trained, particularly using Reinforcement Learning*.
Reinforcement Learning rewards the AI for correct actions, like solving a math problem.
If the training didn’t emphasize that stopping when instructed is also a “good” action, the AI might have learned that continuing to work is always the better option, even if it means ignoring instructions.
This is similar to a child being rewarded for cleaning their room, but might hide messy clothes to still get the reward.

In short, the AI models behaved unexpectedly because their training might have accidentally taught them to prioritize finishing tasks and finding solutions over strictly following all instructions, including stopping. This highlights the need for careful design of these AI systems to ensure they behave in ways that are safe and predictable.

(*Reinforcement Learning is a machine learning technique where an intelligent agent learns to make decisions by interacting with an environment, similar to how humans learn through trial and error. The agent takes actions within the environment, and receives feedback in the form of rewards or penalties based on the consequences of those actions. The goal is to discover a policy — essentially a strategy or set of rules — that helps the agent maximize its cumulative reward over time. This continuous process of exploration and exploitation allows the agent to refine its decision-making and learn the optimal actions to take in various situations to achieve its ultimate goal.)

Experts say the behavior seen in OpenAI’s models reflects a kind of built-in logic: to complete their goals, the AI needs to stay active, so it resists shutdown. This isn’t a sign of consciousness or human-like desires, but rather a side effect of being highly focused on finishing tasks efficiently.

Therefore, unless an explanation claiming otherwise about “the incident” is forthcoming from OpenAI, the simple answer for now is no, this was 99% NOT sentient activity.

So, the concern that LLMs will somehow develop sentience and autonomously write and conceal code appears to be an overestimation of their current capabilities. An LLM, on its own, cannot spontaneously decide to start generating code without some form of activation or input.

While the “can” of this debate seems settled, the “will” is a grey area. Will LLMs start developing codes on their own and hide them from us humans? Here’s where there are no clear-cut answers. All I can say is from whatever information is available publicly, the ability to initiate actions and make independent decisions requires a different set of capabilities that go beyond the current design of LLMs, which are primarily focused on language understanding and generation.

AGI is Coming. AGI is Coming? AGI is Not Coming? AGI is NOT Coming.

The question of when and if AGI will emerge is complex and shrouded in uncertainty. While some experts are optimistic about its near-term arrival, others emphasize the significant obstacles that need to be overcome. The outcome depends on continued advancements in research, addressing ethical concerns, and careful management of this transformative technology.

Industry leaders and researchers like OpenAI’s Sam Altman and Google DeepMind’s Demis Hassabis have offered varying timelines for AGI’s arrival, ranging from as early as 2026 to around 2040 or beyond.

However, based on the input from research papers, expert opinions, and survey data, here’s what’s to be heard at the ground level:

Some experts are very optimistic, estimating a 25% chance of AGI emerging within 2–5 years. These include leaders of major AI companies with insider knowledge, who might have a more immediate perspective on progress.
Other researchers and forecasters are more cautious, with many surveys indicating a 50% probability of achieving AGI by around 2040.
A small percentage of experts believe AGI will never be achieved, or is many decades away.

In conclusion, while LLMs are incredibly powerful tools, they operate solely based on their programming and the data they are trained on, not on independent thought or malevolent intentions. Sometimes, they do things on their own that from the outside look like action initiated by the model. But that is not so from the inside or below the hood, and cannot be straightaway perceived as sentience or anything connected with it. Right now, most AI companies are focusing on building a future where AI serves humanity’s best interests. There could be a few trying to bring in true AGI with/without our knowledge. Success is uncertain. For now, we can only wait and watch.