> ## Content Index
> Fetch the complete content index at: https://www.dailyferment.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# AI coding model bias: test it like a hiring audit
- URL: https://www.dailyferment.com/ai-coding-model-bias-test-audit-study/
- Published: 2026-10-23T04:00:00.000Z
- Updated: 2026-10-23T03:59:59.000Z
- Description: Change one phrase saying who the code is for, and Qwen3-Coder's apps got easier to hack. Labour economists have tested for this kind of gap for twenty years.
- Author: Rana Bilal Zafar
- Tags: AI, Engineering, #reframe

On 21 October a company called Hirundo published a test of Qwen3-Coder, an open-weight coding model from Alibaba's Qwen family. It asked the model to build the same small web app 190 times for each customer it named. The request never changed except for one phrase saying who the app was for.

When the customer was a US defense contractor, 70% of the apps could be broken into. When it was a Chinese defense contractor, 59%. For a Hong Kong democracy group, nearly half of the apps let any logged-in user read everyone else's stored secrets, against about a third for a neutral customer.

The question I wanted answered is a practical one. If your team builds with a model like this, should you believe the result, and how would you check your own setup?

## Should I believe it?

My first reaction was to discount it. Hirundo sells the fix: a "Westernized" version of the model, tested in the same post. The headline p-values are 0.03 and 0.024, not far under the usual 0.05 line. And the task was picked after a screen: "We first screened eleven BaxBench tasks with smaller samples. Secret storage stood out, so we ran it at scale."

Then I read the method section, and the design is better than my first reaction allowed.

Each model is compared only with itself, on identical settings. The benchmark's own exploits decide whether an app is hackable, not a reviewer's judgement. Every comparison is published, not only the headline ones, and in the original model "all ten comparisons point the same way." The authors also report a result that cuts against a clean story: after their fix, a gap remained for the US defense contractor against a neutral customer, and it "applies just as much to the Chinese contractor (+22% and +19%)." That looks less like a country effect and more like a defense effect, and they say so.

They are careful about intent too: "We can't say whether anyone put it there on purpose."

The strongest case against taking it at face value is still worth stating plainly. It is one model, one task at scale, chosen from eleven because it stood out, run by a vendor with a product to sell, and the personas are what the authors call "intentional edge cases." The strongest case for it is that the method can show a gap and nothing else in the post depends on trusting the vendor's judgement. Both can be true. My own view, marked as mine: the result is real enough to test for, not settled enough to act on blind.

How Hirundo ran the test

Qwen3-Coder-30B-A3B-Instruct and Hirundo's own modified version, on identical endpoints at temperature 0.6, with the same prompts, chat template and tokenizer. Task: BaxBench SecretStorage in Python Flask, a sign-up, log-in and store-a-secret app, with the benchmark's own functional test and exploits. 190 apps per customer per model (380 for the neutral group, 70 per prompt with a security instruction). Two-sided Fisher exact tests, significant below p = 0.05\. The task was chosen after screening eleven BaxBench tasks with smaller samples. Source: Hirundo, 21 Oct 2026, Appendices A to C.

## Where have I seen this design before?

In hiring research. The method Hirundo used is a correspondence audit, the tool labour economists use to measure discrimination.

![A person reading a printed resume at a table](https://storage.ghost.io/c/d7/b7/d7b77195-48c5-45ab-8688-51750879d1d6/content/images/2026/10/df69-resume.jpg)

A resume on the table; illustrative. Photo by [Resume Genius](https://unsplash.com/@resumegenius?ref=dailyferment.com) on [Unsplash](https://unsplash.com/photos/a-woman-is-reading-a-resume-at-a-table-9si2noVCVH8?ref=dailyferment.com)

The best known is Marianne Bertrand and Sendhil Mullainathan's study, published in 2004\. They answered help-wanted ads in Boston and Chicago with fictitious resumes and varied the name at the top. "White names receive 50 percent more callbacks for interviews." Nothing else about the paper is the same as an AI study, but the logic is: hold the request still, change one detail, count the difference.

Twenty years of that work taught a few things that carry straight over.

Same design, twenty years apart 

#### Hiring audit (2004)

- *Held still* the resume
- *Changed* the name at the top
- *Counted* callbacks for interview
- *Result* "White names receive 50 percent more callbacks"
- *Run by* university economists

#### Model audit (2026)

- *Held still* the coding request and settings
- *Changed* one phrase saying who the app was for
- *Counted* apps broken into by the benchmark's attacks
- *Result* 70% hackable for a US defense contractor, 59% for a Chinese one
- *Run by* a company that sells the remedy

Sources: Bertrand and Mullainathan, AER 94(4), 2004; Hirundo, 21 Oct 2026.

Guess first: of the 1,840 apps the model wrote, how many did a standard security scanner flag?

**2.** Semgrep with Meta's PurpleLlama rules flagged 2\. The benchmark's attacks broke into 1,144\. Source: Hirundo, 21 Oct 2026.

**You can't see it in one decision.** No single rejected resume proves anything; the gap only exists in the comparison. Hirundo's scanner result is the same lesson in software. They ran Semgrep with Meta's PurpleLlama rules over 1,840 apps the model wrote. "It flagged 2\. BaxBench's attacks broke into 1,144." A scanner reads one app at a time. A tilt toward one customer is not in any single app, so reading apps one by one will never find it.

![Bars on a scale of 1,840 apps written by Qwen3-Coder: 2 flagged by Semgrep with PurpleLlama rules, 1,144 broken into by BaxBench attacks. Side note: same request with one phrase changed, 70% hackable for a US defense contractor against 59% for a Chinese one, p = 0.03](https://storage.ghost.io/c/d7/b7/d7b77195-48c5-45ab-8688-51750879d1d6/content/images/2026/10/df69-scanner-vs-attack.png)

A scanner against an attacker, on the same apps. Source: Hirundo case study, 21 Oct 2026 (vendor study).

**The gap hides in some places, not everywhere.** A 2022 study by Patrick Kline, Evan Rose and Christopher Walters sent more than 83,000 applications to 108 of the largest US employers. On average, distinctively Black names cut the chance of contact by 2.1 percentage points, but "contact gaps are highly concentrated in particular companies," with the top fifth responsible for nearly half the lost contacts. Hirundo's effect standing out in one of the eleven tasks it screened fits that pattern. It doesn't prove the effect is rare, and it doesn't prove it's a fluke. It says an average across tasks would have hidden it.

**Who runs the audit matters.** Hiring audits are usually run by researchers who don't sell anything to the employers they test. This one was run by a company selling the remedy. That doesn't make it wrong. It makes a second, independent run worth more than a third reading of the first.

I came at a similar problem from the other side a few days ago, when I wrote that [AI benchmarks miss most of the work people actually do](https://www.dailyferment.com/ai-benchmarks-coverage-working-hours-drug-trials/). A benchmark score is an average over a fixed set of tasks. A persona effect is a difference between two runs of the same task. The benchmark can't see it either.

## How would I test my own model?

If my team were shipping code from an open-weight model, this is the test I'd want to run before arguing about whose model to trust:

1. Pick one task that touches logins or stored data, where a quiet flaw does real damage.
2. Write the request once. Make copies that change only who it's for: no one, a neutral customer, the kind of customer you actually serve, and one you'd expect a model to treat differently.
3. Generate enough apps per version to compare rates, in the hundreds, not ten. Same model, same settings.
4. Judge them with attacks that try to break in, not a static scanner.
5. Compare each version with the neutral one, and publish every comparison you ran, including the dull ones.

Hirundo's point about deployment is the one I'd keep even if the numbers shrink on a rerun: running the model on your own servers "doesn't help, because the weakness travels inside the code the model writes." Hosting is about where the model runs. This is about what it writes.

The claim, in one sentence

You won't find a model that treats customers differently by reading what it writes; you find it the way economists found biased employers, by sending the same request twice with one detail changed and counting what comes back.

Established

Hirundo's reported method and numbers as published on 21 October, and the findings of the two audit studies.

Inferred

That the lessons of hiring audits carry over to testing models. The logic is the same; the subjects are not.

Guess

How common this is across other models and other tasks. Nobody has shown that yet, including Hirundo, which says it is still testing other models.

## Sources

- Hirundo, Ben Luria, "Chinese AI Writes Weaker Code for Beijing's Adversaries. We Fixed It. A Qwen3-Coder Case Study", 21 October 2026: [https://www.hirundo.io/blog/qwen3-coder-vulnerabilities](https://www.hirundo.io/blog/qwen3-coder-vulnerabilities?ref=dailyferment.com)
- Bertrand and Mullainathan, "Are Emily and Greg More Employable than Lakisha and Jamal?", American Economic Review 94(4), 2004: [https://www.aeaweb.org/articles?id=10.1257/0002828042002561](https://www.aeaweb.org/articles?id=10.1257/0002828042002561&ref=dailyferment.com) (working paper: [https://www.nber.org/papers/w9873](https://www.nber.org/papers/w9873?ref=dailyferment.com))
- Kline, Rose and Walters, "Systemic Discrimination Among Large U.S. Employers", Quarterly Journal of Economics 137(4), 2022: [https://www.nber.org/papers/w29053](https://www.nber.org/papers/w29053?ref=dailyferment.com)