The author ran his own resume through HackerRank's open-sourced scorer three times and got 74, 88, and 90 — a 16-point spread with no human in the loop. He argues the repo reveals a single uncalibrated LLM call with no temperature pinning, no rubric anchoring, no multi-sample averaging, and no cross-check between the justification text and the numeric score.
The editorial argues the real story isn't that one ATS is broken but that HackerRank — a decade-old developer-assessment company with ML engineers — shipped slot-machine determinism on the path between candidates and recruiters. It frames this as the dirty secret of the AI-in-HR wave: pitch decks promise calibrated matching while the implementation is an unanchored prompt with a parsed integer.
HackerRank quietly open-sourced the resume-scoring guts of its applicant tracking system this week, and a developer writing at danunparsed.com decided to do the obvious thing: feed it his own resume, three times in a row, and see what fell out.
The scores were 74, 88, and 90. Same resume. Same job description. Same model. Same code path. Three runs, sixteen-point spread, no human in the loop — and somewhere in the world, a hiring manager is treating one of those numbers as a signal.
The author dug into the repo. The scoring pipeline is roughly what you'd guess if you've ever shipped an LLM feature on a deadline: pull resume text, pull JD text, stuff both into a prompt that asks for a 0–100 score plus a justification, parse the number out of the response, store it. No temperature pinning visible at the call site. No rubric anchoring with calibration examples. No multi-sample averaging. No deterministic post-processing pass. The justification text and the score aren't cross-checked against each other — the model is free to write "strong match across all required skills" and then return 74, and nothing in the pipeline notices.
The HN thread (827 points) landed on the obvious headline — "ATS scoring is broken" — but the more interesting story is what the source code reveals about how production LLM features actually get built inside companies that should know better.
HackerRank is not a scrappy startup. They've been in the developer-assessment business for over a decade. They have ML engineers. They run benchmarks for a living. And the resume scorer — the thing standing between candidates and a recruiter's inbox — is a single uncalibrated LLM call with the determinism characteristics of a slot machine.
This is the dirty secret of the current AI-in-HR wave: most of these products are a prompt, a JSON parser, and a Postgres column. The vendor pitch deck talks about "AI-powered candidate matching" and "96% accuracy on internal benchmarks." The repo shows a `gpt-4o-mini` call with `response_format: json_object` and a prompt that begins with "You are an expert technical recruiter." There is a gap between those two things and the gap is where careers go to die.
What's salvageable here is well-known to anyone who has shipped LLM-as-judge systems for evals: pin the temperature to 0, sample n times and take the median, anchor the rubric with 5–10 worked examples spanning the score range, and use structured outputs with field-level constraints rather than free-form JSON. None of this is research — it's the table stakes that turn a demo into a system, and HackerRank shipped without any of it. The cost of doing it right is roughly 3–5x the inference bill and a week of engineering. The cost of doing it wrong is that someone's career outcome is determined by which token the sampler picked first.
There's a regulatory angle here too that's worth naming. NYC Local Law 144 requires bias audits of automated employment decision tools. The EU AI Act classifies hiring tools as high-risk and demands documented risk management, data governance, and human oversight. A scorer with this much run-to-run variance can't pass either bar honestly — a 16-point spread on the same input means the "bias audit" is measuring the sampler more than the model. Open-sourcing the code didn't create the compliance problem; it just made the compliance problem visible.
If you're building anything where an LLM call produces a number that downstream systems treat as ground truth — risk scores, lead scores, content moderation tiers, code review verdicts — this is your wake-up call to audit your own pipeline against the same checklist.
First, measure your own variance. Pick ten representative inputs, run each through your scorer twenty times, and look at the standard deviation. If your p95 spread on identical inputs is more than 5% of your score range, you don't have a scoring system, you have a random number generator with opinions. Second, separate the "reasoning" call from the "scoring" call. Have the model produce the justification at temperature 0.3 with a structured rubric, then have a deterministic function — not another LLM — turn the rubric checkboxes into the final number. Third, log everything. The HackerRank repo doesn't appear to log the raw model output alongside the parsed score, which means when a candidate complains, there's nothing to audit.
For candidates on the other side of these systems: the actionable takeaway is bleaker. Your resume is being scored by something that scored itself differently three times in a row. Optimizing for one ATS score is like optimizing for one roll of the dice — the rational move is to apply more places, not to A/B test bullet points against a system that can't agree with itself. The keyword-stuffing era of resume advice was at least responding to a deterministic adversary. The LLM era is responding to noise.
The useful thing about open-sourcing a flawed system is that the next vendor in this space now has to answer for it. Greenhouse, Lever, Workday, and the dozen AI-recruiting startups raising at frothy valuations are all running something architecturally similar — they just haven't published the code. Expect a wave of "deterministic scoring" marketing copy in the next two quarters as competitors race to differentiate. Expect, also, a lawsuit: a 16-point variance on a protected-class-correlated input is going to be exhibit A in a disparate-impact case within eighteen months, and the discovery process is going to be ugly for everyone shipping a single uncalibrated LLM call into a hiring decision.
And this + the tendency for AI to "prefer" AI produced code + some other AI biased is why *this is most likely highly illegal to use in the EU due to violating anti discrimination laws in multiple ways.To be clear:- randomly filtering "too many" resumes is pretty much allowed (I
At this point we might as well adopt that joke where you blindly throw away half the resumes because you don't want to hire unlucky people.
> I fail 65% of the time. Same exact resume, different luck.As someone who’s run hiring pipelines for technical roles in the past few years, that’s actually a fantastic number. I objectively hate saying that, but it’s true.35% chance of elevating a technical individual to the next stage with no e
At my company someone has introduced an internal tool that should help understand and give a "score" to design documents from teams.Needless to say, this tool gives scores exactly like the article mentions. Same document, same LLM, same prompt, and different results. It becomes even more r
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.
An alarming number of people don't understand that LLMs work via purely stochastic processes, so I'm happy to see in-depth pieces like this. I'm looking for a job and maybe this is why it's so hard to get a callback these days: resumes are just dumped in some LLM black hole and n