Argues that HackerRank's open-sourced scorer — a single LLM call with no pinned temperature, no seed, and no rubric anchoring — is architecturally identical to nearly every AI screening, code review, and moderation product shipped in the last 18 months. The only difference is that competitors keep their code closed so customers can't observe the variance by hitting the same endpoint repeatedly.
Contends that ATS scorers derived their power from secrecy — candidates had to optimize blindly against guessed heuristics. Publishing the code turns the scorer into a fitness function candidates can run locally, generating thousands of resume variants and shipping the highest-scoring one against the live model.
Ran his own resume through the public repo three times and got scores of 74, 88, and 90 — demonstrating with a trivial reproduction that the same candidate can be ranked very differently on identical input. Frames this as evidence that the scoring function HackerRank sells to enterprises is essentially a coin flip dressed up as a quantitative signal.
HackerRank quietly open-sourced the resume-scoring service behind its applicant tracking integration. Within hours, a developer at danunparsed.com fed his own resume through the public repo three times and posted the results: 74, 88, 90 — a 16-point spread on identical input. The post hit 935 on Hacker News and stayed there.
The repo itself explains the variance without trying to. It's a single LLM call against a generic prompt. No temperature pinned to zero. No seed. No rubric anchoring — the model is asked to score "fit" on a 0–100 scale with no calibrated examples, no decomposed criteria, no per-axis sub-scores that get summed into a deterministic total. The output is whatever logit-sampled token the model produces that turn.
This is the architecture of essentially every LLM-as-judge system shipped in the last eighteen months, and HackerRank's only mistake was letting you see it. Most vendors selling AI screening, AI code review, AI content moderation, and AI fraud triage are running variants of this exact loop. The difference is they don't publish the code, and their customers don't have the access pattern to call the same endpoint three times on the same input.
The immediate story — "resume scorer is non-deterministic" — is the boring read. The interesting read is what changes now that the source is public.
Until this week, an ATS scorer was an oracle: opaque, one-shot, with the scoring function held by the platform. Candidates optimized blindly against guessed heuristics (keyword stuffing, white-text injection, the usual). The asymmetry was the product. HackerRank just gave away the asymmetry. Anyone with a laptop can now run the scorer locally, generate a thousand variants of their resume, A/B them against the live model, and ship the highest-scoring version. The scorer was load-bearing precisely because it was secret; the moment the code is public, it becomes a fitness function in the candidate's gradient descent loop, not the employer's.
This is the same pattern that broke SEO. Google's algorithm was a defensible moat as long as it was a black box. The day enough of it leaked — through patents, papers, and Search Console signals — the equilibrium shifted from "write good content" to "write content that scores well." Open-source ATS is the equivalent move for hiring, except the gradient is sharper because the input space is one PDF, not a whole website.
The second-order story is uglier. The 74/88/90 spread means HackerRank-screened candidates have been getting different scores on different days for reasons unrelated to the resume. Run the same person through the same ATS three times and you can rank them above the cutoff, at the cutoff, or below it — and only one of those outcomes results in a phone screen. Multiply that by every resume the system has processed and the noise floor on "who got the interview" is enormous. Whether that's legally actionable under EEOC disparate-impact analysis is now an empirical question, because the eval harness is reproducible.
The third-order story is the one developers should actually care about. The HackerRank scorer is a textbook "LLM-as-judge" deployment, and LLM-as-judge is now the default eval pattern for most AI products. RAG quality scoring, agent trajectory eval, code review bots, content moderation triage, fraud risk on signups, customer support routing — they all reach for the same primitive: ask a model to grade a thing on a scale. Most do it with the same defaults the HackerRank repo just made visible. Temperature unset (so ~1.0). No seed. No structured output schema forcing a rubric. No few-shot calibration. No reliability check across multiple samples. The HackerRank repo is, from this week onward, the reference implementation for how not to build one of these.
If you ship an LLM evaluator in production, three concrete things change today.
First, stop using float scores as your judge output. A model asked for an integer 0–100 will produce noise proportional to the granularity. The fix is to decompose into discrete axes ("Does the candidate have 3+ years of Python? Y/N"), score each, and sum deterministically in code. Make the LLM do classification, make your code do the math. This collapses most of the variance because the model isn't sampling over a continuous score space anymore.
Second, set temperature to 0 and pin a seed, then run n=5 samples and take the median or majority vote. Temperature 0 isn't actually deterministic on most hosted APIs (the providers batch and the underlying kernels are non-associative on GPUs), but it dramatically tightens the distribution. Multi-sample voting catches the residual drift. If five identical calls give you 74/88/90/82/85, your evaluator isn't an evaluator — it's a random number generator with a writing degree.
Third, build a reliability test into CI. Pick 50 representative inputs, run each through your judge 10 times, and assert that the standard deviation of scores is below some threshold. The HackerRank repo would fail this test on the first input. If your eval system would too, you don't have an evaluator, you have a confident-sounding coinflip — and that distinction matters more when the decision is "interview this person" than when it's "summarize this email."
The meta-move here is that HackerRank, intentionally or not, has handed the industry a standard reference for what "naive LLM-as-judge" looks like. Expect the next wave of eval-tooling startups to position explicitly against it: deterministic rubric scorers, calibrated multi-sample judges, structured-output graders. Expect a paper within six months that uses the HackerRank repo as a canonical baseline to beat. And expect at least one plaintiff's lawyer to read the same code and notice that a 16-point spread on identical input is a discoverable fact pattern. The HackerRank scorer was always this brittle. It just used to be private.
And this + the tendency for AI to "prefer" AI produced code + some other AI biased is why *this is most likely highly illegal to use in the EU due to violating anti discrimination laws in multiple ways.To be clear:- randomly filtering "too many" resumes is pretty much allowed (I
At this point we might as well adopt that joke where you blindly throw away half the resumes because you don't want to hire unlucky people.
> I fail 65% of the time. Same exact resume, different luck.As someone who’s run hiring pipelines for technical roles in the past few years, that’s actually a fantastic number. I objectively hate saying that, but it’s true.35% chance of elevating a technical individual to the next stage with no e
At my company someone has introduced an internal tool that should help understand and give a "score" to design documents from teams.Needless to say, this tool gives scores exactly like the article mentions. Same document, same LLM, same prompt, and different results. It becomes even more r
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.
An alarming number of people don't understand that LLMs work via purely stochastic processes, so I'm happy to see in-depth pieces like this. I'm looking for a job and maybe this is why it's so hard to get a callback these days: resumes are just dumped in some LLM black hole and n