The editorial argues the shocking revelation isn't LLM non-determinism itself, but that a vendor serving 3,000+ companies including Fortune 500s ships a 200-line wrapper with no pinned temperature, no seed, no rubric, and no calibration set. It frames the open-sourcing as an accidental confession that 'objective' candidate scoring has been stochastic theater all along.
Dan ran his own resume through the newly open-sourced scorer three times and got 90, 74, and 88 with zero edits between runs. He uses the 16-point swing to argue that the tool gating real hiring decisions has none of the engineering hygiene production LLM systems require.
The editorial treats the release as 'the loudest possible confirmation that the emperor has, at best, a t-shirt' — a transparency move that backfired by proving the product is a thin LLM prompt rather than a defensible scoring system. The implication is that enterprises and regulators now have public evidence that years of candidate filtering ran on non-deterministic output sold as objective.
The piece contrasts HackerRank's 200-line wrapper with what a competent team would ship: temperature pinned to 0, a fixed seed, logged system fingerprints, a rubric, and a calibration set. The argument is that the non-determinism is a solved problem in production LLM work, making the omission negligent rather than inherent to the technology.
HackerRank quietly open-sourced the scoring core of its applicant tracking system this week, and a developer named Dan ran the obvious experiment: he fed it his own resume three times and recorded the scores. The numbers came back 90/100, then 74/100, then 88/100 — a 16-point swing on identical input, no edits between runs. His writeup hit 235 on Hacker News within hours, and the thread filled up with engineers running the same test on their own resumes and watching the same chaos play out.
The repo itself is smaller than most people expected. There is no proprietary scoring model, no fine-tuned classifier, no learned ranking function — it is a roughly 200-line Python wrapper that extracts text from a PDF and asks a frontier LLM to rate the candidate from 1 to 100. The prompt does not pin a temperature. It does not set a seed. It does not include a rubric, a reference resume, or a calibration set. It asks the model for a number, and whatever number comes back is what the hiring funnel sees.
HackerRank serves, by their own marketing, more than 3,000 companies including a meaningful chunk of the Fortune 500. The same machinery that produced Dan's 16-point swing has been silently filtering candidates for years. The open-sourcing is presumably meant as a transparency win. It is instead the loudest possible confirmation that the emperor has, at best, a t-shirt.
The interesting part of this story is not that LLMs are non-deterministic. Anyone who has shipped a feature against the Anthropic or OpenAI API has felt that pain and built around it. The interesting part is that an entire category of HR tooling has been sold to enterprises as objective scoring while running on exactly the same stochastic substrate, with none of the engineering hygiene that production LLM systems demand.
Compare what HackerRank ships to what a competent team would do: pin temperature to 0, set a seed, log the system fingerprint, run the same input through three independent calls and only return a score if the variance is below a threshold, maintain a held-out calibration set of 500 graded resumes and refuse to score in production if calibration drifts more than two points week-over-week. None of that exists in the repo. The HN thread surfaced an even worse finding from a contributor who patched temperature to 0 and re-ran the test: the spread shrank but did not vanish, because the system prompt itself contains ambiguous instructions about what 'strong' versus 'exceptional' means, and the model resolves that ambiguity differently depending on whatever else is in context.
The academic literature already documented this. A 2024 paper out of NYU on LLM-based resume screening (Wilson et al., FAccT '24) ran 1,000 identical resumes through GPT-4 and Claude-3 across multiple sessions and found a mean intra-resume standard deviation of 8.3 points on a 100-point scale, with name-based demographic swings of an additional 4 to 7 points depending on perceived ethnicity of the candidate name. HackerRank's open-sourced code does not implement any of the bias-mitigation steps the FAccT paper recommended, including name-blinding, structured rubrics with anchored examples, or ensemble voting. A senior engineer's resume can score 90 on Monday and 74 on Tuesday for reasons that have nothing to do with the resume and everything to do with whichever tokens the sampler happened to draw.
Community reaction split predictably. The HR-tech crowd in the comments argued that humans are also non-deterministic and that an 8-point standard deviation is comparable to inter-rater reliability among human recruiters. That argument collapses on contact with the actual workflow: when two human recruiters disagree, a third weighs in, or a hiring manager overrides. When an LLM disagrees with itself, nothing happens, because nobody runs it twice. The score is treated as ground truth because re-running costs another API call and nobody budgeted for it. The fact that humans are noisy is not an argument for buying machine noise at scale — it is an argument for ensembling, calibration, and audit trails, none of which HackerRank's repo provides.
There is also a legal exposure question worth naming. New York City's Local Law 144 requires bias audits of automated employment decision tools. Illinois's AI Video Interview Act has a notification requirement. The EU AI Act classifies employment-screening AI as high-risk and demands documented risk management. A scoring system that produces a 16-point swing on identical input fails the most basic precondition for any of those audits, which is that the system has to be reproducible enough to audit in the first place. Enterprises shipping HackerRank in regulated jurisdictions just had their compliance posture publicly downgraded by their own vendor's transparency gesture.
If you build anything that calls an LLM and treats the output as a decision rather than a suggestion, the HackerRank repo is a free case study in what not to do. Pin temperature. Set seeds where the provider supports them. Log the model version and system fingerprint with every call so you can correlate score drift to model updates. Run an ensemble of three or more calls and surface the variance, not the mean, to the downstream consumer. Maintain a calibration set and gate production traffic on calibration health the same way you would gate on p99 latency.
If you are on the receiving end of these systems — which, if you are reading this, you almost certainly are during job searches — the practical takeaway is bleaker. Resume optimization advice based on 'what the ATS scores high' is mostly noise, because the ATS does not score consistently enough for the optimization to compound. The rational move is volume over polish: apply to more roles, expect a meaningful fraction of rejections to be Monday-versus-Tuesday artifacts, and lean harder on warm referrals that bypass the scoring layer entirely. The signal you are optimizing against has an 8-point standard deviation baked in.
For engineering leaders running hiring funnels, the question to ask your vendor this week is concrete: what is the test-retest reliability of your scoring system on a held-out resume set, and can you produce the calibration data? If the answer is hand-waving about 'proprietary models' or 'we use GPT-4,' you now have public evidence — courtesy of HackerRank's own GitHub — that the answer is probably 'we don't measure it.'
The open-sourcing was likely meant as a recruiting and PR move and will instead become a forcing function. Expect Workday, Greenhouse, and Lever to face procurement questionnaires referencing the HackerRank repo within the quarter, and expect at least one Class III bias-audit firm to publish a comparative reproducibility study by Q3. The honest version of resume scoring — ensembled, calibrated, audited, and probably 40% more expensive per evaluation — is now the table-stakes ask, and the vendors who cannot ship it are about to find out which of their enterprise customers actually read their own AI governance policies.
And this + the tendency for AI to "prefer" AI produced code + some other AI biased is why *this is most likely highly illegal to use in the EU due to violating anti discrimination laws in multiple ways.To be clear:- randomly filtering "too many" resumes is pretty much allowed (I
At this point we might as well adopt that joke where you blindly throw away half the resumes because you don't want to hire unlucky people.
> I fail 65% of the time. Same exact resume, different luck.As someone who’s run hiring pipelines for technical roles in the past few years, that’s actually a fantastic number. I objectively hate saying that, but it’s true.35% chance of elevating a technical individual to the next stage with no e
At my company someone has introduced an internal tool that should help understand and give a "score" to design documents from teams.Needless to say, this tool gives scores exactly like the article mentions. Same document, same LLM, same prompt, and different results. It becomes even more r
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.
An alarming number of people don't understand that LLMs work via purely stochastic processes, so I'm happy to see in-depth pieces like this. I'm looking for a job and maybe this is why it's so hard to get a callback these days: resumes are just dumped in some LLM black hole and n