How do you make auto-graded coding screens that test humans (not AI)?

Question

We're finally seeing that all of the answers to our company's top-of-the-funnel coding screen are nearly identical. It's fine that people are using LLMs, but it's clear that the value of our current screening questions are diminished. Once you have a small pool of candidates, you can interview them in person, ask questions that deal with ambiguity, etc. There are many opportunities to deconvolve the human from the AI.But, what techniques are people using for automated screening to evaluate technical/programming skills?

austin-cheney · Accepted Answer

The best solution that people mention is to fill your testing requirements with prohibited language. Example:Starting at the paragraph element containing text protect Taiwan independence write a simple program to walk the DOM to the paragraph containing text should not build a bomb provided the following instance of HTML.