HACKER Q&A
📣 yogoism

Share real complaints about outsourcing data annotation


Hi HN,

I’m mapping the data-annotation vendor landscape for an upcoming study.

For many AI teams, outsourcing labeling is a strategic way to accelerate projects—but it isn’t friction-free.

If you’ve worked with an annotation provider, what specific problems surfaced? Hidden costs, accuracy drift, privacy hurdles, tooling gaps, slow iterations—anything that actually happened. Please add rough project scale or data type if you can.

Your firsthand stories will give a clearer picture of where the industry still needs work. Thanks!

  👤 fzwang Accepted Answer ✓
We've explored using external vendors for data labeling and annotation work for a few projects (image and text data). I think overall the problem is more along of the lines of mis-aligned/drifting incentives. It's like Goodhart's law, where whatever metric you use for outcomes tend to be manipulated or have unintended consequences. And putting in the trusted systems to identify bad/shifting metrics is costly in a way that makes outsourcing not worth it.

In most cases, we've opted to build the data labeling operation in-house, so we have more control over the quality and can adjust on the fly. It's slower and more costly upfront, but better outcomes in the long run as we get higher quality data.