Because it requires people to know how to read or specify code.
Spec driven development is similar to the no-code movement i.e. the specs are closer to functional requirements than this strict spec or code.
You can argue that you can just ask the LLM to explain back and forth but it can still be too much.