Transformers is great but what's next?

Question

Transformer architecture is the foundation current paradigm of AI. I agree it's great but what's next, do you think there should have been an innovation since then and another competitor maybe?It was 2017 when the architecture was proposed and in 2024, 7 years have passed and we are heading towards "Globalization" as Peter Thiel says it.I would love to know, how far is the next big breakthrough in this Technology that goes from 0 to 1.

soist · Accepted Answer

No one at the commercial AI research labs knows what they're doing. As far as they are concerned there is nothing beyond gradient descent but it's accepted among academic researchers that gradient descent is insufficient for creating anything truly intelligent.

beardyw · Answer

Intelligence which can be produced biologically. It's a whacky idea but it might just work.

purple-leafy · Answer

I know the solution, and to find out yourself I just need your credit card number!

sk11001 · Answer

You can't get guess these things before they happen.
> do you think there should have been an innovation since then and another competitor maybe?
What does "should have" mean? There's no conspiracy to suppress the development of alternative approaches, there's no council or committee which decides these things - people would love the discover something new and amazing which is better, they just haven't been able to do it.

arthurcolle · Answer

probably S4 or something