"Some AI researchers say that the overwhelming focus on scaling large language models and transformers — the architecture underpinning the technology which was created by Google in 2016 — has itself had a limiting effect, coming at the expense of other approaches.
“We are entering a phase of diminishing return with pure LLMs trained with text,” says Yann LeCun, Meta’s chief scientist, who is considered one of the “godfathers” of modern AI. “But we are definitely not hitting a ceiling with deep-learning-based AI systems trained to understand the real world through video and other modalities.”
These so-called world models are trained on elements of the physical world beyond language, and are able to plan, reason and have persistent memory. The new architecture could yet drive forward progress in self-driving cars, robotics or even sophisticated AI assistants.
“There are huge areas for improvement . . . but we need new strategies to get [there],” says Joelle Pineau, the former Meta AI research lead now chief AI officer at start-up Cohere. “Simply continuing to add compute and targeting theoretical AGI won’t be enough.”"
https://www.ft.com/content/d01290c9-cc92-4c1f-bd70-ac332cd40f94