We talk about scaling laws have slowed down as they apply to large language models, but we used to have the same conversations about organizations. If you increase the size of a deep neural network, into the terabytes number of parameters, it has been shown over the last handful of years, you get some exceptional results. But either because of a data ceiling – as the AI snake oil book authors describe wrt predictive models – or some other constraint, foundation models pivoted away from generalized models to heuristic routing to more specialized models and to agentic real world action at a distance, getting out of that artificial metaphorical arm chair.
But companies also, as they grow larger, have trouble scaling and need to figure out better heuristic architectures around those pristine conway two pizza teams.
Large companies try really hard to streamline, aiming towards simplicity, but the tool to do it is inevitably complexity. And it is sad to watch. Processes are used as glue between all of those conway two pizza teams, but that glue is often literally feeling like glue and the whole machine screeches to a crawl.
We need a well oiled machine. How do you get there, haha I wish I knew.
References
- AI Snake Oil book