It has been a few months now since I started using agentic tools for projects and I was curious to dip into what is being said on the topic of productivity gains lately. I’ve written about productivity and productivity gains many times pretty casually, and checking out what Yegor Denisov-Blanch has to say on the matter [1,2], I find that measurement is also equally casual, at least from the 2025-Dec to 2026-Jan point of view. His research involves measuring gains using objective judges, while accounting for rework, but it sounds like it is still not feasible to disentangle other confounding factors when trhying to attribute changes in economic value to pure AI tooling.
My armchair thesis going into this is that the only way to to really measure economic value is by measuring economic value. That sounds reductive but I mean we would need to actually gig-work-ify knowledge work and perhaps software work in particular. Think Uber or Lyft for software outpaces salaried software jobs. And then naturally if gig workers, – call them gigabyte workers? – choose to pay for claude subscriptions, then maybe that would be the cleanest test for sussing out literal ROI. My thinking is that, more broadly, books like Bullshit Jobs and Slow Productivity exist, because society somehow puts no downward pressure on detritus coordination overhead like Scrum, SAFe, and also all of the bullshit internal facing work outputs like reports, dashboards, and all manner of widgets that sink time and money without measurable benefits.
However, discussing this with ChatGPT, Chat brings up that just because someone in a marketplace is willing to pay someone for a software gig service, doesn’t mean they are deriving value from it. And actually, thinking on that some more, I realize wow actually this is very true and profound! After all, enterprise companies choose to pay for all manner of software as a service things without needing to justify the ROI.
And now that I think about it, often quite the opposite happens. Speaking from personal experience I have observed massive cloud migration undertakings with the expressed goals of consolidating cloud cost, that were saving money on paper – on some future unknown date – but were hobbling business units in the process, reducing their ability to produce meaningful business value during the process.
So ok, thinking through this, then, maybe gigifying all of software work would not help to disentangle it from the value.
Also, more to that counter point, Erik Brunjolfsson [3] argues a similar point that following the money w.r.t. reduced head count is also not necessarily a reliable productivity indicator.
Wow okay this is hard!
Another super interesting perspective offered from the interviewer in [1] was to ask Yegor, well what about the senior engineers who create fewer pull requests and affect outcomes in more indirect ways such as by writing documentation, proposals or mentoring others. Yegor’s reply was basically that, right, his research cannot help there. And then I immediately realized that’s exactly the same boat we are in with the broader knowledge worker, including technical and non technical managers, middle managers, and all manner of folks whose influence is even more squishy.
But maybe there are at least two small take-aways I have from hearing these interviews, which is that measurements in aggregate are safer and measuring them relatively is perhaps useful. He was defending his use of proxy measures, basically saying that it is just too difficult to know if what you are building is useful, but at the very least, we can measure the team’s net throughput. And here, he says looking at the team is safer, so as to blend, smooth out not only variation among individuals but also wrt weird coordination side effects. And finally a team can measure itself relative to itself, over time. This still suffers from tricky pre/post effects that don’t control for other exogenous pressures, and so his research also looks into organic experiments where teams within the same company, with different levels of granular token spend , relative to each other change , over time.
So again, wow, this is a really hard problem. I have more questions than before.
References
- interview with Yegor Denisov-Blanch, https://www.youtube.com/watch?v=4KLuvAa_lrk
- talk by Yegor, Denisov-Blanch, https://www.youtube.com/watch?v=JvosMkuNxF8
- Erik Brunjolfsson, https://www.youtube.com/watch?v=u76xdhpF474