Mini reaction on people saying Mythos can find bugs in decade old code. I think similarly I was reading Mythos or maybe a different model is solving Erdos problems previously unsolved, yet it turned out they were solved but just lost into the internet and the model in question sort of performed a lookup as with stack overflow answers.
Especially if folks are saying, Mythos is finding bugs in really old software, that begs the question are old bugs being resurfaced , recycled as novel.
In other words, it is maybe a classic leaky hold out set problem. The training data , or in this case the pretraining data, has these old bugs and they are being re-discovered?
More impressive and more practical to test would be to set a time cutoff for when to look and when not to look. Any code prior to the date of the training data collection may have had bug reports published as part of the pretraining data crawled.
And all of this of course is not mentioning the other issue of false positives and verifiability of bugs found. Can you actually verify it, let alone prove it can be meaningfully exploited (not already patched etc)