I struggle to believe that AI R&D is ingenious

It is difficult for me to comprehend the notion that autonomous R&D through artificial intelligence is truly ingenious. Over the past few months, I have kept an eye on various side projects, and in each instance, the model's judgment appears to be quite poor in a fundamental manner. To illustrate, during a hackathon held over the weekend, I assessed eight papers, all of which scored between 60% and 95% on the Pangram metric.

Regrettably, these papers tended to be highly similar, often lacking in methodological completeness, with a vague yet recurring issue of overlooking the most straightforward analysis.

However, there is one notable exception: certain hill-climbing tasks that have fast feedback loops. For instance, I managed to improve the state-of-the-art (SOTA) on listing all legal Scrabble moves by a factor of ten simply by prompting Astra repeatedly to strive harder, aim for a specific performance level, and systematically explore each method in the literature that seems somewhat related.

Yet, even in this case, two significant breakthroughs stand out: first, instructing the model to consider measuring progress across the entire game rather than per random position, and second, suggesting the model design algorithms for three distinct cases based on the number of blank tiles left in the bag. These actions seemingly cause the number of possible moves to skyrocket.

Additionally, it is worth noting that Astra, and presumably Claude, tends to generate reports that are extremely challenging to comprehend, often requiring a Q&A interrogation to decipher the actual work performed and the key results achieved. It seems there may be a temptation to simply declare "whatever" and let the model run unconstrained.

However, my impression is that the coding of Opus 5.5 (and even Sonnet 5.5) exhibits vastly superior judgment compared to Opus 4.5, which I utilized in January for a different project. This improvement in judgment likely represents a significant stride, but it is still distant from prevailing research trends, as taste and judgment are inherently multidimensional and hard to evaluate or predict.