Reading a Reinforcement Learning Taxonomy Against My Own Tools

A list of reinforcement learning algorithms was provided, but none of them could help the tools being discussed. The source material was a ChatGPT session in PDF form, listing ninety algorithm names across fifteen sections without any descriptions, citations, or results. The author examined their own tools more closely after receiving the list.

None of the algorithms could be applied to the tools because they require a number indicating how well they performed, which the tools do not have. The tool's own verification verdict of "ALL CHECKS PASSED" is not a reward, as it merely means the program meets its own contract. Optimizing this signal trains the tool to produce wrong programs that verify.

Several sections of the list were not assessed against the requirement of having a performance number, so only nine of the fifteen sections were considered. Section ten, which deals with exploration bonuses, was also not applicable as the reward was pseudo-counts and Random Network Distillation, both reward-free.

The tool built 52 requests, with 18 being apt (the program matches the request's intent) and 34 not. The in-distribution reference, which defines what the tool was built for, is the repository's own corpus generator and not curated. The AUC (area under the curve) scores for the four working algorithms were 0.717, 0.739, 0.784, and 0.676, all of which exceeded the floor for a resolvable result of 0.641. The two algorithms not imported were A1_pseudocount and A2_rnd, which are the reasons nothing shipped.