Training a coding model to paint watercolours with TRL and OpenEnv

On August 23, Surya Narreddi shared a captivating video of watercolors painted by a language model. A JavaScript program, generated through p5.brush, a library that augments p5.js with natural drawing tools, brought the artwork to life. The post quickly amassed over 1.5 million views, and a subsequent blog post delved into the training process, which initially focused on close-up flowers before expanding to full compositions.

The author of this account aims to reproduce Narreddi's methodology, ensuring that all components are open-source and published. The project's origins lie in the art and design realm, with the author contributing on the engineering side.

The article replicates Narreddi's approach using TRL and OpenEnv. The reference pool dataset, the RL environment, training scripts, and trained models are all available open-source. The pipeline is executed end-to-end on Hugging Face. Upon launching the two Spaces, the recipe becomes executable with a single command. The article details the process of setting up the environment and scorer model, adjusting reward mix variables, and initiating the execution.

The article's background material is based on Narreddi's thesis, which traces the project's evolution. Initially, text-to-image models were prompted, with the prompt being the primary control factor. However, training the model itself leads to more significant advancements. The model generates approximately 150 lines of JavaScript code that paint the image.

This code can be read, edited, and rerun, allowing for a transparent examination of each brushstroke's decision-making process. The style is derived from a constraint that limits the model's usage to ten methods of the library.

Parallel to this development, Anna Ridler captured thousands of tulips, meticulously labeled each one, and exhibited the dataset as an artwork. Subsequently, she trained a model on this dataset, showcasing a similar approach by curating a manually selected set of images before training against them. While most recent RL work on language models relies on verifiable rewards, such as math problems with known answers, code that passes tests, or cheap-to-run graders, this project ventures into a realm closer to RLHF, where the model learns an aesthetic reward model from human preferences.

However, unlike the typical scenario with a correct answer, the aesthetic preference here is subjective and lacks a definitive correct response. The challenge lies in determining whether RL can be applied to aesthetic judgment.