Palette: teaching a small model to build decks in your design language

Designing a slide deck using a small open-weight model poses challenges, as these models tend to produce consistent, generic results that don't match a specific design language. However, a novel approach aims to create a model trained on synthetic data within a specific design language, allowing it to compete with more advanced systems while keeping ownership of the results.

The process begins by fine-tuning a 20 billion parameter model to build decks in a manner similar to how a human designer would approach it. The training data follows the IBM Design Language and covers around 10,000 slides, including the reasoning behind each design choice. The deck creation process involves two primary roles: the designer and the coder.

The designer handles editorial judgment, such as deciding what each slide should contain and how it should be presented (charts or tables). Meanwhile, the coder is responsible for the geometry, determining the precise placement of elements on the slide.

To address the limitations of small models, a harness is built around the trained model. This harness checks for issues such as overlapping shapes or text running off the slide, making necessary adjustments and rewriting the reasoning behind the changes. By generating a large dataset of decks, each with accompanying explanations for design decisions, the model learns to understand the rules behind choices rather than simply memorizing them.

This method also allows users to provide finer-grained control over specific design choices using directives, which the coder follows during slide creation.

The approach tackles the challenges of small models by breaking down the deck creation process into smaller, more manageable tasks while enhancing performance through parallel processing. This allows for faster generation and greater customization of slide decks, maintaining a consistent design language while ensuring the final product meets the desired specifications.