I built a macOS screensaver that throws rubber ducks at you, directed by a local model

A developer created a macOS screensaver called Mac Attack, which throws harmless effects like rubber ducks at people in the user's room. The screensaver runs as a normal app and as a real screensaver, with all processes happening on the Mac. Apple's Vision framework is used to detect and turn people into cartoon characters. The developer, Laya, is a non-autoregressive typed-decision model that processes body rectangles and provides calibrated probabilities for various actions. By using sampling instead of the top answer, the game feels more dynamic and unpredictable.

Three key measurements were taken: 1) sampling beats argmax, 2) policy belongs in code, not in the prompt, and 3) measure the model you're actually running. Sampling from the calibrated distribution made the game feel more alive, while rewriting the questions to make rare options less likely helped prevent the game from becoming stagnant. Additionally, splitting the model's input into smaller questions increased the diversity of responses.

The challenge of creating a real screensaver was overcome by using a launchd agent to own the camera and serve anonymous person boxes on 127.0.0.1. The screensaver then polls the agent at 10 Hz, runs the game engine, and renders the effects. This architecture ensures the camera is only used when necessary and the screensaver remains fully functional even when the main application isn't running.

Mac Attack is currently an unsigned build, so Gatekeeper will prompt users to open it. The entire source code, architecture notes, and a detailed account of what worked and didn't during development can be found on GitHub.