Postcards from Outside

Postcards from Outside is an open-source project that combines scavenger hunts with technology to encourage people to get outside and explore their surroundings. The app, named Postcards from Outside, allows users to participate in a fall scavenger hunt by telling the app who will be joining them (kids, walking the dog, an easy pace, or adults) and adding their names.

Users can then specify their location if desired. The app builds a short list of things to find based on the user's group, area, and time of year. For example, in New York during October, the app might suggest finding red leaves and pumpkins, but not in Florida.

Once a user finds an item, they take a photo of it and add it to the app. An open-source AI model runs on the phone and checks off the item if it recognizes it. If the AI is unsure, users can try again or count the item anyway, since no one should argue with their phone during a walk. At the end of the hunt, users can save or share their photos as a postcard, complete with a title, date, names, and a 🐾 stamp if a service dog was present.

The app was built using a stack of Vite + React, Transformers.js running an open-weight CLIP model (Xenova/clip-vit-base-patch32) on the phone's CPU inside a Web Worker, OpenStreetMap's Nominatim for optional location lookup, and a plain HTML canvas for the postcard. It is hosted as a static site on DigitalOcean with no backend, accounts, or API keys required.

The development process focused on CPU-only processing, CLIP model selection, testing before building, teaching the AI not to be fooled by false positives, and creating a hand-written list of items to look for based on user preferences, region, and season.

The app has been tested on real walks with a service dog and its creator's mom. The first test showed some false positives, such as a maple leaf being mistaken for a fire hydrant. To fix this issue, background labels like grass, pavement, and the sky were added to every check, allowing the AI to handle ambiguous results better. Another challenge involved a park bench being incorrectly matched with a pavement label.

To address this, items were given the option to opt out of competing with their surroundings, and a minimum confidence threshold of 0.4 was set for the AI's top answer. The final product is a simple, free, offline-first app that encourages outdoor exploration without relying on cloud services.