I am tired of taking screenshots and explaining them to Claude code while working.
Okay so a little context first, I work in an SDE role and work in multiple windows at once like i have many apps open in my mac. I faced a problem of screenshotting everything and explaining the context and their relations to claude code again and again like for little UI changes and such or any bug. I got this idea and started working on it , made a swift app which will track cursor and when…
The author, an SDE (Software Development Engineer), works in an environment where they frequently have multiple windows open on their Mac. This makes it challenging to explain UI changes or bugs to AI agents like Claude, Cursor, or Codex through screenshots and explanations. This led the author to develop a Swift app called Deiko that can track the cursor and automatically draw over the areas the user points to with their cursor.
Deiko uses Whisper to convert voice into text and attaches the timings to the cursor positions and screenshots. The app then creates event files that are stored as context whenever the user performs actions like drawing a circle or pointing to something. At the end of the session, Deiko generates a context using these events, which can be directly dropped into any AI agent.
The app works by keeping two streams - one for the cursor's position and the other for the user's narration. Deiko identifies the points when the user's narration refers to a specific settle (the cursor's position) by looking for pointing words like "this" or "here". This binding process is crucial to understand what the user is pointing at.
For screenshots, Deiko uses AXUIElementCopyElementAtPosition to capture the element under the cursor, including its accessibility label and window title. The app then crops the pixels and runs OCR (Optical Character Recognition) on the crop. All the generated data - screenshots, OCR text, and the user's narration - remains on the Mac. The only data that leaves the system is the audio, which is discarded as soon as the text is generated.
Currently, a trial version of Deiko is available, keyed to a hash of the machine's platform UUID. This means that deleting a plist file doesn't generate a new trial version. The product has its own website on Product Hunt and can be downloaded from Deiko - Point at it. Say it. Hand it to your agent. Deiko is designed to work in any language and can handle two languages at once.
The app stops working when the user stops talking, and it provides a clear brief ready to be dropped into any AI agent. The author is considering providing a free version with a BYO (Bring Your Own) option and a lifetime subscription of around $40.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
This story
This is one outlet's version. Read the fullest account.
