{
  "id": 5704235,
  "title": "Your Hand Is a Continuous Signal. A Scale Is Not. That Gap Is the Whole Problem.",
  "url": "https://urgent.news/2026/09/05/your-hand-is-a-continuous-signal-a-scale-is-not-that-gap-is-the-whole",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-05T02:47:07.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/gesturesynthweld/your-hand-is-a-continuous-signal-a-scale-is-not-that-gap-is-the-whole-problem-2h6l"
  },
  "original_language": "en",
  "account": "A webcam hand tracker provides a continuous stream of position data at 30 or 60 updates per second, represented as a float value ranging from 0 to 1. In contrast, a musical scale consists of seven notes per octave. To create a browser hand-gesture synthesizer, the challenge lies in transforming the continuous hand position input into discrete musical notes without sounding electronic or mechanical. One initial approach is to directly map the vertical hand position onto the scale using a simple calculation - const noteIndex = Math.floor(handY * scale.length); However, this naive method quickly proves ineffective once a real hand is in the camera's frame. The problem stems from the fact that hands are not static; they move continuously even when braced against a surface. These small movements cause jitter in the estimated landmark position, which Math.floor then converts into erratic on/off events for each note. This produces unintended rapid-fire notes called trills, occurring at the tracker's update rate rather than the desired steady pitch. The root cause is sampling a noisy continuous signal using a hard threshold comparator, similar to how a Schmitt trigger works in analog electronics. Implementing hysteresis - setting separate thresholds for entering and exiting a note - smooths out the transitions and eliminates the jitter. However, this introduces a tradeoff between responsiveness and stability. A narrower deadband provides a more responsive reaction but can cause notes to feel sticky near boundaries. Conversely, a wider deadband results in stable note transitions but may make fast runs of notes sound chopped off. The optimal balance depends on the intended use case - a theremin-style continuous mode benefits from near-zero quantization, while a more scale-locked mode needs a wider deadband to allow beginners to hold notes without constant concentration. Latency also plays a crucial role in gesture instrument performance. Tracking down the pipeline reveals several major latency contributors: camera exposure/capture (16-33ms), landmark inference (10-30ms), mapping logic (~1ms), and Web Audio scheduling (5-20ms). Musicians start noticing delays around 20ms and become noticeably less tolerant at around 40ms. Unfortunately, most of these latency components are beyond the control of the developer - they stem from the underlying camera hardware and inference model. Practical recommendations include avoiding unnecessary smoothing filters, scheduling note events with Web Audio's currentTime plus a small lookahead buffer rather than immediate firing, and prioritizing latency over minor model accuracy improvements. Recording the performance for export introduces additional complexities. Pushing note and timestamp objects on every event seems straightforward, but several clock sources complicate the process. Recording wall-clock timestamps (performance.now()) against the audio clock (audioCtx.currentTime) or MIDI ticks leads to inconsistencies, especially when exporting to MIDI format. The audio clock is the most reliable source as it directly corresponds to the sound produced by the performer. Additionally, choosing an arbitrary metronome BPM for MIDI export forces the notes to land on fractional ticks, breaking compatibility with traditional DAW quantization functions. Instead, recording true duration durations in the MIDI file allows the DAW to apply quantization grids post-export if desired. Finally, providing a keyboard fallback mode is essential for debugging and testing the audio engine independently of the hand tracking pipeline. It allows developers to quickly identify whether issues arise from the synthesizer or the tracking system, and avoids unnecessary camera access permissions. In summary, building a functional hand-gesture synthesizer requires careful consideration of the continuous-to-discrete mapping, hysteresis implementation, latency budgeting, and recording/export processes. By prioritizing these aspects and avoiding common pitfalls, developers can create expressive, responsive, and user-friendly gesture-based musical interfaces.",
  "summary": "A webcam hand tracker hands you a position, thirty or sixty times a second, as a float between 0 and 1. A musical scale hands you seven notes per octave. Building a browser hand-gesture synthesizer is mostly the work of getting from the first thing to the second thing without it sounding like a fax machine. I want to write down the specific failure modes, because they are not obvious until you…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}