Stop Wasting LLM Tokens! I Built a Rust CLI to Prune JS/TS Codebases by 80% 🦀🚀

<stop-wasting-llm-tokens-i-built-a-rust-cli-to-prune-js-ts-codebases-by-80-urai-ecma> In an article titled <stop-wasting-llm-tokens-i-built-a-rust-cli-to-prune-js-ts-codebases-by-80-urai-ecma>, the author discusses a Rust CLI tool called <urai-ecma> designed to prune JavaScript and TypeScript codebases by up to 82.7% using an AST-aware compiler engine.

The author explains that when feeding an entire React or Next.js repository into LLMs like Claude 3.5 Sonnet or GPT-4o, over 70% of the tokens are dead weight due to static Tailwind CSS utility classes, imperative loops, and unnecessary formatting logic. This leads to collapsed context windows, slow responses, and hallucinations about core functions.

To address this issue, the author built <urai-ecma>, which parses code into Abstract Syntax Trees (AST) and applies semantic transformations. Unlike traditional file dumpers that include all raw text in the context, <urai-ecma> retains only nodes critical to understanding the application's architecture, such as component declarations, hooks, and JSX layouts.

The tool achieves this through a multi-step process: it ignores files specified in .gitignore, prunes unneeded directories like node_modules and dist, and uses multi-threading with SWC to parse ASTs across CPU cores. The core components include a RouteVisitor for Next.js/Express/NestJS applications, a ReactComponentAnalyzer for props, state, and hooks, a ReactJsxPruner to strip unnecessary Tailwind classes, and a FunctionSummarizerVisitor to preserve structural stubs.

By applying these optimizations, <urai-ecma> reduces a 209,757-token codebase to just 36,000 tokens in milliseconds. The author emphasizes that the goal is not blind concatenation but rather semantic distillation, akin to the Tamil literary tradition of Urai Ezhuthudhal, where scholars distill the core meaning of texts while preserving their architectural depth.