I Tried Needle2 for Local Tool Calling. I Ended Up With llama.cpp + Granite

The author experimented with Needle2, a small, local model for tool calling and structured extraction, in an attempt to create a semantic shell that understands user intent without generating shell commands. While the model performed well initially, challenges arose when handling more complex cases, as the retrieval mechanism for selecting tools became less reliable with larger sets of tools.

The author attempted to bypass retrieval by grouping tools and selecting the strongest candidate, but this led to questions about the comparability of confidence scores, which they could not find definitive answers for.

Additionally, the author emphasized the importance of precise tool descriptions, as a model's performance heavily relies on the clarity of these descriptions. They discovered that a small model like Needle2 is sensitive to tool descriptions, requiring explicit definitions to avoid confusion, especially when neighboring tools have similar functionalities.

The author also evaluated the C++ runtime option for a native shell integration, overcoming challenges related to cross-platform compatibility and the integration of ARM NEON includes and intrinsics. Ultimately, the author opted for integrating llama.cpp and Granite 4 350M, which performed better than expected and exhibited better behavior for handling ambiguous inputs, such as failing gracefully instead of making confident guesses.

This change in the model choice did not significantly impact the overall shell architecture, as the surrounding system was already designed to handle uncertainties effectively.