I let a local LLM control my Home Assistant automations, and it took a wrong turn before I added guardrails

I use a local large language model (LLM) with Home Assistant to interpret complex voice commands. While most requests still go through regular intent matching, a more sophisticated model provides a more natural conversational experience. However, the local model isn't always as accurate as expected, leading to occasional errors.

The model has mistakenly switched off lights instead of lamps, answered questions about a light's state with actions, and sometimes failed to execute commands altogether. To mitigate these issues, I've implemented guardrails within the system. By limiting the model's access to fewer than 25 entities and carefully selecting which scripts and helpers to expose, I've reduced the scope of potential mistakes.

Additionally, I've configured predefined scripts for complex actions and restricted the model's ability to execute certain sensitive commands, such as setting temperature extremes. While these measures have improved the reliability of the setup, I still prefer maintaining a balance between convenience and control.