A free AI chat that costs me nothing: BYOK on Cloudflare Workers AI

There is no free AI chat service that stays free. The usual solution is to purchase credits, but there is an alternative: let the user pay for their own key. Cloudflare Workers AI provides tokens, which are then relayed along with the request. This approach incurs no cost, avoids storing credentials that weren't initially provided, and simplifies abuse management by revoking your token.

This architecture was implemented for the chat on ai.hartwell-labs.pl, taking only an afternoon. The authentication process was concise, consisting of just twenty lines of code. The most challenging aspect was not related to the chat functionality itself, but rather an unrelated issue.

The backend is incredibly simple: a single Worker handles incoming requests without any AI binding. It forwards the caller's credentials directly to the appropriate model. The request body contains configuration details such as provider, API key, account ID, and model. The Worker performs several checks: verifying the presence of an API key, confirming the token's validity by listing the user's accounts, and finally forwarding the request to the appropriate upstream service. The response is then returned to the client as an event stream.

Two key points enhance the user experience: account discovery eliminates the need for a manual account selection step, and the user can simply paste a token and start typing without any additional prompts. The system automatically selects the first account and stores it for future use. Additionally, error messages are crafted to provide actionable guidance instead of overwhelming the user with cryptic codes.

One critical aspect to consider is the handling of structured directives emitted by the model. For instance, a directive like "Zapisuję. [notepad action=new doc= Zakupy mleko chleb [/notepad]" would typically be parsed and processed correctly when the entire response is received. However, due to the streaming nature of the request, incomplete directives could lead to unexpected behavior, such as stray bullet lists appearing in the chat output.

To address this issue, the code includes a function to strip directives from the received text. This function iterates through the text, identifying and removing complete and partial directives, ensuring that only the intended content is displayed to the user. By focusing on the essential aspects of the chat functionality and implementing robust error handling, this solution offers a free and efficient AI chat experience without compromising on security or usability.