New advancements in Generative AI

New advancements in Generative AI have moved beyond being just a chatbot API. The focus has shifted towards agentic workflows, local execution, and structured inputs that respect schemas. Recently, structured outputs have become a significant improvement. Major providers and open-source runtimes now allow passing JSON schemas directly to the inference endpoint, constraining token selection at the logit level to ensure the model cannot output invalid data.

This is demonstrated in the modern OpenAI SDK with Pydantic, where the API errors out if the model tries to return an incorrect data type.

Although running models locally was once a slow and resource-intensive process, advancements in quantized formats like GGUF and engines such as Ollama and llama.cpp have made it possible to run models like Llama 3.1 8B or Mistral 7B on consumer GPUs. For simple tasks like classification, text extraction, or entity recognition, running models locally can be a viable solution, eliminating the need to share user data with third-party APIs.

However, smaller local models still struggle with context windows and instruction following for complex multi-step reasoning. The complexity of the task should dictate the model size, not personal preferences for minimizing cloud usage. Agentic loops have also evolved, with a trend towards simpler, raw agentic loops instead of complex frameworks.

An agentic loop essentially consists of a while loop that calls an LLM, checks if the model wants to call a tool, executes the tool, and feeds the result back into the context. Function calling and basic control flow are sufficient for this purpose.