Meta is back with Muse Glimmer: local, agentic, multimodal, and open source

Meta has unveiled Muse Glimmer, a new multimodal model designed for local, agentic use cases, as part of its continued commitment to open-source Large Language Models (LLMs). This 30B parameter model, distilled from Muse, is released under the Apache 2.0 license, making it accessible for deployment locally, thereby reducing costs and enhancing privacy.

Muse Glimmer supports a range of applications, including coding, document analysis, and personal assistants. It is compatible with various libraries, such as transformers, llama.cpp, vLLM, and Inference Endpoints, providing day-0 support for seamless integration. The model's architecture features a 2B ViT-like image encoder, capable of handling both images and videos.

This encoder transforms images into a shape of 2 frames by 3 channels by 14 by 14, passing them through a linear layer for projection. The vision tower, comprising 50 layers and GELU MLPs, processes the embeddings via three window attention layers and one full attention layer, employing 2D RoPE for queries and keys.

The model's decoder employs speculative decoding drafter, which can expedite generation at the cost of additional memory. This feature proves particularly useful in structured content generation, such as coding tasks. Muse Glimmer's architecture supports multimodal tool calling, enabling it to interact with external tools based on input prompts. For instance, users can request the model to call a weather tool based on an image, demonstrating its versatility in processing various input types.

Installation of Muse Glimmer requires upgrading to the latest version of transformers. The model comes with day-0 support in transformers, offering auto-loading classes for both the main model and the speculative decoding drafter. With llama.cpp, Muse Glimmer can be deployed on NVIDIA, AMD, and Intel GPUs, automatically placing the model on the available accelerator. Users can perform text-only inference, image inference, video inference, and multimodal tool calling using the provided snippets.

For those seeking a managed, autoscaling deployment, Meta offers a Muse Glimmer 30B Inference Endpoints preset, which can be deployed on compatible cloud providers and GPU instances. Once deployed, the model exposes an OpenAI-compatible Chat Completions API, enabling seamless integration with existing systems.