Urgent.News

What's breaking now, across hundreds of outlets.

AI

A Beginner's Guide to Local AI with Inkling, Qwen, and Unsloth

Welcome to the world of local AI! With the launch of open-weight models like Qwen, cloud-based LLM providers now resemble infrastructure or appliance businesses. Concerns over ballooning costs, privacy, and usage limits have made many, like myself, seek a more personal and private solution. That's where setting up a local AI becomes a practical choice.

This guide will walk you through creating your own personal ChatGPT using Unsloth and Inkling or Qwen, all without the hassle of terminal commands, Python environments, or subscriptions.

First, let's ensure your hardware is up to the task. Depending on the VRAM of your graphics card, you can handle smaller 9B-12B models with 8-12GB of VRAM, while 32GB is recommended for larger 27B-35B models. For consumer-grade hardware, smaller models are typically needed. The full-size Qwen model, at 56GB, won't fit on a typical consumer graphics card unless you're using a Mac with unified memory.

However, quantization can help by compressing weights and moving to a lower precision, allowing the model to fit on your GPU and run faster.

Next, we'll install Unsloth Desktop, a user-friendly option for hosting your AI model. Once installed, download a quantized AI model that fits your GPU's VRAM, set your context size, and enable LAN access for all members of your household to use the AI. With the hardware sorted, setting up your local AI takes just 10-15 minutes.

For those interested in Qwen3.8-27B, a 27B model with 4-bit quantization, we'll use an RTX 5090 with 32GB of VRAM. This setup allows for a larger context window while maintaining excellent performance. If you have a less powerful GPU, don't worry - there are still viable options out there. The table below provides a rough idea of what you can start with based on your GPU:

- RTX 5060 Ti 16GB

- RTX 5070 Ti

- RTX 5080

- RTX PRO 2000

- Qwen3-Coder-30B

- Gemma 4 26B/31B

- Qwen3.8-27B

- Inkling 2 Mini

- RTX 3090

- RTX 4090

- RX 7900 XTX

- RTX PRO 2000

- RTX PRO 4500 32GB

- RTX PRO 5000 48GB

- RTX PRO 6000 96GB

Experiment with different models to find one that suits your needs best. You can also use GPUPartner's VRAM Calculator for more precise estimates.

Rewritten by urgent.news; may contain errors — check the original.

Source: gpupartner.com

Read the original at gpupartner.com →

Report a problem with this story

More in AI

Live Coding with an AI Team: A Safe Way to Review a PR (One Writer, One Reviewer, One Publisher)

I ran a small pull request review live with AI agents, and the part that went wrong was never the code. It was the handoff : two agents editing one folder, a bot "reply" that proved nothing, a caption…

  • AI team assists in live coding session for PR review
  • Guidelines established to prevent complications in AI agent handoff
  • Author compiles guidelines into 41-page visual PDF for free access

More from Sunday 22 November →