{
  "id": 21293989,
  "title": "A Beginner's Guide to Local AI with Inkling, Qwen, and Unsloth",
  "url": "https://urgent.news/2026/11/22/a-beginners-guide-to-local-ai-with-inkling-qwen-and-unsloth",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-22T21:01:16.000Z",
  "source": {
    "name": "Hacker News",
    "slug": "hacker-news",
    "url": "https://www.gpupartner.com/blog/beginners-guide-local-ai-inkling-qwen-unsloth"
  },
  "original_language": "en",
  "account": "Welcome to the world of local AI! With the launch of open-weight models like Qwen, cloud-based LLM providers now resemble infrastructure or appliance businesses. Concerns over ballooning costs, privacy, and usage limits have made many, like myself, seek a more personal and private solution. That's where setting up a local AI becomes a practical choice. This guide will walk you through creating your own personal ChatGPT using Unsloth and Inkling or Qwen, all without the hassle of terminal commands, Python environments, or subscriptions.\n\nFirst, let's ensure your hardware is up to the task. Depending on the VRAM of your graphics card, you can handle smaller 9B-12B models with 8-12GB of VRAM, while 32GB is recommended for larger 27B-35B models. For consumer-grade hardware, smaller models are typically needed. The full-size Qwen model, at 56GB, won't fit on a typical consumer graphics card unless you're using a Mac with unified memory. However, quantization can help by compressing weights and moving to a lower precision, allowing the model to fit on your GPU and run faster.\n\nNext, we'll install Unsloth Desktop, a user-friendly option for hosting your AI model. Once installed, download a quantized AI model that fits your GPU's VRAM, set your context size, and enable LAN access for all members of your household to use the AI. With the hardware sorted, setting up your local AI takes just 10-15 minutes.\n\nFor those interested in Qwen3.8-27B, a 27B model with 4-bit quantization, we'll use an RTX 5090 with 32GB of VRAM. This setup allows for a larger context window while maintaining excellent performance. If you have a less powerful GPU, don't worry - there are still viable options out there. The table below provides a rough idea of what you can start with based on your GPU:\n\n- RTX 5060 Ti 16GB\n- RTX 5070 Ti\n- RTX 5080\n- RTX PRO 2000\n- Qwen3-Coder-30B\n- Gemma 4 26B/31B\n- Qwen3.8-27B\n- Inkling 2 Mini\n- RTX 3090\n- RTX 4090\n- RX 7900 XTX\n- RTX PRO 2000\n- RTX PRO 4500 32GB\n- RTX PRO 5000 48GB\n- RTX PRO 6000 96GB\n\nExperiment with different models to find one that suits your needs best. You can also use GPUPartner's VRAM Calculator for more precise estimates.",
  "summary": null,
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}