#055 ML Илья Гусев. Как запускать большие языковые модели локально

Item: #055 ML Илья Гусев. Как запускать большие языковые модели локально
Rating: 4
Author: PostSingularity Institute

#055 ML Илья Гусев. How to run large language models locally

by Machine Learning Podcast (host: Mikhail Kryzhanovsky; guest: Ilya Gusev) · View the channel on PSI

Formal & Physical Sciences AI — Models & Research UYQArtificial intelligence (AI)UYQLNatural language and machine translation

🎙 Machine Learning Podcast (host: Mikhail Kryzhanovsky; guest: Ilya Gusev) 👥 5K 📅 September 26, 2024 ⏱ 41 min 👁 1K 📄 expert opinion 🧭 2026-10-09

Available in: English (current) Français

Keywords

large language modelslocal deploymentquantizationLoRARussian language models

Summary

This podcast episode features Ilya Gusev, a prominent NLP specialist known for open-source projects such as the natasha morphological analyzer, a poetry generator, and the Russian LLM Saiga. The conversation begins with a historical overview of large language models, starting from the 2017 Transformer paper, through BERT and GPT, up to the current landscape. Gusev explains the fundamentals of model parameters, memory requirements, and the role of quantization in enabling local deployment. He details how 8-bit quantization and LoRA adapters have made it feasible to fine-tune and run models with billions of parameters on consumer hardware like a single RTX 3090. The discussion covers various parameter-efficient fine-tuning methods, including prompt tuning and adapters, and their trade-offs. Gusev then shares the origin story of Saiga, a Russian-language model built by fine-tuning LLaMA on synthetic dialogue data generated by ChatGPT. He describes the challenges of creating Russian datasets and the iterative process of improving the model. The episode also touches on the lack of censorship in Saiga compared to ChatGPT, and the importance of instruction tuning and RLHF. Throughout, Gusev provides practical insights into the tools and techniques that democratize access to LLMs, emphasizing the balance between model size, quantization, and quality.

200 words

Critical Evaluation

Value of the Information & Strength of the Argument

The episode offers substantial value by combining a clear historical narrative with practical, hands-on knowledge about running LLMs locally. Gusev’s explanations are grounded in his direct experience building and deploying models, making the information both credible and actionable. He systematically walks through the evolution from early transformers to modern quantized models, highlighting key innovations like 8-bit quantization and LoRA that have lowered the barrier to entry. The argumentation is logical and well-structured: he identifies the problem (large models require immense resources), explains the technical solutions (quantization, adapters), and demonstrates their impact (fine-tuning on a single GPU). He also candidly discusses limitations, such as quality degradation at lower bit widths and the challenges of synthetic data. The dialogue format allows for clarifying questions that deepen the technical explanations without oversimplifying. Overall, the episode provides a robust framework for understanding the current state and practicalities of local LLM deployment.

Scientific Rigor, Source Quality, Title Accuracy

The scientific rigor is high, with references to foundational papers (Transformer, BERT, GPT) and specific techniques (bitsandbytes, GPTQ, LoRA). The description includes links to primary sources: the arXiv paper on self-consuming generative models, the Habr article by Gusev on building a Russian local ChatGPT, and repositories for LM Studio, oobabooga, and Hugging Face TGI. These sources are directly relevant and reputable. The title accurately reflects the content, as the episode indeed focuses on how to run large language models locally, using Saiga as a case study. The discussion is technical but accessible, avoiding hype while acknowledging the rapid progress in the field. The guest’s credentials and open-source contributions further bolster the reliability of the information. No significant contradictions or unsupported claims were noted.

285 words

Title / Content Match

The title accurately reflects the content: the episode focuses on running large language models locally, with a historical overview and practical details, though the guest's Saiga model is discussed as a case study rather than the sole topic.

Quality & Reliability

8/10

The guest is a recognized NLP practitioner (Yandex, Booking.com) with first-hand experience building Russian LLMs (ruTurboAlpaca, Saiga) and open-source tools (natasha, poetry generator). Technical claims about transformer history, quantization, LoRA, and PEFT are accurate and well-explained. Sources cited in the description include primary papers and repositories. No sensationalism or unsupported claims; the discussion is grounded in practical experience and published research.

Key Moments

Markers derived by PSI from the transcript: the creator did not define chapters.

Cited Sources

How to (quickly) make a Russian local ChatGPT — Article by Ilya Gusev detailing the creation of a Russian local ChatGPT, referenced in the description.
LM Studio — Service for running large language models locally, mentioned as a tool.
oobabooga text-generation-webui — Repository for a web interface to work with LLMs locally.
Text Generation Inference (TGI) — Repository for deploying and maintaining LLMs, by Hugging Face.
Self-Consuming Generative Models Go MAD — arXiv paper analyzing the effects of training LLMs on synthetic datasets generated by other LLMs.
Ilya's post with NLP learning materials — Telegram post listing resources for diving into NLP.

Concurring Sources

LoRA: Low-Rank Adaptation of Large Language Models — Original paper on LoRA, which is discussed as a key enabler for local fine-tuning.
LLaMA: Open and Efficient Foundation Language Models — Meta's LLaMA paper, which is the base model for Saiga and many other fine-tuned models.
Training language models to follow instructions with human feedback — InstructGPT paper, which introduced RLHF and is referenced in the discussion of ChatGPT's training.

External References

Contribution & Novelties

The episode provides a comprehensive, practitioner’s perspective on the local deployment of large language models, with a unique focus on the Russian language context. It demystifies the technical hurdles—quantization, memory management, and fine-tuning—by explaining them through the lens of real-world projects like Saiga. The discussion of synthetic data generation and its implications, including the referenced arXiv paper on self-consuming models, adds a critical layer about data quality and model degradation. The guest’s candid remarks on censorship and the iterative nature of model improvement offer valuable insights for both newcomers and experienced practitioners.

Pour aller plus loin :

Transformer (machine learning model) — Foundational architecture for modern LLMs, discussed in the historical overview.
Quantization (signal processing) — General concept applied to neural network compression.
LoRA (Low-Rank Adaptation) — The original paper on LoRA, a key technique for efficient fine-tuning.
Self-Consuming Generative Models Go MAD — Paper on the risks of training on synthetic data, cited in the episode.

157 words

Radar Profile

The radar profile shows high scores in information quantity, quality, and reliability, reflecting the episode's dense, accurate, and well-sourced content. The technical level is moderately high, suitable for an audience with some background in machine learning. The overall shape indicates a strong, trustworthy source that balances depth with accessibility.

Reliability 8/10