International Journal For Multidisciplinary Research

E-ISSN: 2582-2160   •   Impact Factor: 9.24

A Widely Indexed Open Access Peer Reviewed Multidisciplinary Bi-monthly Scholarly International Journal

Call for Paper Volume 8, Issue 6 (November-December 2026) Submit your research before last 3 days of December to publish your research paper in the issue of November-December.

MoSLM: Sparse Orchestration of Specialized Small Language Models

Author(s) Mr. Damodhar Meshram, Mr. Manas Tarare, Ms. Samruddhi Tonghale
Country India
Abstract This proposal presents MoSLM (Mixture of Small Language Models), a task-level orchestration system inspired by Mixture-of-Experts routing. A small orchestrator (1–3B parameters) decomposes requests into typed subtasks and routes each to suitable small expert models for math, code, retrieval QA, tool calling, and general chat; low-confidence subtasks can be escalated to a larger model.
The proposed orchestrator training has two stages: supervised routing from oracle labels, followed by reinforcement learning (GRPO) with a reward that combines correctness, serving cost, latency, and a regularizer intended to discourage expert collapse. Capability embeddings will be used to test whether new experts can be added without retraining the orchestrator.
The study will test the hypothesis that MoSLM can match or beat a monolithic 8–14B model on mixed reasoning, coding, and tool-use benchmarks while reducing measured serving cost per successful query by 40–60%. This is a target, not a result; cost will include orchestration, retrieval, model loading, and fallback. If completed, planned artifacts include the code, router weights, and MixBench.
Keywords Keywords: Small language models; LLM orchestration; task-level routing; expert models; cost-aware inference; reinforcement learning; tool use; retrieval-augmented question answering.
Field Computer > Artificial Intelligence / Simulation / Virtual Reality
Published In Volume 8, Issue 6, November-December 2026
Published On 2026-11-05

Share this