Skeleton-of-Thought
Standard generation is strictly sequential — each token waits for the previous one. Skeleton-of-Thought breaks this bottleneck by first generating a concise outline, then expanding each point independently and in parallel, dramatically reducing latency while producing well-structured responses.
Introduced: Skeleton-of-Thought (SoT) was introduced in 2023 by Ning et al. in “Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding.” The research demonstrated that LLM response latency could be reduced by up to 2x by splitting generation into two phases: first producing a skeleton (outline of key points), then expanding each skeleton point independently. Because the expansion of each point does not depend on the others, these expansions can run in parallel across multiple API calls or batched requests. The technique was tested on GPT-4, Claude, and LLaMA models across diverse question types.
Modern LLM Status: Skeleton-of-Thought has become an important latency optimization technique for production AI systems. The core insight — that many responses can be decomposed into independently expandable sections — influences modern streaming architectures and parallel generation strategies. While native LLM inference is still autoregressive (sequential token generation), SoT’s principle has been adopted in practice through parallel API calls, speculative decoding research, and structured output generation. The technique is particularly valuable for long-form content generation, multi-part answers, and any application where response latency affects user experience.
Outline First, Expand in Parallel
Traditional LLM generation is brutally sequential: every token depends on every previous token. A 2,000-word response requires the model to generate each word one after another, with no opportunity for parallelism. The first sentence must be complete before the second begins, the first paragraph before the second. For complex, multi-part responses, this sequential bottleneck means users wait while the model slowly writes from beginning to end.
Skeleton-of-Thought exploits a key observation: many responses have independently expandable sections. When asked to explain a concept, the model might produce five key points. Each point can be elaborated without knowing the details of the others — only the skeleton (the outline of points) needs to be generated sequentially. Once the skeleton exists, all expansions can happen simultaneously, collapsing a serial process into a parallel one.
Think of it like building a house. The framing (skeleton) must go up first — you need to know where the rooms are. But once the frame is in place, you can wire the electrical, plumb the pipes, and install the drywall all at the same time. You don’t need to finish the kitchen before starting the bathroom. Skeleton-of-Thought applies this same parallel-after-planning principle to text generation.
The beauty of Skeleton-of-Thought is that the speed improvement comes from better organization, not from cutting corners. The skeleton phase forces the model to plan its response structure before writing, which often produces more coherent and well-organized output than standard sequential generation. You get faster responses AND better structure — the outline-first approach is a rare case where the optimization also improves quality.
The Skeleton-of-Thought Process
Three phases from question to parallel-generated response
Generate the Skeleton
The model receives the question and produces a concise outline — a numbered list of key points, sections, or arguments it plans to cover. This skeleton captures the structure of the complete response in a compact form. The skeleton generation is sequential (fast, since it produces only short bullet points) and provides the blueprint for parallel expansion.
Question: “What are the advantages of renewable energy over fossil fuels?”
Skeleton:
1. Environmental impact and emissions reduction
2. Long-term cost economics
3. Energy independence and security
4. Job creation and economic growth
5. Technological scalability
Expand Each Point in Parallel
Each skeleton point is sent as an independent generation request. Because no expansion depends on any other expansion, all of them can run simultaneously. Each expansion prompt includes the original question, the full skeleton (for context), and the specific point to elaborate. The model produces a detailed paragraph or section for each point independently.
Five parallel API calls, each expanding one skeleton point. Point 1 expands into a paragraph about carbon reduction and climate impact. Point 3 expands into a paragraph about reducing dependence on imported fossil fuels. All five complete roughly simultaneously rather than sequentially.
Assemble the Final Response
The expanded sections are collected and concatenated in skeleton order. Optional post-processing can add transitions between sections, an introduction, or a conclusion. The result is a complete, well-structured response generated in significantly less wall-clock time than sequential generation would require.
The five expanded paragraphs are assembled into a cohesive response with the skeleton providing the heading structure. A brief introduction and conclusion are added. Total generation time: roughly the time of the skeleton plus the longest single expansion — not the sum of all expansions.
See the Difference
How parallel expansion dramatically reduces response time
Sequential Generation
Explain the advantages of renewable energy over fossil fuels in detail.
The model generates the entire response token by token, from the first word of the introduction through each advantage in sequence to the conclusion. Total time: sum of all tokens generated. For a 2,000-word response at typical generation speeds, this can take 30–60 seconds.
Skeleton-of-Thought
Phase 1: Outline the key advantages (skeleton). Phase 2: Expand each advantage in parallel.
Phase 1 produces a 5-point skeleton in 2–3 seconds. Phase 2 expands all 5 points simultaneously — each taking 5–8 seconds, but running in parallel. Total time: skeleton time plus the longest expansion, roughly 10–11 seconds. Same quality, up to 2x faster.
Natural Language Works Too
While structured frameworks and contextual labels are powerful tools, LLMs are exceptionally good at understanding natural language. As long as your prompt contains the actual contextual information needed to create, answer, or deliver the response you’re looking for — the who, what, why, and constraints — the AI can produce complete and accurate results whether you use a formal framework or plain conversational language. But even in 2026, with the best prompts, verifying AI output is always a necessary step.
Skeleton-of-Thought in Action
See how outline-first generation accelerates structured responses
“Write a comprehensive guide to setting up a CI/CD pipeline with GitHub Actions.”
Skeleton produced:
1. Prerequisites and Repository Setup
2. Workflow File Configuration
3. Build and Test Stages
4. Deployment Configuration
5. Monitoring and Notifications
6. Troubleshooting Common Issues
All 6 sections are expanded independently and in parallel. Each expansion prompt includes the original question, the full skeleton for structural context, and the specific section to elaborate. Section 1 details repository structure and required permissions. Section 3 covers YAML syntax for build matrices and test runners. Section 6 addresses permission errors, caching issues, and runner timeouts. The final assembled document has clear section headers derived from the skeleton and detailed content from each parallel expansion — produced in the time it would take to sequentially generate just two of the six sections.
“Explain the causes and consequences of the Industrial Revolution for a university-level history course.”
Skeleton produced:
1. Agricultural and Technological Preconditions
2. Key Inventions and Innovations
3. Social and Demographic Transformations
4. Economic Restructuring
5. Environmental Consequences
6. Long-Term Political Impact
Each section is expanded independently with appropriate academic depth. Section 1 covers the enclosure movement and seed drill innovations. Section 3 details urbanization patterns and the emergence of the factory system. Section 5 examines deforestation, air pollution, and resource extraction at industrial scale. The parallel generation produces the full 2,500-word essay-style response in the time it would take to generate a single section sequentially — each expansion runs simultaneously, and the assembled result maintains coherent academic tone throughout.
“Compare React, Vue, and Angular for building a large-scale enterprise web application.”
Skeleton produced:
1. Learning Curve and Developer Experience
2. Performance and Bundle Size
3. Enterprise Ecosystem and Tooling
4. Community and Long-Term Support
5. State Management Approaches
6. Verdict and Recommendation
Each section is expanded independently with specific technical comparisons. Section 2 benchmarks virtual DOM diffing, hydration costs, and tree-shaking effectiveness across all three frameworks. Section 5 compares Redux/Zustand, Pinia/Vuex, and NgRx/Signals with code-level tradeoffs. Section 6 synthesizes the findings into a context-dependent recommendation. The parallel approach produces the full comparison in under 15 seconds rather than the 45+ seconds of sequential generation — particularly valuable when developers need quick, comprehensive framework evaluations.
When to Use Skeleton-of-Thought
Best for multi-section responses where speed and structure both matter
Perfect For
When generating responses with multiple distinct sections or points that can be elaborated independently — reports, guides, analyses, comparisons.
When users are waiting for responses and reducing wall-clock time directly improves the experience — chatbots, real-time assistants, interactive tools.
When the response needs clear organization with headings, sections, or numbered points — the skeleton phase naturally produces well-structured content.
When generating many multi-section responses at scale — the parallel expansion multiplies throughput across the entire pipeline.
Skip It When
Single-paragraph responses or factual lookups where the overhead of skeleton generation exceeds the time saved by parallelism.
Creative writing, storytelling, or flowing prose where each paragraph’s content depends heavily on the specific words of the previous paragraph.
Complex logical arguments where later points build directly on the specific details (not just the topic) of earlier points — these resist independent expansion.
Use Cases
Where Skeleton-of-Thought delivers the most value
Real-Time Chat Assistants
Reduce response latency for complex questions by generating the outline quickly, then expanding sections in parallel while streaming results to the user.
Report Generation
Produce multi-section business reports, research summaries, or analytical documents faster by parallelizing section writing after establishing the structure.
Educational Content
Generate lesson plans, study guides, or course materials with multiple independent sections that can be expanded simultaneously.
API Documentation
Create endpoint documentation, integration guides, or SDK references where each section (authentication, endpoints, errors, examples) is independent.
Product Descriptions
Generate detailed product pages with independently expandable sections: features, specifications, comparisons, and use cases.
Email Campaigns
Draft multi-section newsletters or marketing emails where each content block can be generated independently after establishing the overall theme and structure.
Where Skeleton-of-Thought Fits
From sequential tokens to parallel generation strategies
Skeleton-of-Thought embodies a fundamental computer science principle: identify the serial dependencies in a process, minimize them, and parallelize everything else. In traditional generation, the serial dependency spans the entire response — every token depends on all previous tokens. SoT reduces the serial dependency to just the skeleton, making the bulk of the generation (the expansions) embarrassingly parallel. This same principle drives performance optimization across all of computing, from CPU pipelines to distributed systems.
Related Techniques & Frameworks
Explore complementary decomposition and generation techniques
Accelerate Your AI Outputs
Design structured, parallel-friendly prompts with our Prompt Builder or explore decomposition techniques that organize complex reasoning.