Learning Agent Skills Without Touching Weights: What Microsoft's SkillOpt Changes¶

For / Key Points

For: Developers and technical leads writing SKILL.md files or custom agent instructions for Claude Code, Codex CLI, GitHub Copilot, or similar agent systems.

Key Points:

SkillOpt freezes the model and optimizes the natural-language skill document as trainable external state
Candidate edits are accepted only when they improve held-out validation, turning self-editing into a gated learning loop
The deployed artifact is a compact best_skill.md, with no extra model calls at inference time

Agent skills are easy to write and hard to improve. You can put procedures, verification steps, and output formats into a SKILL.md, but the document does not tell you whether it actually got better. In many teams, the loop stops at manual edits or a one-shot request to a stronger model.

This article answers one question. How can a skill document learn from execution feedback without changing the model weights?

Skills Were Still Hand-Maintained External Memory¶

SkillOpt starts from a practical complaint: current skill creation is mostly hand-written, generated once, or loosely self-revised. The paper argues that none of those patterns behaves like an optimizer that reliably improves the skill from feedback¹.

SkillOpt changes the status of the document, not the model. It treats the skill as trainable external state for a frozen agent. This is not fine-tuning. It is text-space optimization over a natural-language procedure file.

That framing matters for people already writing skills in Claude Code or Codex. A team-maintained SKILL.md can easily become a document that feels right but has no measured improvement loop. SkillOpt adds scores, candidate edits, and an acceptance rule.

The Learning Loop Has Four Actions¶

One SkillOpt step consists of rollout, reflection, edit, and gate. The order is the method. Run the current skill, inspect successes and failures, propose bounded changes, then keep only the candidate that wins validation.

Rollout: The frozen target model runs tasks with the current skill and records messages, tool calls, verifier feedback, metadata, and scores
Reflect: A separate optimizer model analyzes successful and failed minibatches to find reusable procedures
Edit: Add, delete, and replace operations are merged and ranked under a textual edit budget
Gate: A candidate skill becomes current only if it improves held-out selection performance

The project page explicitly maps this to learning terminology. Rollout evidence acts like a forward pass, reflection acts like a language-level backward pass, and the textual learning rate limits how far the skill can move². The weights do not move. The document moves.

Without the gate, this would be ordinary self-editing. With the gate, it becomes propose-and-test optimization. Rejected edits, slow updates, and an optimizer-side meta skill preserve longer-horizon feedback without bloating the deployed artifact².

The ablations support that interpretation. On the project page, removing both meta skill and slow update drops the SpreadsheetBench value from 77.5 to 55.0³. The boring constraints are doing real work.

In ALFWorld, Search Rules Emerged¶

The ALFWorld example shows what SkillOpt is actually learning. In that run, GPT-5.4-mini is the frozen target model and GPT-5.5 is the optimizer model. The selection-gate score rises from 68.6% to 81.4%, while the final hard test score improves from 70.9% to 85.8%⁴.

The accepted rules are not generic encouragement. They are concrete operating rules for household search tasks.

# best_skill.md (ALFWorld excerpt)
- Count any generic target receptacle instance as valid.
- Keep a strict numbered searched set and do not re-check observed locations.
- Broaden search after several misses in one location type.

The important part is that a candidate with better-looking training rollout can still fail selection and be rejected. If the held-out gate does not improve, the candidate does not become the current skill. That is the line between prompt self-improvement and SkillOpt.

The Artifact Is a Portable Single File¶

The deployment shape is the most practical part of SkillOpt. The final artifact is typically a 300- to 2,000-token best_skill.md⁵. At runtime, the target model consumes only that file. The optimizer model and training memory stay behind.

The evaluation reports 52 assessed cells across six benchmarks, seven target models, and three execution harnesses: direct chat, Codex, and Claude Code. SkillOpt is reported as best or tied-best on all 52 cells¹. For GPT-5.5 in direct chat, it improves average no-skill accuracy by +23.5 points.

Harness	Average gain over no-skill GPT-5.5
Direct chat	+23.5
Codex	+21.8
Claude Code	+18.6

This table follows the benchmark-by-benchmark main table on the official project page⁶. The arXiv abstract and README/PyPI overview also report summary values of Codex +24.8 and Claude Code +19.1¹⁵⁷. Here, the benchmark-level table is used because its component values reproduce the displayed averages.

The transfer results are also relevant. The project page reports cross-model transfer at +15.2, Codex-trained SpreadsheetBench skill transfer into Claude Code at +31.8, and a self-optimizer setting at +10.4⁸. Those gains are measured against the destination no-skill baseline, without further target-side optimization.

The Method Works Only Where Validation Exists¶

SkillOpt is not a universal wrapper for every business workflow. Its first requirement is a scoring mechanism. If rollouts cannot be scored and a held-out validation set cannot be defined, the gate has nothing to enforce.

It fits tasks with clear correctness signals. SearchQA, SpreadsheetBench, DocVQA, LiveMathematicianBench, OfficeQA, and ALFWorld are close to that shape⁷. Ambiguous planning, negotiation, editorial taste, and long open-ended workflows require evaluation design first.

The cost also moves from inference to training. Deployment is light, but training needs many rollouts and optimizer calls. That is reasonable for a skill used repeatedly over a long period. It can be overkill for short-lived tasks.

The optimizer model still matters. The project page says stronger optimizer models give the largest gains, while matched target-as-optimizer settings can still discover useful edits when updates are constrained, buffered, and validated⁸. So this is not merely distillation from a stronger model, but the quality of reflection and edit proposals still sets part of the ceiling.

Start With an Eval Set, Not the WebUI¶

SkillOpt v0.1.0 was released on PyPI on June 2, 2026 and requires Python 3.10 or later⁷. The README describes OpenAI, Azure, Claude, Qwen, and MiniMax backends, six built-in benchmarks, and a WebUI dashboard⁵.

pip install skillopt

# If you want the WebUI dependencies
pip install "skillopt[webui]"
python -m skillopt_webui.app

The first practical step is not the UI. It is a small evaluation set. Even 10 cases help if they define the input, expected result, scoring rule, and held-out split.

Without that set, SkillOpt becomes an automated prompt editor. With that set, even manual skill maintenance improves. Propose a small edit, compare it on fixed tasks, and keep it only if it wins.

For local coding agents, Microsoft also publishes SkillOpt-Sleep as a preview. It reviews past Claude Code, Codex, or Copilot sessions at night, replays recurring tasks, and stages only validation-gated skill updates for adoption⁹. Microsoft also links SkillLens as a related project for studying model-generated agent skills¹⁰.

Summary¶

SkillOpt is not just another claim that agents can improve themselves. Its useful contribution is the discipline it brings to natural-language skill documents.

A held-out validation gate and a bounded textual learning rate change the character of self-editing. They turn unlimited rewriting into a measured improvement procedure. Remove those two constraints and automatic prompt improvement can drift quickly.

The idea is usable even without adopting SkillOpt. Before editing a SKILL.md, freeze the tasks you cannot afford to regress. Keep edits small. Retain only changes that beat the current document on that fixed set. Skill quality is governed less by prose polish than by the acceptance mechanism around the prose.