benchmark-optimization-loop

Use it when:: You're asked to speed something up, try dozens of optimizations, or pick the best implementation by repeated measurement.
It does:: Pins down the operation, metric, correctness gate, baseline and search budget, then benchmarks variants that each test one hypothesis on the same input.
You get:: A table of variants with hypothesis, command, time and correctness, and the winner codified in a script, test, config or doc.
Promotes:: A variant only when correctness tests pass, the gain is repeated or explained, rollback is obvious and the change is in source control or a runbook.

TL;DR

ECC's answer to requests like "make it 20x faster": keep the ambition, but make the loop bounded and measurable. Nothing gets optimized until five things exist: the operation, the correctness gate that must stay green, the metric (wall time, p95 latency, rows per second, cost per run, memory or error rate), the current baseline, and a budget of variants, time, spend and data impact. The agent then measures the baseline, finds bottlenecks from evidence, runs variants that each test one hypothesis on the same input shape, rejects any that fail correctness, safety or reproducibility, and promotes the fastest safe one. It codifies the winner and reruns baseline and winner to confirm the gain. For recursive or hyperparameter searches it logs every run to a ledger, compares against the prior accepted winner rather than the last run, keeps a holdout or replay check, and stops when gains fall within noise, correctness fails or the budget runs out. It calls the result the "best measured safe variant", not a global optimum, unless the search was exhaustive, and its final summary gives exact commands and measurements.

Summarizes the skill at d2ef268.

npx skills add https://github.com/affaan-m/ecc --skill benchmark-optimization-loop

skills.sh page Source on GitHub All skills from affaan-m/ecc

Total Downloads: 5,300 on skills.sh, Nov 9, 2026
Repo stars: 299,776 affaan-m/ecc
First seen: May 25, 2026 on skills.sh

Weekly Downloads

446

+69% over the 8 weeks to Nov 9

Weekly values

Week to	Installs
Sep 21	264
Sep 28	164
Oct 5	148
Oct 12	235
Oct 19	413
Oct 26	404
Nov 2	475
Nov 9	446

Category	category:debugging category:software-engineering Find root causes, review code and test what you built. Everything for building software: planning, coding, debugging, design, frameworks, cloud, agents and skills.
Type	type:workflow A multi-step process the agent follows: interview, debug, run a TDD loop, review.
Effect	effect:local Its own steps create or change files, git state or local tools.
Needs	needs:none Instructions only. Works in any agent that supports skills.
Publisher	by:community Everyone else. Unverified vendor claims default to community.
Topics	performance verification

SKILL.md

From affaan-m/ecc at d2ef268. Checked daily, last changed Nov 6, 2026.

Frontmatter

name: benchmark-optimization-loop
description: Convert 'make it faster' requests into a bounded measured optimization loop — baseline first, generate one-hypothesis variants, benchmark each against a correctness gate, and promote the fastest safe variant with reproducible commands. Use when asked to speed something up, try many variants, run recursive optimization, benchmark latency/throughput/cost, or pick the best implementation by repeated measured tests.
license: MIT
metadata:
  origin: ECC
tools: Read, Write, Edit, Bash, Grep, Glob

Benchmark Optimization Loop

Use this skill to convert "make it 20x faster" or "try 50 recursive optimizations" into a bounded measured loop that can actually improve a system.

Required Baseline

Do not optimize until these exist:

the operation being optimized;
the correctness gate that must stay green;
the metric: wall time, p95 latency, rows/sec, cost/run, memory, error rate;
the current baseline;
the search budget: max variants, max time, max spend, max data impact.

If the user asks for an unrealistic target, keep the ambition but make the loop bounded and measurable.

Loop

Measure the baseline.
Identify bottlenecks from evidence.
Generate variants that test one hypothesis each.
Run variants with the same input shape.
Reject variants that fail correctness, safety, or reproducibility.
Promote the fastest safe variant.
Codify the winning path in a script, command, test, config, or doc.
Rerun the baseline and winner to confirm the delta.

Variant Table

Track variants like this:

Variant | Hypothesis | Command | Time | Correct? | Notes
baseline | current path | npm run job | 120s | yes | stable
batch-500 | fewer round trips | npm run job -- --batch 500 | 42s | yes | winner
parallel-8 | more workers | npm run job -- --workers 8 | 31s | no | rate limited

Recursive Search

For recursive or hyperparameter work:

persist every run to a ledger;
compare against the prior accepted winner, not only the previous run;
keep a holdout or replay check;
stop when improvement is within noise, correctness fails, cost exceeds the budget, or the search starts changing more variables than it can explain.

Use phrases like "best measured safe variant" instead of "global optimum" unless the search space was actually exhaustive.

Promotion Gate

A variant cannot become the new default until:

correctness tests pass;
the performance delta is repeated or explained;
rollback is obvious;
the change is encoded in source control or a durable runbook;
the final summary includes exact commands and measurements.