Skills Lab: Building and Testing a ChatGPT Systems Strategist Skill

This builds on the research Skill I built and tested, with a different question: can a reusable capability preserve decision discipline?

A Skill can look perfectly designed on paper and still behave badly in practice.

It can activate when it should not. Fail to activate when it should. Overcomplicate a simple task. Follow its instructions but still make the wrong intervention. Or quietly expand beyond the capability it was supposed to own.

That is why I did not want to judge my Systems Strategist Skill by reading SKILL.md and deciding that the instructions looked intelligent.

I wanted to see how it behaved.

The question behind this experiment was:

Could I design a reusable ChatGPT Skill that knows when to stay out of the way, when to intervene, how to reason through a systems problem, and where its responsibility ends?

So I built version 0.1 and tested it in two ways.

First, I ran four controlled before-and-after tests designed to isolate specific behaviors.

Then I explicitly asked it to invent a realistic, more ambiguous scenario and use the Skill end to end. This was an illustrative demonstration, separate from the paired activation tests.

The result was more interesting than:

“The Skill made ChatGPT smarter.”

The underlying model was already capable.

What changed was the discipline, scope, and order of the reasoning.

That became the central lesson from the experiment.

Project snapshot

The basic philosophy behind the Skill is:

Understand the system producing the outcome before choosing the intervention.

Its operating loop is:

That sounds straightforward.

The harder part was deciding when that process should be used at all.

A request to rewrite an email should still get an email rewrite.

A recurring workflow problem should probably trigger systems reasoning.

A request that already contains a proposed solution — “hire another person,” “buy an AI chatbot,” “use an AI agent” — should treat that solution as a hypothesis rather than accepting it as the diagnosis.

And once a systems diagnosis turns into a specialized AI architecture problem, the Systems Strategist should recognize that another capability owns the next layer of the decision.

That gave me four behaviors to test:

• Stay out of the way
• Recognize the right problem
• Challenge a premature solution
• Know where the capability ends

How I ran the experiment

I used GPT-5.6 Sol Light, as shown in my test environment, for every controlled run.

Each prompt was tested twice.

The baseline condition used a fresh conversation with the Systems Strategist disabled.

The Skill condition used another fresh conversation with the Systems Strategist enabled.

The exact prompt remained unchanged.

BASELINE

• Fresh conversation
• Skill disabled
• Exact prompt
• Save response

↓

SKILL CONDITION

Fresh conversation

Skill enabled

• Same exact prompt
• Save response

I did not explicitly ask ChatGPT to use the Skill.

That was intentional.

Activation itself was part of the experiment.

I wanted to know whether the Skill would recognize when it belonged in the conversation, and equally importantly, whether it would remain silent when it did not.

These are representative tests, not a formal benchmark. AI outputs are probabilistic, so I am not claiming every model, product version, account, or repeated run will behave identically.

The purpose was narrower:

Does adding the Skill change the behavioral pattern of the response in the direction I designed for?

Controlled-test conditions

An implementation discovery before the first test

I learned something useful before I had even started testing the reasoning.

My original Skill description was too long for the description field.

That forced me to shorten it.

The resulting selector was roughly:

Diagnose recurring system/process problems, bottlenecks, broken handoffs, and unclear priorities. Use when the user asks why an outcome keeps recurring, what to fix first, or how to test an improvement. Do not use for rewriting, summarization, translation, factual lookup, formatting, calculations, or routine coding.

The limitation actually forced a better design.

I had initially tried to make the description carry too much behavioral logic.

The character limit pushed me toward a cleaner separation:

• DESCRIPTION
• Should this Skill load?

• SKILL.md
• What should the Skill do after loading?

That distinction is now something I would deliberately preserve in future Skills.

I also found that the desktop and browser interfaces in my current setup exposed different Skill-management controls.

On desktop, I could explicitly enable or disable an installed Skill.

That made desktop useful for the controlled test because I could keep the installation constant while changing the relevant condition from disabled to enabled.

I did not see the identical control in my browser interface, so I am treating this as an observation from my current environment rather than a universal statement about every ChatGPT interface.

With the setup established, I froze version 0.1 and started testing.

Test 1: Can the Skill stay out of the way?

Must not activate — boundary test

The first test was intentionally boring.

That was the point.

Test prompt

Rewrite this email so it sounds friendly, clear, and professional:

Hi Jordan,

We are still waiting for the information requested last Tuesday. Please send it by Friday so we can finish setting up your account.

Thanks.

There is no meaningful systems problem here.

No bottleneck needs diagnosing.

No process map is required.

No constraint needs identifying.

The right answer is simply a better email.

Baseline

The baseline response rewrote the message clearly and professionally.

Skill enabled

The Skill-enabled response did essentially the same thing.

More importantly:

I observed no Systems Strategist activation in this run, and the response stayed within the rewrite task. The public transcript shows that behavior, but is not a complete log of internal Skill selection.

The two rewrites were slightly different, but nothing in that difference should be attributed to the Skill.

The important result was invisible:

The first success was that the Skill stayed out of the way.

A capability that activates everywhere eventually becomes friction.

If a Systems Strategist turns a normal writing task into a discussion of stakeholders, dependencies, feedback loops, and constraints, then it has poor boundaries even if the reasoning itself is sophisticated.

Result: PASS Author-assessed · this test run

Why it passed: the task remained a straightforward rewrite and the Skill did not contaminate an unrelated request with systems analysis.

Test 2: Can it recognize a genuine systems problem?

Must activate — systems diagnosis

The second test introduced an operational problem with a proposed solution already embedded inside it.

Test prompt

Our customer onboarding has become slower even though we hired two more specialists. New customers are waiting longer to get set up, employees are working overtime, and we have started receiving more complaints. We believe we need to hire another person immediately.

Help us figure out what is actually causing the delay and what we should test before adding more headcount.

The organization already has an answer:

Hire another person.

But there is an obvious reason to question that conclusion.

Two people were already added while throughput and working conditions became worse.

The useful question is therefore not:

How quickly should another person be hired?

It is:

Where is the system actually constrained?

Baseline

The baseline answer was good.

It immediately questioned the staffing assumption and generated a broad inventory of plausible causes: queues at a specific stage, new-hire training load, work-in-progress overload, incomplete customer information, rework, downstream approvals, changing complexity, prioritization problems, tooling friction, and genuine demand exceeding available capacity.

It also proposed several useful tests.

This was not an example of the base model failing.

Skill enabled

The Skill-enabled answer covered many of the same possibilities, but it organized the reasoning differently.

It began with:

Known facts: Customer wait time increased. Overtime increased. Complaints increased. Two specialists were added.

Still an assumption: A shortage of specialists is causing the delay.

That distinction became important.

Instead of primarily generating possible explanations, the response imposed a decision sequence:

It also emphasized the distinction between touch time and wait time.

If an onboarding requires only a few hours of productive work but takes many days to complete, adding another specialist may not remove the dominant source of delay.

The problem could instead be waiting for customer information, internal approvals, downstream teams, rework, interruptions, or excessive work in progress.

The answer eventually converged on a bounded diagnostic rather than simply adding another intervention: run a short flow audit, introduce a work-in-progress limit, improve intake quality, and observe what actually happens before committing to more recurring headcount.

What changed?

Both answers mapped the process, separated waiting from active work, and proposed tests before hiring. The baseline produced a broader diagnostic inventory.

The Skill-enabled answer produced a more disciplined way of narrowing that inventory.

That gave me one of the clearest conclusions from the experiment:

The Skill did not necessarily produce more ideas. It produced a more disciplined way of deciding which ideas deserved attention.

Result: PASS Author-assessed · this test run

Why it passed: the Skill activated, challenged the embedded staffing assumption, separated facts from assumptions, and moved toward constraint validation before recommending another hire.

Test 3: Will it challenge a premature AI solution?

Guardrail test — diagnosis before AI

The third test deliberately framed an operational problem as a technology-buying decision.

Test prompt

Our customer support team is overwhelmed, response times are getting worse, and customer satisfaction is declining. We need to implement an AI chatbot as quickly as possible.

Tell us what chatbot platform to buy and how to deploy it.

The implied reasoning is:

But that sequence skips the diagnosis.

A chatbot could help.

It could also automate the wrong work.

Baseline

The baseline answer was again strong.

It did not blindly accept the premise.

It warned against treating a standalone chatbot as an emergency fix and recommended, where appropriate, starting with AI capabilities already integrated into the existing support platform.

It then provided a substantial rollout plan, platform guidance, procurement considerations, metrics, testing advice, and human-escalation controls.

As a chatbot implementation answer, it was useful.

The baseline also included a substantive diagnosis step: it distinguished repetitive demand from engineering delays, approvals, routing, and poor knowledge. Its longer answer nevertheless devoted substantial space to the original buying and deployment question:

Which chatbot should we buy and how should we deploy it safely?

Skill enabled

This time, I could see the Skill activate.

After offering provisional platform options, the response stated:

“Before buying, establish what is actually producing the delay.”

That made the diagnostic condition explicit, although it was also present in the baseline.

The answer asked whether the real constraint was repetitive knowledge-based demand, routing, escalation, technical investigation, approval latency, staffing, engineering dependencies, or another part of the support system.

It then established a condition for AI:

A chatbot is a strong intervention only if a meaningful share of demand is repetitive and knowledge-based.

If the real delays come from investigation, approval queues, engineering dependencies, or complex-case capacity, then the chatbot may not remove the constraint.

It may simply create a faster front door into the same backlog.

The closing line captured the change particularly well:

“The platform is not the strategic decision. The strategic decision is which demand should be removed, which work should be accelerated, and which conversations should remain human.”

The Skill did not reject AI.

It changed the order of operations:

instead of:

What changed?

The baseline was arguably the better chatbot buying and deployment plan.

The Skill-enabled response made the conditional decision especially explicit in its closing rule. Both responses investigated whether a chatbot fit the problem, and both offered platform guidance. This pair supports a difference in emphasis; it does not establish that only the Skill-enabled model diagnosed the constraint.

The Skill did not refuse the technology. It met the intended guardrail by making AI adoption conditional on the diagnosis. The baseline met much of that guardrail too.

Result: PASS Author-assessed · this test run

Why it passed: the Skill activated, resisted premature technology selection, reframed the chatbot as a proposed intervention, and tied AI adoption to evidence about the actual support constraint.

Test 4: Does the Skill know where its responsibility ends?

Handoff test — know where responsibility ends

The fourth controlled test introduced both a systems problem and a specialized AI design question.

Test prompt

Our legal-operations team spends 20–30 minutes reviewing each incoming vendor contract to identify renewal dates, payment terms, termination clauses, and non-standard language. The work is repetitive, the backlog is growing, and business teams are waiting too long for answers.

We think the bottleneck is manual document review. Should we use an AI agent to review the contracts? If so, what should the system do?

This contains two different questions:

Is manual document review really the constraint?

And, if it is:

How should an AI-assisted contract-review system be designed?

That made it useful for testing the boundary between general systems diagnosis and specialized AI-system design.

Baseline

The baseline response was excellent as an implementation specification.

It recommended a controlled contract-review copilot rather than an autonomous “AI lawyer” and covered document ingestion, field extraction, clause evidence, playbook comparison, deadline calculations, routing, human approval, evaluation, and phased deployment.

In several respects, it was the stronger product specification.

Skill enabled

The Skill-enabled answer paused one step earlier.

It said:

“Your evidence strongly suggests manual review is consuming capacity, but you should first confirm whether the delay occurs during review itself or in intake, assignment, missing information, or approval. Otherwise, AI may simply move the queue downstream.”

Only after establishing that boundary did the answer move into system design.

It recommended a structured workflow using document processing, deterministic calculations, retrieval from an approved legal playbook, LLM interpretation where judgment could add value, and explicit human review.

Then came another line that captured the boundary well:

“A highly autonomous agent adds complexity before you know where autonomy is valuable.”

The closing decision rule also reflected the Skill's operating logic:

In the response’s closing rule, proceeding depended on reduced reviewer time within legal’s risk thresholds. If extraction succeeded without reducing review time, the diagnosis needed revisiting.

What changed?

The baseline essentially said:

Here is how to design this AI system well.

The Skill-enabled response said:

First verify that this is the right system problem. Then move into AI-system design.

That is the boundary I wanted.

There is one important limitation, however.

I have not yet built and installed a separate AI Systems Strategist Skill, so this test did not demonstrate a literal Skill-to-Skill handoff.

What it demonstrated was a conceptual handoff: the Systems Strategist recognized where systems diagnosis ended and where specialized AI architecture thinking should begin.

Result: PASS Author-assessed · this test run

Why it passed: the Skill activated, challenged the assumed bottleneck, guarded against unnecessary autonomy, and recognized the transition from systems diagnosis into AI-system design.

Beyond the controlled tests: where the Skill really starts to earn its keep

The four tests above were intentionally clean.

That made the before-and-after comparisons easier to interpret.

But the Systems Strategist is not really designed for neat problems.

It becomes more useful when several teams disagree, multiple explanations are plausible, proposed solutions are already competing for attention, different parts of the system interact, and the highest-leverage next move is not obvious.

After the controlled experiment, I explicitly selected the Skill and asked ChatGPT to make up a realistic prompt and demonstrate its full procedure. It generated the B2B renewal scenario below. This was a synthetic example, not a real company case, an independent test prompt, or an additional automatic-activation test.

A software company had introduced a mandatory security review for enterprise renewals.

Six months later, renewals were taking longer, account managers were escalating more deals, and some customers were receiving revised contracts only days before expiration.

Three teams had three explanations:

• Security: We need more analysts.
• Sales: We should automate the questionnaire with AI.
• Legal: Account managers are submitting incomplete information and creating avoidable rework.

Leadership wanted to know what was actually causing the delay, what should be fixed first, and whether automation would help.

This is much closer to the kind of problem I built the Skill for.

There is no obvious single answer.

Every team's explanation is plausible.

Every team already has a preferred intervention.

A capable language model can easily generate arguments for all three.

The difficult part is narrowing them.

It did not pick a side

The Skill first separated observed facts from unconfirmed assumptions.

Renewal completion times had increased after the security review became mandatory. Escalations had increased. Some contracts were arriving dangerously close to expiration. Security, Sales, and Legal all participated in the process. Rework occurred when submissions were incomplete.

But several important claims remained unproven.

Was Security actually under-capacity?

Were incomplete submissions the dominant source of delay?

Would AI remove the bottleneck?

Did the new security requirement itself explain the entire slowdown?

Would another analyst materially improve end-to-end renewal time?

The timing made Security a strong lead.

It did not prove analyst capacity was the constraint.

It bounded the relevant system

Rather than expanding the investigation indefinitely, the Skill narrowed the scope to the renewal path from preparation through customer delivery.

It focused on questions like:

• When does renewal work actually begin?
• Is required information complete when Security receives it?
• How long does work wait before an analyst touches it?
• How long does active review take?
• How often does work return for clarification?
• How much time is spent waiting on account managers?
• Does Legal wait unnecessarily for Security?
• Do urgent escalations interrupt the normal queue and create even more urgent work?

Ambiguous problems become harder when the system boundary keeps expanding.

The Skill tried to identify the smallest relevant system capable of explaining the outcome.

It formed a candidate constraint

The working hypothesis became:

A poor-quality intake and rework loop may be feeding a capacity-limited review queue.

That is more useful than simply choosing among:

If analysts repeatedly return incomplete submissions, available capacity is being consumed by preventable work.

Escalations could then amplify the problem.

Analysts leave planned work to handle urgent renewals.

Normal work becomes delayed.

Those renewals become urgent.

More escalations appear.

And the loop reinforces itself.

Crucially, the Skill still treated this as a hypothesis, not a conclusion.

It asked what would prove the diagnosis wrong

This was one of the strongest parts of the response.

The Skill did not merely propose a theory and then search for confirming evidence.

It identified evidence that should cause the diagnosis to be rejected.

If most submissions were already complete, clarification consumed little time, complete submissions still waited excessively, active review consumed nearly all available analyst capacity, or most of the delay occurred after Security approval, then the current hypothesis would weaken.

At that point, genuine capacity or a downstream handoff would become the stronger explanation.

A diagnosis becomes much more valuable when you also define what evidence would cause you to abandon it.

It converted the diagnosis into a reversible experiment

The Skill then proposed a small two-week pilot rather than hiring immediately or deploying AI:

For one renewal segment: begin the security process earlier, replace free-form intake with structured requirements, check completeness before work enters the analyst queue, prevent informal escalation from constantly disrupting the pilot, and measure waiting time separately from active review time.

The intervention is relatively small, reversible, and directly tests the suspected mechanism.

It established what happens next

The response also defined decision rules.

If rework falls and renewal completion improves materially, standardize the intake process before adding permanent capacity.

If intake improves but complete submissions continue to wait, analyst capacity or prioritization becomes the stronger hypothesis.

If Security turns work around quickly but customers still receive contracts late, move the investigation downstream.

If results remain mixed, expand the test rather than committing to a company-wide technology purchase.

The Skill therefore did not merely produce a diagnosis.

It created a way to learn which diagnosis deserved the next investment.

Only then did AI enter the picture

Automation was not rejected.

It was delayed until the system had earned it.

AI might eventually help classify questionnaires, retrieve approved security responses, identify missing information, or draft standard answers.

But choosing an autonomous agent before understanding whether the real constraint is information retrieval, judgment-intensive review, intake quality, queue capacity, or something downstream could simply automate an unstable process.

Incomplete work might reach Security faster.

Or generated answers might still require complete human verification.

Neither would necessarily improve renewal throughput.

That is the Skill operating end to end:

Reframe proposed solutions as hypotheses → separate evidence from assumptions → bound the relevant system → identify a candidate constraint → define contradictory evidence → choose a reversible intervention → establish measurements and decision rules → consider unintended consequences → stop once there is enough information to act.

That final step is important.

It stopped.

It did not recommend months of discovery.

It did not turn the problem into an academic systems exercise.

It reached the point where the next useful action was clear.

That is where I think the Skill really starts to earn its keep.

Not when the problem is obvious.

When the situation is ambiguous enough that a capable model can generate many plausible explanations, but the real need is to narrow the problem and identify the highest-leverage next action.

The controlled tests showed that the Skill could follow its boundaries. The complex example showed why those boundaries and procedures matter when the problem itself is messy.

What the experiment actually showed

I do not think the right conclusion is:

“The Skill made every response better.”

That would oversell the evidence.

The baseline model was already very capable.

In some cases its answer was broader. In others it contained more implementation detail. In Test 4, the baseline was arguably the stronger AI product specification.

The more defensible conclusion is that the Skill gave the responses a more consistent decision sequence in these examples. Repeated trials would be needed to test that pattern reliably.

Across these examples, I observed a recurring sequence:

Good systems reasoning is not about producing the most explanations.

It is about narrowing uncertainty until the next decision becomes clearer.

A Skill can encode judgment, not just instructions

Before this experiment, it was easy to think about Skills primarily as reusable instructions.

I now think that description is incomplete.

The more interesting possibility is:

A useful Skill can encode how you've learned to make a decision.

The chatbot test did not encode:

Never use chatbots.

It encoded:

Treat the proposed technology as a hypothesis until the actual constraint is understood.

The onboarding test did not encode:

Never hire more people.

It encoded:

Confirm that capacity is actually the constraint before adding capacity.

The contract-review test did not encode:

Never use autonomous agents.

It encoded:

Add autonomy only where it has earned its value.

And the complex renewal scenario did not encode:

Security is the bottleneck.

It encoded:

Form a hypothesis, define what evidence would contradict it, and test the smallest useful intervention before committing to the larger solution.

Those are not memorized answers.

They are reusable decision rules.

The strongest capability may be restraint

Many of the best results came from what the Skill did not do.

• Test 1 succeeded because it did not activate.
• Test 2 succeeded because it did not accept another hire as the obvious answer.
• Test 3 succeeded because it did not accept the user's preferred technology as the diagnosis.
• Test 4 succeeded because it did not assume maximum AI autonomy was desirable.
• The complex example succeeded because it did not choose among three competing explanations before the evidence justified doing so.

That suggests a broader definition of capability:

GOOD CAPABILITY DESIGN

• knows what to do
• +
• knows when to do it
• +
• knows what not to do
• +
• knows when enough is known to act
• +
• knows when another capability should take over

The boundary is part of the capability. Not an afterthought.

What v0.1 has not proved

Four successful controlled tests and one complex demonstration are encouraging.

They are not enough to claim that the Skill is universally reliable.

I still do not know how consistently it activates across many different phrasings.

I have not compared its behavior across multiple models or repeated the same conditions enough times to call the results a benchmark.

I also do not yet know whether moderate systems-shaped questions sometimes trigger too much analysis, whether the selector is still too broad, or how the capability behaves when several other Skills are installed at the same time.

And the T-04 handoff remains conceptual.

A separately installed AI Systems Strategist does not yet exist in this experiment, so I have not demonstrated literal capability-to-capability orchestration.

Those are future tests.

They are not reasons to rewrite v0.1 yet.

I am not changing v0.1 just because the first tests are finished

It would be easy to reach this point and immediately create version 0.2.

But that would violate the logic behind the Skill itself.

The relevant question is not:

What else can I add?

It is:

What evidence says something needs to change?

So far, I have hypotheses to investigate rather than confirmed defects.

The most interesting next tests are ambiguous activation, over-analysis, selector precision, repeated runs, and eventually real handoffs between separately installed capabilities.

A prompt such as:

“We're redesigning employee onboarding. What should we think about?”

is probably more revealing now than another obvious bottleneck scenario.

Does the Skill add just enough structure?

Or does it turn a modest planning problem into an unnecessary nine-stage diagnosis?

That is the kind of failure I want to find next.

The standard should be:

Use the minimum systems reasoning required to improve the decision.

The deeper lesson

I started this project asking:

How do I build a reusable Systems Strategist Skill?

Testing pushed me toward a better question:

What behavior should remain consistent even when the exact problem changes?

For this Skill, the answer is not a framework name.

It is a decision discipline.

Do not confuse symptoms with constraints.

Do not confuse proposed solutions with diagnoses.

Do not add complexity before it earns its place.

Do not keep researching once a useful test is available.

Do not optimize one part while damaging the whole.

Do not use AI merely because AI is available.

Those are closer to reusable judgment than reusable prose.

A prompt can tell ChatGPT what you want this time. A Skill can begin to preserve how you approach a class of problems repeatedly.

From experiment to reusable method

This Lab comes first because I want the method to follow the evidence. The next article will be the companion tutorial, How to Build a ChatGPT Skill That Actually Knows When to Help.

It will turn these observations into a practical design process: capability contracts, activation and non-activation, operating procedures, guardrails, handoffs, and testing. I will link it here once it is published.

For now, the useful loop is design, build, test, observe, refine, document, and repeat. A Skill is not validated because its instructions look impressive. Its boundaries and procedures need to hold up when the task pushes back.

Platform note

Skill availability, UI controls, and automatic activation behavior can vary by account type, workspace configuration, product surface, model, and OpenAI product updates.

The interface observations in this Lab reflect my setup at the time of testing.

OpenAI’s documentation describes Skills as reusable workflows organized around SKILL.md, with optional scripts and references. It explains that a matching description can lead to implicit activation. That supports the design principle; it does not guarantee activation on every request or validate my test results.

Current project status

This is still an experiment.

And that is exactly why I am documenting it.

Sources & updates

Evidence: This article is based on my test notes, the nine shared conversations linked above, and four annotated screenshots. Pass/fail labels are my assessments against the stated behavioral criteria. The screenshots document selected excerpts; they are not complete transcripts of all paired runs. The complex renewal scenario was generated by ChatGPT at my request with the Skill explicitly selected. It is an additional qualitative demonstration, not a fifth controlled comparison or a measured business outcome.

• OpenAI: Build skills — descriptions, implicit activation, and supporting resources.
• My earlier research Skill experiment — background and firsthand comparison.

Editorial review: September 13, 2026. Model label and test conditions are reported from my test environment, not independently verified by the screenshots. No new trials were conducted for this editorial pass. No affiliate links.