Building More Defensible AI Analytics: How I Rebuilt InsightForge

What changed from V1 to V2—and why the biggest improvement was deciding where AI should stop

Part 3 of the InsightForge series

Earlier in the series:
Part 1 — InsightForge V1
Part 2 — Architecture B / controlled analytics design

I built the first version of InsightForge around a straightforward idea: calculate important facts deterministically, then let AI explain those facts in plain language.

That solved one problem I cared about early on. The model did not need to invent revenue totals, product averages, or other analytical facts. Those values could be calculated first and supplied to the model as grounded context.

But after reviewing V1 more closely, I found a harder question:

Was the answer actually tied to the analytical question the user asked?

That distinction matters.

A number can be correct for a dataset and still be the wrong number for the population, filter combination, or business question a user intended.

InsightForge V2 became an experiment in solving that problem.

In one sentence: I rebuilt the analytical path so that a validated, user-visible request determines the calculation, the calculation passes through an explicit reporting policy, and the resulting claim remains attached to enough evidence to inspect what it actually means.

The goal was not to give AI more analytical authority.

It was to decide where that authority should stop.

What V1 got right

The first InsightForge already had an important control in place.

Its analytical facts were calculated outside the language model. Supporting Data exposed those facts, and the AI layer could interpret them rather than inventing them from scratch.

That was a meaningful step toward grounded analytical answers.

But V1’s deterministic calculations were predefined.

The user’s natural-language question was passed to an interpretation layer alongside those existing facts. Changing the question did not necessarily cause a new deterministic calculation to be constructed around the requested population.

That distinction becomes important when a question asks about an intersection.

Suppose I have:

• overall satisfaction by product;
• overall satisfaction by region;
• counts by gender.

Those facts can all be correct.

But they do not tell me:

Which products have the lowest satisfaction among Male observations in the West during 2026?

That question requires the filters to be applied together before the product-level calculation is performed.

Separate aggregates cannot establish that intersection.

I do not have a preserved V1 response to the exact question I later used to test V2, so I cannot responsibly claim that V1 answered that specific question incorrectly.

What I can show from the frozen V1 implementation is narrower:

The question did not select a new deterministic calculation for that exact requested intersection.

That became the design problem for V2.

The redesign: put a checked request before the calculation

V2 changes the order of operations.

The implemented path is:

Compatible dataset → manual controls or natural-language interpretation → validated analytical request → approved deterministic calculation → result validation and release → immutable Evidence Package → deterministic result → optional bounded AI explanation

Natural language can still make the product easier to use.

But the model does not define the authoritative calculation.

If a user asks:

Which products have the lowest satisfaction among observations marked Male in the West between January 1 and December 31, 2026?

the AI layer can propose:

• Metric: Mean Satisfaction by Product
• Gender: Male
• Region: West
• Start date: January 1, 2026
• End date: December 31, 2026
• Ranking: Lowest first

Those controls remain visible and editable.

And nothing has been calculated yet.

The user still selects Analyze.

Only then does the validated request reach the deterministic engine.

That separation became one of the most important design decisions in V2:

AI can propose an interpretation of intent. The calculation runs from the validated, user-visible request.

Selecting Analyze confirms those visible analytical controls operationally. It does not prove that the system has perfectly understood every nuance of the user’s underlying intent.

The golden test: one question, two different decisions

To test the redesign, I created a 28-row synthetic fixture that could be independently audited by hand.

Twenty of those rows match the Male / West / 2026 request. The remaining eight test exclusion by the selected filters.

The key question was:

Which products have the lowest satisfaction among observations marked Male in the West between January 1 and December 31, 2026?

The independent calculation for the 20 matching observations produces:

At first glance, the answer appears obvious.

Product C has the lowest mathematical mean:

2.00

But V2 has a separate reporting rule for this demonstration: a product requires at least five valid satisfaction observations before it can appear in the released ranking.

Product C has four.

So the calculation and the released analytical claim are separate decisions.

The checked result becomes:

Product B — 2.40
1 product suppressed for insufficient valid observations

And the released wording is:

Product B had the lowest calculated mean satisfaction among releasable products matching the selected filters.

The five-observation threshold is a reporting safeguard for this prototype.

It is not a claim that five observations are statistically sufficient for a business decision.

It is also not a privacy mechanism.

In this prototype, suppression controls whether a product is eligible for the displayed ranking. The underlying synthetic observations remain locally inspectable so the result can be audited.

A system handling restricted or sensitive business data would need separate disclosure controls across row access, exports, evidence views, overlapping queries, storage, and model-provider payloads.

There is another limitation worth making explicit.

Even an eligible group with five observations may still be noisy. When many products are ranked, an extreme value based on a small number of observations may be unstable or partly driven by chance.

The threshold therefore controls release eligibility, not statistical reliability.

That distinction matters because:

A correct calculation is not automatically the same thing as a defensible analytical claim.

Calculation, reporting eligibility, release, evidence and explanation are separate responsibilities.

What changed—and why it matters

The golden test makes the difference concrete. The broader V1-to-V2 changes are:

V2 is intentionally narrower than a general-purpose BI system.

That constraint is part of the experiment.

From Architecture B to the product I actually built

The previous design article described a broader controlled-analytics architecture.

V2 did not implement every part of that proposal.

What survived into the product was the core analytical path:

• a checked request before execution;
• deterministic calculation;
• explicit release states;
• an Evidence Package;
• optional natural-language intent interpretation;
• optional bounded explanation;
• stale-state controls.

Some design ideas were narrowed.

The earlier architecture discussed a more explicit Clarify path. In V2, clarification remains part of the intent state, but visible and editable controls became the primary mechanism for reviewing or correcting a proposed request before Analyze.

The proposed Access & Governed Data layer was not implemented in V2.

That remains outside the scope of this prototype.

The result is smaller than the architecture proposal, but intentionally so:

V2 implements the analytical control path needed to test the idea without pretending the surrounding enterprise platform already exists.

Where AI belongs

V2 uses AI in two places.

Neither is allowed to mutate the authoritative deterministic result.

1. Interpreting the question

A user can describe the analysis naturally.

The model proposes a typed analytical request from the supported vocabulary.

The controls remain visible.

The user can edit them.

Then a separate Analyze action executes the approved deterministic calculation.

If the user asks for something unsupported, the system rejects that analytical request rather than silently substituting another metric.

For example:

What is the highest selling product?

V2 does not replace sales with satisfaction.

It explains that the supported analysis is Mean Satisfaction by Product and leaves the manual controls available.

A bug that made the stale-state problem real

This was not correct on the first implementation.

During testing, I found that if a valid result was already displayed and the user then submitted an unsupported question such as the sales example above, the previous result could remain visible in the browser.

The old calculation had not become the answer to the new question.

But visually, it could look that way.

That was a product defect.

I changed the result-area lifecycle so that beginning a new intent attempt retires the previous current result, evidence, explanation and qualifying rows before the new request is interpreted.

Regression tests were then added around the behavior.

That bug reinforced one of the main lessons of the project:

A correct old answer shown in the wrong context is still a misleading answer.

2. Explaining a checked result

After the deterministic result and Evidence Package exist, the user can optionally ask AI to explain the checked result.

This layer is deliberately constrained.

The explanation path has no write path into the deterministic calculation, ranking or release state.

Suppressed metric values are excluded from the model’s bounded result context.

The generated structured response is checked against the current Evidence Package and the permitted explanation contract.

Tests cover conditions such as:

• a changed winner;
• fabricated customer counts;
• unsupported causal claims;
• unsupported statistical-significance claims;
• leakage of suppressed values;
• stale package identity.

Outputs that violate those controls are withheld rather than presented as the checked explanation.

Those safeguards reduce specific failure modes.

They do not prove that every sentence a language model generates is semantically perfect.

That is why the deterministic result remains visually and logically primary.

If AI is unavailable, the checked analytical result remains usable.

InsightForge does not need AI in order to calculate the supported result. AI makes the controlled analytical system easier to use and explain.

Which AI tests were live?

The acceptance work used both live Azure OpenAI calls and deterministic test doubles.

Live Azure was used to verify the natural-language intent path on supported and unsupported questions, including the golden analytical request, no-filter requests, directional requests, and unsupported sales or causal questions.

The optional result-explanation path was also smoke-tested live against a checked result.

Guardrail cases such as changed winners, malformed or contradictory outputs, unsupported causal claims, statistical claims, provider failures and stale package identity were primarily exercised using controlled fixtures or mocks.

That distinction matters.

A fault-injection test proves that a control responds correctly to the injected failure condition. It does not prove that the live provider naturally produces that failure at a particular rate.

Evidence stays with the claim

The main result is deliberately concise.

But a user can open Why this answer? and inspect the analytical basis beneath it.

That includes:

• the metric definition;
• the exact population;
• matching observations;
• valid observations;
• missing observations;
• reporting eligibility by product;
• the reporting rule;
• source identity;
• warnings;
• standing limitations.

The qualifying observations can also be inspected directly.

And underneath the human-readable evidence is a technical Evidence Package containing fields such as:

• Evidence Package ID;
• Request ID;
• source fingerprint;
• Dataset Contract version;
• Metric Contract version;
• Reporting Policy version;
• calculation version;
• release state;
• warnings;
• limitations.

The goal is not to make every reader inspect technical JSON.

It is to make traceability available when the claim matters.

The package is also immutable.

Changing the source or analytical scope creates new evidence rather than silently rewriting the previous package.

Then I tried data InsightForge did not create

The 28-row fixture is useful because I can audit every important value by hand.

But that raised another question:

Would the same frozen workflow still behave correctly on a larger dataset created independently of InsightForge?

I tested that using the public synthetic retail dataset from the Rameswar555/Retail-Sales-EDA repository.

The tested copy contained 10,000 transactions covering January 1, 2023 through December 31, 2024.

I did not modify the analytical engine.

Instead, I used a small deterministic adapter to select five source fields and rename them into the frozen InsightForge Dataset Contract:

The adapter did not alter individual values.

There is an important semantic qualification here.

For this portability test, I treated the source’s CustomerRating field as the Customer_Satisfaction measure required by the frozen InsightForge contract.

The values already used the accepted numeric range of 1–5, so no rating values needed to be rescaled or transformed.

But range compatibility does not prove that two business concepts are identical.

That mapping is therefore an explicit demonstration assumption, not something established merely by similar column names.

The source repository is publicly accessible and includes the generator used to create the synthetic dataset.

I did not identify an explicit reuse license for that repository, so the original CSV was used locally for validation rather than redistributed as an InsightForge asset.

The adapted source was fingerprinted so the tested copy could be identified precisely.

All 10,000 adapted rows passed Dataset Contract v1.0 with zero violations.

The data contained:

• 50 products;
• 5,109 Female observations;
• 4,891 Male observations;
• East, North, South and West regions;
• ratings from 1.0 through 5.0;
• zero missing satisfaction values.

Testing the larger dataset

I ran several supported analyses and recalculated the results through a separately implemented reference path.

That reference code did not call InsightForge’s calculation functions.

It read the source independently, applied the predicates, grouped by product, summed ratings using Decimal values, represented means exactly for comparison, applied the same declared five-valid-observation release rule, and used decimal round-half-up only for the displayed two-decimal result.

I use the word independent here to mean a separate implementation path rather than a third-party audit.

The reference was still created and reviewed within the same project.

The two paths agreed on:

• qualifying-row counts;
• valid counts;
• missing counts;
• product sums;
• exact ranking;
• reporting eligibility;
• suppression;
• released extreme products;
• displayed means.

Across the full 10,000 observations:

Bed Frame — 3.58

was the lowest calculated mean.

For East:

Cycling Helmet — 4.10

was the highest.

The more interesting test was a filtered intersection:

• Gender: Male
• Region: West
• Dates: January 1 through June 30, 2024
• Ranking: Lowest first

That produced:

378 matching observations

The released result was:

Tea Powder — 3.06
7 product groups suppressed for insufficient valid observations

Tea Powder had exactly five valid observations with a total rating of 15.3:

15.3 ÷ 5 = 3.06

The result passed the prototype’s release threshold.

That does not mean I would treat Tea Powder as a stable business conclusion.

It sits exactly at the minimum valid-observation threshold, and it is the extreme result among many product groups. A real decision would require more evidence about uncertainty, stability and business context.

The independent reference path and InsightForge agreed on the ranking and suppression decisions.

The same Evidence Package machinery and qualifying-row inspection continued to work.

This does not establish production scalability.

It was not a formal load or performance benchmark.

But it provided a useful secondary result:

The analytical workflow was not dependent on the small fixture I designed to demonstrate it.

I also tested the boundaries

Most analytical demos show only the successful path.

That is not enough for the kind of system I wanted to build.

The acceptance suite also exercised conditions where the correct behavior is to reject, retire, withhold or preserve rather than produce another answer.

The altered-value test was deliberately fault-injected.

It does not imply that the live Azure model naturally produced that mistake.

The purpose was to verify that the control behaved correctly when presented with a contradictory explanation.

That is the distinction I care about:

A control should be tested against the failure it is supposed to contain, not merely demonstrated on a successful example.

What V2 proves—and what it does not

InsightForge V2 is not a general-purpose analytics platform.

It currently supports one analytical family:

Arithmetic mean of valid satisfaction scores by Product

It operates against a declared compatible schema with optional Gender, Region and inclusive Date filters.

The demonstration data represents dated sales observations.

Those rows are not verified unique customers.

They are not necessarily unique orders.

The system validates structural compatibility with its contract, but it does not establish that an uploaded source completely or accurately represents the underlying business.

V2 also does not provide:

• causal analysis;
• forecasting;
• statistical-significance testing;
• arbitrary BI queries;
• public arbitrary-file upload;
• production multi-user architecture;
• enterprise permissions or governance;
• privacy guarantees from the reporting threshold.

Those boundaries are intentional.

The goal was not to prove that InsightForge could answer every analytical question.

It was to test whether a narrow analytical claim could remain connected to:

• the visible request that caused the calculation;
• the population used;
• the calculation performed;
• the reporting rule governing release;
• the evidence supporting the result;
• and the limitations constraining what the result means.

For that bounded problem, the answer was yes.

What I learned

The first version of InsightForge taught me that AI-generated analytical explanations become more useful when the underlying facts are calculated outside the model.

V2 pushed the idea further.

The harder problem was not simply grounding a model in correct facts.

It was making sure the right calculation happened for the right visible analytical scope, then preserving enough evidence to understand what could responsibly be claimed from the result.

That changed where I drew the system boundary.

The model can help interpret the request.

It can help explain the checked answer.

But the analytical authority remains with the controlled system in between.

The most important change in InsightForge V2 was not adding more AI.

It was deciding where AI should stop.

InsightForge V3 — Coming Soon

V2 deliberately proves a narrow analytical method before expanding it.

The next phase will explore what happens when that method meets less controlled data.

Areas I expect to investigate include:

• controlled schema adaptation when incoming columns do not exactly match the frozen contract;
• explicit handling of semantic mappings rather than assuming similarly named fields mean the same thing;
• more flexible geography handling across source-defined, national and international taxonomies;
• larger synthetic validation sets;
• and stronger source and lineage integration without giving the AI layer analytical authority it does not need.

The design principle will stay the same:

Complexity should earn its place.

V3 will not start by trying to make InsightForge answer everything.

It will start by asking which additional forms of variability can be supported without weakening the evidence chain that V2 established.

InsightForge V3: coming soon.

Continue the series

Part 1 — InsightForge V1
Read the original InsightForge V1 case study

Part 2 — Designing a More Defensible Architecture for AI Analytics
Read the Architecture B design companion

Part 3 — Building More Defensible AI Analytics: How I Rebuilt InsightForge
You are here.

Part 4 — InsightForge V3
Coming soon.

If you are working on similar problems—AI analytics, evidence-bound outputs, deterministic controls, evaluation or trustworthy AI system design—I’d be interested in comparing approaches.

Get in touch through the Jabez AI contact page