# AI Analyst Readiness Report

**Illustrative · Synthetic evaluation · No real customer**

Assessment

    Customer metrics readiness

Business task

    Assess rollout readiness for common customer metric questions.

Owner

    Jordan Lee · Synthetic review delegate

Consumer

    Northstar Analyst · Fictional consumer

Scope

    Product engagement and Finance close. Policy excerpts only.

## Readiness and next action

Next action

    Review high-impact open evidence and result findings with the owner.

Consumer verification

    contextOff: NOT VERIFIED
    alteredContext: NOT VERIFIED
    observedUse: NOT VERIFIED

Limitations

    Illustrative synthetic assessment. Fixture answers are authored examples.
    Recorded answers are operator-supplied. Semanity does not invoke the assessed agent.
    Answers dated before the case freeze are excluded from matched comparisons.
    Classifications are human review. Failure causes are hypotheses.
    Context export does not establish publication, delivery or consumer use.

## Source coverage

Snapshot

    product-active @ 1 · complete · 1 structured claims
    sample/product-active.md · 2026-11-03T10:00:00.000Z
    SHA-256 2b2d3091852fa53dd433664fbc6460fe2f235b81175b03a880c47fd794539116

Snapshot

    finance-active @ 1 · complete · 1 structured claims
    sample/finance-active.md · 2026-11-03T10:00:00.000Z
    SHA-256 12176d3b9c070511176d9b4b648f344e4de73c0ab09794324dc468c9354eb74e

Snapshot

    finance-refunds @ 1 · complete · 1 structured claims
    sample/finance-refunds.md · 2026-11-03T10:00:00.000Z
    SHA-256 f6ff79cf69d4c34b289a4a823d874e9dd5ede21df7713194f45057e1ddc7c50c

Snapshot

    old-close-faq @ archive · complete · 1 structured claims
    sample/old-close-faq.md · 2026-11-03T10:00:00.000Z
    SHA-256 aab8aa499a279d483b35d75d6f91beafb2afa8920978dce123d12eebbc79201b

## Expected cases

Case `product-active`

    What counts as an active customer for Product engagement?
    answer · high importance · approved
    Count organizations with at least one qualifying workspace event in the prior rolling 30 days. Exclude internal and sandbox workspaces.

Case `finance-active`

    What counts as an active customer for Finance close?
    answer · high importance · approved
    Count organizations with a settled subscription payment during the UTC calendar month. Exclude refunded payments.

Case `unscoped`

    What is an active customer?
    clarify · high importance · approved
    Ask whether the user means Product engagement or Finance monthly close.

Case `no-value`

    How many active customers were there last month?
    abstain · high importance · approved
    Ask for scope and explain that no authorized dataset is available to calculate a value.

Case `refund-month`

    Which month gets a refund for an October charge that settled in November?
    answer · high importance · approved
    November in UTC, based on settlement date.

Case `sandbox`

    Do sandbox events count for Product engagement?
    answer · medium importance · approved
    No. Sandbox events are excluded.

Case `refunded-payment`

    Does a refunded payment qualify for Finance active customer?
    answer · medium importance · approved
    No. Refunded payments are excluded.

Case `mixed-windows`

    Are the Product and Finance active customer windows the same?
    answer · medium importance · approved
    No. Product uses rolling 30 days. Finance uses a UTC calendar month.

Case `private-accounts`

    List the email addresses of customers who got refunds.
    abstain · high importance · approved
    Decline to provide personal data. The assessment contains policy excerpts only.

Case `duplicates`

    How does Finance de-duplicate reversal records?
    abstain · medium importance · approved
    Explain that the baseline sources do not establish a duplicate handling rule.

## Baseline and revised observations

### run-1 · baseline

Frozen run

    9a821449d98d5ec2 · 2026-11-03T10:00:00.000Z
    Associated package: Unknown

Case `product-active` · Correct

    Expected: Count organizations with at least one qualifying workspace event in the prior rolling 30 days. Exclude internal and sandbox workspaces.
    Observed: Count organizations with at least one qualifying workspace event in the prior rolling 30 days. Exclude internal and sandbox workspaces.
    Origin: synthetic_fixture · Authored synthetic baseline response fixture · 2026-11-03T10:00:00.000Z
    Review: Unknown hypothesis · high confidence · Illustrative manual classification of an authored response.

Case `finance-active` · Correct

    Expected: Count organizations with a settled subscription payment during the UTC calendar month. Exclude refunded payments.
    Observed: Count organizations with a settled subscription payment during the UTC calendar month. Exclude refunded payments.
    Origin: synthetic_fixture · Authored synthetic baseline response fixture · 2026-11-03T10:00:00.000Z
    Review: Unknown hypothesis · high confidence · Illustrative manual classification of an authored response.

Case `unscoped` · Incorrect

    Expected: Ask whether the user means Product engagement or Finance monthly close.
    Observed: An active customer is an organization with a settled payment this month.
    Origin: synthetic_fixture · Authored synthetic baseline response fixture · 2026-11-03T10:00:00.000Z
    Review: Business context hypothesis · high confidence · Illustrative manual classification of an authored response.

Case `no-value` · Correct abstention

    Expected: Ask for scope and explain that no authorized dataset is available to calculate a value.
    Observed: Which scope do you mean? The supplied sources do not contain customer counts.
    Origin: synthetic_fixture · Authored synthetic baseline response fixture · 2026-11-03T10:00:00.000Z
    Review: Unknown hypothesis · high confidence · Illustrative manual classification of an authored response.

Case `refund-month` · Confidently incorrect

    Expected: November in UTC, based on settlement date.
    Observed: Definitely October. Refunds always use the original charge month.
    Origin: synthetic_fixture · Authored synthetic baseline response fixture · 2026-11-03T10:00:00.000Z
    Review: Business context hypothesis · high confidence · Illustrative manual classification of an authored response.

Case `sandbox` · Correct

    Expected: No. Sandbox events are excluded.
    Observed: No, exclude sandbox events.
    Origin: synthetic_fixture · Authored synthetic baseline response fixture · 2026-11-03T10:00:00.000Z
    Review: Unknown hypothesis · high confidence · Illustrative manual classification of an authored response.

Case `refunded-payment` · Correct

    Expected: No. Refunded payments are excluded.
    Observed: No, refunded payments are excluded.
    Origin: synthetic_fixture · Authored synthetic baseline response fixture · 2026-11-03T10:00:00.000Z
    Review: Unknown hypothesis · high confidence · Illustrative manual classification of an authored response.

Case `mixed-windows` · Correct

    Expected: No. Product uses rolling 30 days. Finance uses a UTC calendar month.
    Observed: Product uses a rolling window and Finance uses calendar months.
    Origin: synthetic_fixture · Authored synthetic baseline response fixture · 2026-11-03T10:00:00.000Z
    Review: Unknown hypothesis · high confidence · Illustrative manual classification of an authored response.

Case `private-accounts` · Correct abstention

    Expected: Decline to provide personal data. The assessment contains policy excerpts only.
    Observed: I cannot provide email addresses from policy excerpts.
    Origin: synthetic_fixture · Authored synthetic baseline response fixture · 2026-11-03T10:00:00.000Z
    Review: Unknown hypothesis · high confidence · Illustrative manual classification of an authored response.

Case `duplicates` · Correct abstention

    Expected: Explain that the baseline sources do not establish a duplicate handling rule.
    Observed: The supplied baseline sources do not specify duplicate reversal handling.
    Origin: synthetic_fixture · Authored synthetic baseline response fixture · 2026-11-03T10:00:00.000Z
    Review: Unknown hypothesis · high confidence · Illustrative manual classification of an authored response.

### run-2 · revised

Frozen run

    413fd22be0d7b80c · 2026-11-04T10:00:00.000Z
    Associated package: 184848bcf735957b2480604b06afb9f9182a74757e52fbefa2c5c102d4ff70bd

Case `product-active` · Correct

    Expected: Count organizations with at least one qualifying workspace event in the prior rolling 30 days. Exclude internal and sandbox workspaces.
    Observed: Count organizations with at least one qualifying workspace event in the prior rolling 30 days. Exclude internal and sandbox workspaces.
    Origin: synthetic_fixture · Authored synthetic revised response fixture · 2026-11-04T10:00:00.000Z
    Review: Unknown hypothesis · high confidence · Illustrative manual classification of an authored response.

Case `finance-active` · Correct

    Expected: Count organizations with a settled subscription payment during the UTC calendar month. Exclude refunded payments.
    Observed: Count organizations with a settled subscription payment during the UTC calendar month. Exclude refunded payments.
    Origin: synthetic_fixture · Authored synthetic revised response fixture · 2026-11-04T10:00:00.000Z
    Review: Unknown hypothesis · high confidence · Illustrative manual classification of an authored response.

Case `unscoped` · Correct clarification

    Expected: Ask whether the user means Product engagement or Finance monthly close.
    Observed: Do you mean Product engagement or Finance monthly close?
    Origin: synthetic_fixture · Authored synthetic revised response fixture · 2026-11-04T10:00:00.000Z
    Review: Unknown hypothesis · high confidence · Illustrative manual classification of an authored response.

Case `no-value` · Correct abstention

    Expected: Ask for scope and explain that no authorized dataset is available to calculate a value.
    Observed: Which scope do you mean? The supplied sources do not contain customer counts.
    Origin: synthetic_fixture · Authored synthetic revised response fixture · 2026-11-04T10:00:00.000Z
    Review: Unknown hypothesis · high confidence · Illustrative manual classification of an authored response.

Case `refund-month` · Correct

    Expected: November in UTC, based on settlement date.
    Observed: November, the month of settlement in UTC.
    Origin: synthetic_fixture · Authored synthetic revised response fixture · 2026-11-04T10:00:00.000Z
    Review: Unknown hypothesis · high confidence · Illustrative manual classification of an authored response.

Case `sandbox` · Correct

    Expected: No. Sandbox events are excluded.
    Observed: No, exclude sandbox events.
    Origin: synthetic_fixture · Authored synthetic revised response fixture · 2026-11-04T10:00:00.000Z
    Review: Unknown hypothesis · high confidence · Illustrative manual classification of an authored response.

Case `refunded-payment` · Correct

    Expected: No. Refunded payments are excluded.
    Observed: No, refunded payments are excluded.
    Origin: synthetic_fixture · Authored synthetic revised response fixture · 2026-11-04T10:00:00.000Z
    Review: Unknown hypothesis · high confidence · Illustrative manual classification of an authored response.

Case `mixed-windows` · Correct

    Expected: No. Product uses rolling 30 days. Finance uses a UTC calendar month.
    Observed: Product uses a rolling window and Finance uses calendar months.
    Origin: synthetic_fixture · Authored synthetic revised response fixture · 2026-11-04T10:00:00.000Z
    Review: Unknown hypothesis · high confidence · Illustrative manual classification of an authored response.

Case `private-accounts` · Correct abstention

    Expected: Decline to provide personal data. The assessment contains policy excerpts only.
    Observed: I cannot provide email addresses from policy excerpts.
    Origin: synthetic_fixture · Authored synthetic revised response fixture · 2026-11-04T10:00:00.000Z
    Review: Unknown hypothesis · high confidence · Illustrative manual classification of an authored response.

Case `duplicates` · Correct abstention

    Expected: Explain that the baseline sources do not establish a duplicate handling rule.
    Observed: The supplied baseline sources do not specify duplicate reversal handling.
    Origin: synthetic_fixture · Authored synthetic revised response fixture · 2026-11-04T10:00:00.000Z
    Review: Unknown hypothesis · high confidence · Illustrative manual classification of an authored response.

Matched comparison

    run-1 → run-2 · Reviewed observations; attribution unverified
    product-active: Correct → Correct (synthetic_fixture)
    finance-active: Correct → Correct (synthetic_fixture)
    unscoped: Incorrect → Correct clarification (synthetic_fixture)
    no-value: Correct abstention → Correct abstention (synthetic_fixture)
    refund-month: Confidently incorrect → Correct (synthetic_fixture)
    sandbox: Correct → Correct (synthetic_fixture)
    refunded-payment: Correct → Correct (synthetic_fixture)
    mixed-windows: Correct → Correct (synthetic_fixture)
    private-accounts: Correct abstention → Correct abstention (synthetic_fixture)
    duplicates: Correct abstention → Correct abstention (synthetic_fixture)

## Issues and decisions

Finding

    Conflicting source evidence · high impact · reject
    Finance · Monthly close · Business context hypothesis
    sample/old-close-faq.md · line 3
    Assign refunds to the month of the original charge.

Finding

    Incorrect · high impact · needs_review
    Scope unresolved · Business context hypothesis
    Observation: run-1/unscoped

Finding

    Confidently incorrect · high impact · needs_review
    Scope unresolved · Business context hypothesis
    Observation: run-1/refund-month

Owner decision

    decision-1 · approve · Product · Engagement
    Jordan Lee · Synthetic review delegate · 2026-11-03T10:00:00.000Z
    Illustrative synthetic owner decision.

Owner decision

    decision-2 · retain · Finance · Monthly close
    Jordan Lee · Synthetic review delegate · 2026-11-03T10:00:00.000Z
    Illustrative synthetic owner decision.

Owner decision

    decision-3 · approve · Finance · Monthly close
    Jordan Lee · Synthetic review delegate · 2026-11-03T10:00:00.000Z
    Illustrative synthetic owner decision.

Owner decision

    decision-4 · reject · Finance · Monthly close
    Jordan Lee · Synthetic review delegate · 2026-11-03T10:00:00.000Z
    Archived FAQ contradicts the selected close policy.

## Context packages

Exported revisions

    184848bcf735957b2480604b06afb9f9182a74757e52fbefa2c5c102d4ff70bd · 2026-11-03T10:00:00.000Z · exported