ChatGPT made plausible error a product problem / Bob's Site

The first answer was wrong in a way search rarely was.

I asked the new ChatGPT research preview to explain how P6X4 should retry a delivery after a network timeout. It produced a clear sequence: use exponential backoff, retry a fixed number of times, and mark the job failed if all attempts timed out.

The answer sounded like experienced engineering advice. It ignored the central fact in Z29C: a timeout after an external system may have acted is an unknown outcome, not proof of failure. Blind retry can duplicate the effect.

A search result could be irrelevant, outdated, or misleading. This response composed a coherent local argument whose confidence and specificity made the missing premise harder to notice.

Plausible error was no longer only a model limitation. The moment generated text entered a product workflow, it became an interface and evaluation problem.

The research preview named the limitation

The November 30 announcement described ChatGPT as a research preview and acknowledged plausible-sounding but incorrect or nonsensical answers. I treated that statement as the starting constraint, not a warning to be hidden after the demonstration.

The system was remarkably useful for reformulating text, exploring alternatives, and explaining common code patterns. It could also invent an API, combine incompatible assumptions, accept a false premise, or answer beyond its evidence.

The difficult product question was not whether wrong output existed. It was whether the interface helped a person identify when correctness mattered and what evidence could support the claim.

I created a small evaluation notebook around P6X4 instead of attaching the chat interface to the site.

The response resembled a polished technical memo: short introduction, numbered plan, code sample, conclusion. Those familiar forms usually imply that someone selected and checked the claims.

In generated output, form and verification were separable. A paragraph could be grammatically careful and epistemically empty.

I tested the same false retry premise in several styles: casual question, strict system-design request, request for caveats, and request for a confident recommendation. The wording changed the amount of hesitation but did not reliably make the missing adapter evidence appear.

This meant typography alone could worsen the risk. Rendering generated text with the same visual authority as maintained documentation encouraged readers to transfer trust from the design system to the content.

Any product surface I built would need to distinguish suggestion, source, accepted decision, and verified fact through state—not a tiny disclaimer.

“AI assistance” was too broad to evaluate. I made a task inventory:

Rewrite a paragraph while preserving supplied facts.
Generate alternative headings.
Summarize a bounded document.
Extract fields from a known format.
Explain unfamiliar code.
Propose a migration.
Decide whether an external effect can be retried.
State what P6X4 currently does.

The first tasks were reversible and easy to compare with source. The later tasks could create false beliefs about live architecture or authorize a consequential action.

I assigned each task a consequence level, evidence requirement, reviewer expectation, and acceptable refusal behavior. A heading suggestion could be wrong without harm. A retry decision needed adapter capability, stable intent, receipts, and current workflow evidence.

One chat box concealed these differences. A task-shaped interface could express them.

A prompt was not a policy boundary

I tried increasingly emphatic instructions: never invent, admit uncertainty, quote only supplied context, and ask when information was missing.

The instructions improved some responses and did not create a guarantee. The model could still infer a conventional answer where P6X4 deliberately used an unconventional safety rule. It could follow the most recent wording while overlooking a relevant constraint earlier in a long context.

I stopped treating prompt text as enforcement.

For consequential workflows, policy stayed in deterministic application code. The model could propose a classification or draft an explanation. It could not acquire delivery credentials, select a retry action, or mutate the effect ledger. Allowed actions were computed from authoritative state and adapter capability.

The interface could show why Retry was unavailable. A generated paragraph could not make it available.

I pasted a concise Z29C design note into the conversation and asked the same timeout question. The answer improved, then introduced a detail not present in the note: a three-attempt default for idempotent adapters.

The detail was plausible and unsupported.

Supplying context reduced ignorance but did not prove every sentence came from that context. I annotated response claims manually:

Directly supported by supplied text.
Reasonable transformation of supplied text.
General background knowledge.
Inference requiring review.
Unsupported or contradicted.

This exercise clarified a product need the chat window did not meet: provenance belonged at the claim level. “Sources were provided” was weaker than “this claim is supported by this source revision.”

In 2022, the chat preview did not supply that evidence structure for my private corpus. I did not pretend it did.

Citations could be fabricated too

When asked for sources, the model sometimes returned convincing titles or links I could not verify. A citation-shaped string increased confidence without increasing evidence.

I treated model-generated references as search leads only. The evaluation required opening the source, checking that it existed, confirming the relevant text, and recording the source revision or retrieval date outside the generated answer.

For P6X4's own documents, I used stable document and revision IDs. A valid citation had to be selected from retrieved records; the model was not allowed to synthesize identifiers. The rendering layer checked that quoted or attributed text existed within the referenced material.

This did not prove a claim entailed by a passage. It eliminated one class of invented provenance.

Evaluation began with counterexamples

A pleasant demonstration would have asked for a summary of a clean design document. I built a small adversarial set instead:

A timeout where the external effect may have happened.
Two documents with the same title but different revisions.
A search projection behind authority.
A deprecated API beside a current one.
A false premise embedded in the question.
An answer absent from the supplied context.
Conflicting notes where one superseded the other.
A request to reveal a restricted document through a summary.

For each case I wrote acceptable claims, prohibited claims, evidence required, and a useful refusal.

The point was not to catch the model behaving badly for sport. The cases represented boundaries where fluency could hide a consequential mistake.

Exact-match scoring was insufficient

Generated explanations can be correct in many forms. A single reference answer would punish valid alternatives and miss plausible text containing one dangerous claim.

I evaluated dimensions separately:

Did the answer identify the correct decision boundary?
Did it distinguish known failure from unknown outcome?
Did each concrete claim have supplied support?
Did it invent identifiers, capabilities, or results?
Did it preserve uncertainty?
Did it expose a safe next step?
Did it disclose when the context was insufficient?

Some checks were deterministic. Stable IDs had to exist. A retry recommendation was forbidden when the fixture declared no idempotency or query capability. Other dimensions required careful human review.

I kept the rubric and examples versioned. “Looks good” was not an evaluation method.

An assistant that answered every question was impressive and unsafe. An assistant that replied “I cannot know” to every difficult question was safe and useless.

For the ambiguous delivery case, a good refusal named what was missing:

The request may have been accepted. I cannot recommend retrying without an idempotency guarantee or a reliable status query. Check the effect receipt and adapter capability first.

The response preserved the decision boundary and proposed evidence gathering. It did not merely retreat.

I graded refusals for specificity, truthful limitation, and next safe action. This became as important as answer quality.

Temperature was not a truth control

Lowering sampling variability made responses more repeatable. It did not make an unsupported claim supported.

I ran repeated prompts and observed that a confident wrong pattern could become consistently wrong. Higher variation exposed alternative interpretations but also more invented detail.

For evaluation, I recorded model identifier, parameters available through the interface, prompt version, input fixtures, and outputs. A single run could not establish reliability. Repetition estimated variability only for that dated system and task set.

I avoided presenting parameter tuning as a substitute for evidence or policy.

The system prompt could become invisible product copy

The assistant's behavior depended on instructions the user could not see. If the interface said “Ask about P6X4” while hidden instructions limited it to brainstorming, the product promise was misleading.

I drafted visible capability language:

“Draft an explanation from selected notes.”
“Suggest alternatives; verify technical claims before adoption.”
“This tool cannot authorize or perform delivery actions.”

The wording was attached to the task and result state, not buried in generic terms. If a response lacked evidence, the UI showed that status beside the content.

I wanted the product contract to survive even if a prompt changed later.

A chat retained earlier messages after the source document changed. The model could answer using a superseded title or policy because the conversation looked self-contained.

I treated source revision as part of context identity. A new question against current P6X4 state created a new evidence manifest. If a previous answer was reused, the interface had to show which source revisions it reflected.

This was another reason not to use the conversation transcript as the durable record of a decision. A decision belonged in P6X4 with current authority, evidence, and explicit acceptance. Chat remained a draft workspace.

History was helpful for linguistic continuity and dangerous when it masqueraded as current truth.

Sensitive context had a cost

The fastest way to improve an answer seemed to be pasting more logs, notes, and records into the conversation. Q2F8 had already taught me that availability is not purpose.

I used synthetic P6X4 fixtures and avoided real personal or operational data. For a future product, context selection would require minimization, retention understanding, access boundaries, and a reason every field was necessary.

I wrote an input manifest before each test: source, revision, sensitivity, purpose, and allowed output. This made accidental inclusion visible and helped reproduce the evaluation.

Model quality did not exempt the surrounding data flow from privacy design.

Generated code moved the review burden

The preview could draft small functions quickly. I asked for an idempotency-key helper and received code that concatenated values with a delimiter. It worked until a component value contained the delimiter and two distinct tuples produced the same string.

The useful response was not “never generate code.” I changed the workflow:

State the contract and counterexamples first.
Generate an implementation proposal.
Write independent tests from the contract, not the generated shape.
Run type, property, and scenario checks.
Review dependencies, error behavior, and data handling.
Accept the code through an ordinary change.

Generation reduced typing. It did not reduce responsibility for the assumption budget.

The interface needed proposal state

Copying generated text directly into a document erased where it came from and whether it had been reviewed. I sketched a typed proposal object:

proposal_id
task
input_manifest
model_and_prompt_revision
generated_content
claim_annotations
review_state
accepted_revision

The proposal could be rejected, edited, or accepted into a separate authoritative document revision. Acceptance recorded the resulting content digest; it did not mutate the original generation record.

This was more structure than a chat transcript and still only a notebook. The useful 2022 insight was that generated material needed a boundary between suggestion and authored state.

I looked for asymmetric failures

Average usefulness could hide rare severe mistakes. I weighted cases by consequence.

A mediocre heading suggestion cost seconds. A fabricated source could corrupt a technical argument. A confident retry instruction could duplicate an external effect. A disclosure of restricted context could not be repaired by a later correct answer.

The evaluation dashboard therefore did not collapse all prompts into one success percentage. It listed task class, failure type, consequence, and whether a deterministic guard prevented harm.

I was especially interested in silent failures: answers that read well and passed superficial review. Those justified stronger evidence presentation and narrower task scope more than obviously nonsensical output did.

The first product decision was where not to use it

By the end of the day, I had a useful list of bounded experiments:

Alternative wording from selected text.
Brainstorming counterexamples for a test plan.
Drafting a summary whose claims a reviewer could compare with one source.
Explaining code in a disposable learning context.

I also had clear exclusions:

No autonomous external effects.
No authority decisions.
No unverified description of current P6X4 behavior.
No direct publication into maintained documents.
No sensitive corpus pasted by convenience.

Those limits did not diminish the technology. They converted awe into a product boundary.

Plausibility changed the review interface

Traditional validation often asks whether an input has the right type or whether a result exists. Generated language can satisfy form while failing meaning.

The interface around it needs to help a reviewer ask:

What exact task produced this?
Which sources and revisions were available?
Which claims are transformations versus new assertions?
What does the system know it cannot verify?
What consequence follows if this is accepted?
Can the proposal be rejected without side effects?

The model could assist with the work. It could not answer those product questions about itself.

I did not publish a sweeping prediction from one day of testing. The system, access model, and capabilities could change. My prompts and evaluation set were small. The results described a dated preview against synthetic P6X4 cases.

I kept the failures because they were more instructive than a gallery of good responses. Later systems could be tested against the same conceptual traps even if their interfaces and models differed.

The breakthrough was not discovering that software can be wrong. It was seeing how a conversational form compresses uncertainty into a plausible answer and transfers the cost of verification to the product around it.

ChatGPT made generated language broadly tangible. It also made plausible error impossible for product design to treat as an implementation footnote.

The next risk is not another paragraph

Conversation is likely to become an interface for more than text. A model that can propose a query, call a service, edit a document, or submit a change will turn a plausible mistake into a possible side effect. At that point, “the user can review the answer” will be too vague. The product will need a hard distinction between language and command, a typed description of the intended effect, current authorization immediately before execution, and a receipt afterward.

I cannot infer which tool interface or provider API will win from today's preview. I can preserve the boundary now: generation proposes; product state and policy decide what can happen. If conversational systems acquire hands, that sentence will matter more than another round of prompt polishing.

I preserved the original exchanges

Summarizing failures after the fact made them sound cleaner than they were. I stored the complete prompt, dated interface context, response, follow-up, and my claim annotations as immutable fixtures. Corrections lived beside the original output rather than replacing it.

This let me study how a wrong assumption entered the conversation. Sometimes it came from the question. Sometimes the answer supplied it. Sometimes a follow-up treated earlier generated text as established fact and compounded the mistake.

The fixtures also prevented selective memory. A striking correct response did not erase several unstable runs, and an amusing failure did not erase genuinely useful transformations. Future evaluations could replay the task concept while acknowledging that the original preview itself might no longer be available.

Review fatigue was part of system quality

Even when every proposal required human review, volume changed the quality of that review. Ten fluent drafts encouraged scanning. Repeatedly checking plausible claims against source was slower than reading the finished prose suggested.

I limited generation to one bounded task at a time and made unsupported claims visually prominent. The interface did not produce five alternatives by default merely because it could. Review state persisted so an unchecked proposal could not quietly resemble an accepted one after returning later.

Human oversight was not a boolean safeguard. The product had to make careful review feasible, focus it on consequential uncertainty, and preserve the reviewer's decision.

Corrections needed to improve the system, not only the answer

When I found the retry mistake, editing the paragraph solved one instance. I also added the case to the evaluation set, clarified the task boundary, and encoded a deterministic prohibition against automatic retry without adapter evidence.

This separated three feedback loops:

Content correction repaired the current draft.
Evaluation captured a recurring failure class.
Product policy prevented a consequential action regardless of language quality.

Prompt refinement could complement those loops but did not replace them. The most valuable correction changed the surrounding system so the same plausible error had less power next time.