Notes — Research Log
What Happened When We Had
Multiple AIs Review Our OSS.
We had Claude Opus 5 and GPT-5.6 Sol independently review the same document,
then compared results. Where they agreed, we trusted. Where they diverged, we investigated.
What We Did
The setup.
prompt-as-code is an RFC-style syntax standard for treating LLM prompts as code. Published in December 2025, we had iterated it to v0.3.0.
In August 2026, alongside a positioning update toward context engineering, we decided to ship a major update to v0.4.0. But auditing an 1,100-line document solo is not realistic.
So we took a different approach: have different AI models independently review the same document, then cross-reference the results.
Method
Verification design.
We ran the validation in two phases.
| Phase 1 Pattern effectiveness | We ran the same test prompts for SCoT (Structured Chain of Thought), Few-shot, and CoVe (Chain of Verification) on both Opus 5 and GPT-5.6 Sol to confirm whether these patterns still work on 2026-era models. |
|---|---|
| Phase 2 Full document audit | We fed the entire STANDARDS.md to both models and asked each to independently flag factual errors, missing topics, unnecessary content, and technically incorrect claims. Neither model saw the other's output. |
The key: the two models were blind to each other's output. When independent reviewers agree, confidence is high; when they diverge, further investigation is needed — the same principle as human code review.
Phase 1 Results
Do the patterns still work?
Short answer: all three patterns were confirmed effective on both models.
| SCoT XML tags separating reasoning from conclusions |
Both models conformed to <thinking> / <answer> tags. Without the instruction, both said they would default to Markdown headings or freeform text — meaning they do not adopt this structure by default. The pattern has value.
|
|---|---|
| Few-shot Output format control via examples | Both models accurately followed the demonstrated format and explicitly stated they would not use the same format without examples. Even on 2026 models, examples remain effective for output format control. |
| CoVe Self-verification loop | Both models followed the C1–C4 loop structure and self-corrected roughly 40% of factual claims in their own drafts. Interestingly, the corrected items differed between models. |
The CoVe results are particularly telling. On the same question, Opus 5 corrected "Micronaut 4 release date" and "Blazor United naming," while GPT-5.6 Sol corrected "React library/framework classification" and "ambiguity in GitHub ranking methodology." The same pattern caught different blind spots on different models — strong evidence for the pattern's generality.
Phase 2 Results
What we found.
The full-document audit produced findings the models agreed on and findings unique to each.
Both models agreed (high confidence)
| CO-STAR "gold standard" | Calling it an "industry standard" was an overclaim. "A widely adopted framework" is more accurate. On attribution, Opus 5 claimed "Sheila Teo introduced CO-STAR, not GovTech." Fact-checking revealed that GovTech Singapore's team developed it and Sheila Teo popularized it through their GPT-4 Prompt Engineering Competition — a lesson that AI review results themselves need fact-checking. |
|---|---|
| CoVe / CoD language | "Becoming standard" and "reference standard" were overclaims for research techniques. Changed to "a recommended technique" and "an effective method." |
| Extended thinking conflation |
The in-prompt <thinking> tag (an output structuring technique) and API-level extended thinking from Claude or OpenAI o-series (a model feature) were not distinguished.
|
| Security gaps | Only direct prompt injection was covered; no mention of indirect injection via RAG or tool outputs. |
Only one model found it (needs investigation)
Opus 5 focused on structural issues (incomplete table of contents, section ordering, model list inconsistencies). GPT-5.6 Sol dug into technical accuracy — triple-backtick Markdown nesting, a RAG code example bug, imprecise SemVer 0.y.z documentation.
What stood out: GPT-5.6 Sol questioned whether "State everything explicitly" is excessive in an era of agentic models. If agents act autonomously, perhaps prompts should only specify goals and constraints — a challenge to the document's underlying philosophy, not a simple right-or-wrong call.
Human Judgment
What humans decided.
We did not accept every AI finding. Both models listed many "missing topics," but most were outside the document's scope.
prompt-as-code is "a syntax specification for text written as prompts." What you can express in Markdown is the boundary.
| AI said "this is missing" | Human judgment |
|---|---|
| Structured Output / JSON mode | API parameter. Out of scope. |
| Tool definitions / Function calling | API feature. Out of scope. |
| Multimodal prompting | Beyond a text syntax spec. Out of scope. |
| Prompt caching | Infrastructure layer. Out of scope. |
| Persistent context files | Separate topic. |
AI casts a wide net: "you're missing this, and this, and this." But scoping the document is a human job. What we discovered was that what was actually missing was not topics, but a scope declaration — "what this document covers and what it does not."
Result
What we changed.
Based on the validation, v0.4.0 includes these changes:
- 5 factual error corrections (CO-STAR attribution, 3 overclaims, translation template version)
- 6 technical accuracy improvements (extended thinking distinction, indirect injection, code example bug fixes, etc.)
- [Step N] notation rationale rewrite ("models fail to parse numbered lists" replaced with "task decomposition, HITL, and output cross-referencing")
- Structural improvements (completed table of contents, removed stale "NEW" markers)
- README.md and CONTRIBUTING.md synchronized
All changes are visible in the GitHub repository. Raw validation data is included as test-results-v040-*.md.
Takeaway
What this approach teaches.
Three lessons from this approach.
First, independent model agreement is more trustworthy than a single model's confidence. If only Opus 5 had flagged the CO-STAR attribution, it might have ended with "maybe, maybe not." Two models independently raising the same issue gave us the confidence to act.
Second, models have different perspectives. Opus 5 excelled at structural, bird's-eye review. GPT-5.6 Sol was stronger on code-level bugs and technical definition precision. Using only one would leave blind spots.
Third, scope decisions can only be made by humans. AI is good at casting a wide net, but deciding "what to leave out" is the job of someone who understands the document's purpose and audience.
This is also a practical example of the "design-time context engineering" discussed in the previous article. What to have AI review, and how to judge the results — that itself is context design.