Prompt Engineering & Structured Output — Complete Lesson

The Core Concept

False positives are not a model quality problem — they are a prompt precision problem. When you tell a model to "check that comments are accurate," it will flag anything that might be inaccurate, including stylistic issues, subjective interpretations, and local conventions. When you tell it to "flag comments only when the claimed behaviour contradicts actual code behaviour," it has an unambiguous criterion to apply.

Vague vs Explicit Criteria

## Code Review Severity Criteria

### CRITICAL — Report immediately
- Null pointer dereference with no guard
- SQL injection via string concatenation
- Hardcoded credentials or API keys
Example: `db.query("SELECT * FROM users WHERE id=" + userId)`
→ Report as CRITICAL

### HIGH — Report with suggested fix
- Logic errors where output contradicts specification
- Race conditions in shared state
Example: `if (user.role = "admin")` (assignment not comparison)
→ Report as HIGH

### SKIP — Do not report
- Minor style differences from team convention
- Missing JSDoc on private methods
- Variable naming preferences
- Comments that could be more descriptive (but aren't wrong)
Example: `// get user` above getUserById() → SKIP

The False Positive Trust Problem

High false positive rates cause cascading trust failure. When developers learn that a particular review category generates noise, they start ignoring all findings from that category — including the real bugs it correctly identifies. A 30% false positive rate in comment accuracy doesn't just waste time on 30% of reviews — it erodes confidence in the 70% of findings that were correct.

• Write criteria that define which issues to report (bugs, security vulnerabilities, logic errors) and which to skip (style preferences, local patterns, subjective quality)
• Temporarily disable high false-positive categories while improving their prompts — restores developer trust in the remaining categories
• Define explicit severity levels (CRITICAL / HIGH / MEDIUM / SKIP) with concrete code examples for each — eliminates ambiguous classifications
• Never rely on "be conservative" or confidence thresholds as substitutes for categorical scope definitions

Exam Traps for Task 4.1

🔨 Implementation Task

Exam Simulation — Task 4.1

The Core Concept

Few-shot examples demonstrate the reasoning behind a decision, not just the decision itself. A good example shows why one choice was made over a plausible alternative — enabling Claude to apply that same reasoning to inputs it has never seen. Examples that only show inputs and outputs (without explaining the why) improve consistency but not generalisation.

Anatomy of an Effective Few-Shot Example

## Example 1 — Genuine Bug (REPORT)
Code:
  const total = items.reduce((sum, item) => sum + item.price, 0);
  // Returns total including tax

Finding: REPORT as HIGH
Location: checkout.js line 42
Issue: Comment claims tax is included, but price field
  does not include tax per the schema in types.ts
Suggested fix: Either update the comment to "Returns
  subtotal (excluding tax)" or add tax calculation
Why reported: Claimed behaviour contradicts code behaviour.
  This could cause double-counting in tax reporting.

## Example 2 — Style Issue (SKIP)
Code:
  // Get the user from db
  const user = await db.getUser(userId);

Finding: SKIP
Why skipped: Comment accurately describes what the code
  does. "From db" vs "from database" is a wording
  preference, not a behavioural contradiction.

## Example 3 — Ambiguous (Demonstrate the Decision)
Code:
  // Validate input
  if (!user.email) throw new Error('Email required');

Finding: SKIP
Why skipped: Comment describes the general category
  (validation) but not every validation rule. Missing
  specificity is not a contradiction. No false claim.

Key Applications

• Create 2–4 targeted examples for ambiguous scenarios — show reasoning for why one action was chosen over plausible alternatives
• Demonstrate exact desired output format (location, issue, severity, suggested fix) in every example to achieve structural consistency
• Include examples showing acceptable patterns that are NOT issues — this reduces false positives while preserving generalisation to genuine bugs
• For extraction tasks: provide examples from multiple document formats (inline citations, bibliographies, narrative, tables) — single-format examples fail on novel structures
• For extraction of optional fields: include an example showing correct null output when information is absent — prevents hallucinated values

Exam Traps for Task 4.2

🔨 Implementation Task

Exam Simulation — Task 4.2

The Core Concept

Asking Claude to "return JSON" in a prompt is unreliable — it can return markdown-wrapped JSON, prose before/after the JSON, or malformed JSON under pressure. Defining an extraction function via tool_use with a JSON schema guarantees syntax-valid output because the API enforces the schema structurally before returning the response.

tool_choice: Three Modes

tools = [{
    "name": "extract_invoice",
    "description": "Extract structured data from an invoice document",
    "input_schema": {
        "type": "object",
        "properties": {
            "vendor_name":    {"type": "string"},
            "invoice_date":   {"type": ["string", "null"]},  # nullable!
            "total_amount":  {"type": "number"},
            "line_items": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "description": {"type": "string"},
                        "amount": {"type": "number"}
                    }
                }
            },
            "currency": {
                "type": "string",
                "enum": ["USD", "EUR", "GBP", "other"]  # "other" for extensibility
            }
        },
        "required": ["vendor_name", "total_amount", "line_items"]
        # invoice_date is NOT required — may be absent
    }
}]

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    tools=tools,
    tool_choice={"type": "any"},  # ← guarantees structured output
    messages=[{"role": "user", "content": document_text}]
)

Schema Design Principles

• Make fields nullable (optional) when source documents may not contain the information — prevents the model from fabricating values to satisfy a required field
• Use enum fields with an "other" + detail string pattern for extensible categorisation — captures known categories precisely while preserving unknown ones
• Add an "unclear" enum value for fields where the source data is ambiguous — explicit rather than guessing
• Include format normalisation rules in prompts alongside strict schemas — e.g., "dates should be ISO 8601 format" — schemas enforce structure but not format consistency

# ✗ Closed enum — fails on unknown contract types
"contract_type": {"type": "string", "enum": ["NDA", "SLA", "MSA"]}

# ✓ Open enum with fallback — handles unknown types
"contract_type": {
    "type": "string",
    "enum": ["NDA", "SLA", "MSA", "other"]
},
"contract_type_detail": {
    "type": ["string", "null"],
    "description": "Populated when contract_type is 'other'"
}

What Schemas Cannot Do

Tool use schemas enforce syntactic correctness. They cannot detect semantic errors — when the values are syntactically valid but logically wrong:

Exam Traps for Task 4.3

🔨 Implementation Task

Exam Simulation — Task 4.3

The Core Concept

Retry-with-error-feedback is a powerful pattern: rather than sending the same prompt again, you append the specific validation error and ask the model to correct its output. This gives the model the information it needs to fix the error — but only if the error is fixable given the source document.

Retry-with-Error-Feedback

def extract_with_retry(document, max_retries=3):
    messages = [{"role": "user", "content": document}]

    for attempt in range(max_retries):
        response = extract(messages)
        result = parse_tool_result(response)

        # Validate the extraction result
        errors = validate_extraction(result)

        if not errors:
            return result  # ✓ Success

        # ✓ Append assistant response AND specific errors
        messages.append({"role": "assistant", "content": response.content})
        messages.append({
            "role": "user",
            "content": f"""Your extraction has validation errors. Please correct them:

Errors found:
{chr(10).join(f"- {e}" for e in errors)}

Original document for reference:
{document}

Please re-extract with these corrections applied."""
        })

    return result  # Return best attempt after max retries

When Retry Won't Help

The most important distinction in 4.4: retries fix errors caused by format or structural output problems. They cannot fix errors caused by missing information in the source document.

Feedback Loop Design

Exam Traps for Task 4.4

🔨 Implementation Task

Exam Simulation — Task 4.4

The Core Concept

The Message Batches API is designed for workloads that can tolerate async processing — overnight reports, weekly audits, bulk document processing. Its 50% cost savings make it compelling. Its non-deterministic completion time (up to 24 hours) makes it incompatible with any blocking workflow.

Batch API Trade-offs and SLA Calculation

Batch Failure Handling

def process_batch_results(batch_results, original_documents):
    successful = []
    failed = []

    for result in batch_results:
        doc_id = result.custom_id  # ← correlate using custom_id

        if result.result.type == "succeeded":
            successful.append(result)
        else:
            original_doc = original_documents[doc_id]
            failure_reason = result.result.error.type

            if failure_reason == "token_limit_exceeded":
                # ✓ Chunk the document and resubmit
                chunks = chunk_document(original_doc)
                failed.extend([(doc_id, chunk) for chunk in chunks])
            else:
                # ✓ Resubmit with same content
                failed.append((doc_id, original_doc))

    # ✓ Only resubmit failures, not the entire batch
    if failed:
        resubmit_batch(failed)

    return successful

• Match API approach to latency: synchronous for blocking pre-merge checks, batch for overnight/weekly non-urgent workloads
• Resubmit only failed documents identified by custom_id — not the entire batch
• For documents that exceeded context limits: chunk them before resubmitting rather than retrying as-is
• Test on a representative sample before batch-processing large volumes — maximises first-pass success and reduces costly resubmission cycles

Exam Traps for Task 4.5

🔨 Implementation Task

Exam Simulation — Task 4.5

The Core Concept

When Claude generates code and then reviews it in the same session, it retains the reasoning context from generation — making it less likely to question its own architectural decisions. An independent review instance (a fresh session with no generation context) approaches the code with fresh eyes and is measurably more effective at catching subtle issues.

The Self-Review Limitation

Review Architectures

• Use a second independent Claude instance without the generator's reasoning context — not self-review instructions in the same session
• Split large multi-file reviews into per-file local analysis passes plus a separate cross-file integration pass — avoids attention dilution and contradictory findings
• Run verification passes with model-reported confidence scores to enable calibrated routing between automated action and human review

Exam Traps for Task 4.6

🔨 Implementation Task

Exam Simulation — Task 4.6