SEC Semi-Annual Reporting Proposal Tracker (S7-2026-15)

• Unanimous: 7894 (98.5%)
• 2-of-3 majority: 118 (1.5%)
• Fleiss' κ: 0.625 · pairwise Cohen's κ: Primary–Literalist 0.82, Primary–Skeptic 0.61, Literalist–Skeptic 0.50

Each letter card in the table below carries an agreement badge (Unanimous / 2-of-3) and shows all three rater calls on hover.

I then ran the same 3 rubric prompts through ChatGPT-5.5 as an independent second ensemble. The question I wanted to answer: would the stance calls hold up under a different model family? Carlson & Burbano (SMJ 2026) recommend this kind of cross-model check where feasible.

GPT-Majority vs Claude-Majority: 136 / 142 = 96% raw, Cohen's κ = 0.86. Substantial cross-model agreement on the aggregated label.

Per-rubric agreement varies:

• GPT-Primary vs Claude-Primary: κ = 0.81 (substantial)
• GPT-Literalist vs Claude-Literalist: κ = 0.62 (moderate-substantial)
• GPT-Skeptic vs Claude-Skeptic: κ = 0.40 (moderate)

The Skeptic divergence reflects a rubric-conditioning effect. The same "default to Conditional unless unambiguous" instruction yielded 86 Conditional calls in GPT-5.5 versus 37 in Claude Opus 4.7. Same prompt, different operationalization across model families. Aggregate agreement on the majority vote holds; per-rubric agreement is more model-dependent.

6 letters fall outside the cross-model majority match: #2 Fardeen Irani, #3 Anonymous, #13 Skyler Mathis, #43 Steven A. Collazo, #80 Bayo Olabisi, #122 Tal Madison. All 6 push from Claude's Support or Oppose call into GPT's Conditional call. 4 of the 6 already had at least one Claude rater calling Conditional, so the cross-model disagreement concentrates on the hedge-boundary letters Claude's own ensemble was already split on.

I read all 6 of these letters by hand. On every one, Claude-Majority is the call I would have made; GPT-Majority is not. The uniform GPT failure mode: it treats any structural alternative or rhetorical hedge in the letter body as evidence of conditionality, even when the author's stated position is unambiguous. Six letters is a small validation set, but the direction is one-sided.

I ran those same comment letters through four open-weight models that anyone can download and run for free, and compared each model's majority stance call to the label shown on this site.

• phi-4 (14B): 95% raw, Cohen's κ = 0.80
• gemma-3 (27B): 96% raw, κ = 0.84
• Mistral-Small (24B): 97% raw, κ = 0.90
• Qwen2.5 (0.5B, deliberate floor): 70% raw, κ = 0.20

On the same letters where ChatGPT-5.5 reaches κ = 0.86, the three capable open models land in the same band (κ = 0.80 to 0.90), and a free 24-billion-parameter model edges the commercial system. Across the full docket of 1,898 letters the capable models reproduce the site's stance label on 93 to 98 percent. Kappa runs lower on the full docket because it is roughly 97 percent Oppose, so agreement by chance is high and the metric is pushed down even when raw agreement is near-perfect; the 142-letter sample above is less skewed. Qwen, kept as a deliberate floor, is the one that breaks down.

Oppose

Author argues against adoption.

Support

Author explicitly endorses the proposal.

Conditional (in-between / mixed)

Author wants modifications or alternatives (e.g. enhanced auditor assurance, monthly revenue disclosure, every-4-months cadence, qualifying-criteria framework). Would not vote yes on the rule as written.

I validated these classifications against a human expert gold standard and a 14-model benchmark spanning Claude, GPT, Gemini, and four open-weight models. Stance is where the labels are most reliable; entity and rationale are harder, for people and machines both.

• Unanimous: 7758 (96.8%)
• 2-of-3 majority: 253 (3.2%)
• Fleiss' κ: 0.766

Substantially higher agreement than the stance ensemble (κ = 0.625). Majority headline distribution: Individual 7615 / Investment professional 105 / Accountant (CPA) 96 / Industry practitioner 55 / Legal practitioner 37 / Issuer-current 33 / Issuer-former 29 / Academic researcher 26 / Student 9 / Trade association / advocacy organization 6.

The same 3 rubric prompts ran through ChatGPT-5.5 as an independent second ensemble.

GPT-Majority vs Claude-Majority: 120 / 142 = 85%, Cohen's κ = 0.63. Moderate cross-model agreement on the aggregated label.

Per-rubric agreement:

• GPT-Primary vs Claude-Primary: κ = 0.68 (substantial)
• GPT-Self-described vs Claude-Self-described: κ = 0.63 (substantial)
• GPT-Letterhead vs Claude-Letterhead: κ = 0.62 (substantial)

The pattern is systematic. GPT-5.5 has a stronger "Individual" prior than Claude Opus 4.7 across all three rubrics. The biggest split is on writers who sign as "CFO, ACME Corp" or similar institutional role but write in a personal register engaging investor-protection concerns rather than issuer-specific concerns: GPT-Primary classifies as Individual; Claude-Primary classifies as Issuer-current. Both readings are defensible under the rubric. The rubric requires a "speaking-as" judgment, and the two model families weight surface role vs register differently.

Intra-model Fleiss κ is 0.87 for Claude and 0.78 for GPT. Within-model agreement holds for both ensembles; the divergence is across model families.

22 letters fall outside the cross-model majority match. 17 of 22 flow into GPT-Individual from a more specific Claude bucket. 11 of these 22 are already on the contested-letters list internal to Claude's own three-rater ensemble.

The same four open-weight models, scored against the entity bucket shown on this site:

• phi-4 (14B): 81% raw, Cohen's κ = 0.55
• gemma-3 (27B): 78% raw, κ = 0.51
• Mistral-Small (24B): 79% raw, κ = 0.51
• Qwen2.5 (0.5B, deliberate floor): 65% raw, κ = 0.00

Entity is the harder call for every model. The capable open models (κ = 0.51 to 0.55) sit just behind ChatGPT-5.5 (κ = 0.63) and well above the Qwen floor, the same ordering as on stance. Who is writing is a genuine judgment call, and the open and commercial models miss it in the same places.

I validated these classifications against a human expert gold standard and a 14-model benchmark spanning Claude, GPT, Gemini, and four open-weight models. Stance is where the labels are most reliable; entity and rationale are harder, for people and machines both.

Rater 1 — Primary

Balanced read. Codes the rationale families the writer substantively argues, even when not framed in the SEC's exact language.

Rater 2 — Literalist

Strict. Codes a family only when the letter explicitly invokes that framing in surface text.

Rater 3 — Inclusive

Expansive. Codes a family whenever plausibly invoked, including allusive references and arguments the writer relies on without fully developing.

• Unanimous (all three raters produced identical code sets): 7024 (87.7%)
• 2-of-3 majority: 760 (9.5%)
• Split (three different code sets): 227 (2.8%)
• Mean per-code Cohen's κ across pairs: 0.917. Substantial agreement, in line with multi-prompt LLM-annotation benchmarks.

Low-frequency codes have volatile κ (swinging to either extreme on a handful of calls): CC (contractual constraints) on 9 letters (κ 0.93); ICs (intl evidence — supportive) on 9 letters (κ 1.00); OP (OP) on 49 letters (κ 0.91). κ stabilizes as code frequency rises.

The same 3 rubric prompts ran through ChatGPT-5.5 as an independent second ensemble.

GPT-Majority vs Claude-Majority: mean per-code Cohen's κ = 0.44. Moderate cross-model agreement, lower than the stance ensemble (κ = 0.86) and the entity ensemble (κ = 0.63). The ranking is consistent with rationale being the most inferential and multi-label of the three ensembles. Set-level exact match (GPT majority code set identical to Claude majority code set): 37 of 142 letters, 26%. Mean Jaccard similarity between the two majority sets: 0.53.

Per-rubric mean κ across the 20 codes:

• GPT-Primary vs Claude-Primary: κ = 0.483
• GPT-Literalist vs Claude-Literalist: κ = 0.365
• GPT-Inclusive vs Claude-Inclusive: κ = 0.361

Surface-readable codes converge across model families: MF (κ = 0.79), CMP (κ = 0.76), FR (κ = 0.70), LE (κ = 0.70), NR (κ = 0.60). Inferential or umbrella codes diverge: EX (κ = 0.20), IP (κ = 0.32), OP (κ = 0.32), AU (κ = 0.39).

The rubric-conditioning pattern visible in the stance and entity ensembles shows up again. GPT-Inclusive fires 4.50 codes per letter; Claude-Inclusive fires 2.84. Same "code whenever plausibly invoked" instruction, very different operationalization. Claude's three rationale raters stay within a 25% spread of each other (2.25 to 2.84 codes per letter); GPT's three span nearly 2x (2.44 to 4.50). The 3-rater majority κ (0.477) is higher than two of the three matched-rubric κs, which shows that aggregation dampens cross-model variance just as it does within-model.

I validated these classifications against a human expert gold standard and a 14-model benchmark spanning Claude, GPT, Gemini, and four open-weight models. Stance is where the labels are most reliable; entity and rationale are harder, for people and machines both.