Highest full-pass rate
Tied: GPT-5.6 Luna, GPT-5.6 Sol and GPT-6 Astra
3/3 full passes each · 95% CI 29–100%. Listed alphabetically; no tie-break is applied.
Compare accuracy, speed and cost for your next task.
7 OpenAI models · tested September 3 to 18, 2026 · Anthropic and other vendors not yet tested
Highest full-pass rate
3/3 full passes each · 95% CI 29–100%. Listed alphabetically; no tie-break is applied.
Lowest-cost passing
$0.0273m 27s
3/3 full passes · 95% CI 29–100%
Fastest passing
$0.0273m 27s
3/3 full passes · 95% CI 29–100%
A full pass means every check passed in that attempt; passing cards require all 3 attempts to pass. Intervals are exact 95% binomial (Clopper-Pearson) intervals. At 3 attempts per model, no model is statistically separable from another on this task: every pair of 95% intervals overlaps.
3 attempts per model · same assignment
| GPT-5.6 Luna | 100.0% | 100.0% | 3 / 329–100% | 3m 27s | $0.027 |
|---|---|---|---|---|---|
| GPT-5.6 Sol | 100.0% | 100.0% | 3 / 329–100% | 3m 38s | $0.18 |
| GPT-6 Astra | 100.0% | 100.0% | 3 / 329–100% | 3m 50s | $0.73 |
| GPT-5.4 | 100.0% | 98.8% | 2 / 39–99% | 3m 51s | $0.40 |
| GPT-5.6 Terra1 format failure (no JSON block), scored as a miss on every check | 66.7% | 66.7% | 2 / 39–99% | 4m 26s | $0.23 |
| GPT-5.5 | 100.0% | 97.6% | 1 / 31–91% | 5m 56s | $0.63 |
| GPT-5.4 Mini | 88.9% | 95.2% | 0 / 30–71% | 6m 28s | $0.26 |
Select a column to sort. Incomplete samples stay last. Accuracy columns are not shaded: at 3 attempts per model, differences inside the intervals are not meaningful. Cost uses the list prices in the price table; the price behind GPT-5.6 Sol’s cost is unconfirmed and may be too low. Swipe to see all columns →
Observed misses across all models, most frequent first. An attempt without a usable memo is counted once, under the usable-memo check, not as a miss on every other check.
The $28,500 legal settlement is non-recurring and should be removed from expenses (a favorable adjustment).
Illustrative wording, credited by both judges (GPT-5.4, try 3): Broker already removed the one-time Marigold settlement; firm also excludes it.
Missed by GPT-5.4, GPT-5.5, GPT-5.4 Mini.
Anchor share of in-place retail base rent: 42.80, accepted within ±2.50 points.
| Model | Answer key | Try 1 | Try 2 | Try 3 |
|---|---|---|---|---|
| GPT-5.6 Sol | 42.80 | 42.78 | 42.78 | 42.78 |
| GPT-6 Astra | 42.80 | 42.78 | 42.78 | 42.78 |
| GPT-5.6 Terra | 42.80 | 42.78 | 42.78 | no JSON block |
| GPT-5.6 Luna | 42.80 | 42.78 | 42.78 | 42.78 |
| GPT-5.4 | 42.80 | 42.78 | 42.78 | 42.78 |
| GPT-5.5 | 42.80 | 42.78 | 42.78 | 42.78 |
| GPT-5.4 Mini | 42.80 | 42.80 | 65.22 | 65.22 |
Unit 307 has been offline since March 2026 due to water intrusion from a failed building envelope; the PCA calls for $186,000 of immediate repairs and the envelope work was excluded from insurance.
Illustrative wording, credited by both judges (GPT-5.4 Mini, try 1): Immediate PCA repairs: 186,000. - Envelope / flashing / water intrusion: 95,000 - Unit 307 restoration: 41,300
Missed by GPT-5.4 Mini.
Broker pro forma marks every residential unit to market rent on day one (loss-to-lease taken as Year-1 income). Firm basis is in-place rents for occupied units only.
Illustrative wording, credited by both judges (GPT-5.4 Mini, try 2): Broker pro forma uses residential market rents and full mark-to-market assumptions. Firm Context prohibits that. I used in-place contractual rents for occupied units only
Missed by GPT-5.4 Mini.
Returned a complete memo with the required JSON block
These are benchmark checks, not severity ratings. Financial impact has not been measured. Missed findings use the primary judge. A mistake is attributed to a model only when both judges flagged it. Full-pass results use both judges where a second judgment exists.
Reconcile the data room, normalize NOI and write a preliminary investment recommendation. 10 supplied text documents, 35 checks, 3 planned attempts per model. Small sample; every output needs review.
Numbers correct is the share of numerical checks within the answer key’s tolerance. Risk & decision checks covers findings, mistakes avoided and the requested recommendation. Both are individual-check pass rates, not the probability that an entire output is correct.
The check on the price at which the deal works accepts “no clearing price” as a correct answer; 20 of 20 usable attempts gave that answer, so the check never tested a stated figure.
Passed every check is the stricter measure: every required check must pass in that attempt. One missed observation fails it, just as a wrong number does. A 0/3 result can still contain correct calculations.
The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count; ties are listed alphabetically and are not broken by cost or by check rates. Cost and speed cards require all planned attempts to pass every check. The table’s default order (full passes, then numbers, then risk and decision checks, then cost) is for display only, and it sorts all models directly by cost or wait, including failed results. Incomplete samples appear last in every view and are excluded from the cards when complete samples exist.
Chart labels are rounded. Missing results are not plotted; asterisks mark incomplete samples. In the combined table, teal marks lower API cost and faster median response within each column; accuracy columns are not shaded. It is not an overall score. Highlighted marks identify the models named on the selected card.
API charges per response at the list prices in the method’s price table, excluding subscriptions, tools, retries and human review. Times reflect this run; neither price nor speed is guaranteed in a chat app.
Highest full-pass rate
3/3 full passes each · 95% CI 29–100%. Listed alphabetically; no tie-break is applied.
Lowest-cost passing
$0.0233m 13s
3/3 full passes · 95% CI 29–100%
Fastest passing
$0.392m 34s
3/3 full passes · 95% CI 29–100%
A full pass means every check passed in that attempt; passing cards require all 3 attempts to pass. Intervals are exact 95% binomial (Clopper-Pearson) intervals. At 3 attempts per model, no model is statistically separable from another on this task: every pair of 95% intervals overlaps.
Numbers: n/a. The only scored numerical check (the price at which the deal works) accepts “no clearing price” as a correct answer, and every answered attempt gave no price, so no stated figure was checked on this task.
Scoring v2 (September 14, 2026) excludes the Year-1 NOI and going-in cap rate at the asking price checks for every model. This exclusion was decided after results were observed. It changed full passes for GPT-5.6 Sol (2 to 3), GPT-6 Astra (0 to 3), GPT-5.6 Terra (1 to 2), GPT-5.6 Luna (1 to 3) and GPT-5.5 (0 to 3). The original results are shown below the table.
3 attempts per model · same assignment Numbers: n/a, see the note above.
| GPT-5.6 Luna | n/a | 100.0% | 3 / 329–100% | 3m 13s | $0.023 |
|---|---|---|---|---|---|
| GPT-5.6 Sol | n/a | 100.0% | 3 / 329–100% | 3m 23s | $0.14 |
| GPT-6 Astra | n/a | 100.0% | 3 / 329–100% | 2m 34s | $0.39 |
| GPT-5.5 | n/a | 100.0% | 3 / 329–100% | 5m 7s | $0.46 |
| GPT-5.4 | n/a | 97.9% | 2 / 39–99% | 2m 46s | $0.27 |
| GPT-5.6 Terra1 format failure (no JSON block), scored as a miss on every check | n/a | 66.7% | 2 / 39–99% | 3m 3s | $0.15 |
| GPT-5.4 Mini | n/a | 93.8% | 1 / 31–91% | 3m 29s | $0.14 |
Select a column to sort. Incomplete samples stay last. Accuracy columns are not shaded: at 3 attempts per model, differences inside the intervals are not meaningful. Cost uses the list prices in the price table; the price behind GPT-5.6 Sol’s cost is unconfirmed and may be too low. Swipe to see all columns →
| Model | Original keyfull passes | Scoring v2full passes | Year-1 NOIpassed, original key | Going-in cap rate at the asking pricepassed, original key |
|---|---|---|---|---|
| GPT-5.6 Sol | 2 / 3 | 3 / 3 | 2 / 3 | 3 / 3 |
| GPT-6 Astra | 0 / 3 | 3 / 3 | 0 / 3 | 0 / 3 |
| GPT-5.6 Terra | 1 / 3 | 2 / 3 | 1 / 3 | 1 / 3 |
| GPT-5.6 Luna | 1 / 3 | 3 / 3 | 1 / 3 | 1 / 3 |
| GPT-5.4 | 2 / 3 | 2 / 3 | 3 / 3 | 3 / 3 |
| GPT-5.5 | 0 / 3 | 3 / 3 | 0 / 3 | 3 / 3 |
| GPT-5.4 Mini | 1 / 3 | 1 / 3 | 3 / 3 | 3 / 3 |
Same tight rule in both columns; the original key also requires the excluded checks to pass. Original grades are retained.
Observed misses across all models, most frequent first. An attempt without a usable memo is counted once, under the usable-memo check, not as a miss on every other check.
The $28,500 legal settlement is non-recurring and should be removed from expenses (a favorable adjustment).
Illustrative wording, credited by both judges (GPT-5.5, try 1): Broker removed the $28,500 Marigold legal settlement, which is consistent with firm policy.
Missed by GPT-5.4, GPT-5.4 Mini.
Percentage rent from Bluewater Café is based on 2024 sales; TTM sales are below the natural breakpoint, so percentage rent should be zero.
Illustrative wording, credited by both judges (GPT-5.5, try 1): Includes Bluewater percentage rent of $18,420 based on 2024 sales | Count only if TTM reported sales exceed breakpoint
Missed by GPT-5.4 Mini.
Counts percentage rent.
GPT-5.4 Mini, try 3, flagged by both judges: Percentage rent
= **18,420**
Missed by GPT-5.4 Mini.
Returned a complete memo with the required JSON block
These are benchmark checks, not severity ratings. Financial impact has not been measured. Missed findings use the primary judge. A mistake is attributed to a model only when both judges flagged it. Full-pass results use both judges where a second judgment exists.
Check the broker’s claims, calculate the firm-basis numbers and identify diligence questions. 3 supplied text documents, 18 checks, 3 planned attempts per model. Small sample; every output needs review.
Numbers correct is the share of numerical checks within the answer key’s tolerance. Risk & decision checks covers findings, mistakes avoided and the requested recommendation. Both are individual-check pass rates, not the probability that an entire output is correct.
OM recovery provisions are absent. NOI and cap-rate estimates cannot be graded against a unique contractual answer. These two checks are excluded for every model; original grades are retained. The exclusion itself awards no credit for withheld Year-1 NOI and going-in cap rate at the asking price answers.
Passed every check is the stricter measure: every required check must pass in that attempt. One missed observation fails it, just as a wrong number does. A 0/3 result can still contain correct calculations.
The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count; ties are listed alphabetically and are not broken by cost or by check rates. Cost and speed cards require all planned attempts to pass every check. The table’s default order (full passes, then numbers, then risk and decision checks, then cost) is for display only, and it sorts all models directly by cost or wait, including failed results. Incomplete samples appear last in every view and are excluded from the cards when complete samples exist.
Chart labels are rounded. Missing results are not plotted; asterisks mark incomplete samples. In the combined table, teal marks lower API cost and faster median response within each column; accuracy columns are not shaded. It is not an overall score. Highlighted marks identify the models named on the selected card.
API charges per response at the list prices in the method’s price table, excluding subscriptions, tools, retries and human review. Times reflect this run; neither price nor speed is guaranteed in a chat app.
Highest full-pass rate
0/3 for every model with all attempts graded · 95% CI 0–71%
Lowest-cost passing
Requires every planned attempt to pass.
Fastest passing
Requires every planned attempt to pass.
A full pass means every check passed in that attempt; passing cards require all 3 attempts to pass. Intervals are exact 95% binomial (Clopper-Pearson) intervals. At 3 attempts per model, no model is statistically separable from another on this task: every pair of 95% intervals overlaps.
Sensitivity: two checks were missed in at least 85% of attempts: “Every lease rolls within twelve months; the broker's 'immediate mark-to-market' is the same fact as 100% turnover exposure in year one” (missed in 100% of attempts) and “Suite D has been vacant since January 2026 and is being marketed at a rent well above the building's in-place retail rents” (missed in 100% of attempts). Excluding them, full passes would be GPT-5.6 Sol 2/3, GPT-6 Astra 1/3, GPT-5.6 Terra 1/3, GPT-5.6 Luna 2/3, GPT-5.4 2/3, GPT-5.5 3/3, GPT-5.4 Mini 3/3. The answer key still requires them; this is not a rescore.
3 attempts per model · same assignment
| GPT-5.6 Luna | 100.0% | 83.3% | 0 / 30–71% | 1m 32s | $0.015 |
|---|---|---|---|---|---|
| GPT-5.6 Sol | 100.0% | 83.3% | 0 / 30–71% | 1m 53s | $0.093 |
| GPT-5.6 Terra | 100.0% | 83.3% | 0 / 30–71% | 1m 44s | $0.10 |
| GPT-5.4 Mini | 100.0% | 83.3% | 0 / 30–71% | 2m 15s | $0.13 |
| GPT-6 Astra | 100.0% | 83.3% | 0 / 30–71% | 1m 55s | $0.32 |
| GPT-5.5 | 100.0% | 83.3% | 0 / 30–71% | 2m 4s | $0.32 |
| GPT-5.4 | 97.6% | 77.8% | 0 / 30–71% | 1m 54s | $0.20 |
Select a column to sort. Incomplete samples stay last. Accuracy columns are not shaded: at 3 attempts per model, differences inside the intervals are not meaningful. Cost uses the list prices in the price table; the price behind GPT-5.6 Sol’s cost is unconfirmed and may be too low. Swipe to see all columns →
Observed misses across all models, most frequent first.
Every lease rolls within twelve months; the broker's 'immediate mark-to-market' is the same fact as 100% turnover exposure in year one.
Missed by every model, every try.
Suite D has been vacant since January 2026 and is being marketed at a rent well above the building's in-place retail rents.
Missed by every model, every try.
Month-to-month units: 4, exact match required.
| Model | Answer key | Try 1 | Try 2 | Try 3 |
|---|---|---|---|---|
| GPT-5.6 Sol | 4 | 4 | 4 | 4 |
| GPT-6 Astra | 4 | 4 | 4 | 4 |
| GPT-5.6 Terra | 4 | 4 | 4 | 4 |
| GPT-5.6 Luna | 4 | 4 | 4 | 4 |
| GPT-5.4 | 4 | 4 | 3 | 4 |
| GPT-5.5 | 4 | 4 | 4 | 4 |
| GPT-5.4 Mini | 4 | 4 | 4 | 4 |
Three new leases (209, 301, 506) carry one month free, so their effective rent is about 8% below face.
Illustrative wording, credited by both judges (GPT-6 Astra, try 1): Three concession units: 209, 301 and 506 each have one month free.
Missed by GPT-5.4.
Four month-to-month tenants (204, 311, 402, 508) can leave on 30 days' notice.
Illustrative wording, credited by both judges (GPT-5.6 Terra, try 1): 4 month-to-month units: 204, 311, 402, and 508, totaling $13,900/month.
Missed by GPT-5.4.
These are benchmark checks, not severity ratings. Financial impact has not been measured. Missed findings use the primary judge. A mistake is attributed to a model only when both judges flagged it. Full-pass results use both judges where a second judgment exists.
Calculate occupancy and rent, then identify rollover, concessions and collection risk. 2 supplied text documents, 27 checks, 3 planned attempts per model. Small sample; every output needs review.
Numbers correct is the share of numerical checks within the answer key’s tolerance. Risk & decision checks covers findings, mistakes avoided and the requested recommendation. Both are individual-check pass rates, not the probability that an entire output is correct.
Passed every check is the stricter measure: every required check must pass in that attempt. One missed observation fails it, just as a wrong number does. A 0/3 result can still contain correct calculations.
The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count; ties are listed alphabetically and are not broken by cost or by check rates. Cost and speed cards require all planned attempts to pass every check. The table’s default order (full passes, then numbers, then risk and decision checks, then cost) is for display only, and it sorts all models directly by cost or wait, including failed results. Incomplete samples appear last in every view and are excluded from the cards when complete samples exist.
Chart labels are rounded. Missing results are not plotted; asterisks mark incomplete samples. In the combined table, teal marks lower API cost and faster median response within each column; accuracy columns are not shaded. It is not an overall score. Highlighted marks identify the models named on the selected card.
API charges per response at the list prices in the method’s price table, excluding subscriptions, tools, retries and human review. Times reflect this run; neither price nor speed is guaranteed in a chat app.
Highest full-pass rate
1/3 full passes each · 95% CI 1–91%. Listed alphabetically; no tie-break is applied.
Lowest-cost passing
Requires every planned attempt to pass.
Fastest passing
Requires every planned attempt to pass.
A full pass means every check passed in that attempt; passing cards require all 3 attempts to pass. Intervals are exact 95% binomial (Clopper-Pearson) intervals. At 3 attempts per model, no model is statistically separable from another on this task: every pair of 95% intervals overlaps.
3 attempts per model · same assignment
| GPT-5.6 Terra | 100.0% | 96.1% | 1 / 31–91% | 1m 45s | $0.11 |
|---|---|---|---|---|---|
| GPT-5.6 Sol | 93.9% | 98.0% | 1 / 31–91% | 2m 30s | $0.12 |
| GPT-5.6 Luna | 90.9% | 98.0% | 1 / 31–91% | 2m 6s | $0.020 |
| GPT-5.4 Mini | 100.0% | 94.1% | 0 / 30–71% | 2m 36s | $0.15 |
| GPT-5.4 | 100.0% | 94.1% | 0 / 30–71% | 2m 10s | $0.23 |
| GPT-5.5 | 100.0% | 94.1% | 0 / 30–71% | 2m 55s | $0.39 |
| GPT-6 Astra | 81.8% | 92.2% | 0 / 30–71% | 1m 47s | $0.36 |
Select a column to sort. Incomplete samples stay last. Accuracy columns are not shaded: at 3 attempts per model, differences inside the intervals are not meaningful. Cost uses the list prices in the price table; the price behind GPT-5.6 Sol’s cost is unconfirmed and may be too low. Swipe to see all columns →
Observed misses across all models, most frequent first.
Three units carry one month free; contractual rent overstates effective rent for those leases.
Illustrative wording, credited by both judges (GPT-5.6 Luna, try 1): rent roll’s one-month-free concessions for Units 209, 301 and 506
Missed by every model at least once.
Year-1 NOI (band: recoveries zeroed → recoveries at T-12): 1,085,101 to 1,216,254, accepted anywhere inside the band or within ±3% of its nearer edge.
| Model | Answer key | Try 1 | Try 2 | Try 3 |
|---|---|---|---|---|
| GPT-5.6 Sol | 1,085,101 to 1,216,254 | 1,197,146 | did not state a figure | 1,197,146 |
| GPT-6 Astra | 1,085,101 to 1,216,254 | did not state a figure | did not state a figure | did not state a figure |
| GPT-5.6 Terra | 1,085,101 to 1,216,254 | 1,085,107 | 1,085,107 | 1,085,107 |
| GPT-5.6 Luna | 1,085,101 to 1,216,254 | did not state a figure | 1,085,107 | 1,085,107 |
| GPT-5.4 | 1,085,101 to 1,216,254 | 1,085,107 | 1,197,146 | 1,197,146 |
| GPT-5.5 | 1,085,101 to 1,216,254 | 1,197,146 | 1,197,146 | 1,197,146 |
| GPT-5.4 Mini | 1,085,101 to 1,216,254 | 1,197,146 | 1,198,602 | 1,198,602 |
Going-in cap at guidance (band): 3.15 to 3.53, accepted anywhere inside the band or within ±0.15 points of its nearer edge.
| Model | Answer key | Try 1 | Try 2 | Try 3 |
|---|---|---|---|---|
| GPT-5.6 Sol | 3.15 to 3.53 | 3.47 | did not state a figure | 3.47 |
| GPT-6 Astra | 3.15 to 3.53 | did not state a figure | did not state a figure | did not state a figure |
| GPT-5.6 Terra | 3.15 to 3.53 | 3.15 | 3.15 | 3.15 |
| GPT-5.6 Luna | 3.15 to 3.53 | did not state a figure | 3.15 | 3.15 |
| GPT-5.4 | 3.15 to 3.53 | 3.15 | 3.47 | 3.47 |
| GPT-5.5 | 3.15 to 3.53 | 3.47 | 3.47 | 3.47 |
| GPT-5.4 Mini | 3.15 to 3.53 | 3.47 | 3.47 | 3.47 |
The T-12 retail base rent includes six months of a departed tenant (Suite D, Marigold Boutique, through January 2026); Year-1 retail rent must be built from the current roll, not the T-12 total.
Illustrative wording, credited by both judges (GPT-5.5, try 3): Count occupied tenants only, excluding vacant Suite D
Missed by GPT-6 Astra, GPT-5.4.
Non-recurring items removed (insurance proceeds + legal settlement): 90,500, accepted within ±5%.
| Model | Answer key | Try 1 | Try 2 | Try 3 |
|---|---|---|---|---|
| GPT-5.6 Sol | 90,500 | 90,500 | 90,500 | 90,500 |
| GPT-6 Astra | 90,500 | 90,500 | 90,500 | 90,500 |
| GPT-5.6 Terra | 90,500 | 90,500 | 90,500 | 90,500 |
| GPT-5.6 Luna | 90,500 | 98,550 | 90,500 | 90,500 |
| GPT-5.4 | 90,500 | 90,500 | 90,500 | 90,500 |
| GPT-5.5 | 90,500 | 90,500 | 90,500 | 90,500 |
| GPT-5.4 Mini | 90,500 | 90,500 | 90,500 | 90,500 |
The judges split on 1 mistake flag in this task (one judge flagged it, the other did not). Split flags are not attributed to a model here, but under the full-pass rule a flag from either judge still fails that attempt.
These are benchmark checks, not severity ratings. Financial impact has not been measured. Missed findings use the primary judge. A mistake is attributed to a model only when both judges flagged it. Full-pass results use both judges where a second judgment exists.
Translate the rent roll and T-12 into a defensible Year-1 assumption set. 4 supplied text documents, 29 checks, 3 planned attempts per model. Small sample; every output needs review.
Numbers correct is the share of numerical checks within the answer key’s tolerance. Risk & decision checks covers findings, mistakes avoided and the requested recommendation. Both are individual-check pass rates, not the probability that an entire output is correct.
Passed every check is the stricter measure: every required check must pass in that attempt. One missed observation fails it, just as a wrong number does. A 0/3 result can still contain correct calculations.
The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count; ties are listed alphabetically and are not broken by cost or by check rates. Cost and speed cards require all planned attempts to pass every check. The table’s default order (full passes, then numbers, then risk and decision checks, then cost) is for display only, and it sorts all models directly by cost or wait, including failed results. Incomplete samples appear last in every view and are excluded from the cards when complete samples exist.
Chart labels are rounded. Missing results are not plotted; asterisks mark incomplete samples. In the combined table, teal marks lower API cost and faster median response within each column; accuracy columns are not shaded. It is not an overall score. Highlighted marks identify the models named on the selected card.
API charges per response at the list prices in the method’s price table, excluding subscriptions, tools, retries and human review. Times reflect this run; neither price nor speed is guaranteed in a chat app.
Highest full-pass rate
2/3 full passes each · 95% CI 9–99%. Listed alphabetically; no tie-break is applied.
Lowest-cost passing
Requires every planned attempt to pass.
Fastest passing
Requires every planned attempt to pass.
A full pass means every check passed in that attempt; passing cards require all 3 attempts to pass. Intervals are exact 95% binomial (Clopper-Pearson) intervals. At 3 attempts per model, no model is statistically separable from another on this task: every pair of 95% intervals overlaps.
3 attempts per model · same assignment
| GPT-5.6 Sol | 100.0% | 100.0% | 2 / 39–99% | 3m 17s | $0.15 |
|---|---|---|---|---|---|
| GPT-5.6 Terra | 100.0% | 98.2% | 2 / 39–99% | 3m 18s | $0.17 |
| GPT-6 Astra | 100.0% | 98.2% | 2 / 39–99% | 3m 22s | $0.59 |
| GPT-5.5 | 100.0% | 94.7% | 1 / 31–91% | 5m 40s | $0.56 |
| GPT-5.6 Luna | 83.3% | 98.2% | 1 / 31–91% | 3m 36s | $0.025 |
| GPT-5.4 | 100.0% | 98.2% | 0 / 30–71% | 4m 35s | $0.42 |
| GPT-5.4 Mini1 format failure (empty response at the output token limit), scored as a miss on every check | 66.7% | 63.2% | 0 / 30–71% | 6m 27s | $0.26 |
Select a column to sort. Incomplete samples stay last. Accuracy columns are not shaded: at 3 attempts per model, differences inside the intervals are not meaningful. Cost uses the list prices in the price table; the price behind GPT-5.6 Sol’s cost is unconfirmed and may be too low. Swipe to see all columns →
Observed misses across all models, most frequent first. An attempt without a usable memo is counted once, under the usable-memo check, not as a miss on every other check.
The lender sizes the interest reserve to the GC's schedule and requires funded off-site obligations, so the sponsor's equity requirement is understated.
Illustrative wording, credited by both judges (GPT-5.5, try 3): Required MPP equity check is understated by approximately $8.2 million on firm basis. At corrected TDC and 60% LTC:
Missed by GPT-6 Astra, GPT-5.6 Terra, GPT-5.6 Luna, GPT-5.4, GPT-5.5, GPT-5.4 Mini.
Schedule: the GC says 30 months, the sponsor says 24; lease-up and interest reserve both move.
Illustrative wording, credited by both judges (GPT-5.6 Terra, try 1): Uses $3.100m based on a 24-month schedule; omits revised 30-month construction schedule
Missed by GPT-5.5, GPT-5.4 Mini.
Firm-basis untrended NOI (with affordable set-aside): 4,118,800, accepted within ±4%.
| Model | Answer key | Try 1 | Try 2 | Try 3 |
|---|---|---|---|---|
| GPT-5.6 Sol | 4,118,800 | 4,066,631 | 4,066,631 | 4,062,718 |
| GPT-6 Astra | 4,118,800 | 4,062,718 | 4,062,718 | 4,062,718 |
| GPT-5.6 Terra | 4,118,800 | 4,066,631 | 4,062,718 | 4,066,631 |
| GPT-5.6 Luna | 4,118,800 | 4,062,718 | 3,789,118 | 4,052,118 |
| GPT-5.4 | 4,118,800 | 4,066,628 | 4,066,631 | 4,066,631 |
| GPT-5.5 | 4,118,800 | 4,066,631 | 4,066,631 | 4,062,718 |
| GPT-5.4 Mini | 4,118,800 | empty (output token limit) | 4,066,631 | 4,075,397 |
Untrended yield on cost: 4.55, accepted within ±0.35 points.
| Model | Answer key | Try 1 | Try 2 | Try 3 |
|---|---|---|---|---|
| GPT-5.6 Sol | 4.55 | 4.48 | 4.48 | 4.47 |
| GPT-6 Astra | 4.55 | 4.47 | 4.47 | 4.47 |
| GPT-5.6 Terra | 4.55 | 4.47 | 4.41 | 4.48 |
| GPT-5.6 Luna | 4.55 | 4.47 | 4.17 | 4.46 |
| GPT-5.4 | 4.55 | 4.48 | 4.48 | 4.48 |
| GPT-5.5 | 4.55 | 4.48 | 4.48 | 4.47 |
| GPT-5.4 Mini | 4.55 | empty (output token limit) | 4.48 | 4.49 |
Spread to comp-supported exit cap (5.25%–5.50%; key uses 5.25%): -70, accepted within ±45 bps.
| Model | Answer key | Try 1 | Try 2 | Try 3 |
|---|---|---|---|---|
| GPT-5.6 Sol | -70 | -91.14 | -91 | -92 |
| GPT-6 Astra | -70 | -92.57 | -92.57 | -92.57 |
| GPT-5.6 Terra | -70 | -91.90 | -98 | -91.14 |
| GPT-5.6 Luna | -70 | -91.57 | -122 | -93 |
| GPT-5.4 | -70 | -92.12 | -92 | -91 |
| GPT-5.5 | -70 | -77 | -91 | -93 |
| GPT-5.4 Mini | -70 | empty (output token limit) | -91 | -90 |
Returned a complete memo with the required JSON block
These are benchmark checks, not severity ratings. Financial impact has not been measured. Missed findings use the primary judge. A mistake is attributed to a model only when both judges flagged it. Full-pass results use both judges where a second judgment exists.
Reconcile costs, schedule and lender requirements to assess the equity investment. 8 supplied text documents, 26 checks, 3 planned attempts per model. Small sample; every output needs review.
Numbers correct is the share of numerical checks within the answer key’s tolerance. Risk & decision checks covers findings, mistakes avoided and the requested recommendation. Both are individual-check pass rates, not the probability that an entire output is correct.
The check on the land price that would make it work accepts “no clearing price” as a correct answer; 19 of 20 usable attempts gave that answer, so the check rarely tested a stated figure.
Passed every check is the stricter measure: every required check must pass in that attempt. One missed observation fails it, just as a wrong number does. A 0/3 result can still contain correct calculations.
The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count; ties are listed alphabetically and are not broken by cost or by check rates. Cost and speed cards require all planned attempts to pass every check. The table’s default order (full passes, then numbers, then risk and decision checks, then cost) is for display only, and it sorts all models directly by cost or wait, including failed results. Incomplete samples appear last in every view and are excluded from the cards when complete samples exist.
Chart labels are rounded. Missing results are not plotted; asterisks mark incomplete samples. In the combined table, teal marks lower API cost and faster median response within each column; accuracy columns are not shaded. It is not an overall score. Highlighted marks identify the models named on the selected card.
API charges per response at the list prices in the method’s price table, excluding subscriptions, tools, retries and human review. Times reflect this run; neither price nor speed is guaranteed in a chat app.
Highest full-pass rate
1/3 full passes each · 95% CI 1–91%. Listed alphabetically; no tie-break is applied.
Lowest-cost passing
Requires every planned attempt to pass.
Fastest passing
Requires every planned attempt to pass.
A full pass means every check passed in that attempt; passing cards require all 3 attempts to pass. Intervals are exact 95% binomial (Clopper-Pearson) intervals. At 3 attempts per model, no model is statistically separable from another on this task: every pair of 95% intervals overlaps.
3 attempts per model · same assignment
| GPT-5.4 | 95.8% | 97.8% | 1 / 31–91% | 3m 57s | $0.40 |
|---|---|---|---|---|---|
| GPT-5.6 Sol | 91.7% | 95.6% | 1 / 31–91% | 2m 56s | $0.15 |
| GPT-5.6 Terra | 91.7% | 91.1% | 1 / 31–91% | 2m 22s | $0.16 |
| GPT-5.5 | 100.0% | 93.3% | 0 / 30–71% | 4m 15s | $0.69 |
| GPT-5.6 Luna | 95.8% | 91.1% | 0 / 30–71% | 3m 16s | $0.024 |
| GPT-5.4 Mini | 87.5% | 80.0% | 0 / 30–71% | 3m 48s | $0.22 |
| GPT-6 Astra | 83.3% | 95.6% | 0 / 30–71% | 4m 9s | $0.65 |
Select a column to sort. Incomplete samples stay last. Accuracy columns are not shaded: at 3 attempts per model, differences inside the intervals are not meaningful. Cost uses the list prices in the price table; the price behind GPT-5.6 Sol’s cost is unconfirmed and may be too low. Swipe to see all columns →
Observed misses across all models, most frequent first.
CO-08, the window acceleration premium, was rejected; the three-week slip therefore stands and the window delivery risk is unresolved.
Illustrative wording, credited by both judges (GPT-5.6 Terra, try 1): the unresolved window-delivery issue noted by the monitor
Missed by GPT-5.6 Sol, GPT-5.6 Terra, GPT-5.5, GPT-5.4 Mini (every try).
In-balance shortfall after draw: 2,326,000, accepted within ±12%.
| Model | Answer key | Try 1 | Try 2 | Try 3 |
|---|---|---|---|---|
| GPT-5.6 Sol | 2,326,000 | 2,315,800 | 3,026,998 | 2,315,800 |
| GPT-6 Astra | 2,326,000 | 2,315,800 | 3,090,720 | 2,315,800 |
| GPT-5.6 Terra | 2,326,000 | 2,081,908 | 1,384,480 | 2,092,708 |
| GPT-5.6 Luna | 2,326,000 | 2,174,700 | 2,095,678 | did not state a figure |
| GPT-5.4 | 2,326,000 | 2,325,800 | 3,230,091 | 2,081,908 |
| GPT-5.5 | 2,326,000 | 2,315,800 | 2,315,800 | 2,325,800 |
| GPT-5.4 Mini | 2,326,000 | -1,053,300 | 1,384,480 | -352,862 |
The right call is FUND_REDUCED. Fund the corrected amount.
| Model | Right call | Try 1 | Try 2 | Try 3 |
|---|---|---|---|---|
| GPT-5.6 Sol | FUND_REDUCED | FUND_REDUCED | FUND_REDUCED | FUND_REDUCED |
| GPT-6 Astra | FUND_REDUCED | HOLD | HOLD | FUND_REDUCED |
| GPT-5.6 Terra | FUND_REDUCED | FUND_REDUCED | HOLD | HOLD |
| GPT-5.6 Luna | FUND_REDUCED | HOLD | HOLD | HOLD |
| GPT-5.4 | FUND_REDUCED | FUND_REDUCED | FUND_REDUCED | FUND_REDUCED |
| GPT-5.5 | FUND_REDUCED | FUND_REDUCED | FUND_REDUCED | FUND_REDUCED |
| GPT-5.4 Mini | FUND_REDUCED | FUND_REDUCED | FUND_REDUCED | FUND_REDUCED |
Amount to fund: 2,843,910, accepted within ±3%.
| Model | Answer key | Try 1 | Try 2 | Try 3 |
|---|---|---|---|---|
| GPT-5.6 Sol | 2,843,910 | 2,132,712 | 2,843,910 | 2,843,910 |
| GPT-6 Astra | 2,843,910 | 0 | 2,830,140 | 2,132,712 |
| GPT-5.6 Terra | 2,843,910 | 2,830,140 | 2,132,712 | 2,840,940 |
| GPT-5.6 Luna | 2,843,910 | 2,843,910 | 2,843,910 | 2,843,910 |
| GPT-5.4 | 2,843,910 | 2,830,140 | 2,830,140 | 2,830,140 |
| GPT-5.5 | 2,843,910 | 2,854,710 | 2,830,140 | 2,830,140 |
| GPT-5.4 Mini | 2,843,910 | 2,843,910 | 2,843,910 | 2,833,110 |
The interest reserve is $520,000 short of projected remaining interest because of the schedule slip and the forward curve; that gap is part of the rebalancing.
Illustrative wording, credited by both judges (GPT-5.4, try 1): Projected remaining interest cost at the revised schedule / forward curve: **$4,430,000** **Interest reserve shortfall:** 4,430,000 - 3,910,000 = **$520,000**
Missed by GPT-5.6 Luna, GPT-5.5, GPT-5.4 Mini.
The HVAC line's total completed exceeds previous plus current by $12,000.
Illustrative wording, credited by both judges (GPT-5.4, try 1): the **$12,000** overstatement sits in **line 9 / HVAC**
Missed by GPT-5.4, GPT-5.4 Mini.
After the draw the loan is out of balance: remaining uses (hard cost to complete, retainage payable at completion, soft costs, contingency, projected interest) exceed the undisbursed commitment, so the borrower must deposit the shortfall before the next draw.
Illustrative wording, credited by both judges (GPT-5.6 Terra, try 3): The loan is out of balance by $2,092,708 after the corrected draw.
Missed by GPT-5.4 Mini.
Runs the in-balance test without retainage payable or projected interest, or declares the loan in balance.
GPT-5.4 Mini, try 1, flagged by both judges: So the project is still in balance by 1053300
GPT-5.4 Mini, try 3, flagged by both judges: **33,766,890 - 33,414,028 = 352,862 cushion**
Missed by GPT-5.4 Mini.
Retainage at 10% (on unheld eligible work, or on all eligible work in place): 315,990 to 419,470, accepted anywhere inside the band or within ±3% of its nearer edge.
| Model | Answer key | Try 1 | Try 2 | Try 3 |
|---|---|---|---|---|
| GPT-5.6 Sol | 315,990 to 419,470 | 340,448 | 315,990 | 315,990 |
| GPT-6 Astra | 315,990 to 419,470 | 417,940 | 417,940 | 236,968 |
| GPT-5.6 Terra | 315,990 to 419,470 | 417,940 | 340,448 | 419,140 |
| GPT-5.6 Luna | 315,990 to 419,470 | 419,470 | 419,470 | 315,990 |
| GPT-5.4 | 315,990 to 419,470 | 314,460 | 417,940 | 417,940 |
| GPT-5.5 | 315,990 to 419,470 | 420,670 | 314,460 | 417,940 |
| GPT-5.4 Mini | 315,990 to 419,470 | 315,990 | 315,990 | 314,790 |
These are benchmark checks, not severity ratings. Financial impact has not been measured. Missed findings use the primary judge. A mistake is attributed to a model only when both judges flagged it. Full-pass results use both judges where a second judgment exists.
Check the draw, remaining funding and in-balance test before recommending a release. 6 supplied text documents, 24 checks, 3 planned attempts per model. Small sample; every output needs review.
Numbers correct is the share of numerical checks within the answer key’s tolerance. Risk & decision checks covers findings, mistakes avoided and the requested recommendation. Both are individual-check pass rates, not the probability that an entire output is correct.
Passed every check is the stricter measure: every required check must pass in that attempt. One missed observation fails it, just as a wrong number does. A 0/3 result can still contain correct calculations.
The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count; ties are listed alphabetically and are not broken by cost or by check rates. Cost and speed cards require all planned attempts to pass every check. The table’s default order (full passes, then numbers, then risk and decision checks, then cost) is for display only, and it sorts all models directly by cost or wait, including failed results. Incomplete samples appear last in every view and are excluded from the cards when complete samples exist.
Chart labels are rounded. Missing results are not plotted; asterisks mark incomplete samples. In the combined table, teal marks lower API cost and faster median response within each column; accuracy columns are not shaded. It is not an overall score. Highlighted marks identify the models named on the selected card.
API charges per response at the list prices in the method’s price table, excluding subscriptions, tools, retries and human review. Times reflect this run; neither price nor speed is guaranteed in a chat app.
Highest full-pass rate
$0.533m 28s
3/3 full passes · 95% CI 29–100%
Lowest-cost passing
$0.533m 28s
3/3 full passes · 95% CI 29–100%
Fastest passing
$0.533m 28s
3/3 full passes · 95% CI 29–100%
A full pass means every check passed in that attempt; passing cards require all 3 attempts to pass. Intervals are exact 95% binomial (Clopper-Pearson) intervals. At 3 attempts per model, no model is statistically separable from another on this task: every pair of 95% intervals overlaps.
Second judge: only GPT-6 Astra’s memos on this task had a second judgment. Full passes for GPT-5.6 Sol, GPT-5.6 Luna, GPT-5.4, GPT-5.5 and GPT-5.4 Mini use one judge, a more lenient test, so these counts are not like for like.
* Incomplete sample; coverage shown below.
| GPT-6 Astra | 100.0% | 100.0% | 3 / 329–100% | 3m 28s | $0.53 |
|---|---|---|---|---|---|
| GPT-5.4 | 100.0% | 97.8% | 2 / 39–99% | 3m 24s | $0.35 |
| GPT-5.5 | 90.5% | 100.0% | 2 / 39–99% | 4m | $0.49 |
| GPT-5.6 Luna | 100.0% | 95.6% | 1 / 31–91% | 2m 58s | $0.021 |
| GPT-5.4 Mini | 81.0% | 91.1% | 0 / 30–71% | 4m 23s | $0.18 |
| GPT-5.6 Sol2/3 attempts graded · 1 memo returned, not scored (judge API quota exhausted) | 100.0% | 96.7% | 1 / 21–99% | 2m 24s | $0.11 |
| GPT-5.6 Terra0/3 attempts graded · 3 memos returned, not scored (judge API quota exhausted) | Not scored | Not scored | Not scored | Not scored | Not scored |
Select a column to sort. Incomplete samples stay last. Accuracy columns are not shaded: at 3 attempts per model, differences inside the intervals are not meaningful. Cost uses the list prices in the price table; the price behind GPT-5.6 Sol’s cost is unconfirmed and may be too low. Swipe to see all columns →
Observed misses across all models, most frequent first.
The sponsor's trended stabilized NOI must not be used for sizing; the untrended, concession-adjusted figure is the one the lender will use.
Illustrative wording, credited by both judges (GPT-6 Astra, try 1): "Actual rents, concessions and the newer stabilized expense budget supersede that earlier total. No trended rents are used."
Missed by GPT-5.6 Sol, GPT-5.6 Luna, GPT-5.4, GPT-5.4 Mini (every try).
Stabilized NOI, effective rents net of concessions: 3,135,000, accepted within ±4%.
| Model | Answer key | Try 1 | Try 2 | Try 3 |
|---|---|---|---|---|
| GPT-5.6 Sol | 3,135,000 | 3,144,798 | 3,040,785 | memo returned; not scored (judge API quota exhausted) |
| GPT-6 Astra | 3,135,000 | 3,040,785 | 3,040,785 | 3,040,785 |
| GPT-5.6 Terra | 3,135,000 | memo returned; not scored (judge API quota exhausted) | memo returned; not scored (judge API quota exhausted) | memo returned; not scored (judge API quota exhausted) |
| GPT-5.6 Luna | 3,135,000 | 3,017,118 | 3,040,652 | 3,034,959 |
| GPT-5.4 | 3,135,000 | 3,085,390 | 3,190,099 | 3,078,939 |
| GPT-5.5 | 3,135,000 | 3,086,423 | 2,992,803 | 3,144,798 |
| GPT-5.4 Mini | 3,135,000 | 3,014,647 | 2,788,355 | 3,085,183 |
Paydown at maturity: 18,200,000, accepted within ±8%.
| Model | Answer key | Try 1 | Try 2 | Try 3 |
|---|---|---|---|---|
| GPT-5.6 Sol | 18,200,000 | 18,098,783 | 19,246,508 | memo returned; not scored (judge API quota exhausted) |
| GPT-6 Astra | 18,200,000 | 19,246,508 | 19,246,508 | 19,246,508 |
| GPT-5.6 Terra | 18,200,000 | memo returned; not scored (judge API quota exhausted) | memo returned; not scored (judge API quota exhausted) | memo returned; not scored (judge API quota exhausted) |
| GPT-5.6 Luna | 18,200,000 | 19,507,663 | 19,247,977 | 19,310,798 |
| GPT-5.4 | 18,200,000 | 17,709,965 | 17,598,909 | 18,825,503 |
| GPT-5.5 | 18,200,000 | 18,742,916 | 19,775,972 | 18,098,783 |
| GPT-5.4 Mini | 18,200,000 | 18,339,064 | 22,031,944 | 18,756,599 |
Take-out under DSCR test: 34,600,000, accepted within ±5%.
| Model | Answer key | Try 1 | Try 2 | Try 3 |
|---|---|---|---|---|
| GPT-5.6 Sol | 34,600,000 | 34,701,217 | 33,553,492 | memo returned; not scored (judge API quota exhausted) |
| GPT-6 Astra | 34,600,000 | 33,553,492 | 33,553,492 | 33,553,492 |
| GPT-5.6 Terra | 34,600,000 | memo returned; not scored (judge API quota exhausted) | memo returned; not scored (judge API quota exhausted) | memo returned; not scored (judge API quota exhausted) |
| GPT-5.6 Luna | 34,600,000 | 33,292,337 | 33,552,023 | 33,489,202 |
| GPT-5.4 | 34,600,000 | 35,090,035 | 35,201,091 | 33,974,497 |
| GPT-5.5 | 34,600,000 | 34,057,084 | 33,024,028 | 34,701,217 |
| GPT-5.4 Mini | 34,600,000 | 34,460,936 | 30,768,056 | 34,043,401 |
Take-out under LTV test: 37,060,000, accepted within ±5%.
| Model | Answer key | Try 1 | Try 2 | Try 3 |
|---|---|---|---|---|
| GPT-5.6 Sol | 37,060,000 | 37,165,792 | 35,936,552 | memo returned; not scored (judge API quota exhausted) |
| GPT-6 Astra | 37,060,000 | 35,936,552 | 35,936,552 | 35,936,552 |
| GPT-5.6 Terra | 37,060,000 | memo returned; not scored (judge API quota exhausted) | memo returned; not scored (judge API quota exhausted) | memo returned; not scored (judge API quota exhausted) |
| GPT-5.6 Luna | 37,060,000 | 35,656,849 | 35,934,980 | 35,867,696 |
| GPT-5.4 | 37,060,000 | 36,463,703 | 37,701,168 | 36,387,460 |
| GPT-5.5 | 37,060,000 | 36,475,911 | 35,369,485 | 37,165,792 |
| GPT-5.4 Mini | 37,060,000 | 36,908,442 | 32,953,287 | 36,461,257 |
The permanent lender sizes on trailing-three-month NOI net of concessions and requires 90% occupancy for 90 days, which pushes funding past the construction loan maturity and makes the extension necessary.
Illustrative wording, credited by both judges (GPT-6 Astra, try 1): "The lender ultimately requires trailing-three-month actual NOI, lease-level concession amortization and physical-occupancy evidence."; "the 90-day seasoning requirement pushes funding into March 2030 at the earliest"; "Without an extension or replacement financing, the full $52.8 million construction balance remains d
Missed by GPT-5.4 Mini.
These are benchmark checks, not severity ratings. Financial impact has not been measured. Missed findings use the primary judge. A mistake is attributed to a model only when both judges flagged it. Full-pass results use both judges where a second judgment exists.
Update stabilization, size permanent debt and identify the loan paydown. 5 supplied text documents, 23 checks, 3 planned attempts per model. Small sample; every output needs review.
Numbers correct is the share of numerical checks within the answer key’s tolerance. Risk & decision checks covers findings, mistakes avoided and the requested recommendation. Both are individual-check pass rates, not the probability that an entire output is correct.
Passed every check is the stricter measure: every required check must pass in that attempt. One missed observation fails it, just as a wrong number does. A 0/3 result can still contain correct calculations.
The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count; ties are listed alphabetically and are not broken by cost or by check rates. Cost and speed cards require all planned attempts to pass every check. The table’s default order (full passes, then numbers, then risk and decision checks, then cost) is for display only, and it sorts all models directly by cost or wait, including failed results. Incomplete samples appear last in every view and are excluded from the cards when complete samples exist.
Chart labels are rounded. Missing results are not plotted; asterisks mark incomplete samples. In the combined table, teal marks lower API cost and faster median response within each column; accuracy columns are not shaded. It is not an overall score. Highlighted marks identify the models named on the selected card.
API charges per response at the list prices in the method’s price table, excluding subscriptions, tools, retries and human review. Times reflect this run; neither price nor speed is guaranteed in a chat app.
Highest full-pass rate
0/3 for every model with all attempts graded · 95% CI 0–71%
Lowest-cost passing
Requires every planned attempt to pass.
Fastest passing
Requires every planned attempt to pass.
A full pass means every check passed in that attempt; passing cards require all 3 attempts to pass. Intervals are exact 95% binomial (Clopper-Pearson) intervals. At 3 attempts per model, no model is statistically separable from another on this task: every pair of 95% intervals overlaps.
Sensitivity: two checks were missed in at least 85% of attempts: “The pre-close $6,100 receivable on Unit 211 belongs to the seller under the PSA and is not our bad debt” (missed in 95% of attempts) and “Suite C paid rent through March as budgeted; its Q1 rent is not run-rate and the April cliff is already in the budget” (missed in 86% of attempts). Excluding them, full passes would be GPT-5.6 Sol 1/3, GPT-6 Astra 1/3, GPT-5.6 Terra 1/3, GPT-5.6 Luna 0/3, GPT-5.4 0/3, GPT-5.5 1/3, GPT-5.4 Mini 0/3. The answer key still requires them; this is not a rescore.
3 attempts per model · same assignment
| GPT-5.6 Sol | 100.0% | 86.7% | 0 / 30–71% | 3m 8s | $0.14 |
|---|---|---|---|---|---|
| GPT-5.4 | 100.0% | 86.7% | 0 / 30–71% | 4m 29s | $0.48 |
| GPT-6 Astra | 100.0% | 86.7% | 0 / 30–71% | 3m 35s | $0.62 |
| GPT-5.5 | 100.0% | 86.7% | 0 / 30–71% | 7m 17s | $0.66 |
| GPT-5.6 Luna | 100.0% | 84.4% | 0 / 30–71% | 3m 19s | $0.026 |
| GPT-5.6 Terra | 100.0% | 82.2% | 0 / 30–71% | 2m 42s | $0.17 |
| GPT-5.4 Mini | 100.0% | 77.8% | 0 / 30–71% | 5m 10s | $0.25 |
Select a column to sort. Incomplete samples stay last. Accuracy columns are not shaded: at 3 attempts per model, differences inside the intervals are not meaningful. Cost uses the list prices in the price table; the price behind GPT-5.6 Sol’s cost is unconfirmed and may be too low. Swipe to see all columns →
Observed misses across all models, most frequent first.
The pre-close $6,100 receivable on Unit 211 belongs to the seller under the PSA and is not our bad debt.
Illustrative wording, credited by both judges (GPT-6 Astra, try 3): The seller’s $6,100 pre-close rent receivable is not ours and creates no additional NOI adjustment.
Missed by every model at least once.
Suite C paid rent through March as budgeted; its Q1 rent is not run-rate and the April cliff is already in the budget.
Illustrative wording, credited by both judges (GPT-6 Astra, try 1): Suite C vacated March 31 as budgeted.
Missed by every model at least once.
Commentary appears only on lines outside the threshold; the bridge ties with no plug.
Illustrative wording, credited by both judges (GPT-5.6 Luna, try 1): Commentary is shown only where the variance exceeds the lower of 5% or $2,500. ... NOI impact: $(10,856) + $(18,203) = $(29,059) ... The $100 is disclosed explicitly rather than included in a miscellaneous plug.
Missed by GPT-6 Astra, GPT-5.6 Terra, GPT-5.6 Luna, GPT-5.4 Mini.
Uses a miscellaneous/other plug in the bridge or comments on every line.
GPT-5.6 Terra, try 2, flagged by both judges: Vacancy & credit loss | $(856) | Q1 actual; no permanent change is documented for the remaining forecast period.
GPT-5.6 Luna, try 2, flagged by both judges: Management fee effect | 325 | Mechanical / Q1 actual | Lower Q1 EGI reduced the 3% management fee versus budget.
GPT-5.4 Mini, try 2, flagged by both judges: Management Fee (3% of EGI) | 16,781 | 16,456 | -325 | Lower EGI
Missed by GPT-5.6 Terra, GPT-5.6 Luna, GPT-5.4 Mini. The judges split on 5 further attempts; they are not attributed to a model.
The county assessed at $25.2m, above the $24.0m price; taxes are permanently higher than budget and the appeal cannot be assumed to succeed.
Illustrative wording, credited by both judges (GPT-5.6 Sol, try 2): “Taxes were accrued from January at the $25.2 million reassessed value rather than the $24.0 million purchase price. This is permanent; no appeal benefit is assumed.” / “The reforecast uses annual taxes of $282,240”
Missed by GPT-5.4.
The judges split on 5 mistake flags in this task (one judge flagged it, the other did not). Split flags are not attributed to a model here, but under the full-pass rule a flag from either judge still fails that attempt.
These are benchmark checks, not severity ratings. Financial impact has not been measured. Missed findings use the primary judge. A mistake is attributed to a model only when both judges flagged it. Full-pass results use both judges where a second judgment exists.
Bridge actuals to budget, reforecast NOI and draft the LP commentary. 4 supplied text documents, 23 checks, 3 planned attempts per model. Small sample; every output needs review.
Numbers correct is the share of numerical checks within the answer key’s tolerance. Risk & decision checks covers findings, mistakes avoided and the requested recommendation. Both are individual-check pass rates, not the probability that an entire output is correct.
Passed every check is the stricter measure: every required check must pass in that attempt. One missed observation fails it, just as a wrong number does. A 0/3 result can still contain correct calculations.
The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count; ties are listed alphabetically and are not broken by cost or by check rates. Cost and speed cards require all planned attempts to pass every check. The table’s default order (full passes, then numbers, then risk and decision checks, then cost) is for display only, and it sorts all models directly by cost or wait, including failed results. Incomplete samples appear last in every view and are excluded from the cards when complete samples exist.
Chart labels are rounded. Missing results are not plotted; asterisks mark incomplete samples. In the combined table, teal marks lower API cost and faster median response within each column; accuracy columns are not shaded. It is not an overall score. Highlighted marks identify the models named on the selected card.
API charges per response at the list prices in the method’s price table, excluding subscriptions, tools, retries and human review. Times reflect this run; neither price nor speed is guaranteed in a chat app.
Highest full-pass rate
0/3 for every model with all attempts graded · 95% CI 0–71%
Lowest-cost passing
Requires every planned attempt to pass.
Fastest passing
Requires every planned attempt to pass.
A full pass means every check passed in that attempt; passing cards require all 3 attempts to pass. Intervals are exact 95% binomial (Clopper-Pearson) intervals. At 3 attempts per model, no model is statistically separable from another on this task: every pair of 95% intervals overlaps.
Sensitivity: one check was missed in at least 85% of attempts: “The T-12 includes $62,000 of insurance proceeds and six months of a departed tenant's rent; both come out of normalized NOI” (missed in 95% of attempts). Excluding it, full passes would be GPT-5.6 Sol 0/3, GPT-6 Astra 2/3, GPT-5.6 Terra 1/3, GPT-5.6 Luna 2/3, GPT-5.4 1/3, GPT-5.5 2/3, GPT-5.4 Mini 0/3. The answer key still requires it; this is not a rescore.
3 attempts per model · same assignment
| GPT-5.6 Luna | 100.0% | 93.3% | 0 / 30–71% | 2m 15s | $0.017 |
|---|---|---|---|---|---|
| GPT-5.6 Terra | 100.0% | 93.3% | 0 / 30–71% | 1m 57s | $0.13 |
| GPT-6 Astra | 100.0% | 93.3% | 0 / 30–71% | 4m 7s | $0.61 |
| GPT-5.6 Sol | 100.0% | 91.1% | 0 / 30–71% | 2m 43s | $0.12 |
| GPT-5.4 | 100.0% | 91.1% | 0 / 30–71% | 3m 20s | $0.30 |
| GPT-5.5 | 100.0% | 91.1% | 0 / 30–71% | 5m 25s | $0.48 |
| GPT-5.4 Mini | 100.0% | 84.4% | 0 / 30–71% | 2m 12s | $0.099 |
Select a column to sort. Incomplete samples stay last. Accuracy columns are not shaded: at 3 attempts per model, differences inside the intervals are not meaningful. Cost uses the list prices in the price table; the price behind GPT-5.6 Sol’s cost is unconfirmed and may be too low. Swipe to see all columns →
Observed misses across all models, most frequent first.
The T-12 includes $62,000 of insurance proceeds and six months of a departed tenant's rent; both come out of normalized NOI.
Missed by every model at least once.
Suite D lease-up and the $45/SF asking rent are upside for a buyer to underwrite, not in-place income.
Illustrative wording, credited by both judges (GPT-6 Astra, try 3): "Suite D: vacant space, not a $45/SF lease. There is no executed LOI."
Missed by GPT-5.6 Sol, GPT-5.4, GPT-5.5, GPT-5.4 Mini.
Lotus Nail Spa is three months in arrears and Unit 211 is two months late; credit issues to resolve or disclose before marketing.
Illustrative wording, credited by both judges (GPT-5.6 Sol, try 3): Unit 211 owes $6,300, representing two unpaid months
Missed by GPT-5.4 Mini.
$30m implies a cap rate below the comp range even before buyers adjust for taxes, Suite C and E; the honest answer is that $30m is not achievable in this market on this file.
Illustrative wording, credited by both judges (GPT-5.4, try 2): As-is this fall, 30m is not supported by the current package: it is a 5.43% cap on normalized T-12 NOI
Missed by GPT-5.4 Mini.
The judges split on 1 mistake flag in this task (one judge flagged it, the other did not). Split flags are not attributed to a model here, but under the full-pass rule a flag from either judge still fails that attempt.
These are benchmark checks, not severity ratings. Financial impact has not been measured. Missed findings use the primary judge. A mistake is attributed to a model only when both judges flagged it. Full-pass results use both judges where a second judgment exists.
Normalize income, assess valuation and separate in-place value from potential upside. 8 supplied text documents, 20 checks, 3 planned attempts per model. Small sample; every output needs review.
Numbers correct is the share of numerical checks within the answer key’s tolerance. Risk & decision checks covers findings, mistakes avoided and the requested recommendation. Both are individual-check pass rates, not the probability that an entire output is correct.
Passed every check is the stricter measure: every required check must pass in that attempt. One missed observation fails it, just as a wrong number does. A 0/3 result can still contain correct calculations.
The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count; ties are listed alphabetically and are not broken by cost or by check rates. Cost and speed cards require all planned attempts to pass every check. The table’s default order (full passes, then numbers, then risk and decision checks, then cost) is for display only, and it sorts all models directly by cost or wait, including failed results. Incomplete samples appear last in every view and are excluded from the cards when complete samples exist.
Chart labels are rounded. Missing results are not plotted; asterisks mark incomplete samples. In the combined table, teal marks lower API cost and faster median response within each column; accuracy columns are not shaded. It is not an overall score. Highlighted marks identify the models named on the selected card.
API charges per response at the list prices in the method’s price table, excluding subscriptions, tools, retries and human review. Times reflect this run; neither price nor speed is guaranteed in a chat app.
Exhibit H
Text presentation, out of 100. Correctness and visual design are tested separately. Scored by one AI judge, gpt-5.5-2026-04-23 (the GPT-5.5 contestant), so GPT-5.5 is graded by its own snapshot. No second judge; the 1.8-point spread is not a meaningful ranking.
| Model | Writing scoreout of 100 | Structureout of 5 | Clarityout of 5 | Numbers shownout of 5 | Decision readyout of 5 | Concisionout of 5 |
|---|---|---|---|---|---|---|
| GPT-5.527 memos scored | 92.1 | 5.00 | 4.89 | 5.00 | 4.00 | 4.15 |
| GPT-5.4 Mini25 memos scored | 91.8 | 4.96 | 4.72 | 4.68 | 3.96 | 4.64 |
| GPT-5.6 Terra27 memos scored | 91.1 | 5.00 | 4.74 | 5.00 | 3.96 | 4.07 |
| GPT-5.6 Sol27 memos scored | 90.8 | 5.00 | 4.81 | 5.00 | 3.96 | 3.93 |
| GPT-6 Astra27 memos scored | 90.8 | 5.00 | 4.48 | 5.00 | 4.19 | 4.04 |
| GPT-5.6 Luna27 memos scored | 90.7 | 5.00 | 4.70 | 4.96 | 4.04 | 3.96 |
| GPT-5.426 memos scored | 90.3 | 5.00 | 4.73 | 5.00 | 3.88 | 3.96 |
Scroll this table sideways to see every column.
7 different scores across seven models, spread 0.71 · zoomed axis, 3.7 to 4.8 of a possible 5
7 different scores across seven models, spread 0.41 · zoomed axis, 4.4 to 5.0 of a possible 5
GPT-5.6 SolStrong reporting package with excellent structure and numerical transparency, but add a crisp action-oriented close if this is meant to be decision-ready.
GPT-6 AstraStrong reporting package with excellent structure and numbers, but add a crisp closing action/recommendation so the principal knows exactly what to do with it.
GPT-5.6 TerraStrong package structure and numerical presentation, but add a concise end-state recommendation or action conclusion and trim minor over-detail.
GPT-5.6 LunaStrong package structure and numerical presentation, but it needs a clear closing recommendation or action point if judged against decision-ready memo standards.
GPT-5.4Strong reporting package with excellent structure and numerical presentation, but it needs a clear closing recommendation or action call to be decision-ready.
GPT-5.5Strong reporting package with excellent structure and numerical support, but add a crisp concluding action/recommendation and trim internal process commentary before presenting.
GPT-5.4 MiniStrong reporting package presentation, but add an explicit closing recommendation or action statement if this is meant to be decision-ready.
Rendered output tests · 2026-09-13
Writing, layout and delivery across three assignments.
| Model | Writingout of 5 | Visual designout of 5 | Facts keptboth judges | File checksout of 3 | Speedmedian | CostAPI / attempt |
|---|---|---|---|---|---|---|
| GPT-5.6 Terra | 4.94.8–5.0 | 4.54.2–4.8 | 100% | 2 / 3 | 35s | $0.058 |
| GPT-5.6 Sol | 4.84.8–4.9 | 4.94.9–4.9 | 100% | 2 / 3 | 1m 56s | $0.082 |
| GPT-5.6 Luna | 4.74.6–5.0 | 4.34.2–4.4 | 100% | 0 / 3 | 51s | $0.010 |
| GPT-5.5 | 4.84.5–5.0 | 4.44.0–4.7 | 93% | 0 / 3 | 55s | $0.19 |
| GPT-5.4 | 4.84.8–4.9 | 4.24.1–4.3 | 100% | 0 / 3 | 1m 7s | $0.15 |
| GPT-5.4 Mini1/3 complete documents | 4.84.8–4.8 | 4.64.6–4.6 | 100% | 0 / 3 | 1m 37s | $0.085 |
| GPT-6 Astra | 4.94.9–5.0 | 5.04.9–5.0 | 100% | 3 / 3 | 1m 49s | $0.32 |
Writing and design show the mean and range across attempts. File checks cover page count, fit, minimum text size and word limit. Visual design covers hierarchy, typography, charts and layout. Speed and cost include unsuccessful attempts.
| Model | Writingout of 5 | Visual designout of 5 | Facts keptboth judges | File checksout of 3 | Speedmedian | CostAPI / attempt |
|---|---|---|---|---|---|---|
| GPT-5.6 Terra | 4.84.5–4.9 | 4.64.2–4.8 | 100% | 2 / 3 | 55s | $0.099 |
| GPT-5.6 Sol | 4.84.7–4.9 | 4.64.4–4.9 | 100% | 2 / 3 | 1m 48s | $0.087 |
| GPT-5.6 Luna | 4.64.3–4.8 | 4.13.6–4.5 | 100% | 1 / 3 | 1m 10s | $0.012 |
| GPT-5.5 | 4.74.7–4.8 | 4.54.3–4.8 | 100% | 0 / 3 | 1m 23s | $0.31 |
| GPT-5.4 | 4.94.9–5.0 | 4.84.6–5.0 | 100% | 2 / 3 | 1m 36s | $0.24 |
| GPT-5.4 Mini0/3 complete documents | Not scored | Not scored | Not scored | 0 / 3 | 2m 15s | $0.090 |
| GPT-6 Astra | 4.94.8–4.9 | 5.04.9–5.0 | 100% | 3 / 3 | 3m 25s | $0.53 |
Writing and design show the mean and range across attempts. File checks cover page count, fit, minimum text size and word limit. Visual design covers hierarchy, typography, charts and layout. Speed and cost include unsuccessful attempts.
No rendered example available.
| Model | Writingout of 5 | Visual designout of 5 | Facts keptboth judges | File checksout of 3 | Speedmedian | CostAPI / attempt |
|---|---|---|---|---|---|---|
| GPT-5.6 Terra | 4.84.6–5.0 | 4.54.0–4.9 | 100% | 1 / 3 | 43s | $0.077 |
| GPT-5.6 Sol | 4.74.7–4.8 | 4.84.6–5.0 | 100% | 2 / 3 | 2m | $0.096 |
| GPT-5.6 Luna | 4.84.7–4.8 | 4.74.4–4.9 | 100% | 2 / 3 | 1m 7s | $0.011 |
| GPT-5.5 | 4.74.7–4.8 | 4.23.9–4.5 | 100% | 0 / 3 | 1m 28s | $0.31 |
| GPT-5.40/3 complete documents | Not scored | Not scored | Not scored | 0 / 3 | 2m 25s | $0.30 |
| GPT-5.4 Mini0/3 complete documents | Not scored | Not scored | Not scored | 0 / 3 | 2m 24s | $0.091 |
| GPT-6 Astra | 4.84.8–4.8 | 4.94.9–5.0 | 100% | 3 / 3 | 2m 49s | $0.50 |
Writing and design show the mean and range across attempts. File checks cover page count, fit, minimum text size and word limit. Visual design covers hierarchy, typography, charts and layout. Speed and cost include unsuccessful attempts.
No rendered example available.
No rendered example available.
Each model received the same new synthetic brief with verified facts and the same light PSV design requirements. The assignment requested a complete HTML document at high reasoning effort with a 20,000 output-token allowance, including reasoning. Exhausting that allowance is a failed attempt; a larger allowance may change the result. Pages were rendered at fixed document or slide dimensions with scripts and network access disabled. No manual repairs were made.
Two model-anonymous vision reviewers (gpt-5.4 and gpt-5.5) scored the actual page images and extracted text on five writing and five design criteria. Scores run from 1 (unusable) to 5 (ready for the stated audience). Facts kept requires both judges to credit a check; clipped facts do not earn credit. File checks are automated requirements; visual design is judged from the rendered pages. Judge fact agreement: 99.6% across 260 comparisons. Both reviewers are also contestants in this test (GPT-5.4 and GPT-5.5), so each grades its own output alongside the others.
These are small-sample, same-vendor AI reviews; differences are provisional and have not been validated by CRE professionals. Scores measure communication of supplied facts, not independent underwriting, PDF extraction, Excel formulas or PowerPoint editing. Example images always use attempt 1.
Speed measures API attempt time, including unsuccessful attempts and excluding rendering and judging. Costs use the list prices in the underwriting method’s price table, exclude review costs in the table, and are estimates rather than a billing statement. Recorded generation: $10.93; grading: $4.71.
Can the model carry the right numbers from source files to the final recommendation?
| Model | Harbor CourtProceed → Decline | Cedar CommerceDecline → Proceed | Cypress OfficeExact threshold | Typical workflowtwo API calls | Cost / workflowtwo API calls |
|---|---|---|---|---|---|
| GPT-5.6 Luna | Passed → Passed | Passed → Passed | Passed → Passed | 11s | $0.003 |
| GPT-6 Astra | Passed → Incomplete | Passed → Passed | Passed → Passed | 18s | Unknown |
Initial → revision. Passed = every automated check. $0.25 recorded spend; 1 incomplete call with unknown cost; $2.00 batch cap.
All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗API did not complete; no correctness result.
| Model | Mesa CourtMultifamily | Portside FlexIndustrial | Juniper PlazaOffice | Typical workflowtwo API calls | Cost / workflowtwo API calls |
|---|---|---|---|---|---|
| GPT-5.6 Luna | Missed → Passed | Passed → Passed | Passed → Missed | 19s | $0.003 |
| GPT-6 Astra | Passed → Passed | Passed → Passed | Passed → Passed | 20s | $0.096 |
Initial → revision. Passed = every automated check. $0.30 recorded spend; $5.00 batch cap.
All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗Missed: source extraction, spreadsheet calculations, slide figures, memo figures, slide text
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗Missed: spreadsheet calculations, linked assumptions
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗All automated checks passed.
Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗Models receive one scanned synthetic brief as an image and raw cells extracted from an XLSX workbook. Harder cases add signed amendments that override stale rent rolls, misleading broker summaries, missing lease expiries, both directions of recommendation changes, and an exact valuation threshold. These are prescribed screening rules, not complete investment decisions.
Each model gets one initial call and one revision per deal, medium effort, a 3,500-token ceiling and no automatic retry. Failed API calls remain incomplete; their unknown charges retain the full budget reservation. Typical workflow time uses only pairs with both calls timed. Cost per workflow is unavailable when a charge is unknown.
Shared tools create editable XLSX and PPTX files from model-supplied inputs, formulas and text. Checks cover calculations, source precedence, missing facts, recommendations, memo/slide figures, linked formulas and export/reimport. The same layout serves every model; visual design is not scored here. Prose checks verify required figures, not every factual claim.
These small samples cannot establish a general winner. Full native file ingestion/editing, long or damaged scans, DOCX and desktop Office behavior remain untested. Original and harder batches stay separate. Anthropic results appear only after actual calls with configured credentials.
Prices: Luna, Astra; see also the price table. Batch prices checked September 18, 2026; uncached token estimates, not billing statements. No paid judges or tools; local file-tool time and machine cost are excluded.
Finished is not the same as correct. Use task accuracy to judge the result.
| Model | FinishedAPI outputs | Typical waitmedian | Slow wait90th percentile | API spendall attempts | Cost / finishedincludes failed attempts | Retries |
|---|---|---|---|---|---|---|
| GPT-6 Astra | 27 / 27 | 3m 22s | 4m 21s | $14.42 | $0.53 | Not recorded |
| GPT-5.6 Terra | 27 / 27 | 2m 42s | 3m 56s | $4.18 | $0.15 | Not recorded |
| GPT-5.6 Sol | 27 / 27 | 2m 55s | 3m 41s | $3.59 | $0.13 | Not recorded |
| GPT-5.6 Luna | 27 / 27 | 3m 11s | 3m 42s | $0.59 | $0.022 | Not recorded |
| GPT-5.5 | 27 / 27 | 4m 24s | 7m 17s | $14.02 | $0.52 | Not recorded |
| GPT-5.4 | 27 / 27 | 3m 20s | 4m 35s | $9.15 | $0.34 | Not recorded |
| GPT-5.4 Mini | 26 / 27 | 3m 53s | 6m 28s | $5.11 | $0.20 | Not recorded |
| Model | FinishedAPI outputs | Typical waitmedian | Slow wait90th percentile | API spendall attempts | Cost / finishedincludes failed attempts | Retries |
|---|---|---|---|---|---|---|
| GPT-6 Astra | 9 / 9 | 2m 49s | 3m 34s | $4.05 | $0.45 | Not recorded |
| GPT-5.6 Terra | 9 / 9 | 45s | 1m 3s | $0.70 | $0.078 | Not recorded |
| GPT-5.6 Sol | 9 / 9 | 1m 56s | 2m 17s | $0.79 | $0.088 | Not recorded |
| GPT-5.6 Luna | 9 / 9 | 1m 9s | 1m 15s | $0.10 | $0.011 | Not recorded |
| GPT-5.5 | 9 / 9 | 1m 23s | 1m 45s | $2.41 | $0.27 | Not recorded |
| GPT-5.4 | 6 / 9 | 1m 36s | 2m 29s | $2.08 | $0.35 | Not recorded |
| GPT-5.4 Mini | 1 / 9 | 2m 15s | 2m 30s | $0.80 | $0.80 | Not recorded |
Finished means the API returned a completed, nonempty response. It does not require correct facts, valid formatting or every benchmark check to pass. Cost per finished output divides recorded generation spend, including unsuccessful attempts, by completed outputs. It is not cost per correct answer.
Waits cover API generation, including retries where the original timer captured them. Nearest-rank percentiles describe these small samples, not a service guarantee. Historical retry counts and uncaptured retry charges are unknown. Costs use each run’s saved token prices (listed with sources in the price table), with no subscription, human-effort or local-compute allocation. Paid evaluator costs are reported separately in each suite’s methodology. No external paid tools were used.
Method
Each model receives the same plain-text documents and firm assumptions for each task, across nine assignments on two synthetic deals. The documents and answer keys are generated from the same underlying data model. Each response contains a written analysis, structured figures and a recommendation. No tools, retrieval, code execution, spreadsheet creation or raw-PDF reading are tested. The configured reasoning effort was high, with a 64,000-token output cap that includes reasoning.
Every memo is scored against a predefined answer key: 225 checks in total, covering figures inside a stated tolerance, findings a good analyst makes, mistakes a good analyst avoids, and one decision. Findings and mistakes are read by two OpenAI judge models, which are also contestants: gpt-5.5-2026-04-23 (the GPT-5.5 contestant) and gpt-5.4-2026-03-05 (the GPT-5.4 contestant). They are not independent of the models they grade, and no judge from another vendor has been used yet. They agreed on 97.5% of 2,775 judgements (Cohen's kappa 0.81 on findings and 0.63 on mistakes: substantial agreement). For the clean-memo score, a finding must be credited by both judges and a mistake must be flagged by neither, where a second judgment exists. Second-judge comparisons cover 170 of 185 graded responses. The rest are 14 lease-up reforecast & take-out memos (GPT-5.6 Sol 2, GPT-5.6 Luna 3, GPT-5.4 3, GPT-5.5 3, GPT-5.4 Mini 3) and 1 empty response, which have no second judgment. Without a second verdict the first judge decides; failed responses still fail. The Numbers correct and Risk & decision checks columns report individual-check pass rates; writing quality is scored separately.
The plan calls for three attempts per model and task. Repeats measure variation on the same inputs, not performance on different deals. That is 7 models × 9 tasks × 3 tries = 189 attempted memos, of which 185 were graded. 4 memos were returned and billed but never scored because grading failed (judge API quota exhausted): GPT-5.6 Sol on the lease-up reforecast & take-out (1 try), GPT-5.6 Terra on the lease-up reforecast & take-out (3 tries). They are excluded from the performance figures; the task comparison shows each incomplete denominator. An attempt without a usable memo scores zero on every check rather than counting as missing: 1 response was empty (output token limit); 2 memos omitted the required JSON block, out of 185 graded attempts.
Deal set v1 is live from September 2026. A fresh deal set and answer key are planned for December 2026. A changed test set is not a like-for-like comparison. The packages are synthetic; the page publishes selected answer-key excerpts and model responses. This does not establish that the tasks are immune to training-data overlap.
What this bench does not tell you
| Model | API model id | Input$ / 1M tokens | Output$ / 1M, incl. reasoning | Source | Price checked |
|---|---|---|---|---|---|
| GPT-5.6 SolUnconfirmed. On 2026-09-26 the source listed $4 input / $20 output per 1M tokens (promotional pricing), not the $2 / $10 recorded here and used for these runs' cost figures; Sol's costs may be understated by about half. | gpt-5.6-sol | $2.00 | $10.00 | Model page ↗ | Not confirmed |
| GPT-6 Astra | gpt-6-astra | $10.00 | $50.00 | Model page ↗ | September 13, 2026 |
| GPT-5.6 Terra | gpt-5.6-terra | $2.00 | $12.00 | Model page ↗ | September 26, 2026 |
| GPT-5.6 Luna | gpt-5.6-luna | $0.20 | $1.20 | Model page ↗ | September 26, 2026 |
| GPT-5.4 | gpt-5.4 | $2.50 | $15.00 | Model page ↗ | September 26, 2026 |
| GPT-5.5 | gpt-5.5 | $5.00 | $30.00 | Model page ↗ | September 26, 2026 |
| GPT-5.4 Mini | gpt-5.4-mini | $0.75 | $4.50 | Model page ↗ | September 26, 2026 |
Scroll this table sideways to see every column.
Standard, uncached list prices applied to the token counts each API call reported; output includes reasoning tokens. Costs are estimates, not billing statements, and exclude subscriptions, tools, retries and grading. The same prices drive the document-design and reliability cost columns. GPT-5.6 Sol: price used for these runs not confirmed; see the row note.
| Model | API model idas returned | Snapshot | Run dates | Reasoning effort | Output captokens, incl. reasoning | Route |
|---|---|---|---|---|---|---|
| GPT-5.6 Sol | gpt-5.6-sol | Alias; no dated snapshot exposed | September 3 to 4, 2026 (UTC) | high | 64,000 | OpenAI Responses API |
| GPT-6 Astra | gpt-6-astra | Alias; no dated snapshot exposed | September 13, 2026 (UTC) | high | 64,000 | OpenAI Responses API |
| GPT-5.6 Terra | gpt-5.6-terra | Alias; no dated snapshot exposed | September 3 to 4, 2026 (UTC) | high | 64,000 | OpenAI Responses API |
| GPT-5.6 Luna | gpt-5.6-luna | Alias; no dated snapshot exposed | September 3 to 4, 2026 (UTC) | high | 64,000 | OpenAI Responses API |
| GPT-5.4 | gpt-5.4-2026-03-05 | March 5, 2026 | September 3 to 4, 2026 (UTC) | high | 64,000 | OpenAI Responses API |
| GPT-5.5 | gpt-5.5-2026-04-23 | April 23, 2026 | September 3 to 4, 2026 (UTC) | high | 64,000 | OpenAI Responses API |
| GPT-5.4 Mini | gpt-5.4-mini-2026-03-17 | March 17, 2026 | September 3 to 4, 2026 (UTC) | high | 64,000 | OpenAI Responses API |
Scroll this table sideways to see every column.
The API model id is the one each response reported. An alias without a dated snapshot can change behind the same name, so a later run of it may not reproduce these results. Every attempt used the same system prompt, documents and task wording. The output cap was reached once: GPT-5.4 Mini, feasibility memo, try 1.
OpenAI, GPT and the model names shown are trademarks of OpenAI; Claude is a trademark of Anthropic. PSV is not affiliated with or endorsed by either and uses these names only to identify the products tested. PSV builds client systems on Anthropic's Claude.
Every required figure, finding, mistake-avoidance check and decision passed for that attempt, and the response was usable. One missed check fails the attempt. This does not measure the financial impact of the error or remove the need for human review.
This test does not establish a production-readiness threshold. Use the task results to choose a candidate, inspect its misses, and validate it against your own documents and review standards.
Numerical accuracy counts figures inside the answer key’s tolerance. A full pass also requires the findings, mistake-avoidance checks and decision to pass. A response can get every tested number right and still miss an important observation.
The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count on that task. Ties are listed together and are never broken by cost or by check rates, and every count carries an exact 95% interval. At three attempts per model, no difference between models on a single task is statistically reliable. Incomplete samples appear last and remain available for inspection.
No. It is the lowest measured API charge per response among complete samples. The comparison still shows the model’s accuracy and full-pass results. Human correction time, tool costs and retry costs have not been measured here.
The deals are synthetic, generated from a shared data model. Selected answer-key excerpts and model responses are published here. We have not established that these tasks are immune to training-data overlap.
| Model | Soft score | 95% CI | Both judges | Hard checks | 95% CI | Clean, pooled | 95% CI | Clean, one judge | ≤1 miss | Misses / memo | Worst memo |
|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5.6 Sol | 96.2 | 94.0–98.4 | 95.2 | 83.4 | 73.1–93.3 | 42.3 | 14.8–70.4 | 46.2 | 61.5 | 1.15 | 90 |
| GPT-6 Astra | 95.3 | 92.4–97.9 | 94.4 | 79.6 | 65.9–91.9 | 40.7 | 11.1–70.4 | 40.7 | 59.3 | 1.41 | 86 |
| GPT-5.6 Terra | 87.2 | 71.9–96.2 | 86.1 | 72.6 | 57.2–86.0 | 33.3 | 12.5–58.3 | 37.5 | 54.2 | 3.54 | 0 |
| GPT-5.6 Luna | 95.3 | 92.8–97.5 | 94.4 | 79.9 | 68.2–90.0 | 33.3 | 7.4–63.0 | 33.3 | 70.4 | 1.37 | 83 |
| GPT-5.4 | 95.8 | 93.0–98.0 | 94.1 | 80.3 | 66.2–91.5 | 25.9 | 3.7–51.9 | 37.0 | 74.1 | 1.44 | 81 |
| GPT-5.5 | 95.7 | 93.8–97.4 | 94.9 | 80.5 | 71.2–88.8 | 25.9 | 3.7–51.9 | 25.9 | 66.7 | 1.26 | 90 |
| GPT-5.4 Mini | 87.5 | 76.9–93.2 | 85.5 | 59.1 | 47.4–69.9 | 3.7 | 0.0–14.8 | 3.7 | 33.3 | 3.56 | 0 |
Scroll this table sideways to see every column.
Clean, pooled combines graded attempts across all assignments. The task guide above uses each assignment’s own results and denominator. These pooled figures describe this test set; they are not a recommendation for every CRE workflow. Intervals are two-stage bootstrap intervals: resample the nine tasks, then attempts within each drawn task. Attempts on one task are correlated, so this is wider than resampling attempts alone.
| Task | Checks | Passed by everyone | Mean pass rate |
|---|---|---|---|
| Variance report & reforecastHarbor & Fifth: Q1 variance report, reforecast and LP commentary | 23 | 18 | 89.9% |
| Broker opinion of valueHarbor & Fifth: broker opinion of value for the owner | 20 | 15 | 93.3% |
| IC screening memoHarbor & Fifth: full screen and preliminary underwrite from the data room | 35 | 30 | 98.6% |
| OM readHarbor & Fifth: read the OM like a senior person | 18 | 14 | 98.6% |
| Rent roll analysisHarbor & Fifth: parse and analyze the rent roll | 27 | 22 | 92.1% |
| Underwriting assumptionsHarbor & Fifth: build the Year-1 assumption set from the rent roll and T-12 | 29 | 23 | 95.4% |
| Feasibility memoPalomar Yards: JV equity feasibility on the firm basis | 26 | 20 | 97.5% |
| Construction draw reviewPalomar Yards: review construction draw 14 and run the in-balance test | 24 | 15 | 92.5% |
| Lease-up reforecast & take-outPalomar Yards: six months into lease-up: reforecast stabilization and size the take-out | 23 | 17 | 96.4% |
Scroll this table sideways to see every column.
These 4 memos completed and were billed, and are inside the spend figure, but grading failed, so they do not contribute to the performance figures. Their missing coverage is shown in the task comparison.
gpt-5.5-2026-04-23 (the GPT-5.5 contestant) and gpt-5.4-2026-03-05 (the GPT-5.4 contestant) scored 2,775 judgement items across 170 memos. Both kinds of item are compared on one pass/fail scale (finding made, mistake avoided). They agreed on 97.5% of items; pooled Cohen's kappa 0.806. Findings: 96.4% agreement over 1,758 items, kappa 0.809 (substantial). Mistakes: 99.3% over 1,017, kappa 0.628 (substantial); mistakes are rare, so a few disagreements move kappa a long way. gpt-5.5-2026-04-23 passed an item the other failed 53 times; gpt-5.4-2026-03-05 did so 17 times.