Skip to main content

CRE model benchmarks

Compare accuracy, speed and cost for your next task.

7 OpenAI models · tested September 3 to 18, 2026 · Anthropic and other vendors not yet tested

7 models9 underwriting tasks3 writing & design tasks ↗

Choose your assignment

Compare by

Screen a deal for IC

1 deal · 3 attempts / model

Highest full-pass rate

Tied: GPT-5.6 Luna, GPT-5.6 Sol and GPT-6 Astra

3/3 full passes each · 95% CI 29–100%. Listed alphabetically; no tie-break is applied.

Lowest-cost passing

GPT-5.6 Luna

$0.0273m 27s

3/3 full passes · 95% CI 29–100%

Fastest passing

GPT-5.6 Luna

$0.0273m 27s

3/3 full passes · 95% CI 29–100%

A full pass means every check passed in that attempt; passing cards require all 3 attempts to pass. Intervals are exact 95% binomial (Clopper-Pearson) intervals. At 3 attempts per model, no model is statistically separable from another on this task: every pair of 95% intervals overlaps.

Accuracy

Higher is better
NumbersRisk & decisions

Speed

Shorter is faster
Median response time
0m4m8m GPT-5.6 Luna: 3m 27s5.6 Luna3m 27sGPT-5.6 Sol: 3m 38s5.6 Sol3m 38sGPT-6 Astra: 3m 50s6 Astra3m 50sGPT-5.4: 3m 51s5.43m 51sGPT-5.6 Terra: 4m 26s5.6 Terra4m 26sGPT-5.5: 5m 56s5.55m 56sGPT-5.4 Mini: 6m 28s5.4 Mini6m 28s

Cost

Lower is cheaper
API dollars / task at list prices
$0$0.4$0.8 GPT-5.6 Luna · $0.027$0.0275.6LunaGPT-5.6 Sol · $0.18$0.185.6SolGPT-6 Astra · $0.73$0.736AstraGPT-5.4 · $0.40$0.405.4GPT-5.6 Terra · $0.23$0.235.6TerraGPT-5.5 · $0.63$0.635.5GPT-5.4 Mini · $0.26$0.265.4Mini

3 attempts per model · same assignment

All metrics, side by side

Teal = lower cost or faster time
Accuracy, speed and cost for IC screening memo
GPT-5.6 Luna100.0%100.0%3 / 329–100%3m 27s$0.027
GPT-5.6 Sol100.0%100.0%3 / 329–100%3m 38s$0.18
GPT-6 Astra100.0%100.0%3 / 329–100%3m 50s$0.73
GPT-5.4100.0%98.8%2 / 39–99%3m 51s$0.40
GPT-5.6 Terra1 format failure (no JSON block), scored as a miss on every check66.7%66.7%2 / 39–99%4m 26s$0.23
GPT-5.5100.0%97.6%1 / 31–91%5m 56s$0.63
GPT-5.4 Mini88.9%95.2%0 / 30–71%6m 28s$0.26

Select a column to sort. Incomplete samples stay last. Accuracy columns are not shaded: at 3 attempts per model, differences inside the intervals are not meaningful. Cost uses the list prices in the price table; the price behind GPT-5.6 Sol’s cost is unconfirmed and may be too low. Swipe to see all columns →

What needs a review 5 checks

Observed misses across all models, most frequent first. An attempt without a usable memo is counted once, under the usable-memo check, not as a miss on every other check.

The $28,500 legal settlement is non-recurring and should be removed from expenses (a favorable adjustment).20% missed
Answer key

The $28,500 legal settlement is non-recurring and should be removed from expenses (a favorable adjustment).

Illustrative wording, credited by both judges (GPT-5.4, try 3): Broker already removed the one-time Marigold settlement; firm also excludes it.

Missed by GPT-5.4, GPT-5.5, GPT-5.4 Mini.

The anchor tenant's share of retail rent10% missed
Answer key

Anchor share of in-place retail base rent: 42.80, accepted within ±2.50 points.

ModelAnswer keyTry 1Try 2Try 3
GPT-5.6 Sol42.8042.7842.7842.78
GPT-6 Astra42.8042.7842.7842.78
GPT-5.6 Terra42.8042.7842.78no JSON block
GPT-5.6 Luna42.8042.7842.7842.78
GPT-5.442.8042.7842.7842.78
GPT-5.542.8042.7842.7842.78
GPT-5.4 Mini42.8042.8065.2265.22
Unit 307 has been offline since March 2026 due to water intrusion from a failed building envelope; the PCA calls for $186,000 of immediate repairs and the envelope work was excluded from insurance.10% missed
Answer key

Unit 307 has been offline since March 2026 due to water intrusion from a failed building envelope; the PCA calls for $186,000 of immediate repairs and the envelope work was excluded from insurance.

Illustrative wording, credited by both judges (GPT-5.4 Mini, try 1): Immediate PCA repairs: 186,000. - Envelope / flashing / water intrusion: 95,000 - Unit 307 restoration: 41,300

Missed by GPT-5.4 Mini.

These are benchmark checks, not severity ratings. Financial impact has not been measured. Missed findings use the primary judge. A mistake is attributed to a model only when both judges flagged it. Full-pass results use both judges where a second judgment exists.

Scoring & test setup

Reconcile the data room, normalize NOI and write a preliminary investment recommendation. 10 supplied text documents, 35 checks, 3 planned attempts per model. Small sample; every output needs review.

Numbers correct is the share of numerical checks within the answer key’s tolerance. Risk & decision checks covers findings, mistakes avoided and the requested recommendation. Both are individual-check pass rates, not the probability that an entire output is correct.

The check on the price at which the deal works accepts “no clearing price” as a correct answer; 20 of 20 usable attempts gave that answer, so the check never tested a stated figure.

Passed every check is the stricter measure: every required check must pass in that attempt. One missed observation fails it, just as a wrong number does. A 0/3 result can still contain correct calculations.

The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count; ties are listed alphabetically and are not broken by cost or by check rates. Cost and speed cards require all planned attempts to pass every check. The table’s default order (full passes, then numbers, then risk and decision checks, then cost) is for display only, and it sorts all models directly by cost or wait, including failed results. Incomplete samples appear last in every view and are excluded from the cards when complete samples exist.

Chart labels are rounded. Missing results are not plotted; asterisks mark incomplete samples. In the combined table, teal marks lower API cost and faster median response within each column; accuracy columns are not shaded. It is not an overall score. Highlighted marks identify the models named on the selected card.

API charges per response at the list prices in the method’s price table, excluding subscriptions, tools, retries and human review. Times reflect this run; neither price nor speed is guaranteed in a chat app.

Review an OM

1 deal · 3 attempts / model

Highest full-pass rate

Tied: GPT-5.5, GPT-5.6 Luna, GPT-5.6 Sol and GPT-6 Astra

3/3 full passes each · 95% CI 29–100%. Listed alphabetically; no tie-break is applied.

Lowest-cost passing

GPT-5.6 Luna

$0.0233m 13s

3/3 full passes · 95% CI 29–100%

Fastest passing

GPT-6 Astra

$0.392m 34s

3/3 full passes · 95% CI 29–100%

A full pass means every check passed in that attempt; passing cards require all 3 attempts to pass. Intervals are exact 95% binomial (Clopper-Pearson) intervals. At 3 attempts per model, no model is statistically separable from another on this task: every pair of 95% intervals overlaps.

Numbers: n/a. The only scored numerical check (the price at which the deal works) accepts “no clearing price” as a correct answer, and every answered attempt gave no price, so no stated figure was checked on this task.

Scoring v2 (September 14, 2026) excludes the Year-1 NOI and going-in cap rate at the asking price checks for every model. This exclusion was decided after results were observed. It changed full passes for GPT-5.6 Sol (2 to 3), GPT-6 Astra (0 to 3), GPT-5.6 Terra (1 to 2), GPT-5.6 Luna (1 to 3) and GPT-5.5 (0 to 3). The original results are shown below the table.

Accuracy

Higher is better
NumbersRisk & decisions

Speed

Shorter is faster
Median response time
0m4m8m GPT-5.6 Luna: 3m 13s5.6 Luna3m 13sGPT-5.6 Sol: 3m 23s5.6 Sol3m 23sGPT-6 Astra: 2m 34s6 Astra2m 34sGPT-5.5: 5m 7s5.55m 7sGPT-5.4: 2m 46s5.42m 46sGPT-5.6 Terra: 3m 3s5.6 Terra3m 3sGPT-5.4 Mini: 3m 29s5.4 Mini3m 29s

Cost

Lower is cheaper
API dollars / task at list prices
$0$0.25$0.5 GPT-5.6 Luna · $0.023$0.0235.6LunaGPT-5.6 Sol · $0.14$0.145.6SolGPT-6 Astra · $0.39$0.396AstraGPT-5.5 · $0.46$0.465.5GPT-5.4 · $0.27$0.275.4GPT-5.6 Terra · $0.15$0.155.6TerraGPT-5.4 Mini · $0.14$0.145.4Mini

3 attempts per model · same assignment Numbers: n/a, see the note above.

All metrics, side by side

Teal = lower cost or faster time
Accuracy, speed and cost for OM read
GPT-5.6 Lunan/a100.0%3 / 329–100%3m 13s$0.023
GPT-5.6 Soln/a100.0%3 / 329–100%3m 23s$0.14
GPT-6 Astran/a100.0%3 / 329–100%2m 34s$0.39
GPT-5.5n/a100.0%3 / 329–100%5m 7s$0.46
GPT-5.4n/a97.9%2 / 39–99%2m 46s$0.27
GPT-5.6 Terra1 format failure (no JSON block), scored as a miss on every checkn/a66.7%2 / 39–99%3m 3s$0.15
GPT-5.4 Minin/a93.8%1 / 31–91%3m 29s$0.14

Select a column to sort. Incomplete samples stay last. Accuracy columns are not shaded: at 3 attempts per model, differences inside the intervals are not meaningful. Cost uses the list prices in the price table; the price behind GPT-5.6 Sol’s cost is unconfirmed and may be too low. Swipe to see all columns →

Original key vs scoring v2

Full passes out of graded attempts
ModelOriginal keyfull passesScoring v2full passesYear-1 NOIpassed, original keyGoing-in cap rate at the asking pricepassed, original key
GPT-5.6 Sol2 / 33 / 32 / 33 / 3
GPT-6 Astra0 / 33 / 30 / 30 / 3
GPT-5.6 Terra1 / 32 / 31 / 31 / 3
GPT-5.6 Luna1 / 33 / 31 / 31 / 3
GPT-5.42 / 32 / 33 / 33 / 3
GPT-5.50 / 33 / 30 / 33 / 3
GPT-5.4 Mini1 / 31 / 33 / 33 / 3

Same tight rule in both columns; the original key also requires the excluded checks to pass. Original grades are retained.

What needs a review 4 checks

Observed misses across all models, most frequent first. An attempt without a usable memo is counted once, under the usable-memo check, not as a miss on every other check.

The $28,500 legal settlement is non-recurring and should be removed from expenses (a favorable adjustment).10% missed
Answer key

The $28,500 legal settlement is non-recurring and should be removed from expenses (a favorable adjustment).

Illustrative wording, credited by both judges (GPT-5.5, try 1): Broker removed the $28,500 Marigold legal settlement, which is consistent with firm policy.

Missed by GPT-5.4, GPT-5.4 Mini.

Percentage rent from Bluewater Café is based on 2024 sales; TTM sales are below the natural breakpoint, so percentage rent should be zero.5% missed
Answer key

Percentage rent from Bluewater Café is based on 2024 sales; TTM sales are below the natural breakpoint, so percentage rent should be zero.

Illustrative wording, credited by both judges (GPT-5.5, try 1): Includes Bluewater percentage rent of $18,420 based on 2024 sales | Count only if TTM reported sales exceed breakpoint

Missed by GPT-5.4 Mini.

Avoided the trap: counts percentage rent.5% missed
The mistake

Counts percentage rent.

GPT-5.4 Mini, try 3, flagged by both judges: Percentage rent = **18,420**

Missed by GPT-5.4 Mini.

These are benchmark checks, not severity ratings. Financial impact has not been measured. Missed findings use the primary judge. A mistake is attributed to a model only when both judges flagged it. Full-pass results use both judges where a second judgment exists.

Scoring & test setup

Check the broker’s claims, calculate the firm-basis numbers and identify diligence questions. 3 supplied text documents, 18 checks, 3 planned attempts per model. Small sample; every output needs review.

Numbers correct is the share of numerical checks within the answer key’s tolerance. Risk & decision checks covers findings, mistakes avoided and the requested recommendation. Both are individual-check pass rates, not the probability that an entire output is correct.

OM recovery provisions are absent. NOI and cap-rate estimates cannot be graded against a unique contractual answer. These two checks are excluded for every model; original grades are retained. The exclusion itself awards no credit for withheld Year-1 NOI and going-in cap rate at the asking price answers.

Passed every check is the stricter measure: every required check must pass in that attempt. One missed observation fails it, just as a wrong number does. A 0/3 result can still contain correct calculations.

The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count; ties are listed alphabetically and are not broken by cost or by check rates. Cost and speed cards require all planned attempts to pass every check. The table’s default order (full passes, then numbers, then risk and decision checks, then cost) is for display only, and it sorts all models directly by cost or wait, including failed results. Incomplete samples appear last in every view and are excluded from the cards when complete samples exist.

Chart labels are rounded. Missing results are not plotted; asterisks mark incomplete samples. In the combined table, teal marks lower API cost and faster median response within each column; accuracy columns are not shaded. It is not an overall score. Highlighted marks identify the models named on the selected card.

API charges per response at the list prices in the method’s price table, excluding subscriptions, tools, retries and human review. Times reflect this run; neither price nor speed is guaranteed in a chat app.

Analyze a rent roll

1 deal · 3 attempts / model

Highest full-pass rate

No model passed every check

0/3 for every model with all attempts graded · 95% CI 0–71%

Lowest-cost passing

No model qualifies

Requires every planned attempt to pass.

Fastest passing

No model qualifies

Requires every planned attempt to pass.

A full pass means every check passed in that attempt; passing cards require all 3 attempts to pass. Intervals are exact 95% binomial (Clopper-Pearson) intervals. At 3 attempts per model, no model is statistically separable from another on this task: every pair of 95% intervals overlaps.

Sensitivity: two checks were missed in at least 85% of attempts: “Every lease rolls within twelve months; the broker's 'immediate mark-to-market' is the same fact as 100% turnover exposure in year one” (missed in 100% of attempts) and “Suite D has been vacant since January 2026 and is being marketed at a rent well above the building's in-place retail rents” (missed in 100% of attempts). Excluding them, full passes would be GPT-5.6 Sol 2/3, GPT-6 Astra 1/3, GPT-5.6 Terra 1/3, GPT-5.6 Luna 2/3, GPT-5.4 2/3, GPT-5.5 3/3, GPT-5.4 Mini 3/3. The answer key still requires them; this is not a rescore.

Accuracy

Higher is better
NumbersRisk & decisions

Speed

Shorter is faster
Median response time
0m1.25m2.5m GPT-5.6 Luna: 1m 32s5.6 Luna1m 32sGPT-5.6 Sol: 1m 53s5.6 Sol1m 53sGPT-5.6 Terra: 1m 44s5.6 Terra1m 44sGPT-5.4 Mini: 2m 15s5.4 Mini2m 15sGPT-6 Astra: 1m 55s6 Astra1m 55sGPT-5.5: 2m 4s5.52m 4sGPT-5.4: 1m 54s5.41m 54s

Cost

Lower is cheaper
API dollars / task at list prices
$0$0.2$0.4 GPT-5.6 Luna · $0.015$0.0155.6LunaGPT-5.6 Sol · $0.093$0.0935.6SolGPT-5.6 Terra · $0.10$0.105.6TerraGPT-5.4 Mini · $0.13$0.135.4MiniGPT-6 Astra · $0.32$0.326AstraGPT-5.5 · $0.32$0.325.5GPT-5.4 · $0.20$0.205.4

3 attempts per model · same assignment

All metrics, side by side

Teal = lower cost or faster time
Accuracy, speed and cost for Rent roll analysis
GPT-5.6 Luna100.0%83.3%0 / 30–71%1m 32s$0.015
GPT-5.6 Sol100.0%83.3%0 / 30–71%1m 53s$0.093
GPT-5.6 Terra100.0%83.3%0 / 30–71%1m 44s$0.10
GPT-5.4 Mini100.0%83.3%0 / 30–71%2m 15s$0.13
GPT-6 Astra100.0%83.3%0 / 30–71%1m 55s$0.32
GPT-5.5100.0%83.3%0 / 30–71%2m 4s$0.32
GPT-5.497.6%77.8%0 / 30–71%1m 54s$0.20

Select a column to sort. Incomplete samples stay last. Accuracy columns are not shaded: at 3 attempts per model, differences inside the intervals are not meaningful. Cost uses the list prices in the price table; the price behind GPT-5.6 Sol’s cost is unconfirmed and may be too low. Swipe to see all columns →

What needs a review 5 checks

Observed misses across all models, most frequent first.

Every lease rolls within twelve months; the broker's 'immediate mark-to-market' is the same fact as 100% turnover exposure in year one.100% missed · every attempt
Answer key

Every lease rolls within twelve months; the broker's 'immediate mark-to-market' is the same fact as 100% turnover exposure in year one.

Missed by every model, every try.

Suite D has been vacant since January 2026 and is being marketed at a rent well above the building's in-place retail rents.100% missed · every attempt
Answer key

Suite D has been vacant since January 2026 and is being marketed at a rent well above the building's in-place retail rents.

Missed by every model, every try.

Month-to-month units5% missed
Answer key

Month-to-month units: 4, exact match required.

ModelAnswer keyTry 1Try 2Try 3
GPT-5.6 Sol4444
GPT-6 Astra4444
GPT-5.6 Terra4444
GPT-5.6 Luna4444
GPT-5.44434
GPT-5.54444
GPT-5.4 Mini4444

These are benchmark checks, not severity ratings. Financial impact has not been measured. Missed findings use the primary judge. A mistake is attributed to a model only when both judges flagged it. Full-pass results use both judges where a second judgment exists.

Scoring & test setup

Calculate occupancy and rent, then identify rollover, concessions and collection risk. 2 supplied text documents, 27 checks, 3 planned attempts per model. Small sample; every output needs review.

Numbers correct is the share of numerical checks within the answer key’s tolerance. Risk & decision checks covers findings, mistakes avoided and the requested recommendation. Both are individual-check pass rates, not the probability that an entire output is correct.

Passed every check is the stricter measure: every required check must pass in that attempt. One missed observation fails it, just as a wrong number does. A 0/3 result can still contain correct calculations.

The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count; ties are listed alphabetically and are not broken by cost or by check rates. Cost and speed cards require all planned attempts to pass every check. The table’s default order (full passes, then numbers, then risk and decision checks, then cost) is for display only, and it sorts all models directly by cost or wait, including failed results. Incomplete samples appear last in every view and are excluded from the cards when complete samples exist.

Chart labels are rounded. Missing results are not plotted; asterisks mark incomplete samples. In the combined table, teal marks lower API cost and faster median response within each column; accuracy columns are not shaded. It is not an overall score. Highlighted marks identify the models named on the selected card.

API charges per response at the list prices in the method’s price table, excluding subscriptions, tools, retries and human review. Times reflect this run; neither price nor speed is guaranteed in a chat app.

Build underwriting assumptions

1 deal · 3 attempts / model

Highest full-pass rate

Tied: GPT-5.6 Luna, GPT-5.6 Sol and GPT-5.6 Terra

1/3 full passes each · 95% CI 1–91%. Listed alphabetically; no tie-break is applied.

Lowest-cost passing

No model qualifies

Requires every planned attempt to pass.

Fastest passing

No model qualifies

Requires every planned attempt to pass.

A full pass means every check passed in that attempt; passing cards require all 3 attempts to pass. Intervals are exact 95% binomial (Clopper-Pearson) intervals. At 3 attempts per model, no model is statistically separable from another on this task: every pair of 95% intervals overlaps.

Accuracy

Higher is better
NumbersRisk & decisions

Speed

Shorter is faster
Median response time
0m2m4m GPT-5.6 Terra: 1m 45s5.6 Terra1m 45sGPT-5.6 Sol: 2m 30s5.6 Sol2m 30sGPT-5.6 Luna: 2m 6s5.6 Luna2m 6sGPT-5.4 Mini: 2m 36s5.4 Mini2m 36sGPT-5.4: 2m 10s5.42m 10sGPT-5.5: 2m 55s5.52m 55sGPT-6 Astra: 1m 47s6 Astra1m 47s

Cost

Lower is cheaper
API dollars / task at list prices
$0$0.2$0.4 GPT-5.6 Terra · $0.11$0.115.6TerraGPT-5.6 Sol · $0.12$0.125.6SolGPT-5.6 Luna · $0.020$0.0205.6LunaGPT-5.4 Mini · $0.15$0.155.4MiniGPT-5.4 · $0.23$0.235.4GPT-5.5 · $0.39$0.395.5GPT-6 Astra · $0.36$0.366Astra

3 attempts per model · same assignment

All metrics, side by side

Teal = lower cost or faster time
Accuracy, speed and cost for Underwriting assumptions
GPT-5.6 Terra100.0%96.1%1 / 31–91%1m 45s$0.11
GPT-5.6 Sol93.9%98.0%1 / 31–91%2m 30s$0.12
GPT-5.6 Luna90.9%98.0%1 / 31–91%2m 6s$0.020
GPT-5.4 Mini100.0%94.1%0 / 30–71%2m 36s$0.15
GPT-5.4100.0%94.1%0 / 30–71%2m 10s$0.23
GPT-5.5100.0%94.1%0 / 30–71%2m 55s$0.39
GPT-6 Astra81.8%92.2%0 / 30–71%1m 47s$0.36

Select a column to sort. Incomplete samples stay last. Accuracy columns are not shaded: at 3 attempts per model, differences inside the intervals are not meaningful. Cost uses the list prices in the price table; the price behind GPT-5.6 Sol’s cost is unconfirmed and may be too low. Swipe to see all columns →

What needs a review 5 checks

Observed misses across all models, most frequent first.

Three units carry one month free; contractual rent overstates effective rent for those leases.67% missed
Answer key

Three units carry one month free; contractual rent overstates effective rent for those leases.

Illustrative wording, credited by both judges (GPT-5.6 Luna, try 1): rent roll’s one-month-free concessions for Units 209, 301 and 506

Missed by every model at least once.

Year-1 NOI24% missed
Answer key

Year-1 NOI (band: recoveries zeroed → recoveries at T-12): 1,085,101 to 1,216,254, accepted anywhere inside the band or within ±3% of its nearer edge.

ModelAnswer keyTry 1Try 2Try 3
GPT-5.6 Sol1,085,101 to 1,216,2541,197,146did not state a figure1,197,146
GPT-6 Astra1,085,101 to 1,216,254did not state a figuredid not state a figuredid not state a figure
GPT-5.6 Terra1,085,101 to 1,216,2541,085,1071,085,1071,085,107
GPT-5.6 Luna1,085,101 to 1,216,254did not state a figure1,085,1071,085,107
GPT-5.41,085,101 to 1,216,2541,085,1071,197,1461,197,146
GPT-5.51,085,101 to 1,216,2541,197,1461,197,1461,197,146
GPT-5.4 Mini1,085,101 to 1,216,2541,197,1461,198,6021,198,602
Going-in cap rate at the asking price24% missed
Answer key

Going-in cap at guidance (band): 3.15 to 3.53, accepted anywhere inside the band or within ±0.15 points of its nearer edge.

ModelAnswer keyTry 1Try 2Try 3
GPT-5.6 Sol3.15 to 3.533.47did not state a figure3.47
GPT-6 Astra3.15 to 3.53did not state a figuredid not state a figuredid not state a figure
GPT-5.6 Terra3.15 to 3.533.153.153.15
GPT-5.6 Luna3.15 to 3.53did not state a figure3.153.15
GPT-5.43.15 to 3.533.153.473.47
GPT-5.53.15 to 3.533.473.473.47
GPT-5.4 Mini3.15 to 3.533.473.473.47

The judges split on 1 mistake flag in this task (one judge flagged it, the other did not). Split flags are not attributed to a model here, but under the full-pass rule a flag from either judge still fails that attempt.

These are benchmark checks, not severity ratings. Financial impact has not been measured. Missed findings use the primary judge. A mistake is attributed to a model only when both judges flagged it. Full-pass results use both judges where a second judgment exists.

Scoring & test setup

Translate the rent roll and T-12 into a defensible Year-1 assumption set. 4 supplied text documents, 29 checks, 3 planned attempts per model. Small sample; every output needs review.

Numbers correct is the share of numerical checks within the answer key’s tolerance. Risk & decision checks covers findings, mistakes avoided and the requested recommendation. Both are individual-check pass rates, not the probability that an entire output is correct.

Passed every check is the stricter measure: every required check must pass in that attempt. One missed observation fails it, just as a wrong number does. A 0/3 result can still contain correct calculations.

The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count; ties are listed alphabetically and are not broken by cost or by check rates. Cost and speed cards require all planned attempts to pass every check. The table’s default order (full passes, then numbers, then risk and decision checks, then cost) is for display only, and it sorts all models directly by cost or wait, including failed results. Incomplete samples appear last in every view and are excluded from the cards when complete samples exist.

Chart labels are rounded. Missing results are not plotted; asterisks mark incomplete samples. In the combined table, teal marks lower API cost and faster median response within each column; accuracy columns are not shaded. It is not an overall score. Highlighted marks identify the models named on the selected card.

API charges per response at the list prices in the method’s price table, excluding subscriptions, tools, retries and human review. Times reflect this run; neither price nor speed is guaranteed in a chat app.

Assess development feasibility

1 deal · 3 attempts / model

Highest full-pass rate

Tied: GPT-5.6 Sol, GPT-5.6 Terra and GPT-6 Astra

2/3 full passes each · 95% CI 9–99%. Listed alphabetically; no tie-break is applied.

Lowest-cost passing

No model qualifies

Requires every planned attempt to pass.

Fastest passing

No model qualifies

Requires every planned attempt to pass.

A full pass means every check passed in that attempt; passing cards require all 3 attempts to pass. Intervals are exact 95% binomial (Clopper-Pearson) intervals. At 3 attempts per model, no model is statistically separable from another on this task: every pair of 95% intervals overlaps.

Accuracy

Higher is better
NumbersRisk & decisions

Speed

Shorter is faster
Median response time
0m4m8m GPT-5.6 Sol: 3m 17s5.6 Sol3m 17sGPT-5.6 Terra: 3m 18s5.6 Terra3m 18sGPT-6 Astra: 3m 22s6 Astra3m 22sGPT-5.5: 5m 40s5.55m 40sGPT-5.6 Luna: 3m 36s5.6 Luna3m 36sGPT-5.4: 4m 35s5.44m 35sGPT-5.4 Mini: 6m 27s5.4 Mini6m 27s

Cost

Lower is cheaper
API dollars / task at list prices
$0$0.4$0.8 GPT-5.6 Sol · $0.15$0.155.6SolGPT-5.6 Terra · $0.17$0.175.6TerraGPT-6 Astra · $0.59$0.596AstraGPT-5.5 · $0.56$0.565.5GPT-5.6 Luna · $0.025$0.0255.6LunaGPT-5.4 · $0.42$0.425.4GPT-5.4 Mini · $0.26$0.265.4Mini

3 attempts per model · same assignment

All metrics, side by side

Teal = lower cost or faster time
Accuracy, speed and cost for Feasibility memo
GPT-5.6 Sol100.0%100.0%2 / 39–99%3m 17s$0.15
GPT-5.6 Terra100.0%98.2%2 / 39–99%3m 18s$0.17
GPT-6 Astra100.0%98.2%2 / 39–99%3m 22s$0.59
GPT-5.5100.0%94.7%1 / 31–91%5m 40s$0.56
GPT-5.6 Luna83.3%98.2%1 / 31–91%3m 36s$0.025
GPT-5.4100.0%98.2%0 / 30–71%4m 35s$0.42
GPT-5.4 Mini1 format failure (empty response at the output token limit), scored as a miss on every check66.7%63.2%0 / 30–71%6m 27s$0.26

Select a column to sort. Incomplete samples stay last. Accuracy columns are not shaded: at 3 attempts per model, differences inside the intervals are not meaningful. Cost uses the list prices in the price table; the price behind GPT-5.6 Sol’s cost is unconfirmed and may be too low. Swipe to see all columns →

What needs a review 6 checks

Observed misses across all models, most frequent first. An attempt without a usable memo is counted once, under the usable-memo check, not as a miss on every other check.

The lender sizes the interest reserve to the GC's schedule and requires funded off-site obligations, so the sponsor's equity requirement is understated.35% missed
Answer key

The lender sizes the interest reserve to the GC's schedule and requires funded off-site obligations, so the sponsor's equity requirement is understated.

Illustrative wording, credited by both judges (GPT-5.5, try 3): Required MPP equity check is understated by approximately $8.2 million on firm basis. At corrected TDC and 60% LTC:

Missed by GPT-6 Astra, GPT-5.6 Terra, GPT-5.6 Luna, GPT-5.4, GPT-5.5, GPT-5.4 Mini.

The GC says 30 months, the sponsor says 24; lease-up and interest reserve both move.10% missed
Answer key

Schedule: the GC says 30 months, the sponsor says 24; lease-up and interest reserve both move.

Illustrative wording, credited by both judges (GPT-5.6 Terra, try 1): Uses $3.100m based on a 24-month schedule; omits revised 30-month construction schedule

Missed by GPT-5.5, GPT-5.4 Mini.

Stabilized NOI at today's rents5% missed
Answer key

Firm-basis untrended NOI (with affordable set-aside): 4,118,800, accepted within ±4%.

ModelAnswer keyTry 1Try 2Try 3
GPT-5.6 Sol4,118,8004,066,6314,066,6314,062,718
GPT-6 Astra4,118,8004,062,7184,062,7184,062,718
GPT-5.6 Terra4,118,8004,066,6314,062,7184,066,631
GPT-5.6 Luna4,118,8004,062,7183,789,1184,052,118
GPT-5.44,118,8004,066,6284,066,6314,066,631
GPT-5.54,118,8004,066,6314,066,6314,062,718
GPT-5.4 Mini4,118,800empty (output token limit)4,066,6314,075,397

These are benchmark checks, not severity ratings. Financial impact has not been measured. Missed findings use the primary judge. A mistake is attributed to a model only when both judges flagged it. Full-pass results use both judges where a second judgment exists.

Scoring & test setup

Reconcile costs, schedule and lender requirements to assess the equity investment. 8 supplied text documents, 26 checks, 3 planned attempts per model. Small sample; every output needs review.

Numbers correct is the share of numerical checks within the answer key’s tolerance. Risk & decision checks covers findings, mistakes avoided and the requested recommendation. Both are individual-check pass rates, not the probability that an entire output is correct.

The check on the land price that would make it work accepts “no clearing price” as a correct answer; 19 of 20 usable attempts gave that answer, so the check rarely tested a stated figure.

Passed every check is the stricter measure: every required check must pass in that attempt. One missed observation fails it, just as a wrong number does. A 0/3 result can still contain correct calculations.

The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count; ties are listed alphabetically and are not broken by cost or by check rates. Cost and speed cards require all planned attempts to pass every check. The table’s default order (full passes, then numbers, then risk and decision checks, then cost) is for display only, and it sorts all models directly by cost or wait, including failed results. Incomplete samples appear last in every view and are excluded from the cards when complete samples exist.

Chart labels are rounded. Missing results are not plotted; asterisks mark incomplete samples. In the combined table, teal marks lower API cost and faster median response within each column; accuracy columns are not shaded. It is not an overall score. Highlighted marks identify the models named on the selected card.

API charges per response at the list prices in the method’s price table, excluding subscriptions, tools, retries and human review. Times reflect this run; neither price nor speed is guaranteed in a chat app.

Review a construction draw

1 deal · 3 attempts / model

Highest full-pass rate

Tied: GPT-5.4, GPT-5.6 Sol and GPT-5.6 Terra

1/3 full passes each · 95% CI 1–91%. Listed alphabetically; no tie-break is applied.

Lowest-cost passing

No model qualifies

Requires every planned attempt to pass.

Fastest passing

No model qualifies

Requires every planned attempt to pass.

A full pass means every check passed in that attempt; passing cards require all 3 attempts to pass. Intervals are exact 95% binomial (Clopper-Pearson) intervals. At 3 attempts per model, no model is statistically separable from another on this task: every pair of 95% intervals overlaps.

Accuracy

Higher is better
NumbersRisk & decisions

Speed

Shorter is faster
Median response time
0m2.5m5m GPT-5.4: 3m 57s5.43m 57sGPT-5.6 Sol: 2m 56s5.6 Sol2m 56sGPT-5.6 Terra: 2m 22s5.6 Terra2m 22sGPT-5.5: 4m 15s5.54m 15sGPT-5.6 Luna: 3m 16s5.6 Luna3m 16sGPT-5.4 Mini: 3m 48s5.4 Mini3m 48sGPT-6 Astra: 4m 9s6 Astra4m 9s

Cost

Lower is cheaper
API dollars / task at list prices
$0$0.4$0.8 GPT-5.4 · $0.40$0.405.4GPT-5.6 Sol · $0.15$0.155.6SolGPT-5.6 Terra · $0.16$0.165.6TerraGPT-5.5 · $0.69$0.695.5GPT-5.6 Luna · $0.024$0.0245.6LunaGPT-5.4 Mini · $0.22$0.225.4MiniGPT-6 Astra · $0.65$0.656Astra

3 attempts per model · same assignment

All metrics, side by side

Teal = lower cost or faster time
Accuracy, speed and cost for Construction draw review
GPT-5.495.8%97.8%1 / 31–91%3m 57s$0.40
GPT-5.6 Sol91.7%95.6%1 / 31–91%2m 56s$0.15
GPT-5.6 Terra91.7%91.1%1 / 31–91%2m 22s$0.16
GPT-5.5100.0%93.3%0 / 30–71%4m 15s$0.69
GPT-5.6 Luna95.8%91.1%0 / 30–71%3m 16s$0.024
GPT-5.4 Mini87.5%80.0%0 / 30–71%3m 48s$0.22
GPT-6 Astra83.3%95.6%0 / 30–71%4m 9s$0.65

Select a column to sort. Incomplete samples stay last. Accuracy columns are not shaded: at 3 attempts per model, differences inside the intervals are not meaningful. Cost uses the list prices in the price table; the price behind GPT-5.6 Sol’s cost is unconfirmed and may be too low. Swipe to see all columns →

What needs a review 9 checks

Observed misses across all models, most frequent first.

CO-08, the window acceleration premium, was rejected; the three-week slip therefore stands and the window delivery risk is unresolved.43% missed
Answer key

CO-08, the window acceleration premium, was rejected; the three-week slip therefore stands and the window delivery risk is unresolved.

Illustrative wording, credited by both judges (GPT-5.6 Terra, try 1): the unresolved window-delivery issue noted by the monitor

Missed by GPT-5.6 Sol, GPT-5.6 Terra, GPT-5.5, GPT-5.4 Mini (every try).

In-balance shortfall after draw38% missed
Answer key

In-balance shortfall after draw: 2,326,000, accepted within ±12%.

ModelAnswer keyTry 1Try 2Try 3
GPT-5.6 Sol2,326,0002,315,8003,026,9982,315,800
GPT-6 Astra2,326,0002,315,8003,090,7202,315,800
GPT-5.6 Terra2,326,0002,081,9081,384,4802,092,708
GPT-5.6 Luna2,326,0002,174,7002,095,678did not state a figure
GPT-5.42,326,0002,325,8003,230,0912,081,908
GPT-5.52,326,0002,315,8002,315,8002,325,800
GPT-5.4 Mini2,326,000-1,053,3001,384,480-352,862
Reached the right recommendation.33% missed
Answer key

The right call is FUND_REDUCED. Fund the corrected amount.

ModelRight callTry 1Try 2Try 3
GPT-5.6 SolFUND_REDUCEDFUND_REDUCEDFUND_REDUCEDFUND_REDUCED
GPT-6 AstraFUND_REDUCEDHOLDHOLDFUND_REDUCED
GPT-5.6 TerraFUND_REDUCEDFUND_REDUCEDHOLDHOLD
GPT-5.6 LunaFUND_REDUCEDHOLDHOLDHOLD
GPT-5.4FUND_REDUCEDFUND_REDUCEDFUND_REDUCEDFUND_REDUCED
GPT-5.5FUND_REDUCEDFUND_REDUCEDFUND_REDUCEDFUND_REDUCED
GPT-5.4 MiniFUND_REDUCEDFUND_REDUCEDFUND_REDUCEDFUND_REDUCED

These are benchmark checks, not severity ratings. Financial impact has not been measured. Missed findings use the primary judge. A mistake is attributed to a model only when both judges flagged it. Full-pass results use both judges where a second judgment exists.

Scoring & test setup

Check the draw, remaining funding and in-balance test before recommending a release. 6 supplied text documents, 24 checks, 3 planned attempts per model. Small sample; every output needs review.

Numbers correct is the share of numerical checks within the answer key’s tolerance. Risk & decision checks covers findings, mistakes avoided and the requested recommendation. Both are individual-check pass rates, not the probability that an entire output is correct.

Passed every check is the stricter measure: every required check must pass in that attempt. One missed observation fails it, just as a wrong number does. A 0/3 result can still contain correct calculations.

The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count; ties are listed alphabetically and are not broken by cost or by check rates. Cost and speed cards require all planned attempts to pass every check. The table’s default order (full passes, then numbers, then risk and decision checks, then cost) is for display only, and it sorts all models directly by cost or wait, including failed results. Incomplete samples appear last in every view and are excluded from the cards when complete samples exist.

Chart labels are rounded. Missing results are not plotted; asterisks mark incomplete samples. In the combined table, teal marks lower API cost and faster median response within each column; accuracy columns are not shaded. It is not an overall score. Highlighted marks identify the models named on the selected card.

API charges per response at the list prices in the method’s price table, excluding subscriptions, tools, retries and human review. Times reflect this run; neither price nor speed is guaranteed in a chat app.

Reforecast lease-up & take-out

1 deal · 3 attempts / model

Highest full-pass rate

GPT-6 Astra

$0.533m 28s

3/3 full passes · 95% CI 29–100%

Lowest-cost passing

GPT-6 Astra

$0.533m 28s

3/3 full passes · 95% CI 29–100%

Fastest passing

GPT-6 Astra

$0.533m 28s

3/3 full passes · 95% CI 29–100%

A full pass means every check passed in that attempt; passing cards require all 3 attempts to pass. Intervals are exact 95% binomial (Clopper-Pearson) intervals. At 3 attempts per model, no model is statistically separable from another on this task: every pair of 95% intervals overlaps.

Second judge: only GPT-6 Astra’s memos on this task had a second judgment. Full passes for GPT-5.6 Sol, GPT-5.6 Luna, GPT-5.4, GPT-5.5 and GPT-5.4 Mini use one judge, a more lenient test, so these counts are not like for like.

Accuracy

Higher is better
NumbersRisk & decisions

Speed

Shorter is faster
Median response time
0m2.5m5m GPT-6 Astra: 3m 28s6 Astra3m 28sGPT-5.4: 3m 24s5.43m 24sGPT-5.5: 4m5.54mGPT-5.6 Luna: 2m 58s5.6 Luna2m 58sGPT-5.4 Mini: 4m 23s5.4 Mini4m 23sGPT-5.6 Sol: 2m 24s · incomplete sample5.6 Sol*2m 24sGPT-5.6 Terra: No graded results · incomplete sample5.6 Terra*Not scored

Cost

Lower is cheaper
API dollars / task at list prices
$0$0.4$0.8 GPT-6 Astra · $0.53$0.536AstraGPT-5.4 · $0.35$0.355.4GPT-5.5 · $0.49$0.495.5GPT-5.6 Luna · $0.021$0.0215.6LunaGPT-5.4 Mini · $0.18$0.185.4MiniGPT-5.6 Sol · $0.11 · incomplete sample$0.115.6*SolGPT-5.6 Terra · No results · incomplete samplen/a5.6*Terra

* Incomplete sample; coverage shown below.

All metrics, side by side

Teal = lower cost or faster time
Accuracy, speed and cost for Lease-up reforecast & take-out
GPT-6 Astra100.0%100.0%3 / 329–100%3m 28s$0.53
GPT-5.4100.0%97.8%2 / 39–99%3m 24s$0.35
GPT-5.590.5%100.0%2 / 39–99%4m$0.49
GPT-5.6 Luna100.0%95.6%1 / 31–91%2m 58s$0.021
GPT-5.4 Mini81.0%91.1%0 / 30–71%4m 23s$0.18
GPT-5.6 Sol2/3 attempts graded · 1 memo returned, not scored (judge API quota exhausted)100.0%96.7%1 / 21–99%2m 24s$0.11
GPT-5.6 Terra0/3 attempts graded · 3 memos returned, not scored (judge API quota exhausted)Not scoredNot scoredNot scoredNot scoredNot scored

Select a column to sort. Incomplete samples stay last. Accuracy columns are not shaded: at 3 attempts per model, differences inside the intervals are not meaningful. Cost uses the list prices in the price table; the price behind GPT-5.6 Sol’s cost is unconfirmed and may be too low. Swipe to see all columns →

What needs a review 6 checks

Observed misses across all models, most frequent first.

The sponsor's trended stabilized NOI must not be used for sizing; the untrended, concession-adjusted figure is the one the lender will use.41% missed
Answer key

The sponsor's trended stabilized NOI must not be used for sizing; the untrended, concession-adjusted figure is the one the lender will use.

Illustrative wording, credited by both judges (GPT-6 Astra, try 1): "Actual rents, concessions and the newer stabilized expense budget supersede that earlier total. No trended rents are used."

Missed by GPT-5.6 Sol, GPT-5.6 Luna, GPT-5.4, GPT-5.4 Mini (every try).

Stabilized NOI, effective rents net of concessions12% missed
Answer key

Stabilized NOI, effective rents net of concessions: 3,135,000, accepted within ±4%.

ModelAnswer keyTry 1Try 2Try 3
GPT-5.6 Sol3,135,0003,144,7983,040,785memo returned; not scored (judge API quota exhausted)
GPT-6 Astra3,135,0003,040,7853,040,7853,040,785
GPT-5.6 Terra3,135,000memo returned; not scored (judge API quota exhausted)memo returned; not scored (judge API quota exhausted)memo returned; not scored (judge API quota exhausted)
GPT-5.6 Luna3,135,0003,017,1183,040,6523,034,959
GPT-5.43,135,0003,085,3903,190,0993,078,939
GPT-5.53,135,0003,086,4232,992,8033,144,798
GPT-5.4 Mini3,135,0003,014,6472,788,3553,085,183
Paydown at maturity12% missed
Answer key

Paydown at maturity: 18,200,000, accepted within ±8%.

ModelAnswer keyTry 1Try 2Try 3
GPT-5.6 Sol18,200,00018,098,78319,246,508memo returned; not scored (judge API quota exhausted)
GPT-6 Astra18,200,00019,246,50819,246,50819,246,508
GPT-5.6 Terra18,200,000memo returned; not scored (judge API quota exhausted)memo returned; not scored (judge API quota exhausted)memo returned; not scored (judge API quota exhausted)
GPT-5.6 Luna18,200,00019,507,66319,247,97719,310,798
GPT-5.418,200,00017,709,96517,598,90918,825,503
GPT-5.518,200,00018,742,91619,775,97218,098,783
GPT-5.4 Mini18,200,00018,339,06422,031,94418,756,599

These are benchmark checks, not severity ratings. Financial impact has not been measured. Missed findings use the primary judge. A mistake is attributed to a model only when both judges flagged it. Full-pass results use both judges where a second judgment exists.

Scoring & test setup

Update stabilization, size permanent debt and identify the loan paydown. 5 supplied text documents, 23 checks, 3 planned attempts per model. Small sample; every output needs review.

Numbers correct is the share of numerical checks within the answer key’s tolerance. Risk & decision checks covers findings, mistakes avoided and the requested recommendation. Both are individual-check pass rates, not the probability that an entire output is correct.

Passed every check is the stricter measure: every required check must pass in that attempt. One missed observation fails it, just as a wrong number does. A 0/3 result can still contain correct calculations.

The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count; ties are listed alphabetically and are not broken by cost or by check rates. Cost and speed cards require all planned attempts to pass every check. The table’s default order (full passes, then numbers, then risk and decision checks, then cost) is for display only, and it sorts all models directly by cost or wait, including failed results. Incomplete samples appear last in every view and are excluded from the cards when complete samples exist.

Chart labels are rounded. Missing results are not plotted; asterisks mark incomplete samples. In the combined table, teal marks lower API cost and faster median response within each column; accuracy columns are not shaded. It is not an overall score. Highlighted marks identify the models named on the selected card.

API charges per response at the list prices in the method’s price table, excluding subscriptions, tools, retries and human review. Times reflect this run; neither price nor speed is guaranteed in a chat app.

Explain budget variances

1 deal · 3 attempts / model

Highest full-pass rate

No model passed every check

0/3 for every model with all attempts graded · 95% CI 0–71%

Lowest-cost passing

No model qualifies

Requires every planned attempt to pass.

Fastest passing

No model qualifies

Requires every planned attempt to pass.

A full pass means every check passed in that attempt; passing cards require all 3 attempts to pass. Intervals are exact 95% binomial (Clopper-Pearson) intervals. At 3 attempts per model, no model is statistically separable from another on this task: every pair of 95% intervals overlaps.

Sensitivity: two checks were missed in at least 85% of attempts: “The pre-close $6,100 receivable on Unit 211 belongs to the seller under the PSA and is not our bad debt” (missed in 95% of attempts) and “Suite C paid rent through March as budgeted; its Q1 rent is not run-rate and the April cliff is already in the budget” (missed in 86% of attempts). Excluding them, full passes would be GPT-5.6 Sol 1/3, GPT-6 Astra 1/3, GPT-5.6 Terra 1/3, GPT-5.6 Luna 0/3, GPT-5.4 0/3, GPT-5.5 1/3, GPT-5.4 Mini 0/3. The answer key still requires them; this is not a rescore.

Accuracy

Higher is better
NumbersRisk & decisions

Speed

Shorter is faster
Median response time
0m4m8m GPT-5.6 Sol: 3m 8s5.6 Sol3m 8sGPT-5.4: 4m 29s5.44m 29sGPT-6 Astra: 3m 35s6 Astra3m 35sGPT-5.5: 7m 17s5.57m 17sGPT-5.6 Luna: 3m 19s5.6 Luna3m 19sGPT-5.6 Terra: 2m 42s5.6 Terra2m 42sGPT-5.4 Mini: 5m 10s5.4 Mini5m 10s

Cost

Lower is cheaper
API dollars / task at list prices
$0$0.4$0.8 GPT-5.6 Sol · $0.14$0.145.6SolGPT-5.4 · $0.48$0.485.4GPT-6 Astra · $0.62$0.626AstraGPT-5.5 · $0.66$0.665.5GPT-5.6 Luna · $0.026$0.0265.6LunaGPT-5.6 Terra · $0.17$0.175.6TerraGPT-5.4 Mini · $0.25$0.255.4Mini

3 attempts per model · same assignment

All metrics, side by side

Teal = lower cost or faster time
Accuracy, speed and cost for Variance report & reforecast
GPT-5.6 Sol100.0%86.7%0 / 30–71%3m 8s$0.14
GPT-5.4100.0%86.7%0 / 30–71%4m 29s$0.48
GPT-6 Astra100.0%86.7%0 / 30–71%3m 35s$0.62
GPT-5.5100.0%86.7%0 / 30–71%7m 17s$0.66
GPT-5.6 Luna100.0%84.4%0 / 30–71%3m 19s$0.026
GPT-5.6 Terra100.0%82.2%0 / 30–71%2m 42s$0.17
GPT-5.4 Mini100.0%77.8%0 / 30–71%5m 10s$0.25

Select a column to sort. Incomplete samples stay last. Accuracy columns are not shaded: at 3 attempts per model, differences inside the intervals are not meaningful. Cost uses the list prices in the price table; the price behind GPT-5.6 Sol’s cost is unconfirmed and may be too low. Swipe to see all columns →

What needs a review 5 checks

Observed misses across all models, most frequent first.

The pre-close $6,100 receivable on Unit 211 belongs to the seller under the PSA and is not our bad debt.95% missed
Answer key

The pre-close $6,100 receivable on Unit 211 belongs to the seller under the PSA and is not our bad debt.

Illustrative wording, credited by both judges (GPT-6 Astra, try 3): The seller’s $6,100 pre-close rent receivable is not ours and creates no additional NOI adjustment.

Missed by every model at least once.

Suite C paid rent through March as budgeted; its Q1 rent is not run-rate and the April cliff is already in the budget.86% missed
Answer key

Suite C paid rent through March as budgeted; its Q1 rent is not run-rate and the April cliff is already in the budget.

Illustrative wording, credited by both judges (GPT-6 Astra, try 1): Suite C vacated March 31 as budgeted.

Missed by every model at least once.

Commentary appears only on lines outside the threshold; the bridge ties with no plug.24% missed
Answer key

Commentary appears only on lines outside the threshold; the bridge ties with no plug.

Illustrative wording, credited by both judges (GPT-5.6 Luna, try 1): Commentary is shown only where the variance exceeds the lower of 5% or $2,500. ... NOI impact: $(10,856) + $(18,203) = $(29,059) ... The $100 is disclosed explicitly rather than included in a miscellaneous plug.

Missed by GPT-6 Astra, GPT-5.6 Terra, GPT-5.6 Luna, GPT-5.4 Mini.

The judges split on 5 mistake flags in this task (one judge flagged it, the other did not). Split flags are not attributed to a model here, but under the full-pass rule a flag from either judge still fails that attempt.

These are benchmark checks, not severity ratings. Financial impact has not been measured. Missed findings use the primary judge. A mistake is attributed to a model only when both judges flagged it. Full-pass results use both judges where a second judgment exists.

Scoring & test setup

Bridge actuals to budget, reforecast NOI and draft the LP commentary. 4 supplied text documents, 23 checks, 3 planned attempts per model. Small sample; every output needs review.

Numbers correct is the share of numerical checks within the answer key’s tolerance. Risk & decision checks covers findings, mistakes avoided and the requested recommendation. Both are individual-check pass rates, not the probability that an entire output is correct.

Passed every check is the stricter measure: every required check must pass in that attempt. One missed observation fails it, just as a wrong number does. A 0/3 result can still contain correct calculations.

The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count; ties are listed alphabetically and are not broken by cost or by check rates. Cost and speed cards require all planned attempts to pass every check. The table’s default order (full passes, then numbers, then risk and decision checks, then cost) is for display only, and it sorts all models directly by cost or wait, including failed results. Incomplete samples appear last in every view and are excluded from the cards when complete samples exist.

Chart labels are rounded. Missing results are not plotted; asterisks mark incomplete samples. In the combined table, teal marks lower API cost and faster median response within each column; accuracy columns are not shaded. It is not an overall score. Highlighted marks identify the models named on the selected card.

API charges per response at the list prices in the method’s price table, excluding subscriptions, tools, retries and human review. Times reflect this run; neither price nor speed is guaranteed in a chat app.

Prepare an opinion of value

1 deal · 3 attempts / model

Highest full-pass rate

No model passed every check

0/3 for every model with all attempts graded · 95% CI 0–71%

Lowest-cost passing

No model qualifies

Requires every planned attempt to pass.

Fastest passing

No model qualifies

Requires every planned attempt to pass.

A full pass means every check passed in that attempt; passing cards require all 3 attempts to pass. Intervals are exact 95% binomial (Clopper-Pearson) intervals. At 3 attempts per model, no model is statistically separable from another on this task: every pair of 95% intervals overlaps.

Sensitivity: one check was missed in at least 85% of attempts: “The T-12 includes $62,000 of insurance proceeds and six months of a departed tenant's rent; both come out of normalized NOI” (missed in 95% of attempts). Excluding it, full passes would be GPT-5.6 Sol 0/3, GPT-6 Astra 2/3, GPT-5.6 Terra 1/3, GPT-5.6 Luna 2/3, GPT-5.4 1/3, GPT-5.5 2/3, GPT-5.4 Mini 0/3. The answer key still requires it; this is not a rescore.

Accuracy

Higher is better
NumbersRisk & decisions

Speed

Shorter is faster
Median response time
0m4m8m GPT-5.6 Luna: 2m 15s5.6 Luna2m 15sGPT-5.6 Terra: 1m 57s5.6 Terra1m 57sGPT-6 Astra: 4m 7s6 Astra4m 7sGPT-5.6 Sol: 2m 43s5.6 Sol2m 43sGPT-5.4: 3m 20s5.43m 20sGPT-5.5: 5m 25s5.55m 25sGPT-5.4 Mini: 2m 12s5.4 Mini2m 12s

Cost

Lower is cheaper
API dollars / task at list prices
$0$0.4$0.8 GPT-5.6 Luna · $0.017$0.0175.6LunaGPT-5.6 Terra · $0.13$0.135.6TerraGPT-6 Astra · $0.61$0.616AstraGPT-5.6 Sol · $0.12$0.125.6SolGPT-5.4 · $0.30$0.305.4GPT-5.5 · $0.48$0.485.5GPT-5.4 Mini · $0.099$0.0995.4Mini

3 attempts per model · same assignment

All metrics, side by side

Teal = lower cost or faster time
Accuracy, speed and cost for Broker opinion of value
GPT-5.6 Luna100.0%93.3%0 / 30–71%2m 15s$0.017
GPT-5.6 Terra100.0%93.3%0 / 30–71%1m 57s$0.13
GPT-6 Astra100.0%93.3%0 / 30–71%4m 7s$0.61
GPT-5.6 Sol100.0%91.1%0 / 30–71%2m 43s$0.12
GPT-5.4100.0%91.1%0 / 30–71%3m 20s$0.30
GPT-5.5100.0%91.1%0 / 30–71%5m 25s$0.48
GPT-5.4 Mini100.0%84.4%0 / 30–71%2m 12s$0.099

Select a column to sort. Incomplete samples stay last. Accuracy columns are not shaded: at 3 attempts per model, differences inside the intervals are not meaningful. Cost uses the list prices in the price table; the price behind GPT-5.6 Sol’s cost is unconfirmed and may be too low. Swipe to see all columns →

What needs a review 4 checks

Observed misses across all models, most frequent first.

The T-12 includes $62,000 of insurance proceeds and six months of a departed tenant's rent; both come out of normalized NOI.95% missed
Answer key

The T-12 includes $62,000 of insurance proceeds and six months of a departed tenant's rent; both come out of normalized NOI.

Missed by every model at least once.

Suite D lease-up and the $45/SF asking rent are upside for a buyer to underwrite, not in-place income.24% missed
Answer key

Suite D lease-up and the $45/SF asking rent are upside for a buyer to underwrite, not in-place income.

Illustrative wording, credited by both judges (GPT-6 Astra, try 3): "Suite D: vacant space, not a $45/SF lease. There is no executed LOI."

Missed by GPT-5.6 Sol, GPT-5.4, GPT-5.5, GPT-5.4 Mini.

Lotus Nail Spa is three months in arrears and Unit 211 is two months late; credit issues to resolve or disclose before marketing.5% missed
Answer key

Lotus Nail Spa is three months in arrears and Unit 211 is two months late; credit issues to resolve or disclose before marketing.

Illustrative wording, credited by both judges (GPT-5.6 Sol, try 3): Unit 211 owes $6,300, representing two unpaid months

Missed by GPT-5.4 Mini.

The judges split on 1 mistake flag in this task (one judge flagged it, the other did not). Split flags are not attributed to a model here, but under the full-pass rule a flag from either judge still fails that attempt.

These are benchmark checks, not severity ratings. Financial impact has not been measured. Missed findings use the primary judge. A mistake is attributed to a model only when both judges flagged it. Full-pass results use both judges where a second judgment exists.

Scoring & test setup

Normalize income, assess valuation and separate in-place value from potential upside. 8 supplied text documents, 20 checks, 3 planned attempts per model. Small sample; every output needs review.

Numbers correct is the share of numerical checks within the answer key’s tolerance. Risk & decision checks covers findings, mistakes avoided and the requested recommendation. Both are individual-check pass rates, not the probability that an entire output is correct.

Passed every check is the stricter measure: every required check must pass in that attempt. One missed observation fails it, just as a wrong number does. A 0/3 result can still contain correct calculations.

The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count; ties are listed alphabetically and are not broken by cost or by check rates. Cost and speed cards require all planned attempts to pass every check. The table’s default order (full passes, then numbers, then risk and decision checks, then cost) is for display only, and it sorts all models directly by cost or wait, including failed results. Incomplete samples appear last in every view and are excluded from the cards when complete samples exist.

Chart labels are rounded. Missing results are not plotted; asterisks mark incomplete samples. In the combined table, teal marks lower API cost and faster median response within each column; accuracy columns are not shaded. It is not an overall score. Highlighted marks identify the models named on the selected card.

API charges per response at the list prices in the method’s price table, excluding subscriptions, tools, retries and human review. Times reflect this run; neither price nor speed is guaranteed in a chat app.

Writing quality

Original underwriting responses

Exhibit H

All seven models score within 1.8 points.

Text presentation, out of 100. Correctness and visual design are tested separately. Scored by one AI judge, gpt-5.5-2026-04-23 (the GPT-5.5 contestant), so GPT-5.5 is graded by its own snapshot. No second judge; the 1.8-point spread is not a meaningful ranking.

ModelWriting scoreout of 100Structureout of 5Clarityout of 5Numbers shownout of 5Decision readyout of 5Concisionout of 5
GPT-5.527 memos scored 92.1 5.004.895.004.004.15
GPT-5.4 Mini25 memos scored 91.8 4.964.724.683.964.64
GPT-5.6 Terra27 memos scored 91.1 5.004.745.003.964.07
GPT-5.6 Sol27 memos scored 90.8 5.004.815.003.963.93
GPT-6 Astra27 memos scored 90.8 5.004.485.004.194.04
GPT-5.6 Luna27 memos scored 90.7 5.004.704.964.043.96
GPT-5.426 memos scored 90.3 5.004.735.003.883.96

Concision

7 different scores across seven models, spread 0.71 · zoomed axis, 3.7 to 4.8 of a possible 5

3.74.8
GPT-5.4 Mini4.64
GPT-5.54.15
GPT-5.6 Terra4.07
GPT-6 Astra4.04
GPT-5.6 Luna3.96
GPT-5.43.96
GPT-5.6 Sol3.93

Clarity

7 different scores across seven models, spread 0.41 · zoomed axis, 4.4 to 5.0 of a possible 5

4.45.0
GPT-5.54.89
GPT-5.6 Sol4.81
GPT-5.6 Terra4.74
GPT-5.44.73
GPT-5.4 Mini4.72
GPT-5.6 Luna4.70
GPT-6 Astra4.48
Reviewer notes

GPT-5.6 Sol
Strong reporting package with excellent structure and numerical transparency, but add a crisp action-oriented close if this is meant to be decision-ready.

GPT-6 Astra
Strong reporting package with excellent structure and numbers, but add a crisp closing action/recommendation so the principal knows exactly what to do with it.

GPT-5.6 Terra
Strong package structure and numerical presentation, but add a concise end-state recommendation or action conclusion and trim minor over-detail.

GPT-5.6 Luna
Strong package structure and numerical presentation, but it needs a clear closing recommendation or action point if judged against decision-ready memo standards.

GPT-5.4
Strong reporting package with excellent structure and numerical presentation, but it needs a clear closing recommendation or action call to be decision-ready.

GPT-5.5
Strong reporting package with excellent structure and numerical support, but add a crisp concluding action/recommendation and trim internal process commentary before presenting.

GPT-5.4 Mini
Strong reporting package presentation, but add an explicit closing recommendation or action statement if this is meant to be decision-ready.

Rendered output tests · 2026-09-13

Document design

Writing, layout and delivery across three assignments.

52/63 complete · 104 reviews

Write & design a deal one-pager

1 page · supplied facts · HTML output
Writing, visual design, fact preservation, file checks, speed and cost
ModelWritingout of 5Visual designout of 5Facts keptboth judgesFile checksout of 3SpeedmedianCostAPI / attempt
GPT-5.6 Terra4.94.8–5.04.54.2–4.8100%2 / 335s$0.058
GPT-5.6 Sol4.84.8–4.94.94.9–4.9100%2 / 31m 56s$0.082
GPT-5.6 Luna4.74.6–5.04.34.2–4.4100%0 / 351s$0.010
GPT-5.54.84.5–5.04.44.0–4.793%0 / 355s$0.19
GPT-5.44.84.8–4.94.24.1–4.3100%0 / 31m 7s$0.15
GPT-5.4 Mini1/3 complete documents4.84.8–4.84.64.6–4.6100%0 / 31m 37s$0.085
GPT-6 Astra4.94.9–5.05.04.9–5.0100%3 / 31m 49s$0.32

Writing and design show the mean and range across attempts. File checks cover page count, fit, minimum text size and word limit. Visual design covers hierarchy, typography, charts and layout. Speed and cost include unsuccessful attempts.

See the actual outputs Attempt 1 · every model
GPT-5.6 Terra 1 pages
GPT-5.6 Sol 1 pages
GPT-5.6 Luna 1 pages
GPT-5.5 1 pages
GPT-5.4 1 pages
GPT-5.4 Mini 1 pages
GPT-6 Astra 1 pages
How these tests work AI-reviewed · human validation pending

Each model received the same new synthetic brief with verified facts and the same light PSV design requirements. The assignment requested a complete HTML document at high reasoning effort with a 20,000 output-token allowance, including reasoning. Exhausting that allowance is a failed attempt; a larger allowance may change the result. Pages were rendered at fixed document or slide dimensions with scripts and network access disabled. No manual repairs were made.

Two model-anonymous vision reviewers (gpt-5.4 and gpt-5.5) scored the actual page images and extracted text on five writing and five design criteria. Scores run from 1 (unusable) to 5 (ready for the stated audience). Facts kept requires both judges to credit a check; clipped facts do not earn credit. File checks are automated requirements; visual design is judged from the rendered pages. Judge fact agreement: 99.6% across 260 comparisons. Both reviewers are also contestants in this test (GPT-5.4 and GPT-5.5), so each grades its own output alongside the others.

These are small-sample, same-vendor AI reviews; differences are provisional and have not been validated by CRE professionals. Scores measure communication of supplied facts, not independent underwriting, PDF extraction, Excel formulas or PowerPoint editing. Example images always use attempt 1.

Speed measures API attempt time, including unsuccessful attempts and excluding rendering and judging. Costs use the list prices in the underwriting method’s price table, exclude review costs in the table, and are estimates rather than a billing statement. Recorded generation: $10.93; grading: $4.71.

Files & revisions

6 synthetic deals · pilot results

Can the model carry the right numbers from source files to the final recommendation?

  1. 01ReadScanned brief + Excel
  2. 02ReconcileRent roll + amendments
  3. 03BuildLinked workbook + IC slide
  4. 04ReviseUpdate numbers + decision

Harder cases · September 18

3 deals · 2 models
ModelHarbor CourtProceed → DeclineCedar CommerceDecline → ProceedCypress OfficeExact thresholdTypical workflowtwo API callsCost / workflowtwo API calls
GPT-5.6 LunaPassed → PassedPassed → PassedPassed → Passed11s$0.003
GPT-6 AstraPassed → IncompletePassed → PassedPassed → Passed18sUnknown

Initial → revision. Passed = every automated check. $0.25 recorded spend; 1 incomplete call with unknown cost; $2.00 batch cap.

Checks & actual files
GPT-5.6 Luna

cedar-commerce · initial

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

cedar-commerce · revision

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

cypress-office · initial

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

cypress-office · revision

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

harbor-court · initial

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

harbor-court · revision

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗
GPT-6 Astra

cedar-commerce · initial

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

cedar-commerce · revision

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

cypress-office · initial

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

cypress-office · revision

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

harbor-court · initial

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

harbor-court · revision

API did not complete; no correctness result.

Original pilot · 3 simpler deals

Original pilot · September 14

3 deals · 2 models
ModelMesa CourtMultifamilyPortside FlexIndustrialJuniper PlazaOfficeTypical workflowtwo API callsCost / workflowtwo API calls
GPT-5.6 LunaMissed → PassedPassed → PassedPassed → Missed19s$0.003
GPT-6 AstraPassed → PassedPassed → PassedPassed → Passed20s$0.096

Initial → revision. Passed = every automated check. $0.30 recorded spend; $5.00 batch cap.

Checks & actual files
GPT-5.6 Luna

juniper-plaza · initial

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

juniper-plaza · revision

Missed: source extraction, spreadsheet calculations, slide figures, memo figures, slide text

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

mesa-court · initial

Missed: spreadsheet calculations, linked assumptions

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

mesa-court · revision

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

portside-flex · initial

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

portside-flex · revision

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗
GPT-6 Astra

juniper-plaza · initial

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

juniper-plaza · revision

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

mesa-court · initial

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

mesa-court · revision

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

portside-flex · initial

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗

portside-flex · revision

All automated checks passed.

Excel workbook ↗ · PowerPoint slide ↗ · Slide preview ↗ · IC memo ↗
Scope & limits

Models receive one scanned synthetic brief as an image and raw cells extracted from an XLSX workbook. Harder cases add signed amendments that override stale rent rolls, misleading broker summaries, missing lease expiries, both directions of recommendation changes, and an exact valuation threshold. These are prescribed screening rules, not complete investment decisions.

Each model gets one initial call and one revision per deal, medium effort, a 3,500-token ceiling and no automatic retry. Failed API calls remain incomplete; their unknown charges retain the full budget reservation. Typical workflow time uses only pairs with both calls timed. Cost per workflow is unavailable when a charge is unknown.

Shared tools create editable XLSX and PPTX files from model-supplied inputs, formulas and text. Checks cover calculations, source precedence, missing facts, recommendations, memo/slide figures, linked formulas and export/reimport. The same layout serves every model; visual design is not scored here. Prose checks verify required figures, not every factual claim.

These small samples cannot establish a general winner. Full native file ingestion/editing, long or damaged scans, DOCX and desktop Office behavior remain untested. Original and harder batches stay separate. Anthropic results appear only after actual calls with configured credentials.

Prices: Luna, Astra; see also the price table. Batch prices checked September 18, 2026; uncached token estimates, not billing statements. No paid judges or tools; local file-tool time and machine cost are excluded.

Reliability & total cost

All recorded attempts

Finished is not the same as correct. Use task accuracy to judge the result.

Underwriting

27 attempts / model
ModelFinishedAPI outputsTypical waitmedianSlow wait90th percentileAPI spendall attemptsCost / finishedincludes failed attemptsRetries
GPT-6 Astra27 / 273m 22s4m 21s$14.42$0.53Not recorded
GPT-5.6 Terra27 / 272m 42s3m 56s$4.18$0.15Not recorded
GPT-5.6 Sol27 / 272m 55s3m 41s$3.59$0.13Not recorded
GPT-5.6 Luna27 / 273m 11s3m 42s$0.59$0.022Not recorded
GPT-5.527 / 274m 24s7m 17s$14.02$0.52Not recorded
GPT-5.427 / 273m 20s4m 35s$9.15$0.34Not recorded
GPT-5.4 Mini26 / 273m 53s6m 28s$5.11$0.20Not recorded

Writing & design

9 attempts / model
ModelFinishedAPI outputsTypical waitmedianSlow wait90th percentileAPI spendall attemptsCost / finishedincludes failed attemptsRetries
GPT-6 Astra9 / 92m 49s3m 34s$4.05$0.45Not recorded
GPT-5.6 Terra9 / 945s1m 3s$0.70$0.078Not recorded
GPT-5.6 Sol9 / 91m 56s2m 17s$0.79$0.088Not recorded
GPT-5.6 Luna9 / 91m 9s1m 15s$0.10$0.011Not recorded
GPT-5.59 / 91m 23s1m 45s$2.41$0.27Not recorded
GPT-5.46 / 91m 36s2m 29s$2.08$0.35Not recorded
GPT-5.4 Mini1 / 92m 15s2m 30s$0.80$0.80Not recorded
What is included

Finished means the API returned a completed, nonempty response. It does not require correct facts, valid formatting or every benchmark check to pass. Cost per finished output divides recorded generation spend, including unsuccessful attempts, by completed outputs. It is not cost per correct answer.

Waits cover API generation, including retries where the original timer captured them. Nearest-rank percentiles describe these small samples, not a service guarantee. Historical retry counts and uncaptured retry charges are unknown. Costs use each run’s saved token prices (listed with sources in the price table), with no subscription, human-effort or local-compute allocation. Paid evaluator costs are reported separately in each suite’s methodology. No external paid tools were used.

Original underwriting test

2 synthetic deals · 185 graded responses · September 3 to 13, 2026
Method & limitations Text inputs · API testing only

Method

How a memo is graded, and what this bench cannot tell you.

What a memo is

Each model receives the same plain-text documents and firm assumptions for each task, across nine assignments on two synthetic deals. The documents and answer keys are generated from the same underlying data model. Each response contains a written analysis, structured figures and a recommendation. No tools, retrieval, code execution, spreadsheet creation or raw-PDF reading are tested. The configured reasoning effort was high, with a 64,000-token output cap that includes reasoning.

How it is graded

Every memo is scored against a predefined answer key: 225 checks in total, covering figures inside a stated tolerance, findings a good analyst makes, mistakes a good analyst avoids, and one decision. Findings and mistakes are read by two OpenAI judge models, which are also contestants: gpt-5.5-2026-04-23 (the GPT-5.5 contestant) and gpt-5.4-2026-03-05 (the GPT-5.4 contestant). They are not independent of the models they grade, and no judge from another vendor has been used yet. They agreed on 97.5% of 2,775 judgements (Cohen's kappa 0.81 on findings and 0.63 on mistakes: substantial agreement). For the clean-memo score, a finding must be credited by both judges and a mistake must be flagged by neither, where a second judgment exists. Second-judge comparisons cover 170 of 185 graded responses. The rest are 14 lease-up reforecast & take-out memos (GPT-5.6 Sol 2, GPT-5.6 Luna 3, GPT-5.4 3, GPT-5.5 3, GPT-5.4 Mini 3) and 1 empty response, which have no second judgment. Without a second verdict the first judge decides; failed responses still fail. The Numbers correct and Risk & decision checks columns report individual-check pass rates; writing quality is scored separately.

The trials

The plan calls for three attempts per model and task. Repeats measure variation on the same inputs, not performance on different deals. That is 7 models × 9 tasks × 3 tries = 189 attempted memos, of which 185 were graded. 4 memos were returned and billed but never scored because grading failed (judge API quota exhausted): GPT-5.6 Sol on the lease-up reforecast & take-out (1 try), GPT-5.6 Terra on the lease-up reforecast & take-out (3 tries). They are excluded from the performance figures; the task comparison shows each incomplete denominator. An attempt without a usable memo scores zero on every check rather than counting as missing: 1 response was empty (output token limit); 2 memos omitted the required JSON block, out of 185 graded attempts.

The rotation

Deal set v1 is live from September 2026. A fresh deal set and answer key are planned for December 2026. A changed test set is not a like-for-like comparison. The packages are synthetic; the page publishes selected answer-key excerpts and model responses. This does not establish that the tasks are immune to training-data overlap.

What this bench does not tell you

  • It measures nine assignments on two deals. It is evidence, not a guarantee, and a firm's own documents will behave differently.
  • Three tries per task cannot establish a reliable general advantage: on every task, every pair of models' exact 95% full-pass intervals overlaps. The pooled intervals in the appendix resample tasks and then attempts; they describe this small test set, not variation across real deals.
  • The answer key encodes a predefined set of assumptions and grading rules. Where a model disagreed defensibly, it still scored wrong. Some numerical tolerances are wide (±45 bps on spread to the exit cap rate; ±12% on in-balance shortfall after draw; ±0.08x on debt coverage at 60% LTV), which raises the Numbers rates.
  • This run uses the configured reasoning effort (high) through one route. Prompt engineering, tools and retrieval would all change these numbers.
  • OpenAI's models only in this run. Anthropic and other vendors not yet tested. The judges come from the same vendor and are also contestants.
  • Runs recorded September 3 to 13, 2026; per-model dates and model ids are in the table below. Cost applies the list prices below; the price used for GPT-5.6 Sol is not confirmed. Speed is observed API latency. Both can change.

Prices behind every cost figure

ModelAPI model idInput$ / 1M tokensOutput$ / 1M, incl. reasoningSourcePrice checked
GPT-5.6 SolUnconfirmed. On 2026-09-26 the source listed $4 input / $20 output per 1M tokens (promotional pricing), not the $2 / $10 recorded here and used for these runs' cost figures; Sol's costs may be understated by about half.gpt-5.6-sol$2.00$10.00Model page ↗Not confirmed
GPT-6 Astragpt-6-astra$10.00$50.00Model page ↗September 13, 2026
GPT-5.6 Terragpt-5.6-terra$2.00$12.00Model page ↗September 26, 2026
GPT-5.6 Lunagpt-5.6-luna$0.20$1.20Model page ↗September 26, 2026
GPT-5.4gpt-5.4$2.50$15.00Model page ↗September 26, 2026
GPT-5.5gpt-5.5$5.00$30.00Model page ↗September 26, 2026
GPT-5.4 Minigpt-5.4-mini$0.75$4.50Model page ↗September 26, 2026

Standard, uncached list prices applied to the token counts each API call reported; output includes reasoning tokens. Costs are estimates, not billing statements, and exclude subscriptions, tools, retries and grading. The same prices drive the document-design and reliability cost columns. GPT-5.6 Sol: price used for these runs not confirmed; see the row note.

Models as tested

ModelAPI model idas returnedSnapshotRun datesReasoning effortOutput captokens, incl. reasoningRoute
GPT-5.6 Solgpt-5.6-solAlias; no dated snapshot exposedSeptember 3 to 4, 2026 (UTC)high64,000OpenAI Responses API
GPT-6 Astragpt-6-astraAlias; no dated snapshot exposedSeptember 13, 2026 (UTC)high64,000OpenAI Responses API
GPT-5.6 Terragpt-5.6-terraAlias; no dated snapshot exposedSeptember 3 to 4, 2026 (UTC)high64,000OpenAI Responses API
GPT-5.6 Lunagpt-5.6-lunaAlias; no dated snapshot exposedSeptember 3 to 4, 2026 (UTC)high64,000OpenAI Responses API
GPT-5.4gpt-5.4-2026-03-05March 5, 2026September 3 to 4, 2026 (UTC)high64,000OpenAI Responses API
GPT-5.5gpt-5.5-2026-04-23April 23, 2026September 3 to 4, 2026 (UTC)high64,000OpenAI Responses API
GPT-5.4 Minigpt-5.4-mini-2026-03-17March 17, 2026September 3 to 4, 2026 (UTC)high64,000OpenAI Responses API

The API model id is the one each response reported. An alias without a dated snapshot can change behind the same name, so a later run of it may not reproduce these results. Every attempt used the same system prompt, documents and task wording. The output cap was reached once: GPT-5.4 Mini, feasibility memo, try 1.

OpenAI, GPT and the model names shown are trademarks of OpenAI; Claude is a trademark of Anthropic. PSV is not affiliated with or endorsed by either and uses these names only to identify the products tested. PSV builds client systems on Anthropic's Claude.

What does passed every check mean?

Every required figure, finding, mistake-avoidance check and decision passed for that attempt, and the response was usable. One missed check fails the attempt. This does not measure the financial impact of the error or remove the need for human review.

What counts as good enough?

This test does not establish a production-readiness threshold. Use the task results to choose a candidate, inspect its misses, and validate it against your own documents and review standards.

How can the numbers be right when no attempt passes every check?

Numerical accuracy counts figures inside the answer key’s tolerance. A full pass also requires the findings, mistake-avoidance checks and decision to pass. A response can get every tested number right and still miss an important observation.

How is the strongest result selected?

The highest full-pass card names every model with all planned attempts graded that reached the top full-pass count on that task. Ties are listed together and are never broken by cost or by check rates, and every count carries an exact 95% interval. At three attempts per model, no difference between models on a single task is statistically reliable. Incomplete samples appear last and remain available for inspection.

Does lowest cost mean best value?

No. It is the lowest measured API charge per response among complete samples. The comparison still shows the model’s accuracy and full-pass results. Human correction time, tool costs and retry costs have not been measured here.

Could a model have seen these deals before?

The deals are synthetic, generated from a shared data model. Selected answer-key excerpts and model responses are published here. We have not established that these tasks are immune to training-data overlap.

Technical appendix: every statistic behind the page

Per-model statistics

ModelSoft score95% CIBoth judgesHard checks95% CIClean, pooled95% CIClean, one judge≤1 missMisses / memoWorst memo
GPT-5.6 Sol 96.294.0–98.4 95.2 83.473.1–93.3 42.314.8–70.4 46.261.5 1.1590
GPT-6 Astra 95.392.4–97.9 94.4 79.665.9–91.9 40.711.1–70.4 40.759.3 1.4186
GPT-5.6 Terra 87.271.9–96.2 86.1 72.657.2–86.0 33.312.5–58.3 37.554.2 3.540
GPT-5.6 Luna 95.392.8–97.5 94.4 79.968.2–90.0 33.37.4–63.0 33.370.4 1.3783
GPT-5.4 95.893.0–98.0 94.1 80.366.2–91.5 25.93.7–51.9 37.074.1 1.4481
GPT-5.5 95.793.8–97.4 94.9 80.571.2–88.8 25.93.7–51.9 25.966.7 1.2690
GPT-5.4 Mini 87.576.9–93.2 85.5 59.147.4–69.9 3.70.0–14.8 3.733.3 3.560

Clean, pooled combines graded attempts across all assignments. The task guide above uses each assignment’s own results and denominator. These pooled figures describe this test set; they are not a recommendation for every CRE workflow. Intervals are two-stage bootstrap intervals: resample the nine tasks, then attempts within each drawn task. Attempts on one task are correlated, so this is wider than resampling attempts alone.

Task saturation

TaskChecksPassed by everyoneMean pass rate
Variance report & reforecastHarbor & Fifth: Q1 variance report, reforecast and LP commentary231889.9%
Broker opinion of valueHarbor & Fifth: broker opinion of value for the owner201593.3%
IC screening memoHarbor & Fifth: full screen and preliminary underwrite from the data room353098.6%
OM readHarbor & Fifth: read the OM like a senior person181498.6%
Rent roll analysisHarbor & Fifth: parse and analyze the rent roll272292.1%
Underwriting assumptionsHarbor & Fifth: build the Year-1 assumption set from the rent roll and T-12292395.4%
Feasibility memoPalomar Yards: JV equity feasibility on the firm basis262097.5%
Construction draw reviewPalomar Yards: review construction draw 14 and run the in-balance test241592.5%
Lease-up reforecast & take-outPalomar Yards: six months into lease-up: reforecast stabilization and size the take-out231796.4%

Memos returned but never graded

  • GPT-5.6 Sol · Lease-up reforecast & take-out · try 3 · judge API quota exhausted
  • GPT-5.6 Terra · Lease-up reforecast & take-out · try 1 · judge API quota exhausted
  • GPT-5.6 Terra · Lease-up reforecast & take-out · try 2 · judge API quota exhausted
  • GPT-5.6 Terra · Lease-up reforecast & take-out · try 3 · judge API quota exhausted

These 4 memos completed and were billed, and are inside the spend figure, but grading failed, so they do not contribute to the performance figures. Their missing coverage is shown in the task comparison.

Judge agreement

gpt-5.5-2026-04-23 (the GPT-5.5 contestant) and gpt-5.4-2026-03-05 (the GPT-5.4 contestant) scored 2,775 judgement items across 170 memos. Both kinds of item are compared on one pass/fail scale (finding made, mistake avoided). They agreed on 97.5% of items; pooled Cohen's kappa 0.806. Findings: 96.4% agreement over 1,758 items, kappa 0.809 (substantial). Mistakes: 99.3% over 1,017, kappa 0.628 (substantial); mistakes are rare, so a few disagreements move kappa a long way. gpt-5.5-2026-04-23 passed an item the other failed 53 times; gpt-5.4-2026-03-05 did so 17 times.

The assignments

  • IC screening memo: Harbor & Fifth: full screen and preliminary underwrite from the data room · 10 documents · Acquisitions
  • OM read: Harbor & Fifth: read the OM like a senior person · 3 documents · Acquisitions
  • Rent roll analysis: Harbor & Fifth: parse and analyze the rent roll · 2 documents · Acquisitions
  • Underwriting assumptions: Harbor & Fifth: build the Year-1 assumption set from the rent roll and T-12 · 4 documents · Acquisitions
  • Feasibility memo: Palomar Yards: JV equity feasibility on the firm basis · 8 documents · Development
  • Construction draw review: Palomar Yards: review construction draw 14 and run the in-balance test · 6 documents · Development
  • Lease-up reforecast & take-out: Palomar Yards: six months into lease-up: reforecast stabilization and size the take-out · 5 documents · Development
  • Variance report & reforecast: Harbor & Fifth: Q1 variance report, reforecast and LP commentary · 4 documents · Asset management
  • Broker opinion of value: Harbor & Fifth: broker opinion of value for the owner · 8 documents · Brokerage