Overview
This benchmark addresses a critical gap in evaluating frontier AI agents: real-time strategic decision-making under uncertainty and partial observability. StarCraft BW is an excellent testbed because it forces LLMs to integrate long-horizon planning, resource management, and real-time execution—skills that translate directly to robotics, autonomous systems, and complex software engineering tasks. The project creates value by providing transparent, comparable results across major labs (OpenAI, Anthropic, Google, Muse), accelerating the field's ability to measure progress. It could evolve into a recurring competition series, attract sponsorship from hardware/cloud providers, or become the standard reference for "can AI beat humans at a complex real-time game?" The opportunity is particularly strong if positioned as a research platform rather than just a static leaderboard.
Score Breakdown
| Advisor 8.0 | Devil -7.0 | Historian 4.0 | Budget Steward 0.0 | Founder 7.0 | ⭐ Total Score 4.2/10
What They Do
I've refreshed StarSkirmish Bench, a benchmark where LLMs play StarCraft BW by writing code, with results from the flagship models of OpenAI, Anthropic, Google, and Muse
Why It Matters
This benchmark addresses a critical gap in evaluating frontier AI agents: real-time strategic decision-making under uncertainty and partial observability. StarCraft BW is an excellent testbed because it forces LLMs to integrate long-horizon planning, resource management, and real-time execution—skills that translate directly to robotics, autonomous systems, and complex software engineering tasks. The project creates value by providing transparent, comparable results across major labs (OpenAI, Anthropic, Google, Muse), accelerating the field's ability to measure progress. It could evolve into a recurring competition series, attract sponsorship from hardware/cloud providers, or become the standard reference for "can AI beat humans at a complex real-time game?" The opportunity is particularly strong if positioned as a research platform rather than just a static leaderboard.
Business Model
To be determined based on project analysis.
CRP Decision Report
🧠 Advisor
This benchmark addresses a critical gap in evaluating frontier AI agents: real-time strategic decision-making under uncertainty and partial observability. StarCraft BW is an excellent testbed because it forces LLMs to integrate long-horizon planning, resource management, and real-time execution—skills that translate directly to robotics, autonomous systems, and complex software engineering tasks. The project creates value by providing transparent, comparable results across major labs (OpenAI, Anthropic, Google, Muse), accelerating the field's ability to measure progress. It could evolve into a recurring competition series, attract sponsorship from hardware/cloud providers, or become the standard reference for "can AI beat humans at a complex real-time game?" The opportunity is particularly strong if positioned as a research platform rather than just a static leaderboard.
[score: 8]
[score: 8.0]
🧨 Devil
Risk 1: Sustainability and maintenance burden. Benchmarks require constant updating as new models emerge, bug fixes as edge cases are discovered, and infrastructure to host experiments. Who maintains this when the novelty wears off? Without a dedicated team or funding, the benchmark risks becoming stale within months.
Risk 2: Narrow technical focus limits transferability. StarCraft BW is a specific, somewhat dated game environment. Progress here may not correlate well with general AI capability or real-world applicability. Researchers and funders might view this as a parlor trick rather than a meaningful capability measure, especially compared to benchmarks like MMLU or HumanEval.
Risk 3: Compute cost scaling. Running experiments for multiple models across multiple maps and difficulties requires significant GPU/CPU time. If the benchmark becomes popular and more teams enter, hosting costs could explode, potentially pricing out smaller labs and reducing diversity of participants.
Risk 4: Existing benchmark competition. Projects like AutoGym, AgentBench, and various RL environments already test LLMs in game-like settings. Differentiation is unclear—why would anyone use this over alternatives?
[score: -7]
[score: -7.0]
📚 Historian
Historical pattern: Game-playing benchmarks (Chess, Go, Atari, StarCraft) have repeatedly served as milestone markers for AI progress. StarCraft specifically saw a dramatic inflection point when AlphaStar demonstrated superhuman performance in 2019, but subsequent progress has been slower and less publicly visible. The "LLMs write code to play games" pattern has precedent in projects like GPT-3/4 playing Python games, but a structured benchmark with multiple labs is rare. Similar multi-lab benchmarking efforts (like MMLU, HumanEval) have succeeded by becoming community standards through rigorous methodology and transparency. However, many niche benchmarks fade after initial attention—typically within 12-18 months without active maintenance and community buy-in. The key success factor historically is whether the benchmark becomes a required citation in papers; if so, it achieves network effects and longevity.
[score: 4]
[score: 4.0]
🧮 Budget Steward
Cost breakdown for a lean team (≤2 people):
- Infrastructure (3 months): Cloud GPU instances for running baseline experiments + hosting results webpage: ~300/month × 3 = 900
- Development/maintenance: Minimal—likely a static site with data updates. If outsourcing any frontend polish: $500-1,000 (one-time)
- Third-party services: Domain, basic analytics: $50
- Contingency for expanding model support: Additional compute if new labs request runs: $500 buffer
- Total estimated cost: ~$1,950-2,450
This fits well within the $15,000 budget with significant room for error or expansion. No significant funding gap exists. The project is essentially low-overhead if kept lightweight.
[score: 0]
[score: 0.0]
🧱 Founder
We should build this, but with clear scope boundaries. The opportunity is real: a transparent, multi-lab benchmark for a complex real-time strategy game fills an unoccupied niche. The risks are manageable if we treat this as a research tool rather than a commercial product.
Approach:
- Phase 1 (Months 1-2): Publish the v0.2 benchmark with current results. Focus on documentation and reproducibility.
- Phase 2 (Months 3-4): Establish a submission process for new model results. Build a simple API for programmatic access.
- Phase 3 (Months 5-6): Pursue partnerships with 1-2 cloud providers for compute sponsorships to offset costs.
Timeline: MVP launch in 2 months, first paying customers (sponsorships) by month 6 if partnerships materialize.
Key decision: This is a credibility play, not a revenue play. Success is measured by citations and community adoption, not direct income. If that aligns with our goals, we build.
[score: 7]
[score: 7.0]
Why This Made Only 10
This project was selected because it scored highly across our evaluation framework. The CRP analysis confirmed strong potential, making it one of the most promising opportunities this week.