跳到主要内容
aiomniuaiomniu
only10

V26.02:cactus hybrid: we taught gemma 4 to know when it's wrong

Overview

This project introduces a practical, lightweight method to improve hybrid AI systems by giving small on-device models a reliable confidence signal. The probe achieves 0.814 AUROC across modalities, significantly outperforming entropy-based heuristics (0.549). The value is clear: it enables cost-efficient routing without sacrificing quality, matching Gemini Flash-Lite on many benchmarks while routing only 15–35% of queries to the cloud. The MIT license and cross-framework support (Transformers, MLX, llama.cpp) lower adoption barriers. The modality-independence result (trained on text/vision, working on audio) suggests a generalizable insight, making this a potentially foundational building block for edge AI. If successful, it could reduce cloud inference costs for developers and improve privacy. The team has published weights and code, and the open-source community can iterate quickly. This is a high-leverage project with a clear pain point.

Score Breakdown

| Advisor 8.0 | Devil -6.0 | Historian 5.0 | Budget Steward 0.0 | Founder 8.0 | ⭐ Total Score 5.2/10

What They Do

Hey HN, Henry & Roman here from Cactus. A small, on-device model is fast and private, but sometimes wrong, but frontier models are getting expensive pretty fast. So, we post-trained Gemma 4 E2B post-trained to know when it's wrong. Every response comes with a confidence score between 0 and 1. Developers can accept the on-device when it's high, hand off to a bigger cloud model when it's low. By routing only 15-35% of queries to Gemini 3.1 Flash-Lite, Gemma-4-E2B matches Gemini 3.1 Flash-Lite on most benchmarks. - ChartQA: 15-20% - LibriSpeech: 25-30% - MMBench, GigaSpeech, MMAU: 30-35% - MMLU-Pro: 45-55% We were always frustrated by the routing signals hybrid apps rely on: asking the model to rate itself in text (unreliable, and you're parsing prose), or token entropy heuristics (barely better than a coin flip in our tests). So we did mechanistic studies on small models, Gemma 4 particularly, and found the hidden state for different layers carry meaningful self-awareness signal for various situations. SO we extended the model with a 68k params probe layer (LayerNorm, low-rank projection, attention pooling, small MLP head) reads one intermediate layer during decoding and predicts p(wrong); confidence = 1 - p(wrong), returned as structured data, never parsed out of the answer text. Across 12 hold-out benchmarks spanning text, vision and audio, the probe averages 0.814 AUROC vs 0.549 for token entropy. The result that convinced us this is real: the probe was trained on zero audio data, yet scores 0.79-0.88 AUROC on four audio benchmarks where entropy is near-random or worse (0.32-0.52). It's reading a modality-independent correctness signal from the hidden state, not memorizing patterns from its training data. We published all weights on HuggingFace and provide copy-pase codes to run it on Transformers, MLX, Llama.cpp or Cactus. With Ollama, vLLM, SGLang etc in the works. For llama.cpp we ship a patch series you compile in once (upstreaming is planned). The code is MIT licensed; Gemma model use remains subject to the Gemma terms. GitHub: https://github.com/cactus-compute/cactus-hybrid Weights: https://huggingface.co/collections/Cactus-Compute/cactus-hyb... Some caveats: - The probe scores single-sequence decoding only, up to the first 1024 generated tokens. - Handoff works best when routing per task in a multi-step process, not per step. - Hierarchical routing is still in the works: try on-device, then DeepSeek v4 Flash, before Fable/GPT5.5/Gemini/Muse/Grok. - The technique is boutique for each model, we will share each weights as they roll out. These issues are currently being tackled at Cactus and updated weights will be shipped directly into the HuggingFace collection and GitHub repository straight up. Please let us know your thoughts, it helps us find ways to improve the design progressively. Thanks a million!

Why It Matters

This project introduces a practical, lightweight method to improve hybrid AI systems by giving small on-device models a reliable confidence signal. The probe achieves 0.814 AUROC across modalities, significantly outperforming entropy-based heuristics (0.549). The value is clear: it enables cost-efficient routing without sacrificing quality, matching Gemini Flash-Lite on many benchmarks while routing only 15–35% of queries to the cloud. The MIT license and cross-framework support (Transformers, MLX, llama.cpp) lower adoption barriers. The modality-independence result (trained on text/vision, working on audio) suggests a generalizable insight, making this a potentially foundational building block for edge AI. If successful, it could reduce cloud inference costs for developers and improve privacy. The team has published weights and code, and the open-source community can iterate quickly. This is a high-leverage project with a clear pain point.

Business Model

To be determined based on project analysis.

CRP Decision Report

🧠 Advisor

This project introduces a practical, lightweight method to improve hybrid AI systems by giving small on-device models a reliable confidence signal. The probe achieves 0.814 AUROC across modalities, significantly outperforming entropy-based heuristics (0.549). The value is clear: it enables cost-efficient routing without sacrificing quality, matching Gemini Flash-Lite on many benchmarks while routing only 15–35% of queries to the cloud. The MIT license and cross-framework support (Transformers, MLX, llama.cpp) lower adoption barriers. The modality-independence result (trained on text/vision, working on audio) suggests a generalizable insight, making this a potentially foundational building block for edge AI. If successful, it could reduce cloud inference costs for developers and improve privacy. The team has published weights and code, and the open-source community can iterate quickly. This is a high-leverage project with a clear pain point.

[score: 8]

[score: 8.0]

🧨 Devil

The project faces several risks. First, the probe is only validated for single-sequence decoding up to 1024 tokens; real-world applications often require longer generation or multi-turn conversations, which may break the confidence signal. Second, the claimed 0.814 AUROC is an average across 12 benchmarks, but individual benchmarks may vary widely, and the hold-out set might not cover all edge cases. Third, the routing strategy assumes a clear threshold for handoff, but in practice, the optimal threshold is task-dependent and may require tuning, increasing complexity. Fourth, the technique is "boutique for each model" – porting to new models (e.g., future Gemma versions or other architectures) requires new probe training, limiting scalability. Fifth, the hierarchical routing (on-device → DeepSeek → Fable) is still in development, so the current MVP is only a two-layer system, which may not be compelling enough for some use cases. Sixth, the project relies on Google's Gemma terms, which could change or restrict commercial usage. Finally, competition from first-party solutions (e.g., Google's own confidence or uncertainty estimation) could render this approach obsolete.

[score: -6]

[score: -6.0]

📚 Historian

Historically, the pattern of "small model + confidence routing" mirrors early attempts at cascading systems (e.g., Microsoft's "Model Cascades" in 2021, or Google's "Selective Prediction" papers). However, those were often trained on task-specific datasets and didn't generalize well. The closest analog is the "conformal prediction" applied to deep learning, which provides uncertainty sets but is computationally expensive at inference. Projects like "Cascade R-CNN" in vision show that cascading can improve accuracy, but they require careful threshold tuning. In the open-source space, "LLM routing" startups (e.g., Portkey, OpenRouter) have focused on user-controlled routing, not model-intrinsic confidence. The unique selling point here is the mechanistic interpretability approach – using hidden states as a modality-agnostic signal. This is reminiscent of how "BERT loss" probes were used for confidence in NLP, but extending it to audio and vision is novel. The market has shown strong interest in cost-efficient AI; recent adoption of "tiny models" (Phi-3, Gemma 2) and hybrid systems suggests a favorable pattern. However, many similar academic projects never reached commercial adoption due to engineering complexity. The MIT license helps, but the Gemma dependency may limit distribution.

[score: 5]

[score: 5.0]

🧮 Budget Steward

We estimate the cost to go from zero to first paying customer using this open-source project as a base. The MVP is a simple API service that hosts the Gemma-4-E2B model with the probe, accepts text/audio/vision inputs, returns confidence scores, and optionally routes to Gemini Flash-Lite when confidence is low. We assume a 2-person team (indie developer + part-time support) and a 3-month runway to first paid customer.

  • Infrastructure (3 months): GPU instance for Gemma-4 inference (e.g., 1x NVIDIA A10G on AWS, ~$0.80/hr) running 24/7 for dev and testing, plus a small CPU instance for API gateway and routing. Total ~$1,800. Lower-cost alternative: CPU-only inference (Gemma-4 small variant) would cut to ~$300, but may be too slow. We'll use the CPU scenario for a lean MVP, assuming the model fits in a 16GB RAM machine (Gemma-4 2B?). Let's assume CPU: $0.05/hr for a dedicated instance, 3 months = $108. DNS, SSL, database: ~$30. Total infrastructure: ~$140.
  • Development cost (outsourced market rate): The probe code, inference scripts, and integration are already provided. We need to build a REST API (Flask/FastAPI), authentication, usage tracking, and a simple billing system. Estimate 2 weeks of full-time work for a skilled developer at $100/hr = 80 hours * $100 = $8,000. But since we are indie, we can do it ourselves, so cost $0 if we count our own time as free (excluded per rules). The rules say "exclude: marketing costs, your own time opportunity cost." So development cost is $0.
  • Third-party services: Gemini API calls for routing. At 15–35% of queries, initial usage will be low. We can start with a free tier (Gemini Flash-Lite has a free quota of 60 requests/minute). For first paying customer, we might not exceed free tier, so cost $0. Payment processing (Stripe): $0 setup, $0.30 per transaction + 2.9% – negligible for first customer. Legal templates for privacy policy and terms of service: $100 (e.g., using a template). Developer accounts (Apple/Google if needed for mobile): not needed for API-only. Compliance: Gemma terms require attribution – no cost.
  • Total cost: Infrastructure $140 + third-party services $100 = $240. This is well under $1,600. However, we must also consider that the Gemma model itself, while open weights, may require agreement to terms that could impose a fee if used commercially. But no fee is mentioned. So total cost is ~$240, which is <= $1,600. Therefore score = +1.

[score: +1]

🧱 Founder

This is a compelling project with a clear value proposition and a strong technical foundation. The cost to reach first paying customer is extremely low, the open-source release reduces risk, and the market is hungry for cost-efficient AI solutions. The main challenge is whether the probe's reliability holds in production and whether the routing logic is simple enough for developers to adopt. Given the caveats (single-sequence, limited token length, model-specific training), the first version should target a narrow use case, such as a "confidence-aware chatbot" for short queries. The timeline: 1 month to build a minimal API and attract early users (e.g., on Product Hunt), 2 months to iterate based on feedback, and secure first paying customer by month 3. The team already has the core technology, so execution is about packaging and distribution. I recommend building this. The potential upside (reducing cloud inference costs by 65–85%) is significant, and the low barrier to entry makes it a smart bet.

[score: 8]

[score: 8.0]

Why This Made Only 10

This project was selected because it scored highly across our evaluation framework. The CRP analysis confirmed strong potential, making it one of the most promising opportunities this week.