Create hypothesis generation agent and benchmark techniques #24

Summary

Build a dedicated hypothesis generation agent that can use multiple strategies, compare their effectiveness, and evolve the best approach over time. Benchmark literature-based, generative, and hybrid techniques against real backtest results.

This is the "Hypothesis Agent" from docs/RESEARCH_AGENT_DESIGN.md Phase 2, extended to support multiple generation methods.

Motivation

We need to evaluate which hypothesis generation approach works best:

• Literature-based (Phase 1: search papers, extract ideas)
• Generative (Issue Experiment: Generative RecSys + HyDE for hypothesis generation #23: HyDE-style synthetic hypotheses)
• Grid-aware (Extend parameter grid analytically)
• Hybrid (Combine all three with weighted voting)

Instead of guessing, build an agent that:

1. Generates hypotheses using each technique
2. Backtests all proposals
3. Tracks which techniques produce winning hypotheses
4. Learns over time which approach works best per regime

Task

1. Design HypothesisAgent interface supporting multiple generators:

   • LiteratureBasedGenerator (existing Phase 1)
   • GenerativeHyDEGenerator (Issue Experiment: Generative RecSys + HyDE for hypothesis generation #23)
   • GridAnalyticalGenerator (new: extend grid heuristically)
   • EnsembleGenerator (vote across all three)

2. Implement hypothesis generation pipeline:

   • Input: regime context, market data, prior results
   • Output: scored hypotheses with confidence intervals

3. Benchmark on real data:

   • Run each technique on 5-10 historical regimes
   • Measure: Sharpe, generalization gap, discovery rate (novel hypotheses)
   • Track: false positive rate, backtest time

4. Create reporting dashboard:

   • Which technique wins per regime?
   • Technique accuracy over time (learning curve)
   • Cost/benefit trade-offs

Files to Create/Modify

• src/agent/hypothesis_agent.py — Main agent orchestrating generators (NEW)
• src/agent/generators/base.py — Generator interface (NEW)
• src/agent/generators/literature.py — Literature-based (from Phase 1)
• src/agent/generators/generative.py — Generative HyDE (from Experiment: Generative RecSys + HyDE for hypothesis generation #23)
• src/agent/generators/analytical.py — Grid analytical extension (NEW)
• src/agent/generators/ensemble.py — Voting ensemble (NEW)
• scripts/benchmark_hypothesis_techniques.py — Evaluation script (NEW)
• docs/HYPOTHESIS_AGENT_BENCHMARK.md — Results report (NEW)

Acceptance Criteria

• HypothesisAgent loads and orchestrates 4+ generators
• Each generator produces hypotheses in standard format
• Benchmark runs on 5-10 regimes without errors
• Generates comparison metrics (Sharpe, gap, discovery rate)
• Dashboard/report shows which technique wins per regime
• Learning curves tracked (does technique improve over time?)
• Extensible: can add new generators easily

Success Metrics

• Which technique has highest avg Sharpe? (target: >0.55)
• Which discovers most novel hypotheses? (target: >0.8 novelty rate)
• False positive rate per technique? (target: <10%)
• Fastest technique? (target: <2min per regime)
• Best hybrid weighting? (find optimal ensemble blend)

Related

• Depends on: Implement Research Agent Phase 1: Literature Discovery Tool #22 (Literature Agent Phase 1 foundation)
• Enables: Experiment: Generative RecSys + HyDE for hypothesis generation #23 (HyDE generative approach), future Phase 4
• Related: docs/RESEARCH_AGENT_DESIGN.md

Priority

🟠 MEDIUM - Experimental, but shapes the core hypothesis engine. Do after Phase 1-3 are solid.

Notes

This is the meta-research layer: instead of committing to one technique, build an agent that tries all and learns which works. This will inform whether generative (Issue #23) is worth pursuing or if literature-based suffices.

Could be implemented iteratively:

• Phase A: Literature + Grid analytical (simpler, faster)
• Phase B: Add generative + ensemble voting
• Phase C: Live learning (update weightings based on realized Sharpe)