evalstack
Open-source LLM evaluation framework — Python SDK + CI plugin, LLM-as-judge rubrics, regression detection, side-by-side run + event diff. Gates prompt/model changes in CI before they ship. Braintrust alternative.
AI/ML Platform Engineer in Boston. I build production AI systems, streaming data platforms, and the MLOps glue between them.
Four projects shipped in the last week, each targeting a real ML platform / MLOps gap: LLM evaluation in CI, RAG-grounded NL→SQL, production LLM observability & monitoring, and a self-hostable data platform for feature engineering. All MIT-licensed, all demo-able locally.
Open-source LLM evaluation framework — Python SDK + CI plugin, LLM-as-judge rubrics, regression detection, side-by-side run + event diff. Gates prompt/model changes in CI before they ship. Braintrust alternative.
Alpha · Demo liveNatural-language analytics for founders + PMs — schema-aware, RAG-grounded, sqlglot-safety-gated SQL generation against DuckDB. EXPLAIN-based cost preview, single-page chat UI, key-free demo mode.
Alpha · Demo liveProduction LLM monitoring for inference in the wild — a drop-in @traced decorator captures latency / tokens / cost / errors. FastAPI collector, pluggable storage (SQLite or ClickHouse), p50/p95/p99 dashboard. Auto-detects OpenAI + Anthropic response shapes.
MVP · Working end-to-endEnd-to-end open-source data platform template — the feature-engineering backbone under ML/AI systems. Postgres → Debezium / snapshot CDC → Iceberg-style Parquet → dbt (bronze/silver/gold) → DuckDB queries. Single make demo runs the full path.