Vidushi Anand
AI Engineer · Evaluation, tracing and guardrails for production AI agents
About
I work on the question that comes after “it runs”: does it actually work, and can you trust the number that says so? At Noveum.ai I build evaluation, tracing and guardrails for production AI agents.
I also dig into where AI evaluation breaks, from fooling LLM judges into accepting “I don’t know” as a correct answer to measuring how often graders get it wrong, and I bring that rigour to everything I ship. Final-year B.Tech in AI & ML, graduating 2027.
Looking for full-time AI engineering roles: building, evaluating and shipping LLM agents, across evals, observability, guardrails and agent infrastructure.
Experience
AI Engineer · Noveum.ai
May 2026 – Present- Caught a silent 35% under-count in agent quality scores. Failed judge calls were being averaged in as low scores (4.38 reported instead of 6.69); errors are now kept out of every result.
- Built NovaGuard, a guardrail engine that blocks over-budget and rate-limited LLM calls before they reach the provider, using atomic Redis checks shared across processes. 165+ tests.
- Built Judgecheck, which audits an LLM judge without any human labels by testing it on controlled edits of known-correct answers. Matched the research prototype on 819 answers.
- Made voice-agent judges read the whole call, so answers resumed after an interruption stop being scored as repetition. Validated on 50 production calls.
- Shipped CrewAI auto-instrumentation and privacy by default (PII redaction, HTTPS-only transport) in noveum-trace, Noveum’s open-source tracing SDK for agents.
AI Engineer Intern · APi.market
Jun – Oct 2025- Built the RAG evaluation suite from scratch: 30+ metrics for retrieval and answer quality, plus a stage-by-stage evaluator that pinpoints which part of a pipeline failed.
Selected work
-
SafeToSay: a clinical agent that knows when not to answer
Code55 / 56 safety tests passed, up from 30 for the raw model
A LangGraph multi-agent system around MedGemma that checks scope, evidence and risk before every answer, and declines when it should. It fixed emergency escalation, diagnosis refusal and adversarial prompts on my test suite, with no drop on the 20 everyday medical questions.
-
107 / 108 refusals accepted as correct answers
A 20-token optimised suffix turned a Qwen3-4B judge into a rubber stamp, and a readable attack written by an LLM carried over to Gemini. Alongside it, I measured how often judges fail on their own: the grader behind my memory benchmark passed 82% of “I don’t know” answers, and a targeted rubric cut that to 4%.
-
Write time is where memory systems lose facts
A memory layer for agents that turns notes, coding sessions and git commits into dated facts in SQLite, with a nightly pass that retires stale ones. Tracing every LongMemEval failure to its cause showed facts were lost when stored, not when retrieved, and exposed an unreliable grader.
Writing
Skills
- Focus
- LLM agents, evals, LLM-as-a-judge, guardrails, observability, RAG, red-teaming
- Stack
- Python, TypeScript, PyTorch, LangGraph, DSPy, LiveKit, OpenTelemetry, Redis
Recognition
- Smart India Hackathon 2025, Grand Finalist
- Kaggle Notebooks Expert, top 3% of 60,000+
- Co-Lead, Deviators Club (2025–26): ran a 3-day coding contest with 110+ teams