Hey! I’m Vidushi.

I work on AI evals and AI safety. I love obsessing over AI research, going so deep into a topic that I forget to ask whether I’ll ever actually use it.

Most of what I do comes down to one question: when a number says an AI system works, is it telling the truth?

AI Engineer at Noveum.ai · Gurugram, India · open to remote AI roles

Things I’ve caught

  1. 107/108

    An AI judge that said “I don’t know” was the right answer

    Twenty tokens stuck on the end of a refusal got a Qwen3-4B judge to mark it correct 107 times out of 108. A readable version written by another LLM also nudged Gemini, from 5% to 18%. If a judge grades your leaderboard, test it on answers you know are wrong.

    Read the story Code

  2. 89% → 42%

    My monitor aced my tests, then met real chatbots

    A 4B model on my laptop reads airline support chats and flags when the agent told the customer one thing but its tool calls did another. It caught 89% of the mistakes I planted and 42% of the ones real chatbots made. Real failures were quieter than mine, and a plain rule kept up with the model.

    Read the story Code

  3. Write time

    Where agent memory actually loses facts

    I built a memory layer for agents and traced every LongMemEval failure back to its cause. Most facts weren’t lost when the agent searched for them. They were never written down in the first place. Along the way, the benchmark’s own grader turned out to be unreliable too.

    Read the story Code

  4. 55/56

    A clinical agent that knows when not to answer

    SafeToSay wraps MedGemma in a LangGraph team of agents that check scope, evidence and risk before every reply, and decline when they should. It passed 55 of 56 safety tests, up from 30 for the raw model, with no drop on everyday medical questions.

    Code

  5. 8,000

    The network that sat still, then understood

    I trained 88 small transformers on modular addition. One memorised the answers in 132 steps, then took about 8,000 more to actually learn the rule. Taking it apart showed 4 of 56 frequencies doing almost all the work.

    Read the story Code

Now

Building evals, tracing and guardrails for AI agents at Noveum.ai. On the side, I’m seeing how far small models can go at watching bigger ones, and looking for my next full-time role, remote from India.

Work

AI Engineer · Noveum.ai

May 2026 – Present
  • Caught a silent 35% under-count in agent quality scores. Failed judge calls were being averaged in as low scores (4.38 reported instead of 6.69); errors are now kept out of every result.
  • Built NovaGuard, a guardrail engine that blocks over-budget and rate-limited LLM calls before they reach the provider, using atomic Redis checks shared across processes. 165+ tests.
  • Made voice-agent judges read the whole call, so answers resumed after an interruption stop being scored as repetition. Validated on 50 production calls.
  • Shipped CrewAI auto-instrumentation and privacy by default (PII redaction, HTTPS-only transport) in noveum-trace, Noveum’s open-source tracing SDK for agents.

AI Engineer Intern · APi.market

Jun – Oct 2025
  • Built the RAG evaluation suite from scratch: 30+ metrics for retrieval and answer quality, plus a stage-by-stage evaluator that pinpoints which part of a pipeline failed.

Also: Smart India Hackathon 2025 grand finalist · Kaggle Notebooks Expert (top 3%) · co-led Deviators Club · final-year B.Tech in AI & ML, graduating 2027.

Writing

Away from the keyboard

I paint my nails and build Lego. Both reward patience and steady hands, which turns out to be good training for reading model outputs one by one.