why the best ai agents built from scratch, not frameworks?
primitives, primitives and primitives.
hey there!
in this blog post i am gonna talk about why building ai agents from scratch is high alpha over frameworks (obviously in long term and as complexity increases)
frameworks are comfy. they’re plug-n-play no? they give you tools, loops, observability and a full of “just change your JSON” advice. though in most of the cases, when it comes to building real ai agents (the kind that actually think, act and recover), it becomes a battle.
frameworks?
let’s talk a bit about frameworks. they gives you a structure. you just have to bolt in a few functions and yeah, your ‘agent’ runs. but real world tasks are messy (or let’s say not linear). they involve broken apis, hallucinated tools, weird user states and a whole lot of “wtf just happened.” many times building from scratch forces you to model the mind of the agent, not just wire up its limbs lol.
frameworks abstract too early.
frameworks assume you know the shape of your problem. but when you’re building agents, you’re actually discovering that shape on the go. frameworks try to hand you a solution template before you even understand the problem deeply.
scratch = control. and control = debuggability
one of the most underrated things about scratch-built agents? you actually understand why they took that weird tool-use loop or why they got stuck in an infinite retry. you can trace it. log it. rethink it.
on the other hand, frameworks bury that in decorators, logs, and “agent loops” with 3 layers of abstraction.
most agent frameworks treat llms like divine oracles.
give it a goal, let it call tools, observe, loop again (assuming if the llm always knows best). but llms are stochastic, context-sensitive beasts. they make mistakes. they forget. they reward hack. without strong priors and logic scaffolds (which you need to build), they may flop.
use tools late, and only if you’ve to
start small. build things with the most basic setup.
understand how everything actually works. go in rabbit holes.
once you know the pain points, once you see patterns repeat, then maybe real for a framework or abstraction.
but only if it saves real time. and only if it doesn’t hide too much.
understand before you abstract.
frameworks can speed up development, but they often add hidden complexity if you don’t deeply understand their internals. in many cases, all you need is a simple LLM API call.
fundamentals for ai systems.
best ai systems rely on a deep, first principles understanding of the core mechanics. let’s say model interactions, data flow, evaluation, and prompt engg. in many cases, starting with basic SDKs and minimal libraries forces this mastery. this is something very foundational, can enable effective debugging, optimization and nice execution flow later.
bhindi (bhindi.io) is a great example of this. it was built from scratch in pure JS and Python, without relying on heavy frameworks. that approach forced a clearer understanding of what it actually takes to build agentic products.
there certainly MAY be many use cases for them (I didn’t have to do that, but you can do most of the stuff without ever coming close to any of these libs/databases)
agno is awesome if you don’t want to deal with dependencies.
interfacing.
python or typescript are immensely valued while building these agents. you can start interacting with your chosen model api using sdk. this is straight path to learn mechanics of an api call, how to structure requests and handle responses (without intervening abstractions!!)
as the application develops, you’ll write repetitive boilerplate code. when this pattern becomes a clear pain point, switch to some abstraction library that solves this specific problem (pydantic, vercel ai sdk)
some more certain advantages in building from scratch
built for prod from day 0
when you build your agent from scratch, it's already tailored to the environment it’ll live in. like, no extra adapters. no "hacks to make it work." it just works natively, tightly coupled with your infra and user flows.
low-level thinking scales better long-term
you touch the wiring. you know what talks to what. this unlocks something. you can optimize for latency, cost, observability, and scale without praying the framework doesn’t break.
you design small, reusable building blocks
not just "useful components," but things that match your exact ux, task flow, or product logic. you’re not squeezing ideas into a rigid mold but you’re shaping the mold as you go.
if your LLM use case needs search, just build a lightweight tool for it and loop it into your agent via a basic API call. by clearly defining your product’s inputs, outputs, and outcomes, you often find that a minimalist, framework-free approach in pure python (or your language of choice) is more than enough.
framework != easy shipping anyways. many lack opinionated infra pipelines. you want to deploy? good luck wrapping their loop logic into a real CI/CD pipe that runs in production with observability.
frameworks aren’t evil. some are great for quick hacks, mvps, or just understanding the space. they help to move fast at the start.
but once you start caring about things like scale, latency, uptime, and debugging real issues, most frameworks start to fall apart. you end up fighting the tool more than building the product.
building from scratch isn’t about being fancy or “engineer-pilled.” it’s about owning the full stack, knowing what’s talking to what, and having the ability to fix it when shit breaks.
frameworks can help you explore. scratch is how you ship.
next in the series, i may come up with a blog on building agents from scratch (with some hacks or so). until then you can explore these.
will brown's deep research agent from scratch
anthropic's building effective agents blog
i’ve talked about RL, agents and beyond with will brown and tokenbender.
playlist link: GroundZero AI Talks
hmu or tag me about your thoughts/experiences.
x/twitter: @himanshustwts
checkout groundzero ai: @groundzero_ai
cyaa!
Really amazing blog