From Random Slideshows to Agents: Building Kaavad, Part 1

Yo Yo Yo, How are you how’s it going🕺, I was going through current trends in AI and got to know after prompts, loops and graphs people are building their own agentic system or harness. So I thought it would be nice to build my own. But the personal coding harness was a little bit odd for me as I somehow love Claude Code 🙆‍♂️🙆‍♂️. Whatever things it gave are best for me. So I found this idea where let’s try to automate video generation with our harness considering how low cost we can get. Will try to use free models almost and of course will add support to high level models so we can check how much system can take load.

Disclaimer: The post is written by humans so assume it is not perfect in tone ,words etc.. grammar is not an issue it is fixed by AI.

Let’s get started………..

The ideation 🙆‍♂️…

My whole idea is to somehow automate video generation with some animation with these tools. I call this tool kaavad(A travelling storyteller, the Kaavadiya Bhat, carries it from village to village). Indeed it’s a storyteller and that’s what our app will do: it will be given some idea, find the script, make an audio, make an animation and combine all this and make the final output. And I know making video in today’s world is just giving one prompt to any high level model. But we are making it because I was not able to find any other solution for now. Also I want tools to work as per my own need so why not build this only.

And till now the whole thing is just a vibe coded project, till this point it just works. I have not done any r&d on how it will work and 100 things which a normal dev needs to consider while building world class, the best ,the scalable , the god like product. For me kaavad it’s just a tool which will help to automate many things from scripting to editing to final output. But even though it’s vibe coded I have made sure it doesn’t leak data (Like I can just hope for now atlist), or just take all resources of the system like the basic checklist is there.

So my end goals are

Kaavad will generate the ideas as per my input
It will generate end to end script and human will be in loop at each point to improve it
Once the script is ready we will have a flow that just makes audio and video. Also out of the box it will create shorts/reels video with main video.

The Starting 🙆‍♂️….

So, I started the project by remembering Opus 5.5. Because it’s the best model there which I can afford for now at least. And started the project by gathering my initial requirements. I selected python + next.js combo. Never mind I don’t care at that point like I should use rust to make running time Xx times faster. I am ok with using python and next.js.

I started generating one claude doc for this. And I must say one of the best tools till this point of claude, like this whole feat of doc is pretty much good. I can directly talk to the claude in chat and it simply improves productivity 🏃‍♂️‍➡️.

The first version of the app was like there was a job which calls a free llm to get the idea by fetching some random ideas of a niche from wikipedia. And for audio it was selecting a random audio. For video it just combined some random images and that’s it. And I was like hmmm. hmmmmmm. What to do now 🥷. I iterated many versions of it. It was like there are some jobs which are connected and just run one by one. But doing this thing with free models was taking some time and not that good.

I remember there was one video where my voice was like lululu……ldsxcjefdcnxuuuunhj.

AI added more feats while I was asking it to fix issues with the app. But some of them are cool like it added support for selecting multiple niches even when we are struggling with one type of video creation. But yes with time it fixed many things. But still at the end all are free models and my workflow was like the old 1900s era of AI where the tool/kaavad was not doing much, it was just fetching random ideas, generating random scripts from search and generating videos.

So how I (I+AI) fixed the thing…..

The Updating 🙆‍♂️…

Now I was at a point where I was thinking of adding some logic by hand rather than totally relying on llms. Because most of the models which I was using were free. And by free I mean not any open weight model which I am running locally on GPU 🧜‍♀️, I was just testing this with open router free models.

So I decided to try to build feedback loops with agents now. I redesigned / restructured the system with ai by asking ai to add ai agents in my system to work with ai. (ai,ai,ai,ai,ai,ai…….)

At the end I came up, sorry we (me+ai) came up with an approach to let’s add support for creating animations on our own using models (some basic animation). And let’s divide this process from one by one random task to a proper agentic system where we will have multiple agents, feedback cycles so the end output can be made better.

I started with plans to have Scout, Writer, Audio, Visuals and Editor as agents in the system and on top of them there was one main agent, Critic, present at almost all the steps. Like it checks ideas generated by Scout, it rates the idea and if it is below a given limit then Scout needs to regenerate the ideas. Same it check for script, audio, visuals etc.

Each agents works as follows.

A1 Idea researcher: reads your lessons, looks up trends, writes video ideas, ranks them and drops ones too close to videos already made.
A2 Critic: scores the script and rewrites weak sections, checks the ideas, voice and scenes, sends failed work back to its agent, and runs the quality checks before approval and upload.
A3 Scriptwriter: builds the fact sheet, writes the script and the Shorts, runs the checks and makes the title and description.
A4 Voice: records the narration and makes the captions.
A5 Visuals: plans an animated scene for each line, times it to the voice and finds pictures by name.
A6 Editor: cuts and renders the video, Shorts and thumbnails, then uploads them to YouTube as private.
A7 Supervisor: pulls stats, restarts stuck jobs, cleans up old files, sends reports, retries failed stages and checks that the models still work.

And each agent has its own memory, it stores the feedback it receives from humans. I am still working on making this feedback processing more accurate, let’s see how far I will reach.

The whole system right now looks like below 👩‍🎨.

Cool right. Now it’s not just one by one random job calls where we don’t have control on how to process each step’s output without human integration. With the agents in place I have also enhanced Editor and Visuals to use llms that can generate some basic motions. But will change all things with time, like I hope we will.

And after all these changes, I got some cool videos, adding some examples below. For now it will work great for me. It fits. And we built a small agentic system or harness if we want to call it. As it has agents, tools and feedback and small memory.

The conclusion 🙆‍♂️….

So what is next, the above system is generating decent videos I guess, not at all great if I continue using free models. So we will explore more in the upcoming part about what more features we can add to this and how much more we can enhance our harness. We will also make some videos with Claude or GPT models to see what we can make. Right now many things are hardcoded in code as per my needs but will make everything dynamic if needed.

And thanks if you reached here, happy agentic coding…

Byeeee have a safe life ahead with AI 🤖