I built a real-time identification system for my farm. Here is what it does and why.

By Rolf Versluis · Published [DATE] on Uncommon Knowledge in Business and Life

My farm is a business. I am not always on it. I had trail cameras before this. Every time a branch blew in the breeze, I got a picture. After a few months of alerts, I stopped looking. That is what off-the-shelf products did for me.

I built a different kind of system. It does not record footage so I can go back and review what happened after something was stolen or broken. It tells me, in real time, what is moving on the property right now. Vehicle, person, animal, or nothing worth my attention. Then I decide what to do with the information.

The project is open source at github.com/blockops1/ai_camera_monitor. MIT-licensed, runs on a Mac Mini or an old PC, the README has the install instructions. This post is about why I built it the way I did, how the four-stage pipeline works, and what I learned running vision models on motion events at volume.

What the system actually does

A Reolink camera watches the property. When it detects motion, it fires an HTTP webhook. The webhook triggers a small listener that captures four images from the camera's RTSP ring buffer. Those four images are the input.

First stage: a YOLO gate. YOLO is a small convolutional neural network that runs locally on the listener machine. It looks at each of the four images and asks one question: is there a vehicle, a person, or an animal in this frame, or is this just motion that is not actually anything? On a typical day the camera fires about 3000 motion events. YOLO drops the false positives. Only a handful of events make it past the gate, which means the expensive model never gets called for events that do not matter. That is the cost-control move that keeps the system cheap to run 24 hours a day.

Second stage: a pairwise differential. When YOLO says yes to a vehicle, person, or animal, the system computes a pairwise differential across the four images to figure out exactly where the motion is. That gives it two tight cropped boxes around the moving object. The crops, not the full frame, are what get passed forward.

Third stage: the LLM. Qwen3-VL on port 8093, local. It receives the two crops and a specific prompt tailored to what YOLO classified the object as. If YOLO said vehicle, the prompt asks for make, model, color, body style. If YOLO said person, the prompt asks for clothing, posture, distinguishing features. If YOLO said animal, the prompt asks for species, size, behavior. Structured output comes back.

What the LLM sees that YOLO cannot is the part that makes the matcher work. YOLO recognizes seven animal classes. The LLM recognizes hundreds, and it picks up individual patterns: the shape of an ear, the length of a snout, the color and pattern of fur. For vehicles, YOLO sees "vehicle." The LLM sees "black Ford F-150 pickup with a bent passenger-side mirror." For people, YOLO sees "person." It sees "adult in a green jacket, male, mid-50s, walking toward the gate." That detail is what makes the matcher work downstream.

The LLM is authoritative. If YOLO said vehicle and the LLM says person, the alert gets classified as person. If YOLO said person and it says nothing visible, the alert gets dropped. The system does not trust the cheap classifier over the expensive one. That is the whole reason false positives do not get through.

Fourth stage: a small text model on port 8081. It takes the structured output and writes the human-readable alert body. This is what makes the alert read like a person wrote it instead of "VEHICLE DETECTED AT 10:22." After that, the alert goes to Telegram through the bot API. Motion event to phone, the whole pipeline runs in about two seconds.

Why I built this instead of buying it

My first instinct is always to go find some open-source software and implement that. Agents are good at that.

For this surveillance problem, the leading open-source project out there is called Frigate. It does motion detection plus object classification well. It will tell you a vehicle is on the property. It will not tell you whether the vehicle is yours. It runs as a complete system with its own UI, its own dashboard, its own assumptions about what you want. Good product. Not built for the question I needed answered: is this mine, or not mine?

Same problem with the trail cameras. They will tell you motion happened. They will not tell you whether the motion is your truck or a stranger's truck, your neighbor walking their dog or someone you do not know, a deer passing through or a coyote testing the fence. Signal-to-noise too low. You start ignoring the alerts. The system becomes useless.

The missing piece was the matcher. For vehicles, the matcher scores the LLM output against enrolled vehicle signatures (make, model, color, distinguishing features). For people, it scores against enrolled clothing patterns and posture. For animals, it scores against species and individual markings. Three stores, three kinds of match logic.

That is the 5% that decides if this is mine. The 5% that runs the prompts tailored to what YOLO saw. The 5% that runs everything locally so the images never leave the property.

It is also the part no vendor will sell you. Too specific to your property, your cameras, your lighting, your definition of "known."

The design call worth highlighting

Single most important decision in the pipeline: the LLM is authoritative over YOLO. Not the default. Most systems trust the cheap classifier. The cheap classifier is fast and usually right. When it is wrong, the system fires a bad alert.

In this system, the LLM gets the final say. YOLO says vehicle, the LLM says person, the alert is for a person. YOLO says person, the LLM says nothing visible, the alert gets dropped. The system pays the cost of the more expensive model on every event that gets past YOLO, in exchange for not trusting YOLO's judgment when it matters.

The reason this works is depth. YOLO knows seven animal classes. The LLM knows hundreds, plus individual patterns. YOLO sees a vehicle. The LLM sees the make, model, color, and a specific feature like a bent mirror. The expensive model sees more because it was trained on more, and the cheap model sees less because it was optimized for speed over depth. Trusting the cheap model over the expensive one would be backwards.

What runs locally and what could run in the cloud

For my setup, both models run locally. Qwen3-VL on the Mac Mini for the vision work. A small chat model on the same machine for the text formatting. The four images from the camera never leave the property. The Telegram message is the only thing that goes out over the internet.

This works because I have the hardware. A Mac Mini with enough memory to run both models. If you do not have that, the same architecture runs with cloud models. Point the vision endpoint at a hosted vision API. Point the text endpoint at a hosted chat API. Prompts are written so they work either way. The README has the configuration.

Trade-off is privacy and cost. Local means the property images stay on the property. Cloud means a hosted provider sees every frame. For some properties that is fine. For mine it is not. The architecture supports both. Pick the one that matches your situation.

What this actually took

A few months of iteration. At first, the LLM was getting slammed with motion alerts. It mistook shadows for trucks. The text formatter wrote "VEHICLE DETECTED" no matter what the LLM said.

I kept a log of the wrong cases. Each one pointed at a small fix. The YOLO confidence threshold needed adjusting. The differential crop had to hug the motion tighter. The enrolled store needed the truck added. The text formatter needed to pull the actual classification from the LLM output instead of using a hardcoded template. Each fix taught the system something the next alert would benefit from.

The tuning was not a one-time event. After a few months of operating, the false positives were gone. The system identified what was at the property, took a picture, and sent it to me. No tire alerts. No shadow alerts. No branch-blowing-in-the-wind alerts.

What I learned building this

Pattern is the same one I keep finding with AI agents in general. The 95% is the open-source pieces. The 5% is the design calls that turn the pieces into your system. Design calls only get made by someone who knows what they want the system to do.

For me, the system is essentially as good as a person watching all the security cameras 24 hours a day and only telling me when something for real happens. There is no way I could afford to hire a person to do that. I do have an agent that wrote a program to do it.

The code is at github.com/blockops1/ai_camera_monitor. My address is localhermesagent.com.