I am testing Frigate for a couple months now. It is a very ambitious project and I would love to see it succeed.
Here are the observations:
* You don’t actually need hardware decoding or a Coral, but they do help. You will of course need to provision more CPU horse-power for NVR.
* Motion detection uses the usual implementation from OpenCV. Unfortunately this algorithm is not very good in my experience. Many things I would consider as motion are missed (false negative), many things I would not consider motion are being detected (false positive). These factors mean that one is tempted to go ham on masking to filter out false positives, which then leads to further false negatives. I’m genuinely surprised the motion algorithm that’s implemented in OpenCV is still the state of art of what’s available openly.
* Object detection is somewhat knee-capped by the models available publicly. They are not very good either. Frigate has built its behaviour around these models with an assumption that these models are largely pretty accurate, which in my experience has ended up with quite a few missed recordings for important events, which led me to switch to create recordings based on motion (I’m not in a very densely populated area and reviewing the recordings isn’t too onerous.)
* Support for coral is… shaky at best. There are some indications that the production of these devices has largely stopped (and finding them to purchase is hard and expensive,) and maintenance of the drivers and libraries to interface with coral seems to be minimal or non-existent to the point where some Linux distributions have started dropping the relevant packages from their repositories. On the upside, running these models on the CPU isn’t that expensive, especially considering that the models are invoked very sparingly.
I’m currently thinking of moving over to continuous recording, perhaps trying out moonfire-nvr or mayhaps handwriting a gstreamer pipeline. Simple software -> fewer failure modes.
(NB: I worked at a computer vision startup in the past, my views are naturally influenced by that experience.)
> I’m currently thinking of moving over to continuous recording, perhaps trying out moonfire-nvr or mayhaps handwriting a gstreamer pipeline. Simple software -> fewer failure modes.
Moonfire's author here. Please do give it a try! Right now it's a little too simple even for me, lacking any real support for motion or events. [1] But I'd like to keep that simple server core that just handles the recording and database functionality, while allowing separate processes to handle Frigate-like computer vision stuff or even just on-camera motion detection, and enhance the UI and add stuff like MQTT/HA integration to support that well. I'd definitely welcome help with those areas. (And UI is really not an area of expertise of mine, as you can see from e.g. this bug: <https://github.com/scottlamb/moonfire-nvr/issues/286>.)
For now I actually run Moonfire and Frigate side-by-side. They're almost complete opposites in terms of what they support, but I find both are useful.
[1] The database schema has the concept of "signals" (timeseriesed enums like motion/still/unknown or door open/door closed/unknown), but my code to populate that based on camera or alarm system events is in my separate "playground" of half-finished stuff, and the crappy UI for it is rotting in one of my working copies. I'd like Moonfire's database/API layer to also have a more Frigate-like concept of "events" and one of "object tracks".
A killer feature would be a time series showing the rate of change from frame to frame. That would allow someone to jump to the more interesting parts of a video.
I'd love that, but it's hard to collaborate given differences in schedules, working styles, architectural direction, etc. I briefly asked Blake about collaboration back in 2020, and he was open to it, but (understandably) only in areas where we can work independently. (He mentioned wanting frontend help, which is about as far from what Moonfire and I are good at as can be.) I'm not sure what that'd be right now. I see Moonfire's server as basically a DBMS/engine for streaming video. If at some point, Frigate folks are interested in basically replacing their database and (now go2rtc) with Moonfire's, I'm absolutely open to discussing it. But Moonfire would need some feature work to avoid regressing anything they care about, with some discussions around what the API would look like and such. And I'm not moving real fast unfortunately (mostly due to lack of my time and lack of Rust-savvy collaborators).
So more realistically, I see Frigate as something I use in the short- to medium-term and take inspiration from in the long term. I'm not above essentially copying their motion and object tracking algorithms into a plugin, with proper attribution of course.
Frigate is neat because it challenged my idea of what the "minimum viable product" for an NVR could be. When I first looked at it, IIRC it didn't really have any UI at all and didn't support continuous recording. Instead, it just saved events as .mp4 files and published metadata over MQTT for a Home Assistant-based UI. It hadn't occurred to me that would be a useful system, but of course it was.
> Motion detection uses the usual implementation from OpenCV. Unfortunately this algorithm is not very good in my experience.
In frigate 0.13 (currently in beta) the motion detection has been fully rewritten, which has been a large improvement in my and other's experience. We also have docs now that walk users through tuning the motion detectoin.
This is along with many other changes along what you are describing like object tracking and improvements to initial object detection when motion is first detected.
I found motion detection to be the easy part when building my NVR. I just used trial and error and scipy filters and eventually found something I'm happy with.
Handwriting a GST pipeline is pretty much what I did. I start with frame differences(I only decode the keyframes that happen every few seconds, so motion detection has to work in a single frame to have good response time).
Then I do a greyscale erosion to suppress small bits of noise and prioritize connected regions.
After that I take the average value of all pixels, and I subtract it, to suppress the noise floor, and also possibly some global uniform illumination changes.
Then I square every pixel, to further suppress large low intensity background noise stuff, and take the average of those squares.
I mostly only run object detection after motion is detected, and I have a RAM buffer to capture a few seconds before an event occurs.
NVR device code(In theory this can be imported and run from a few like python script), but it needs some cleanup and I've never tried it outside the web server.
My CPU object detection is OK, but the public, fast, easy to run models and my limited understanding of them is the weak point. I wound up doing a bunch of sanity check post filters and I'm sure it could be done much better with better models and better pre/post filtering.
Some of this code is older, before I was more serious about this specific code, and moving to type annotations has been pretty much the big project of the year for me for everything personal, among other "Eliminate everything hacky" projects, going back into 10yo code and cleaning up tons of stuff.
My bigger priority has been moving from Mako to Jinja2, especially for some particularly horrid templates that could not be highlighted or formatted because there's not many good Mako tools, JSON schema validation, but I definitely agree type annotations are critical.
VS Code is smart enough to catch a lot of stuff sans annotations though, so you can get by with a lot of nonsense, especially when half your time is just fighting GStreamer and you're not paying as much attention to the python side.
There's nothing better than GST that I've ever seen for dealing with media without actually having to touch the performance critical stuff in your own code, but it is not easy to debug stuff buried in autogenerated python bindings to C code, especially with an extra RPC layer to use a background process and defend against segfaults.
There's also lots of other weird stuff, like imports not at the top of the file, meant to support systems where some module wasn't available, and generally all kinds of cleanup that's slowly happening.
> Support for coral is… shaky at best. There are some indications that the production of these devices has largely stopped (and finding them to purchase is hard and expensive,) and maintenance of the drivers and libraries to interface with coral seems to be minimal or non-existent to the point where some Linux distributions have started dropping the relevant packages from their repositories.
I've recently gone through the process of trying to install pycoral on Rocky Linux 9. I had to build from source, and there was some challenge because documentation for the build process was sparse. There was some conflicting information about files I had to edit, values I had to set, what was supported and what wasn't.
> Object detection is somewhat knee-capped by the models available publicly. They are not very good either.
Yep, the current models are based on ImageNet which is of course wildly different content to the typical security camera. It's no surprise that its recognition is often pretty poor, especially from the typical ceiling angles that cameras are mounted at
Here are the observations:
* You don’t actually need hardware decoding or a Coral, but they do help. You will of course need to provision more CPU horse-power for NVR. * Motion detection uses the usual implementation from OpenCV. Unfortunately this algorithm is not very good in my experience. Many things I would consider as motion are missed (false negative), many things I would not consider motion are being detected (false positive). These factors mean that one is tempted to go ham on masking to filter out false positives, which then leads to further false negatives. I’m genuinely surprised the motion algorithm that’s implemented in OpenCV is still the state of art of what’s available openly. * Object detection is somewhat knee-capped by the models available publicly. They are not very good either. Frigate has built its behaviour around these models with an assumption that these models are largely pretty accurate, which in my experience has ended up with quite a few missed recordings for important events, which led me to switch to create recordings based on motion (I’m not in a very densely populated area and reviewing the recordings isn’t too onerous.) * Support for coral is… shaky at best. There are some indications that the production of these devices has largely stopped (and finding them to purchase is hard and expensive,) and maintenance of the drivers and libraries to interface with coral seems to be minimal or non-existent to the point where some Linux distributions have started dropping the relevant packages from their repositories. On the upside, running these models on the CPU isn’t that expensive, especially considering that the models are invoked very sparingly.
I’m currently thinking of moving over to continuous recording, perhaps trying out moonfire-nvr or mayhaps handwriting a gstreamer pipeline. Simple software -> fewer failure modes.
(NB: I worked at a computer vision startup in the past, my views are naturally influenced by that experience.)