Cats, Pi, and Machine Learning
Introduction
There is a cat that wanders around my neighbourhood. I wanted to build something that would notify me whenever it came to my backyard.
I thought to myself, if I can get a picture of my backyard as the input I can process the image by checking if there is a cat in it and send a notification to my phone as the output.
For the picture input, I attached a camera module to a raspberry pi 4. For the processing, I wrote some code that would run on the pi, which would periodically take an image and run object detection on it. For the output, I relied on sending a message via Signal to my phone.
The hardware
The raspberry pi 4 has a dedicated slot to connect the camera module. It also has a slot for display output that looks identical to the camera slot. The camera slot is the one that is closer to the USB/Ethernet slots. The board also has the text ‘camera’ and ‘display’ embedded to label the slots. If the camera does not work check to make sure it is not connected to the display port. This image might be helpful to locate the camera slot https://miro.medium.com/max/1200/1*2xIz14qQgQ81lZlo3XXdrQ.png . Always power the pi off before connecting or disconnecting the camera module. After connecting the camera your pi should look something like https://projects-static.raspberrypi.org/projects/getting-started-with-picamera/dbf2d9575be4756f79e4293a047a8a531d340710/en/images/pi-camera-attached.jpg [the blue side of the ribbon faces the USB port]
Once connected, ssh into the pi and use sudo raspi-config
to enable the camera in the interface settings. If raspi-config
does not resolve use sudo apt install raspi-config
to install it.
How do you check if the camera hardware is hooked up correctly? The command to check that is vcgencmd get_camera
if you see detected=1
then everything is working - if it returns detected=0
the cable is not connected properly to the raspberry pi board.
The software
The core loop of the application is to periodically take a picture and perform object detection on it and based on the results send the notification. I added an http server in there as well to see the last picture taken in my browser. All the code was written in kotlin and the pi was running AdoptOpenJDK-16.0.1+9
JVM
Taking an image
To take pictures I used the library https://github.com/Hopding/JRPiCam
. It allows taking a picture as a file saved on disk as well as getting the buffered image object purely in memory. I opted for the latter, which doesn’t require periodic cleanup of all the files created. I wanted the camera taking pictures every minute - this would end up creating more than a thousand image files every day.
The camera is surprisingly configurable providing, all kinds of, tuning options such as exposure, brightness, resolution, and rotations/flips to name a few. I didn’t do anything fancy and created a basic camera object
1 | val piCamera = RPiCamera("/home/pi/cats") |
The bufferedStill
is of type BufferedImage
, which is more than just an array of bytes of the frame. I needed to code an adapter that would convert the BufferedImage
to a Vert.x Buffer
class which could then be easily moved around over the network [for the detection request]
The adapter ended up being only a few lines using javax.imageio.*
1 | fun imageToBuffer(bufferedImage: BufferedImage): Buffer { |
That takes care of getting the picture, in a convenient format that we can work with, of the backyard into the application.
To trigger the image capture periodically, I used Vert.x’s setPeriodic
. Vert.x will also be used to create the admin/debug http server and the web client that makes the notification and detection requests
1 | val vertx = Vertx.vertx() |
Http server
Didn’t go too crazy with the debug http server. One GET
endpoint that returns the most recently captured image as the response.
1 | val router = Router.router(vertx) |
Object detection
To detect whether an image has a cat in it, I used a pre-trained SSD object detection model from Onnx hub. It would take the image as an input and return the class labels with their probabilities of anything detected as the result. The kotlin deep learning library [https://github.com/Kotlin/kotlindl] from Jetbrains came in handy as it provides all the primitives required to achieve object detection based on an input image.
The detection can be done as follows
1 | private val modelHub = ONNXModelHub(cacheDirectory = File("cache/pretrainedModels")) |
The detection does not happen on the pi itself. The detection model is exposed as an API on my server. The API docs are available here
The gist is that the endpoint takes in an image and returns the detected object as a json response
1 | # input |
Now we need some code that runs on the Pi, which will take the image coming in from the camera and make the above request to get the detected objects. Once again, we use Vert.x to do that by creating a web client and making a post request
1 | val webclient = WebClient.create(vertx) |
After that, it is parsing the http response and checking if any detected object has the class label cat
with probability > 0.7
. If yes, send out the notification [if something else was detected - send a notification for that as well].
Notification
A while back I deployed signal’s rest api [ https://github.com/bbernhard/signal-cli-rest-api ] to my server, which is what I used here to send a signal message as a notification when a cat was detected in the captured image. The base64_attachments
allows me to put the base64 encoded bytes of the captured image, which will then also be sent as an attachment.
1 | val body = JsonObject() |
Final result
I got the image https://imgur.com/a/vYs1W1L with the message cat detected with probability 0.954
on my phone when the cat came by to visit. Everything worked as expected!
I also added a cooldown to the signal message notification so I wouldn’t continuously get spammed with messages if the cat decided to linger around.
Other work
Want to know what reddit is saying about cats? Search across 2.9 million+ comments => https://conversations.aawadia.dev/find?query=cats&selectedCategory=all