Cats, Pi, and Machine Learning

linux, machine-learning, raspberrypi, vertx

Introduction

There is a cat that wanders around my neighbourhood. I wanted to build something that would notify me whenever it came to my backyard.

I thought to myself, if I can get a picture of my backyard as the input I can process the image by checking if there is a cat in it and send a notification to my phone as the output.

For the picture input, I attached a camera module to a raspberry pi 4. For the processing, I wrote some code that would run on the pi, which would periodically take an image and run object detection on it. For the output, I relied on sending a message via Signal to my phone.

The hardware

The raspberry pi 4 has a dedicated slot to connect the camera module. It also has a slot for display output that looks identical to the camera slot. The camera slot is the one that is closer to the USB/Ethernet slots. The board also has the text ‘camera’ and ‘display’ embedded to label the slots. If the camera does not work check to make sure it is not connected to the display port. This image might be helpful to locate the camera slot https://miro.medium.com/max/1200/1*2xIz14qQgQ81lZlo3XXdrQ.png . Always power the pi off before connecting or disconnecting the camera module. After connecting the camera your pi should look something like https://projects-static.raspberrypi.org/projects/getting-started-with-picamera/dbf2d9575be4756f79e4293a047a8a531d340710/en/images/pi-camera-attached.jpg [the blue side of the ribbon faces the USB port]

Once connected, ssh into the pi and use sudo raspi-config to enable the camera in the interface settings. If raspi-config does not resolve use sudo apt install raspi-config to install it.

How do you check if the camera hardware is hooked up correctly? The command to check that is vcgencmd get_camera if you see detected=1 then everything is working - if it returns detected=0 the cable is not connected properly to the raspberry pi board.

The software

The core loop of the application is to periodically take a picture and perform object detection on it and based on the results send the notification. I added an http server in there as well to see the last picture taken in my browser. All the code was written in kotlin and the pi was running AdoptOpenJDK-16.0.1+9 JVM

Taking an image

To take pictures I used the library https://github.com/Hopding/JRPiCam. It allows taking a picture as a file saved on disk as well as getting the buffered image object purely in memory. I opted for the latter, which doesn’t require periodic cleanup of all the files created. I wanted the camera taking pictures every minute - this would end up creating more than a thousand image files every day.

The camera is surprisingly configurable providing, all kinds of, tuning options such as exposure, brightness, resolution, and rotations/flips to name a few. I didn’t do anything fancy and created a basic camera object

1
2
3
4
5
6
7
8
9
10
11
12
val piCamera = RPiCamera("/home/pi/cats")
  .setExposure(Exposure.AUTO)
  .setHeight(720)
  .setWidth(1280)
  .setBrightness(60)
  .setTimeout(10)

// take a picture and return the image as an in memory object
val bufferedStill: BufferedImage = piCamera.takeBufferedStill()

// ▼ to take an image saved on disk use
// val image: File = piCamera.takeStill("pic.jpg");

The bufferedStill is of type BufferedImage, which is more than just an array of bytes of the frame. I needed to code an adapter that would convert the BufferedImage to a Vert.x Buffer class which could then be easily moved around over the network [for the detection request]

The adapter ended up being only a few lines using javax.imageio.*

1
2
3
4
5
6
7
fun imageToBuffer(bufferedImage: BufferedImage): Buffer {
  val bos = ByteArrayOutputStream()
  ImageIO.write(bufferedImage, "jpg", bos)
  return Buffer.buffer(bos.toByteArray())
}

val imageBuffer = imageToBuffer(bufferedStill)

That takes care of getting the picture, in a convenient format that we can work with, of the backyard into the application.

To trigger the image capture periodically, I used Vert.x’s setPeriodic. Vert.x will also be used to create the admin/debug http server and the web client that makes the notification and detection requests

1
2
3
4
5
val vertx = Vertx.vertx()
// Every 60 seconds update the bufferedStill with the latest captured frame
vertx.setPeriodic(60_000) {
  bufferedStill = piCamera.takeBufferedStill()
}

Http server

Didn’t go too crazy with the debug http server. One GET endpoint that returns the most recently captured image as the response.

1
2
3
4
5
6
7
8
9
val router = Router.router(vertx)

router.get("/image").handler { it.response().end(imageToBuffer(bufferedStill)) }

vertx.createHttpServer()
  .requestHandler(router)
  .listen(9094)
  .onSuccess { println("server started on ${it.actualPort()}") }
  .onFailure { it.printStackTrace() }

Object detection

To detect whether an image has a cat in it, I used a pre-trained SSD object detection model from Onnx hub. It would take the image as an input and return the class labels with their probabilities of anything detected as the result. The kotlin deep learning library [https://github.com/Kotlin/kotlindl] from Jetbrains came in handy as it provides all the primitives required to achieve object detection based on an input image.

The detection can be done as follows

1
2
3
4
5
6
private val modelHub = ONNXModelHub(cacheDirectory = File("cache/pretrainedModels"))
private val detectionModel = modelHub.loadPretrainedModel(ONNXModels.ObjectDetection.SSD)

fun detectObjects(fileName: String): List<DetectedObject> {
  return detectionModel.detectObjects(File(fileName), 10)
}

The detection does not happen on the pi itself. The detection model is exposed as an API on my server. The API docs are available here

The gist is that the endpoint takes in an image and returns the detected object as a json response

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# input
curl -v --request POST \
  --url https://api.aawadia.dev/ml/v1/object-detect \
  --header 'Accept-Encoding: gzip' \
  --header 'Content-Type: multipart/form-data' \
  --form image=@/path/to/image.png

# output
{
  "result" : [ {
    "classLabel" : "cat",
    "probability" : 0.9659778,
    "xmax" : 0.8196616,
    "xmin" : 0.21427771,
    "ymax" : 0.7083117,
    "ymin" : 0.065882534
  }, {
    "classLabel" : "bowl",
    "probability" : 0.7917733,
    "xmax" : 0.7438286,
    "xmin" : 0.059709966,
    "ymax" : 0.88642734,
    "ymin" : 0.48115438
  } ]
}

Now we need some code that runs on the Pi, which will take the image coming in from the camera and make the above request to get the detected objects. Once again, we use Vert.x to do that by creating a web client and making a post request

1
2
3
4
5
6
7
8
9
10
11
val webclient = WebClient.create(vertx)
val imageData = imageToBuffer(bufferedStill)
val fileUpload = MultipartForm.create().binaryFileUpload("image", "filename.jpg", imageData, "jpg")

val httpResponse = webClient.postAbs("https://api.aawadia.dev/ml/v1/object-detect")
  .putHeader("Accept-Encoding", "gzip")
  .putHeader("Content-Type", "multipart/form-data")
  .expect(ResponsePredicate.SC_OK)
  .sendMultipartForm(fileUpload)
  .onFailure { it.printStackTrace() }
  .await()

After that, it is parsing the http response and checking if any detected object has the class label cat with probability > 0.7. If yes, send out the notification [if something else was detected - send a notification for that as well].

Notification

A while back I deployed signal’s rest api [ https://github.com/bbernhard/signal-cli-rest-api ] to my server, which is what I used here to send a signal message as a notification when a cat was detected in the captured image. The base64_attachments allows me to put the base64 encoded bytes of the captured image, which will then also be sent as an attachment.

1
2
3
4
5
val body = JsonObject()
  .put("base64_attachments", JsonArray().add(Base64.getEncoder().encodeToString(imageData.bytes)))
  .put("message", msg)
  .put("number", "from-number")
  .put("recipients", JsonArray(listOf("to-number")))

Final result

I got the image https://imgur.com/a/vYs1W1L with the message cat detected with probability 0.954 on my phone when the cat came by to visit. Everything worked as expected!

I also added a cooldown to the signal message notification so I wouldn’t continuously get spammed with messages if the cat decided to linger around.

Other work

Want to know what reddit is saying about cats? Search across 2.9 million+ comments => https://conversations.aawadia.dev/find?query=cats&selectedCategory=all