Highway into a service

kotlin, tcp, vertx

Introduction

I have been building micro-services for a few years now and a piece that is extremely important to get right is visibility into the service. This includes throughput, response times, and error rates but something else that I like to have is the ability to tweak/dump things in the service live i.e. without having to redeploy the entire thing. This could be dumping out in memory caches or changing feature flags to take different paths in the code logic etc.

It might be considered hacky or bad practice but with reasonable caution exercised it can be a useful tool to have. One way that I have seen done is to have debug http endpoints that trigger certain actions - I want to show a similar approach but with pure TCP instead - this will allow us to ‘telnet into our service` and execute commands.

We will expose a class that exposes a function attachCommand that takes a command name, description, and a handler to the developer.

The usage will look something like this

1
2
3
4
val highway = Highway()
val cache = Caffeine.newBuilder().build<Int, Int>()
repeat(10) { cache.put(it, it) }
highway.attachCommand("dump.cache", "Dump the caffeine cache to stdout") { /* handler that returns Any */ }

Highway

We will hold our commands in a HashMap defined as val commands = HashMap<String, Pair<String, () -> Any>>(). The key is the command name and the value is a pair of description string and the handler. () -> Any means a function that takes in no inputs and can return any type.

We can use typealias to make the declaration a bit easier to understand. typealias HighwayHandler = () -> Any as the type-alias and then our HashMap becomes val commands = HashMap<String, Pair<String, HighwayHandler>>() - a bit easier to read.

Our TCP server will created using Vert.x. The init function of our class will setup the TCP server and attach the handlers.

The init block looks something as follows

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
vertx
  .createNetServer() // start the TCP server
  .connectHandler { // onConnect execute this function
    RecordParser.newDelimited("\n", it) // delimit incoming buffers based on new lines
      .handler { buffer ->
        when (val cmd = buffer.toString().trim()) {
          "help" -> it.write(printHelp()) // if help command print out help i.e. command name and description
          else -> { // execute the command if registered in our hashmap and return the response + execution time taken
            var response: String
            val time = measureTime { response = "${commands[cmd]?.second?.invoke().toString()}\n" }
            it.write("$response\nTime Taken: $time\n")
          }
        }
      }
  }.listen(port) // listen on the specified port

The RecordParser is a convenience class provided by the Vert.x library that allows us to parse out incoming read streams. TCP is a two way stream of bytes - we need to provide a protocol on top of it to distinguish the individual commands coming in from the developer’s terminal. The three most common approaches are to delimit based on newlines, blank lines, or length prefix. We will use a simple new line delimited protocol. RecordParser takes the delimiter [\n in this case] and the incoming NetSocket and we provide the handler that should be executed on each delimited byte buffer - great!

We convert the buffer to a string and if it is a help message then we print out all the registered commands and their descriptions else we invoke the handler on the registered command name.

The attachCommand and printHelp functions are as follows

1
2
3
4
5
6
7
8
9
fun attachCommand(commandName: String, description: String, handler: () -> Any) {
  vertx.runOnContext { commands[commandName] = Pair(description, handler) } // make sure we are accessing the hashmap from the vertx event loop
}

fun printHelp(): String {
  val helpMessage = StringBuilder()
  commands.forEach { (k, v) -> helpMessage.append("Command Name: $k - Description: ${v.first}\n") }
  return helpMessage.toString()
}

Demo

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
➜  ~ telnet localhost 9191
Trying ::1...
Connected to localhost.
Escape character is '^]'.

help
Command Name: dump.cache - Description: dump the cache

dump.cache
{0=0, 1=1, 2=2, 3=3, 4=4, 5=5, 6=6, 7=7, 8=8, 9=9}

Time Taken: 2.31ms # pre jvm warmup lol

dump.cache
{0=0, 1=1, 2=2, 3=3, 4=4, 5=5, 6=6, 7=7, 8=8, 9=9}

Time Taken: 44.6us

sendMessage # command that doesn't exist
null

Time Taken: 41.5us

The entire code snippet can be found here: https://gitlab.com/-/snippets/2105096