A few days ago we noticed some API calls were taking longer than we expected. We were looking for a quick way to grab some analytics data to see at what point the calls were slowing down. But how to store history and visualize where the problems exist? How about monitoring API response times and posting them via MQTT to the platform itself?
Leveraging an existing Play! app we were currently using for system monitoring I was able to plug in a series of API calls run by a cron-type job scheduler. In this case the API metrics module runs every 10 minutes. See below for some Java pseudo code of this method.
// metrics message map
Map<String,Long> metrics = new Map<String, Long>();
// -- account call -------------------------------------- long initialStartTime = System.currentTimeMillis(); long startTime = initialStartTime; // << account API call here >> long stopTime = System.currentTimeMillis(); Logger.info("Account time:" + (stopTime - startTime)); metrics.put("api_account_call", stopTime - startTime); // -- group call ---------------------------------------- startTime = System.currentTimeMillis(); // << group API call here >> stopTime = System.currentTimeMillis(); Logger.info("Group time:" + (stopTime - startTime)); metrics.put("api_group_call", stopTime - startTime); // -- multipresent calls -------------------------------- for (String s : group.getThings()) { startTime = System.currentTimeMillis(); // << group API call here >> stopTime = System.currentTimeMillis(); Logger.info("Thing " + thingCount + "call time:" + (stopTime - startTime)); metrics.put("api_thing_" + thingCount + "_call", stopTime - startTime); } Logger.info("Total time:" + (stopTime - initialStartTime)); metrics.put("api_total", stopTime - initialStartTime); NotificationManager.pubResponseTimeMetric(metrics);
As you can see I gathered up the response times in a Map called metrics. This map will be the payload of the MQTT message sent to the platform. Let’s see what pubResponseTimeMetric does.
public static void pubResponseTimeMetric(Map<String, Long> metricMap) {
String payload = new Gson().toJson(metricMap);
Logger.info("Publishing a metrics message: " + payload);
Singleton
.getInstance()
.getOpsMqttClient()
.publishNoWait(
ConfigFactory.load().getString(
"api.metrics_topic"), payload, 0, false);
}This method makes use of a Singleton MQTT client that is constantly connected. Behind the scenes this MQTT client is an instance of the Eclipse Paho Java MQTT client. Here is an example MQTT message containing the API response times:
{
"api_thing_1_call": "174",
"api_group_call": "84",
"api_total": "463",
"api_thing_2_call": "96",
"api_account_call": "105"
}With this setup we get a message like the one above published every 10 minutes.
But how to visualize and make use of this data? It turns out we’ve made some good progress filling out the functionality in our developer portal.
After signing in, I navigate to the Past tab. This provides a visual way to pull up any past data published to our MQTT broker and platform.
Image may be NSFW.
Clik here to view.
By entering my stuff and thing (from the MQTT topic space), the _ wildcard for whatevers, 100 for the quantity and an appropriate timeframe I’m able to pull a list of 100 messages. Cool enough but check out the next step. By simply clicking on one of the values a chart is built visually showing the response times.
Image may be NSFW.
Clik here to view.
How cool is that? With zero work on my part (after publishing the metric messages) I have a graph showing my response times on 10 minute intervals.
That huge spike at the right? That’s when we were doing a maintenance activity and all API calls were proxied. It just jumps right out.