Distributing data is hard
Distributed systems are hard because they introduce the tradeoff of increased deployment complexity in exchange for linear horizontal scalability. With resource exhaustion from the physical servers, a “just add more nodes” attitude generally gets you out of trouble. If we were to talk of a stateless system that only focuses on compute, we would only have to monitor resource usage — scale accordingly and call it a day.
However, Apache Pinot is not a stateless platform. It is a distributed OLAP database. On top of our physical resource monitoring requirements, we also need to be aware of how data is distributed and how usage patterns affect the overall user experience.
Scaling a distributed system
I may be kicking open doors here, but a simple question has always helped me start from somewhere. When it comes to investigating degraded user experience caused by latency, can I observe high resource usage on all or some nodes of the system?
A negative answer to that question is the worst, since it is an indicator that you might be facing a software bug or slow external dependency from your system that you are not monitoring. How quickly you can identity the root cause will greatly depend upon how well you are equipped to monitor your external dependencies.
This post will explore how you can easily setup a monitoring stack for Apache Pinot, that will help you troubleshoot occurrences where your system is under provisioned or where data is not evenly distributed. The investigation will generally begin with:
Is resource usage high on all nodes (both system and JVM) or is it only a few nodes?
When all your servers are running hot and the system operates in the expected throughput for a given scale, the answer becomes “just throw more nodes at it”. If that is not the case, it usually means that it is time to roll up the sleeves, dig in and investigate what usage patterns cause some specific nodes of the system to go under high utilization.
Preparing an observability stack for Apache Pinot
Going through such a decision tree, highlights that we need system resource level metrics as well data level metrics from Apache Pinot. Without both of them, you will only be making wild guesses about what is wrong with the system. There’s loads of solutions and technologies out there that will help you out with system resource metrics. As for data level metrics, there’s no dark magic here. Any metrics that can be contextualized to specific data sets need to be emitted by the originating system that you are monitoring. Conveniently, Apache Pinot generates an extensive set of data metrics all exposed through JMX. If your metric system can consume JMX, you should be able to integrate without hurdles.
JMX Exporter for Apache Pinot
Making life easier with Prometheus
The Apache Pinot team packaged out of the box support for Prometheus, the ubiquitous cloud native metric monitoring toolkit. While not forcing you into a specific monitoring infrastructure, you can opt-in options to activate the Prometheus JMX exporter that is packaged within the Apache Pinot Docker image. The image also comes with a pre-configured exporter configuration.
Just add these JVM arguments to the JAVA_OPTS environment variables of your Apache Pinot containers and Prometheus will be able to scrape your instances:
Using Prometheus to scrape metrics from JMX Exporter
Not using Docker?
No problem, just bring your own Prometheus JMX Exporter and your own jmx exporter config. Just ensure you adapt your JAVA_OPTS environment variable with the correct paths for these two files.
Let’s prepare a demo environment
To demonstrate the Apache Pinot integration with Prometheus, I have prepared a sample repository containing a docker-compose deployment pre-configured with Pinot, Prometheus, Grafana and a simple Smoke Test application that emits events into Pinot through Kafka. Prometheus supports more than a dozen of discovery mechanisms such as Consul and Kubernetes. For the sake of simplicity, the example that follows will showcase static configurations referring to our Pinot components manually configured in a prometheus.yml file. Look out for the Apache Pinot reference documentation for a tutorial about Metric discovery on Kubernetes.
- Java 8 or more recent
Download the source code
$ git clone https://github.com/daniellavoie/monitoring-apache-pinot
Taking a look at the deployment manifest
The docker-compose manifest’s notable configurations are the JAVA_OPTS environment variable for all pinot components and the volume mount to the prometheus scrape configuration.