We're building a custom Prometheus exporter using Quarkus Jakarta, focusing on avoiding high cardinality and ensuring efficient metrics. Buckle up for some metric magic!

Why Custom Exporters? Aren't We Reinventing the Wheel?

Before we dive in, let's address the elephant in the room: why bother with a custom exporter when there are plenty of off-the-shelf solutions?

  • Tailored metrics: Your app is unique, and sometimes you need metrics that don't come out of the box.
  • Performance optimization: Custom exporters let you fine-tune what you're measuring, potentially reducing overhead.
  • Avoiding metric explosion: With great power comes great responsibility – and the ability to avoid high cardinality pitfalls.

Setting Up the Quarkus Jakarta Project

First things first, let's get our Quarkus project up and running. If you're new to Quarkus, think of it as the superhero version of Jakarta EE – faster than a speeding bullet and more powerful than a locomotive.

Create a new Quarkus project using the following Maven command:

mvn io.quarkus:quarkus-maven-plugin:2.16.5.Final:create \
    -DprojectGroupId=com.example \
    -DprojectArtifactId=custom-prometheus-exporter \
    -DclassName="com.example.ExporterResource" \
    -Dpath="/exporter"
Bash

Now, let's add the necessary dependencies to our pom.xml:

<dependencies>
    <dependency>
        <groupId>io.quarkus</groupId>
        <artifactId>quarkus-resteasy-reactive</artifactId>
    </dependency>
    <dependency>
        <groupId>io.quarkus</groupId>
        <artifactId>quarkus-micrometer-registry-prometheus</artifactId>
    </dependency>
</dependencies>
XML

The Heart of the Matter: Building the Exporter

Now that we've got our project set up, let's create our custom exporter. We'll focus on a hypothetical scenario where we're monitoring a complex e-commerce system.

package com.example;

import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Tags;
import jakarta.inject.Inject;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

import java.util.Random;

@Path("/exporter")
public class ExporterResource {

    @Inject
    MeterRegistry registry;

    private final Random random = new Random();

    @GET
    @Produces(MediaType.TEXT_PLAIN)
    public String hello() {
        // Simulate some metrics
        recordOrderMetrics();
        recordUserMetrics();
        return "Metrics updated!";
    }

    private void recordOrderMetrics() {
        // Instead of recording per-product metrics (high cardinality),
        // we'll record aggregate metrics per category
        String[] categories = {"Electronics", "Clothing", "Books", "Home"};
        for (String category : categories) {
            double orderValue = 100 + random.nextDouble() * 900; // Random order value between 100 and 1000
            registry.gauge("ecommerce.order.value", Tags.of("category", category), orderValue);
        }
    }

    private void recordUserMetrics() {
        // Instead of recording per-user metrics, we'll use buckets
        int activeUsers = 1000 + random.nextInt(9000); // Random number between 1000 and 10000
        String userBucket = activeUsers < 5000 ? "low" : "high";
        registry.gauge("ecommerce.active.users", Tags.of("load", userBucket), activeUsers);
    }
}
Java

Avoiding the High Cardinality Trap

High cardinality is the boogeyman of Prometheus setups. It's like inviting everyone in your city to a house party – things are bound to get out of hand. Here's how we're avoiding it:

  • Categorization: Instead of tracking metrics for each individual product (which could be thousands), we're grouping them into categories.
  • Bucketing: For user metrics, we're using a simple "low" or "high" bucket instead of tracking each user individually.

Remember, the goal is to have enough detail to be useful without drowning in data.

Ensuring Efficient Metrics

Efficiency isn't just about avoiding high cardinality. Here are some other tips to keep your metrics lean and mean:

  1. Use labels judiciously: Labels are powerful but can quickly lead to metric explosion if overused.
  2. Aggregate where possible: Sometimes a sum or average is more useful than individual data points.
  3. Choose appropriate metric types: Gauges, counters, and histograms each have their place. Use them wisely.
  4. Set retention policies: Not all metrics need to be kept forever. Set appropriate retention periods in Prometheus.

Testing Your Exporter

Before we pat ourselves on the back, let's make sure this thing actually works. Run your Quarkus application:

./mvnw quarkus:dev
Bash

Now, hit the endpoint to generate some metrics:

curl http://localhost:8080/exporter
Bash

Finally, check out your metrics at:

curl http://localhost:8080/q/metrics
Bash

You should see something like this:

# HELP ecommerce_order_value 
# TYPE ecommerce_order_value gauge
ecommerce_order_value{category="Electronics"} 543.21
ecommerce_order_value{category="Clothing"} 321.54
ecommerce_order_value{category="Books"} 123.45
ecommerce_order_value{category="Home"} 987.65

# HELP ecommerce_active_users 
# TYPE ecommerce_active_users gauge
ecommerce_active_users{load="high"} 7523.0

The Proof is in the Pudding: Analyzing Your Metrics

Now that we've got our exporter up and running, let's talk about what we can do with these metrics. Here are some Prometheus queries you might find useful:

# Average order value across all categories
avg(ecommerce_order_value)

# Total number of active users
sum(ecommerce_active_users)

# Highest order value by category
max(ecommerce_order_value) by (category)
PromQL

These queries give you a taste of the insights you can glean from your carefully crafted metrics.

Wrapping Up: Lessons Learned

Building a custom Prometheus exporter isn't just about slapping some metrics together and calling it a day. It's an art form that requires careful consideration of what you're measuring and how you're measuring it. Here are the key takeaways:

  • Always be on guard against high cardinality. It's the silent killer of Prometheus setups.
  • Aggregate and categorize where possible to keep your metrics meaningful and manageable.
  • Choose your labels carefully. They're powerful, but with great power comes great responsibility.
  • Test, iterate, and refine. Your first attempt probably won't be perfect, and that's okay.

Remember, the goal is to have metrics that provide actionable insights, not just a sea of numbers. Happy monitoring!

Food for Thought

"The goal of computing is insight, not numbers." - Richard Hamming

As you continue to refine your custom exporter, keep this quote in mind. It's not about how many metrics you can generate, but about the insights you can derive from them.

Additional Resources

Want to dive deeper? Check out these resources:

Now go forth and export those metrics like a pro! And remember, in the world of monitoring, less is often more. Unless we're talking about uptime – then more is definitely more.