Prometheus Monitor

On Sun, 19 Jun 2022, by @lucasdicioccio, 616 words, 7 links, 0images.

Prometheus-monitor is a low-configuration web application. Run Prometheus-Monitor to inspect data from services which expose metrics in the Prometheus-format.

Intended use cases

Whether you are a developer running software locally or an SRE investigating production hiccups, you sometimes need to inspect how a specific job behaves.

If you organization is mature, services probably provide logs and metrics that you collect and expose in a dashboard. In some situations, however, such platforms may feel too limited or too vast. In particular, monitoring platforms are meant to aggregate information across jobs and across time, thus if your problem is to identify how a single job operates at short time-scale you may feel limited or overwhelmed.

If you are a developer you likely find that configuring a whole metrics and dashboard infrastrucure is too much overhead for development purposes. As a results, counters rarely get added and low-hanging fruits around monitoring are missed. Rightfully, developers find a low return-on-efforts to pro-actively add metrics while developing new features.

Prometheus-monitor’s primary goal is to address such niche use cases where you need or want to focus on individual jobs or individual metrics. In particular, it is useful when your diagnostics have little repetition from one situation to another: point it to /metrics endpoints and you get some live graphs to see.

How to run it?

Prometheus-Monitor is a primitive JavaScript application so requirements are pretty low. The main operational caveat is that if the JavaScript runs in a web-page: HTTP GET queries to Prometheus endpoints are subject to CORS authorizations. The recommended way to circumvent CORS is to install the application as a browser extension.

You can install the Firefox extension. An icon appears in your extension toolbar, press-it and it will open a static HTML page with the application JavaScript.

Directly in the browser

The application also runs on this page if you scroll a bit below. However since the JavaScript for the application runs in-browser and CORS-limitations apply (you can proxy around – see the Troubleshooting section at the bottom of this page for some help).

Chrome extension

The submission and review process for Chrome is longer than for Firefox. My extension is currently in review. Once (if) approved, the usage will be the same as for Firefox.

in progress

bugs

  • fix bug with zombie polling threads when we go ‘too fast’

small improvements

  • some tooltips on buttons
  • prevent/highlight/keep-around exporter sources when the panel is removed and zoomed metric exist
  • styling of labels
  • group-rows by metric-name in previews
  • add headers on ajax calls

large improvements

  • chart more than a timeseries on a panel
  • add minimalistic functions that could translate to Prometheus (e.g., log, unit-changes)

Troubleshooting CORS

The Prometheus project shows a DEMO instance , do not hammer it please.

‘Failed to fetch’ an URL that works from my browser, but gives CORS errors in the console

You may want to use a so-called ‘CORS-proxy’ (i.e., an HTTP web proxy that adds CORS-authorization headers).

modify your code

If you can mod the binary you are running, the easiest way likely is to set a CORS-header allowing this page to get data on the metrics endpoint you want to probe.

open proxies

You may find open-proxies but you have no way to tell what they will do with your in-flight requests: cacheing, rate-limitations, data-collection. Plus open-proxies will only be able to access public endpoints (i.e., not for dev environments).

I’ve found the All Origins open proxy to work if you want to probe public endpoints (like the prometheus demo).

https://api.allorigins.win/raw?url=https://prometheus.demo.do.prometheus.io/metrics

run the proxy locally

If you shop around on GitHub you will find some solutions. For instance I’ve found GoBetween to work as claimed in the README. If you manage to run this package locally on port 3000 you can try adding:

http://localhost:3000/pipe/https://prometheus.demo.do.prometheus.io/metrics

serve the JS directly from your endpoint

This is the solution I take in prodapi, just serve the JavaScript file and include it in some HTML that you serve as well. Optionally, the HTML can have a div with ID metrics to select the script insertion place.

 <div id="metrics"></div>
 <script src="/js/prometheus-monitor.js"></script>

You’ll need some CSS to style it, but this will come later (when I’m out of alpha mode).