Monitoring Wonderland Survey - Visualization

A picture tells more than a …

Now that you’ve collected all the metrics you wanted or even more , it’s time to make them useful by visualizing them. Every respecting metrics tool provides a visualization of the data collected. Older tools tended to revolve around creating RRD graphics from the data. Newer application are leveraging javascript or flash frameworks to have the data updated in realtime and rendered by the browser. People are exploring new ways of visualizing large amounts of data efficiently. A good example is Visualizing Device Utilization by Brendan Gregg. or Multi User - Realtime heatmap using Nodejs

Several interesting books have been written about visualization:

• Designing with Data
• Visualize this
• Information Dashboard Design - Effective Communication
• Design by Nature
• Data Visualizations
• Chapter on visualization in Big Data Glossary Book
• The visual Display of Quantative Information
• Envisioning Information
• Visual and Statistical Thinking

Dashboard written for specific metric tools

Graphite

Graphs are Graphite’s killer feature, but there’s always room for improvement:

• Graphiti - https://github.com/paperlesspost/graphiti an alternative well designed UI. To see it in action watch this presentation Metrics And you
• Pencil - https://github.com/fetep/pencil
• RI Pienaar has created Gdash - Graphite: version control, add graphs dsl, easy bookmarks
• Charcoal - Charcoal: Simple Graphite Templates
• Graphite - Jquery - https://github.com/prestontimmons/graphitejs - if you want to do it all in Javascript

Grockets - Realtime streaming graphite data via socket.io and node.js

Opentsdb

Graphs in Opentsdb are based on Gnuplot

• Opentsdb- Dashboard in Nodejs - https://github.com/clover/opentsdb-dashboard
• Otus - https://github.com/otus/otus - Web Dashboard build on top of Hadoop/Opentsdb for monitoring hadoop cluster -

Ganglia

• The New Ganglia Web - 2 is pretty slick!

Collectd

• Visage - Web Interface to collectd - RRD
• a CollectD viewer by John Bergmans usine Websockets - AMQP - Collectd - realtime view: http://bergmans.com/WebSocket/collectdViewer.html

Nagios

Nagios also has a way to visualize metrics in it’s UI

• PNP4Nagios

• N2RRD

• RRD2Graph

• Visual Timeline - Nagios Vis - Simile

Overall integration

With all these different systems creating graphs, the nice folks from Etsy have provided a way to navigate the different systems easily via their dashboard - https://github.com/etsy/dashboard

I also like the Idea of Embeddable Graphs as http://explainum.com implements it

Development frameworks for visualization

Generic data visualization

There are many javascript graphing libraries. Depending on your need on how to visualize things, they provide you with different options. The first list is more a generic graphic library list

• Protovis-js : http://code.google.com/p/protovis-js
• Processing-js: http://processingjs.org/
• Raphael-js: http://raphaeljs.com/
• Flare: http://flare.prefuse.org/
• Google Fusion Tables : http://www.google.com/fusiontables
• Polymaps: http://polymaps.org/ex/
• Yahoo UI elements: http://developer.yahoo.com/yui
• Gephi: http://gephi.org
• Graphiz: http://www.graphviz.org

Time related libraries

To plot things many people now use:

• Flot: http://code.google.com/p/flot/
• Ruby interface to Flot: https://github.com/pbosetti/flotr

For timeseries/timelines these libraries are useful:

• Simile Timeline - http://www.simile-widgets.org/timeline/

• Simile Timeline in Google Charts - http://code.google.com/apis/chart/interactive/docs/gallery/annotatedtimeline.html

• Dygraphs - http://dygraphs.com/ - that produces produces interactive, zoomable charts of time serie

• Rickshaw - https://github.com/shutterstock/rickshaw : A JavaScript toolkit for creating interactive real-time graphs

• D3 - Data Driven Documents - http://mbostock.github.com/d3 . To see it in action check out Cube -https://github.com/square/cube/wiki, a tool that uses D3, Redis for realtime visualizations.

And why not have Javascript generate/read some RRD graphs :

• Javascript RRD Graph - https://github.com/manuelluis/jsrrdgraph
• Javascript for reading/interpreting RRD files - http://javascriptrrd.sourceforge.net
• Pure javascript RRD file manipulation implementation - https://github.com/tjfontaine/javascript-rrd

Annotations of events in timeseries:

On your graphs you often want event annotated. This could range from plotting new puppet runs , tracking your releases to everything that you do in the proces of managing your servers. This is what John Allspaw calls Ops-Metametrics

These events are usually marked as vertical lines.

• RRD Vertical - works for Cacti, Munin, Collectd … - http://blog.vuksan.com/2010/06/28/overlay-deploy-timeline-on-your-ganglia-graphs

• Ganglia - Overlay Events: http://ganglia.info/?p=382

• Graphite - Draws as infinite: http://readthedocs.org/docs/graphite/en/latest/functions.html

• Graphite - Events to facilitate this: https://github.com/agoddard/graphite-events

• Opentsbd - has a feature request for annotations but is not yet implemented

Dependencies graphs

One thing I was wondering is that with all the metrics we store in these tools, we store the relationships between them in our head. I researched for tools that would link metrics or describe a dependency graph between them for navigation.

We could use Depgraph - Ruby library to create dependencies - based n graphviz to draw a dependency tree, but we obviously first have to define it. Something similar to the Nagios dependency model (without the strict host/service relationship of course)

Conclusion

With all the libraries to get data in and out and the power of javascript graphing libraries we should be able to create awesome visualizations of our metrics. This inspired me and @lusis to start thinking about creating a book on Metrics/Monitoring graphing patterns. Who knows …