RRD file format

CloverDX Platform
1

Hi,

I’m just finding out about CloverETL and I’m wondering if it is possible to read the RRD database created by rrdtool:
http://oss.oetiker.ch/rrdtool/

RRD stands for Round Robin Database. RRD files are in a binary format which just stores measurement number like bandwidth of a router.

Since I’m new with the ETL principle I was wondering if it is common to retrieve data from databases like RRD?

What I want to achieve is to generate reports of data usage per router or whatever a collect measurements from.

Thank you in advance!

2

Clover does not directly support RRD file format (there is unfortunately no Java lib by the RRD author which could be used to develop “native” connector).

However, RRD contains rrddump tool which converts RRD files into XML. Once converted, XML can be parsed using Clover’s XPathReader or XMLExtract. The latter can be used to parse XML of any size since it uses SAX method.
Therefore you might develop Clover’s graph where, in 1st phase you use SysExecute component tu run the conversin (execute “rrdump”) and in 2nd phase use XMLExtract to process the data.

3

Hi dpavlis,

Clover does not directly support RRD file format (there is unfortunately no Java lib by the RRD author which could be used to develop “native” connector).

“dpavlis”

You’re right about the RRD author not offering native Java support. But I’ve found this site which does offer a native Java implementation the read/write RRD files:
https://rrd4j.dev.java.net/
According to the site it can read the RRD files with the exact same result as with the original tools.

However, RRD contains rrddump tool which converts RRD files into XML. Once converted, XML can be parsed using Clover’s XPathReader or XMLExtract. The latter can be used to parse XML of any size since it uses SAX method.
Therefore you might develop Clover’s graph where, in 1st phase you use SysExecute component tu run the conversin (execute “rrdump”) and in 2nd phase use XMLExtract to process the data.

“dpavlis”

To use rrddump to convert the RRDs to XML was an alternative I’ve also thought of. It is not really my purpose to generate graphs. What I want is to use the measurements in combination with my networkadministration so I can generate reports about bandwidth usage per port for example. Is it possible to store all these measurements from RRD in the Clover.ETL database to generate such reports? Or is this not normally done like this?

Which method do you prefer? Now that you know there’s a possibility to read RRDs with Java natively. Or use rrddump in a sort of workflow?

Thanks!

4

Hi !

Yes, we have found rrd4j.dev.java.net project as well. May be sometimes in the future we will create a specialized reader for RRD format (if there is enough demand for it).
Nonetheless, the rrdump seems to me currently as a viable alternative.

As for your question - CloverETL does not have any internal database. You can create transformation graph which will read RRD data (converted into XML) and then do whatever you need - combine it with other data, store it into database or create some other data format (perhaps XML).

My suggestion would be to create a transformation graph with 2 phases. In first phase, use SystemExecute to convert your .RRD data into XML (call “rrdump”). In second phase, use XMLExtract component to parse the XML data.

5

Hello Shot,
you can do the transformation in a single phase (formally): to not write and read the xml file you don’t need, connect the output edge to the SystemExecute component. Then the output from the process is sent to the next component directly (in our case to the reader, which can read data from a flat file as well as from the input edge). How to set fileURL attribute to read data from input port can be found here: fileURL attribute for readers. For small files, you can omit the recordDelimiter attribute in metadata - then the whole file is stored in one data field. Please see the following example:

<?xml version="1.0" encoding="UTF-8"?>
<Graph author="avackova" created="Fri Jul 24 10:28:00 CEST 2009" guiVersion="0.0.0.devel" id="1248426276555" licenseType="Evaluation license." modified="Fri Jul 24 12:20:44 CEST 2009" modifiedBy="avackova" name="xmlSh" revision="1.47">
<Global>
<Metadata id="Metadata0">
<Record fieldDelimiter="|" name="recordName1" recordDelimiter="\n" type="delimited">
<Field name="bar" type="string"/>
<Field name="baz" type="string"/>
</Record>
</Metadata>
<Metadata id="Metadata1" previewAttachmentCharset="ISO-8859-1">
<Record name="recordName2" previewAttachmentCharset="ISO-8859-1" recordDelimiter="\n" type="delimited">
<Field eofAsDelimiter="true" name="field1" type="string"/>
</Record>
</Metadata>
<Property id="GraphParameter10" name="input" value="filename.rrd"/>
<Property fileURL="workspace.prm" id="GraphParameter0"/>
</Global>
<Phase number="0">
<Node append="false" enabled="enabled" guiHeight="0" guiName="SystemExecute" guiWidth="0" guiX="47" guiY="49" id="SYS_EXECUTE0" type="SYS_EXECUTE">
<attr name="command"><![CDATA[rrdtool dump ${input}]]></attr>
</Node>
<Node debugPrint="true" enabled="enabled" guiHeight="0" guiName="Trash" guiWidth="0" guiX="418" guiY="121" id="TRASH2" type="TRASH"/>
<Node enabled="enabled" guiHeight="0" guiName="Trash" guiWidth="0" guiX="420" guiY="8" id="TRASH3" type="TRASH"/>
<Node enabled="enabled" guiHeight="0" guiName="XMLExtract" guiWidth="0" guiX="228" guiY="48" id="XML_EXTRACT0" sourceUri="port:$0.field1:stream" type="XML_EXTRACT">
<attr name="mapping"><![CDATA[ <Mapping element="TestScenario" outPort="0" xmlFields="ident;description" cloverFields="bar;baz">
   <Mapping element="FunctionalTest" outPort="1" xmlFields="ident;graphFile" cloverFields="bar;baz"/>
 </Mapping>
]]></attr>
</Node>
<Edge fromNode="SYS_EXECUTE0:0" guiBendpoints="" id="Edge1" inPort="Port 0 (input)" metadata="Metadata1" outPort="Port 0 (output of command)" toNode="XML_EXTRACT0:0"/>
<Edge fromNode="XML_EXTRACT0:0" guiBendpoints="" id="Edge2" inPort="Port 0 (in)" metadata="Metadata0" outPort="Port 0 (out)" toNode="TRASH3:0"/>
<Edge fromNode="XML_EXTRACT0:1" guiBendpoints="" id="Edge3" inPort="Port 0 (in)" metadata="Metadata0" outPort="Port 1 (out)" toNode="TRASH2:0"/>
</Phase>
</Graph>
6

Thank you all for the quick replies. As I’m new to CloverETL (more like still orientating), I’ll take your word for it when you mentioned these technical details :wink:
I just wanted to know if it was possible to read RRDs and if it was common to do so. At least I know it’s possible.

This a bit off topic but I was a bit surrprised to read from dpavlis that CloverETL doesn’t have an internal database. So every report or graph needs to be generated from the original source? I thought CloverETL worked with an (ROLAP or somekind) database which makes it possible to quickly make all kinds of cross sections without retrieving any data from the original sources.

The thing is that I’m still looking around for good ETL software and haven’t really dug into the architecture of CloverETL. I’m still orientating. But tips/hints are welcome of course :smiley: