Support for User Defined Data Types

CloverDX Platform
1

We are very interested in using Clover as the basis for an ETL tool for non specialist. We would like to build on top of Clover to bring ETL like features in a very friendly user interface. However one stumbling bloc is that w would like to be able to add additional user defined data types to Clover. These data types can be static meaning they are setup once by the programmers of the tool. Then non-specialist ETL users make use of them.

Here’s the rational.

Currently Clover supports data types long, string, number, date etc. When configuring Clover it can establish if an input source does have data matching these types and will complain if it does not.

The CTL 2.0 language does the same with respect to data types contained in rows and used as parameters and return values of functions.

All this helps the ETL designer from making type errors.

We would like to be able to add additional user defined data types for example a social insurance number (SIN) data type, a currency data type and so forth. Added user defined data types is beneficial because it would enable custom display formatting for example a currency might be rendered $1.24 USD. Also functions written for these data types would get validated in the CTL 2.0 language.

Oracle Data Warehouse builder enables the ETL developer to work with user defined datatypes (defined in a database). In the ETL tool you work with your user defined types. We would like to be able to do something similar with Clover.

We are mostly interested in the Clover Engine and the Eclipse GUI tool that is used to edit the graphs.

Question 1) Are there plans to support user defined data types?

Question 2) How could we use Clover to approximate having user defined data types?

Question 3) Supposing we added our own data types directly to the source code of the Clover ETL engine would they be reflected in the GUI designer or would we also have to modify the source code of the Eclipse GUI tooling?

Thank you
Jean-Claude

2

Hi,

adding real custom data types into clover is not an easy task, especially if it should be also understood by the CTL interpreter/compiler and the GUI. For the engine, it is actually quite simple as you would just implement your data type to reflect what org.jetel.data.DataField base class requires - mostly it revolves around (de)serializing.
For the GUI - mostly the metadata editor, that would require also some coding. The CTL would be probably the most complex one as it also allows compiling directly to Java code.

Nonetheless, most of the “custom data type” features you mentioned is already provided by clover. For example the SIN number is probably just a string with some validating pattern - clover StringDataField allows validating pattern to be defined. Similarly for the currency - clover’s decimal field with proper format would do the trick. If this is what you need, you don’t really need to change clover internally, just to provide some kind of wrapper or proxy on top of existing functionality to make it a bit easier for the end user.

As for the full support for user data types - we have been considering it a few times and so far decided that this is not a priority. It might appear in Clover 4.0 which will be major overhaul of current clover.

If you would like to discuss this in a bit more detail, let us know at info@cloveretl.com.

David Pavlis