Need URI from XMLWriter to HTTP connector

CloverDX Platform
1

Hi,
I’m trying to create a Graph that generates an XML string which is pushed to a REST web service which is the interface for a data store. The REST web service I’m working with requires the URI for the XML I want to post to be specified as a request parameter in the URL.

Currently, I have an XMLWriter write out the XML as a string to the output port. However, in the HTTP connector, I need to be able to reference a field to pass as the URI parameter. For example:

http://host:port/documents?uri=/test/blah.xml

(and the XML string is in the request body)

It seems like the XMLWriter can only push one field through its output port. Is there any way for the XMLWriter to also send another field through the output port that represents the URI?

If there’s no easy out of the box mechanism for this, it seems like I could add a Reformat component to parse the XML and construct a URI to pass along as a 2nd field through to the HTTP connector. But even then, it doesn’t seem like CTL supports any sort of XML parsing so I’d have to write it in Java.

Please let me know there is some easier way of handling this use case!

Thanks,
Peter

2

Hi Peter,

Let me go back to your question. You need to parse the URI out from the XML that you create in XMLWriter. Is that correct? If so, you may use the following way. If generated XML contains the URI/URL, it has to „exist“ before it is sent to XMLWriter. In this case, you can save it to dictionary and load it in reformat (that receives data from XMLWriter and send them to HTTPConnector). Using dictionary, you can easily send the URI/URL into record field without parsing the XML. For you convenience, I attached sample graph. Please let me know, if this helps you.

3

Thanks for the response. I am not very familiar with the Dictionary. Is this an object that is shared globally across the execution of the Graph? Or is there a single Dictionary object associated with each record that flows through the Graph?

I’m asking because each record will have its own URI. For example, a record may have the following key-value pairs:

Name: John Smith
Address: 123 Main Street, New York, NY 10001
Phone (666) 666-6666
URI: /info/john-smith.xml

The XMLWriter will construct an XML document like this:

/info/john-smith.xml John Smith 123 Main Street, New York, NY 10001 (666) 666-6666

When this gets to the HTTP connector, I need to make an HTTP request to the web service where it parses the URI from the element and submits the request to the URL:

http://host:port/documents?uri=/info/john-smith.xml

with the request body containing the XML.

If there is a separate dictionary object associated with each record going through the Graph, then this will work perfectly. However, it is meant to be a single globally-scoped key-value pair store shared across the entire execution of the graph, I don’t think this will work for me and will need another solution.

Thanks,
Peter

4

Hi Peter,

Now I think I understand what you need. There is one component that may help you. It is component Combine (which is available in CloverETL Designer version 3.3.0). Basically it takes one record from all input records and merge them together (it maps input data to output port). Please see attached example.

5

Thanks. I think this might work for me but I’ll have to test it. This solution basically assumes that there are no components/logic between the SimpleCopy and Combine components that change the ordering of the records, correct? Since there’s no key parameter on the Combine component, it seems to assume that you’ve designed the Graph such that the two input records are guaranteed to be related, without requiring a key definition.

This seems slightly “hacky” to me. I think it would be a great enhancement to be able to have the XMLWriter pass out more than one field through its output port, so that this use case is supported in a more intuitive manner.

Thank you very much for your help and I will respond back to this thread if this doesn’t work for some reason.

Peter

6

Hi Peter,

Your assumption is correct. However, for the use case you described above, this would be correct design. Data are sent from DataGenerator to SimpleCopy that sends these records to all output ports in the same order. XMLWriter does not change the order of incoming records, therefore these records (formatted as XML) are sent to Combine in the same order as they were sent from DataGenerator/SimpleCopy. On the other hand, in order to make the design clear, you should attempt to put as little components in between SimpleCopy and Combine as possible.