Page 1 of 1

XML Extract Datamapping Issues.

Posted: Wed Nov 11, 2015 6:30 am
by siddhant
Hi guys,

From last couple of day. I am having this issues. I have a XML file which has id tag and below it has a host tag as given below


<video>
<id>434223529</id>
<host id="70865">Amazon.com</host>
<url>http://www.amazon.com/piv-apk-play?asin=B0112O85ZE</url>
<image>http://ecx.images-amazon.com/images/I/516IkZIDS0L._SX320_SY240_.jpg</image>
<updatedAt>2015-09-04</updatedAt>

when i map the id tag in xml extract component. Instead of getting the data in id which is 434223529 tag i am getting,the data of id i.e.70865 which is the attribute host tag.

To avoid this I tried to ues XMLReader but my XMl File is about 413 MBs so i am getting an java heap space error.
Please Help

Thank You.

Re: XML Extract Datamapping Issues.

Posted: Mon Nov 16, 2015 10:40 pm
by slechtaj
Hi Siddhant,

I assume the issue is caused by the fact you use auto-mapping in your XMLExtract mapping. Auto-mapping maps the fields (no matter if it's element or attribute) to output metadata fields with the same name. All you need to do is to turn off the auto mapping and map the elements and attributes manually to the output fields as required. See the below screenshot for more information.

MapXMLExtractFields.png
MapXMLExtractFields.png (71.84 KiB) Viewed 8595 times


Hope it helps!

Re: XML Extract Datamapping Issues.

Posted: Mon Dec 21, 2015 3:11 pm
by siddhant
Thank you sooooooooooooooooo much..

I require your one more help pls


I am reading a file from the source below

https://api-public.guidebox.com/v1.43/U ... ovie/19975

using jsonExtract

I have the issue that, each time to load a new record , the number that is 19975 at the end of that URL must change dynamically, it will be provided by my databases actually they are the movie id's .
Is there any way by which i can do that

so actually my url must be like this

https://api-public.guidebox.com/v1.43/U ... Biv/movie/{value returned form database table}.

and each time i load a new set of data i want to write it simultaniously on a jason file on my drive

Re: XML Extract Datamapping Issues.

Posted: Mon Jan 04, 2016 4:07 pm
by slechtaj
Hi Siddhant,

You can prepare the URL with a reformat and send it through the edge to the input port of the JSONExtract. In File URL attribute you will use the input port with processing type source. Resulting value of the File URL component will look like this:

Code: Select all

port:$0.yourUrlField:source

Re: XML Extract Datamapping Issues.

Posted: Tue Jan 05, 2016 10:53 am
by siddhant
Hi slechtaj,
Thank You So much once again.

I have manage to prepare those url any how, but now the issue i am facing is quite serious . There are almost 94000 such url's
which i am giving an input to my JSON EXTRACT component in its file URL property.

But when I run the graph JSON EXTRACT fails for any of the random url , giving me error "Url not reachable, check if the URL is Valid or not". but when I checked that URL on the WEB browser it was accessable.

But in this case my whole graph fails, to avoid this faliure and to re access that failed URL from JSON EXTRACT what can i do?
please suggest me some thing as early as possible...........

Re: XML Extract Datamapping Issues.

Posted: Tue Jan 05, 2016 2:17 pm
by siddhant
hi slechtaj,

I had one more query regarding accessing the same urls from different components in the same graph.
Is it possible that i can hit 1 lakh pre prepared urls in database from many different JSON EXTRACT Component at a time
will there be any dead lock situation

Re: XML Extract Datamapping Issues.

Posted: Tue Jan 05, 2016 4:47 pm
by slechtaj
Hi,

I'm sorry I didn't realize you need to process remote file. In this case, you can get the content of the file using HTTPConnector. Simply pass the URL into the HTTPConnector using Input mapping attribute and then map the content of the file to the output field using the Output mapping attribute. In JSONExtract make sure the processing type is set to discrete.

Re: XML Extract Datamapping Issues.

Posted: Wed Jan 06, 2016 10:41 am
by siddhant
Hi slechtaj,
I have done according to what you said. The graph is working now. Thank you soooo much.
I am highly grateful for your quick response.

Re: XML Extract Datamapping Issues.

Posted: Wed Jan 06, 2016 2:15 pm
by siddhant
Hi Slechtaj,

I had a query regarding the mapping of nested child elements in JSON EXTRACT component.
I have to map all the child elements to a single output port.
I am attaching a graph and my source file below. Please have a look.
This file contains many child elements which are further nested, I want to map them all to the output port 0 .
is it possibble??? please let me know as early as possible

Thank You....
Copy of new-graph.grf
(1.84 KiB) Downloaded 266 times





Copy of new-graph.grf
(1.84 KiB) Downloaded 266 times

Re: XML Extract Datamapping Issues.

Posted: Tue Jan 12, 2016 3:10 pm
by slechtaj
Hi Siddhant,

I am not sure if I understand what you need. However, if your goal is to get all the information from the XML file into one Clover record, you will have to use multiple output ports at first and then join them together using a key (in your case it might be id of the record).
You should also keep in mind two things:
  • First you have to always start the mapping from the most nested object (for example link attribute of the facebook object. Each nested object has access to its parent object values.
  • Second JSONExtract uses SAX parser, therefore if an object has multiple attributes (e.g. facebook element has facebook_id and link) you should set the mapping on the last attribute, otherwise you wouldn't be able to map the value of the other attributes.


I have prepared an example that should help you understand the logic - see the attached graph.

Re: XML Extract Datamapping Issues.

Posted: Mon Jan 18, 2016 8:38 am
by siddhant
Hi slechtaj,

Thank You soooo much for your reply on my data mapping issue query.
Here now i am facing a new issue which i am not able to understand.
my graph fails when i am trying to load major data load in database ,my graph fails. i am not able to understand the error logs .
I am attaching the error logs below ,can you please help me to figure out what is the issueand how can i resolve it.
Error_Log.txt
(94.64 KiB) Downloaded 175 times

Re: XML Extract Datamapping Issues.

Posted: Tue Jan 19, 2016 8:27 am
by siddhant
Hi slechtaj,

I had a query regarding error handling in cover ETL. Is there any possible way in which i may be able to trap the error thrown by any of the component ie my dbOutputTable or JSONExtract component in my graph and continue the execution of graph to fetch data from web server and insert it in to database.i am fetching bulk data from web servers and suddenly i get json parser error and my graph fails . is there any way by which i can resume the graph from point of faliure

Thank You,
Siddhant Dilip Satav

Re: XML Extract Datamapping Issues.

Posted: Wed Jan 27, 2016 10:10 pm
by slechtaj
Hi Siddhant,

To you first question, the log is from a parent graph so it contains only very limited information about the child graph. However, it seems you are facing some kind of network connection issues. I would recommend you first checking if you are able to do the same action from another client application.

Regarding the second question, Clover components of this type are usually able to handle this situations, you can define the amount of failed records on a component (DBOutputTable) and attach an output port to collect these records for further processing. When it comes to the JSONExtract, or XMLExtract components, you should always make sure at first if the document you want to read is valid and matches the structure your mapping is tailored to.

Hope it helps.