JSON reader mapping for Twitter JSON

CloverDX Platform
1

Hi,

I am trying to fetch tweets and user details from tweeter JSON file.

For a simple JSON
{
“md5”: “fa4c6baa0812e5b5c80ed8885e55a8a6”,
“original”: “example_text”
}

Mapping is :-

But what mapping should we do when there are other internal objects/arrays like

{
“metadata”: {
“resulttype”: “recent”,
“isolanguagecode”: “en”
},
“md5”: “fa4c6baa0812e5b5c80ed8885e55a8a6”,
“original”: “example_text”
}

Please guide me on this …

Thanks

Sandeep Thakur

2

Hi Sandeep,

The nested information should be of course sent to another output port, it can be done like this:

<Context xpath="/root/object" outPort="0">
  <Mapping cloverField="md5" xpath="md5"/>
  <Mapping cloverField="original" xpath="original"/>
    <Context xpath="//metadata" outPort="1">
      <Mapping cloverField="resulttype" xpath="resulttype" />
      <Mapping cloverField="isolanguagecode" xpath="isolanguagecode"/>
    </Context>
</Context>

Regards,

3

Thanks a lot!!

This resolved my problem…

There is new one I encountered:- if there are special characters involved in the “values” like

“source”: “Twitter for Windows Phone”

ETL throws an error during reading the json file.

Is there any way to escape these character “<” “>” across the file?

Regards

Sandeep Thakur

4

Further please find attached, mapping.xml,test.json and error_logs for above mentioned error…

Please guide me how to remove this parsing error…

Thanks

Sandeep

5

I am sorry but if the JSON file contains the line you posted, it is not a valid JSON file and our readers can not process it. The issue is not in angle brackets, it is in quote marks delimiting both strings and attributes of html elements. If Twitter returns such output, you could report the issue there.

FYI, here is an useful tool for JSON validation: http://jsonformatter.curiousconcept.com/

Regards,

6

Hi,

I have resolved this problem with following code…


Context xpath="/root/array" outPort="0">
         <Mapping cloverField="id" xpath="id" />
         <Mapping cloverField="text" xpath="text" />
         <Mapping cloverField="created_at" xpath="created_x005fat" />
         <Mapping cloverField="number1" xpath="geo/coordinates[1]"/>
         <Mapping cloverField="number2" xpath="geo/coordinates[2]"/>        
</Context>