Http connector response file

Support/help with CloverETL (4.9) and CloverDX (5.0 or newer) implementation problems

gaurav_kapoor
Posts: 1
Joined: Mon Jul 23, 2018 6:11 pm

Http connector response file

Postby gaurav_kapoor » Mon Jul 23, 2018 6:14 pm

Hi,

I am using http connector to make a API call and receive the response file in tmp directory. Since its a big dataset, more than 200k records but I see Clover is limiting it to 100,000 records. I am unable to find the property which limits this dataset. Where can I change this property to pull full dataset.

jandikovae
Posts: 41
Joined: Fri Nov 04, 2016 8:51 am

Re: Http connector response file

Postby jandikovae » Fri Aug 03, 2018 9:19 am

Hi,

Off the top of my head, I am not aware of any parameter or functionality that would limit the number of records or size of response content while using especially HTTPConnector. As I might have not fully understood your use case, would it be possible to provide us with an example of your graph or more detailed explanation what you would like to achieve? (Please internalize your metadata and/or other dependencies to the graph and feel free to remove any sensitive data from your graph.)

Generally, CloverETL allows limiting the number of records in Reader components. Therefore if you process the HTTP response content in a FlatFileReader, for instance, check whether you have any value in the "Max number of records" Property in the reader. (In such a case the graph would look like as in the following forum post https://forum.cloveretl.com/viewtopic.php?t=7407).

In addition, you can control also the max size of each record and other useful parameters by changing so-called Engine properties. For more information see the following documentation page:
http://doc.cloveretl.com/documentation/UserGuide/index.jsp?topic=/com.cloveretl.gui.docs/docs/cloveretl-default-settings.html

Have a nice day, Eva
---
Eva Jandikova
CloverCARE Support
CloverDX

Visit us online at http://www.cloverdx.com


cron