I have a need to consume data from a RESTful web service that returns JSON, simple enough. Here’s the catch – the HTTP connection is persistent meaning that it never closes. The JSON data will continue to stream in until the connection is forcibly killed by the client (CloverETL) or the server (the web service).
Are there any mechanics in CloverETL that would allow this JSON data to “flow” through the grf as data comes available? What are my options here?
Hello,
HTTPConnector component doesn’t close the connection explicitly, so it should work according your requirements.
Thank you for your reply. It is helpful that the HTTP Connector component doesn’t end the connection, but there’s a problem: I need the data collected to start flowing to the output edge (and on to other components) while the HTTP Connector is still collecting data (i.e. the HTTP Connector hasn’t received a HTTP200 flagging end of transmission).
The services I am consuming from actually has two variations of connection – one that sends 10 JSON responses and that’s all and one that is persistent and never closes. I am able to move data beyond my HTTP_Connector component for the first one, but not the second.
Any thoughts? Is a custom HTTP Connector needed here?
I thought I replied earlier, but my post is not showing on the site.
The HTTP Connector helps because it allows the connection to stay open, but it will not put any data on the output edge until the HTTP200 successful response is received and socket closed. This is my issue. I need data to flow across my graph while the HTTP Connector continues to collect JSON “records” in the response. Any ideas?
Hello,
try to set edge after the HttpConnector component as directFastPropagate (see Types of Edges), then all data, that the component receives should be sent immediately to the next component.
Agata,
I work with Dan and I tested the proposed fix. However it did not seem to make a difference. Is there something else I can try?
Hello guys,
unfortunately HttpConnector really sends the response to the output field after receiving the full response. But it writes the response continuously, when writing it to the output file. If you ran the graph on CloverETL Server, you would send the response from the HttpConnector to the file and read it by another graph that would be started by File event listener.
Anyway I’ve created an issue in our bug tracking system for this feature (https://bug.javlin.eu/browse/CL-1919). You can browse it with guest/guest credentials.