Search found 16 matches

by anyeone
Fri May 11, 2018 3:38 pm
Forum: CloverDX (formerly CloverETL) Support
Topic: Best practices for health checks against Clover Server / Execution History filters question
Replies: 4
Views: 1712

Re: Best practices for health checks against Clover Server / Execution History filters question

Addendum to the question: if the security check jobflow is the best practice, is there a way we can make it not require a username & password when being executed through http? My devops guru says route 53 doesn't support it so would need to put up a proxy to attach the auth header, but I'm wonde...
by anyeone
Thu May 10, 2018 8:34 pm
Forum: CloverDX (formerly CloverETL) Support
Topic: Best practices for health checks against Clover Server / Execution History filters question
Replies: 4
Views: 1712

Best practices for health checks against Clover Server / Execution History filters question

Greetings, We recently had an interesting situation where the Tomcat server hosting our Clover server was up, but Clover itself had gone down. Our internal job error messaging protocols didn't run because of course Clover wasn't running, and our AWS health checks on the VM accessibility didn't fire ...
by anyeone
Tue Nov 21, 2017 4:40 pm
Forum: CloverDX (formerly CloverETL) Support
Topic: Optimizing for ExtSort with very large file
Replies: 3
Views: 2470

Re: Optimizing for ExtSort with very large file

The graphs involved are extremely complicated, and the ExtSort occurs many places in the process so it will be kind of hard to send it in a usable way for you. We upped the JVM heap to 8GB and that seems to have resolved the issue at least for now, but it seems to me that there should be a way to op...
by anyeone
Tue Nov 14, 2017 12:39 am
Forum: CloverDX (formerly CloverETL) Support
Topic: Optimizing for ExtSort with very large file
Replies: 3
Views: 2470

Optimizing for ExtSort with very large file

Hi all, I have a graph that keeps failing with GC out of memory error on the ExtSort routine. There are millions of records in the file that need to be sorted and the file itself is over 2GB. Is there a way I should configure the ExtSort itself to avoid this problem? I was looking at this page in th...
by anyeone
Thu Apr 13, 2017 6:37 pm
Forum: CloverDX (formerly CloverETL) Support
Topic: Nested JSON Extract
Replies: 2
Views: 2440

Re: Nested JSON Extract

Try the Denormalizer component for turning multiple records into fewer. I'm not sure what you're actually asking in your second question but there are a number of Joiner components that you might be able to use depending on what you're trying to do.
by anyeone
Thu Apr 13, 2017 6:35 pm
Forum: CloverDX (formerly CloverETL) Support
Topic: How to make HTTPConnector POST request with JSON from JSONWriter
Replies: 2
Views: 2791

Re: How to make HTTPConnector POST request with JSON from JSONWriter

Hi there, I've been posting to a lot of REST services from my Clover projects so hopefully I can assist. If you have the JSON write to a port, you should be able to send it as input to a Reformat object that has the HTTPConnector_Request as its output metadata. Then, map the JSON to the requestConte...
by anyeone
Thu Apr 13, 2017 5:31 pm
Forum: CloverDX (formerly CloverETL) Support
Topic: Partitioning JSONWriter output not working & hiding elements from the xml
Replies: 2
Views: 2220

Re: Partitioning JSONWriter output not working & hiding elements from the xml

I still don't have the partitioning working, but I found another solution to the two issues (not quite as elegant, I fear) but I thought I would share. Instead of partitioning, in my main jobflow I use an aggregate to get the list of distinct catalog names that appears in both the category and the p...
by anyeone
Wed Apr 12, 2017 6:23 pm
Forum: CloverDX (formerly CloverETL) Support
Topic: Partitioning JSONWriter output not working & hiding elements from the xml
Replies: 2
Views: 2220

Partitioning JSONWriter output not working & hiding elements from the xml

I've been trying to get data to partition into multiple files using a JSONWriter and though I thought I had it distilled into a simple case, but it is not working properly in two ways. 1) I have flattened my data into a single input port that has a field called "catalog_id" that is the par...
by anyeone
Tue Apr 04, 2017 5:24 pm
Forum: CloverDX (formerly CloverETL) Support
Topic: Reading xml file out of zip file issues
Replies: 3
Views: 3447

Re: Reading xml file out of zip file issues

Thanks, I had gone forward with other work on these jobs and graphs so once I get the branch with the attempted zip file processing merged in with the latest and have all the latest logs. I'll send you all the requested info. I don't have a problem with posting here, it's not super sensitive, but th...
by anyeone
Fri Mar 31, 2017 3:02 am
Forum: CloverDX (formerly CloverETL) Support
Topic: Reading xml file out of zip file issues
Replies: 3
Views: 3447

Reading xml file out of zip file issues

Greetings, I have a zip file that contains two XML files that I'll call the catalog file and the price file. In my jobflow I move the zip file from a remote location to the DATAIN_DIR and then attempt to read from each of the XML files using XML Extract in a couple different places. According to thi...
by anyeone
Mon Mar 20, 2017 8:45 pm
Forum: CloverDX (formerly CloverETL) Support
Topic: MoveFiles error
Replies: 2
Views: 2650

Re: MoveFiles error

Hi Pavel, That wasn't it but your suggestion helped me figure it out. I added a ListFiles component beforehand to try to view the contents of the directory I was trying to write to. It turns on that when I *manually* sftp to that server and log in as the user in question it drops me in that user's f...
by anyeone
Thu Mar 16, 2017 10:26 pm
Forum: CloverDX (formerly CloverETL) Support
Topic: MoveFiles error
Replies: 2
Views: 2650

MoveFiles error

Greetings, I'm trying to use the MoveFiles component in a job flow (in a Server project) to move a file from the data-out directory to a remote SFTP location to archive it. I am able to create the SFTP connection and it validates. However, when I run the jobflow, I get an error: Component [MoveFiles...
by anyeone
Fri Feb 24, 2017 4:59 pm
Forum: CloverDX (formerly CloverETL) Support
Topic: Type issue mapping parameters to ExecuteGraph
Replies: 2
Views: 2177

Re: Type issue mapping parameters to ExecuteGraph

That's how I've been handling it, thanks for clearing it up!
by anyeone
Thu Feb 23, 2017 11:36 pm
Forum: CloverDX (formerly CloverETL) Support
Topic: Type issue mapping parameters to ExecuteGraph
Replies: 2
Views: 2177

Type issue mapping parameters to ExecuteGraph

Hi, I'm trying to use the ExecuteGraph component to execute a graph that contains several integer parameters that I wish to set from the calling JobFlow. The issue that I am repeatedly seeing is that even when I declare the parameters as an integer on the child graph, the Input mapping of the Execut...
by anyeone
Thu Feb 23, 2017 4:30 pm
Forum: CloverDX (formerly CloverETL) Support
Topic: Dynamic number of partitions
Replies: 3
Views: 2499

Re: Dynamic number of partitions

Perfect! Thanks, gentlemen!

cron