Memory/Space Estimating Tool?

Support/help with CloverETL (4.9) and CloverDX (5.0 or newer) implementation problems

pintail
Posts: 33
Joined: Wed Aug 27, 2014 6:58 pm

Memory/Space Estimating Tool?

Postby pintail » Tue Feb 03, 2015 9:52 pm

Does anyone have a memory and/or space estimating tool? I'm looking to estimate how much temp space and memory space I'll need to process my graphs once I take some of the record filters off of them that I've used for development to limit the counts going through.

Thanks in advance for any help

imriskal
Posts: 399
Joined: Wed Aug 15, 2012 8:18 am

Re: Memory/Space Estimating Tool?

Postby imriskal » Thu Feb 05, 2015 12:37 pm

Hi pintail,

Sorry but we have no such estimation tool. I think it is impossible for the computer to estimate those things without running the graph first. I can give you a few tips though.

1) Memory space depends on the number of input records and their size and the way you process them. (the components used, CTL functions used, efficiency of processing, number of graph phases, ...) Especially components like sorters, joiners or some others which keep all records in memory can have high influence on the overall memory requirements.
2) Temp space depends on the number of debug edges, buffered edges and size of the buffers, the number of temp files you create and the number of records stored in them.
3) Graph run log shows the amount of memory used. Exploring a graph log from a run in development environment, you can get a hint about the memory requirements for production.
4) You can also use tools like jvisualvm to see and examine JVM behavior during the graph run. If you see the results for a graph with 100 records processed and the other one for 200 records, you should be able to estimate the results for 300 records.

I know I provided you only with a high level view on the topic so do not hesitate to ask specific things you are interested in.
---
Lubos Imriska
CloverCARE Support
CloverDX

Visit us online at http://www.cloverdx.com

Sudharshan
Posts: 1
Joined: Fri Oct 16, 2020 10:47 am

Re: Memory/Space Estimating Tool?

Postby Sudharshan » Fri Oct 16, 2020 1:18 pm

How to overcome the following issue "Temp space E:\CloverDXServer.5.2\temp\clover_temp_node01 is almost full".Can you please sort me out this problem.

dpavlis
Posts: 189
Joined: Sat Mar 10, 2007 8:12 pm

Re: Memory/Space Estimating Tool?

Postby dpavlis » Mon Oct 19, 2020 9:34 pm

Hi,

Since it looks like you are using CloverDX Server then TEMP space configuration options are available in our documentation.
David Pavlis
CloverCARE Support
CloverDX | Rapid Data Integration

Visit us online at http://www.cloverdx.com