Excel Output

CloverDX Platform
1

Hi

I have a graph that writes multiple tabs (around 8) into Excel. The output excel size is around 3mb. But I keep hitting JVM heap space issue when using spreadsheetwriter or XLSdatawriter. I was able to run this now by increase the jvm size from 256mb to 1gb. My question - Is there any better way to do this to use as minimum space as possible. 1gb sounds like a lot of memory to write into a 3mb excel file.

Below is the part of execution summary

INFO [WatchDog] - -----------------------** Summary of Phases execution **---------------------
INFO [WatchDog] - Phase# Finished Status RunTime(sec) MemoryAllocation(KB)
INFO [WatchDog] - 0 FINISHED_OK 3 96958
INFO [WatchDog] - 1 FINISHED_OK 2 129966
INFO [WatchDog] - 2 FINISHED_OK 2 170021
INFO [WatchDog] - 3 FINISHED_OK 2 209959
INFO [WatchDog] - 4 FINISHED_OK 2 248970
INFO [WatchDog] - 5 FINISHED_OK 2 288984
INFO [WatchDog] - 6 FINISHED_OK 2 328735
INFO [WatchDog] - 7 FINISHED_OK 14 439756
INFO [WatchDog] - 8 FINISHED_OK 2 400166
INFO [WatchDog] - 9 FINISHED_OK 7 402445

500Mb seems to work now, but I am not convinced this would take so much space. Each node has an input that reads from datasource and writes into a tab.

Thanks,
Praveen Boddu

2

Hi Praveen,

XLSdatawriter handles excel files by DOM method which is adding huge overhead to data size. This explains high memory usage.

In commercial version of CloverETL since version 3.3 is available new component http://doc.cloveretl.com/documentation/ … riter.html which under some constraints is able to do streamed writing. Therefore uses minimum amount of memory.