Samba (java.lang.OutOfMemoryError) - File-size limitation?

CloverDX Platform
1

Good day,

I have, unfortunately, run into another problem using samba to get files from windows shared folders.

When I try to move files of size lower then 3,8 MB everything works well, but If the file is larger then 3,8 MB file move operation starts, components are initialized successfuly, but after creating terget file and copying approx. 3,8MB of source file content to target file, the process is interrupted by following exception:

at org.jetel.graph.Node.createNodeException(Node.java:566)
at org.jetel.graph.Node.run(Node.java:549)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:713)
at jcifs.UniAddress.lookupServerOrWorkgroup(UniAddress.java:173)
at jcifs.UniAddress.getAllByName(UniAddress.java:290)
at jcifs.UniAddress.getByName(UniAddress.java:245)
at jcifs.smb.Dfs.getTrustedDomains(Dfs.java:62)
at jcifs.smb.Dfs.resolve(Dfs.java:167)
at jcifs.smb.SmbFile.resolveDfs(SmbFile.java:671)
at jcifs.smb.SmbFile.send(SmbFile.java:773)
at jcifs.smb.SmbFileInputStream.readDirect(SmbFileInputStream.java:181)
at jcifs.smb.SmbFileInputStream.read(SmbFileInputStream.java:142)
at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
at sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:630)
at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:668)
at org.jetel.util.stream.StreamUtils.copy(StreamUtils.java:95)
at org.jetel.component.fileoperation.DefaultOperationHandler.copyFile(DefaultOperationHandler.java:83)
at org.jetel.component.fileoperation.DefaultOperationHandler.copyInternal(DefaultOperationHandler.java:146)
at org.jetel.component.fileoperation.DefaultOperationHandler.copy(DefaultOperationHandler.java:187)
at org.jetel.component.fileoperation.FileManager.copy(FileManager.java:342)
at org.jetel.component.fileoperation.DefaultOperationHandler.move(DefaultOperationHandler.java:229)
at org.jetel.component.fileoperation.FileManager.move(FileManager.java:436)
at org.jetel.component.fileoperation.FileManager.move(FileManager.java:369)
at com.opensys.cloveretl.component.MoveFiles.executeOperation(Unknown Source)
at com.opensys.cloveretl.component.MoveFiles.executeOperation(Unknown Source)
at com.opensys.cloveretl.component.AbstractFileOperation.mainExecuteOperation(Unknown Source)
at com.opensys.cloveretl.component.AbstractFileOperation.execute(Unknown Source)
at org.jetel.graph.Node.run(Node.java:505)

The source file is not deleted, because process is interrupted before it finishes full copy - these are good news.
Of course, we are working with huge data, therefore limitation approx. 3,8 MB max. file size for move operation is more then ridiculous and leaves smile on my face :smiley: because this can’t be bug, it must be something trivial to fix, but I can’t figure it out…

I searched options in Clover Designer but I did not find any settings for file size limitation in “Remote files” section and there is no “Samba” section, therefore I guess that this is samba config (smb.conf) or some server limitations…

I am not even saying that this must be limitation of size of file.

But it is strange, that error speaks about OutOfMemory problem with creating new thread, should have been some insufficient privileges, permision denied or similar sort of exception message If it is caused by limitation of some sort - I assure you, that I ran it with no other running processes/graphs etc. therefore unsufficient memory is out of question as a reason of this problem.

Any hint is apppreciated.
In advance, I thank you very much for help.

Best regards

Tomáš Šimčík
Data Integration & Analytics Specialist
2nd level support

Global Payments Europe
V Olšinách 80/626
100 00 Praha 10
www.globalpaymentsinc.com

2

Hi Tomas,

„java.lang.OutOfMemoryError: unable to create new native thread“ is a standard exception thrown by Java. The issue might be caused by several reasons. The most common reasons for this issue are:

  • Insufficient user’s limits

  • Native memory depletion

Could you provide us with the following information?

  • Does the graph run on the CloverETL Server?

  • What Java version you use (32/64 bit)?

  • User resource limits (Ulimit) set for the app-server user

You may find some tips to solve the issue in this article: http://javaeesupportpatterns.blogspot.c … e-new.html
Especially Problem diagnostic chapter is something what might bring you to the solution.
Another useful source of information may be here: http://www.mastertheboss.com/jboss-moni … ive-thread

If you would like to do some more detailed research, for application monitoring, you may use JVisualVM that you can find in the bin folder of your JDK.

Hope this helps.

3

Hi,

Thanks for reply - I am sorry for not responding for quiet some time…

Answers:
a) graph runs on server (executed via designer - not scheduled)
b) 64
c)
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 256675
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 256675
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Thanks for help

4

btw I read several topics on memory depletion, but on 64 bit system, and while no other processes run on server, there is no way that this could be caused by native depletion of memory…

I am successfuly running graphs that use-up to 20MB of memory (generating prefix sequences with over 25 mil. of records from scratch and/or using fast sort on it with flagged “in memory”) while other processes are running… and I had no problems.

But whenever I try to move or copy files with samba this error keeps occuring if file size exceeds 3,8MB.
sftp, for example, does not have this problem.

I really need to move ahead with this, because there are many projects waiting due to this problem.

I even thought about installing sftp server on windows to exclude samba and use sftp, but that is not solution and it has its troubles as well, because windows servers are, yet again, administrated by internal IT departement.

It is much more simple to just allow shared folder then request installation of sftp listener…

Any help is much appreciated.
Kind regards

Tomáš Šimčík
Data Integration & Analytics Specialist
2nd level support

Global Payments Europe
V Olšinách 80/626
100 00 Praha 10
www.globalpaymentsinc.com

5

Hi Tomas,

I tried to reproduce the issue on my side. I tried small files and big files. I tried to move file in scope of localhost, local network and Internet in both directions. I tried Windows host and Linux host. I tried CloverETL version 3.5.0 and 4.0.0-M1. No luck so far, everything works fine.

Could you please describe your network environment? Do you use Windows domains? Is the shared smb directory placed on Windows or Linux machine? Which version of CloverETL do you have? Does at least one direction work? (move from smb to local folder or from local folder to smb)

We discussed this issue internally and found an article where setting the following two properties helped in the similar situation:

-Djcifs.resolveOrder=DNS
-Djcifs.smb.client.dfs.disabled=true

See the whole article here: http://www.ryanchapin.com/fv-b-4-648/ja … tream.html

Let me know with the answers and try the aforesaid workaround, please.

Regards,

6

Hi,

just note: parameters mentioned by Lubos must be passed to JVM of Server during application server initialization. How to do that depends on application server you use. If you are not sure how to do that, please let us know about app. server you use and we will advice.

To check you set this properties correctly you can go into CloverETL Server web console, tab Configuration->System info->System properties. There should be mentioned properties listed as key-value pair.