MoveFiles error

CloverDX Platform
1

Greetings,

I’m trying to use the MoveFiles component in a job flow (in a Server project) to move a file from the data-out directory to a remote SFTP location to archive it.

I am able to create the SFTP connection and it validates. However, when I run the jobflow, I get an error:
Component [MoveFiles:MOVE_FILES] finished with status ERROR. (In0: 1 recs, Out0: 0 recs)
Failed to move sandbox://SDX/./data-out/categories_4_2 … 00000.json to sftp://{redacted}:***@{redacted}/4/archive/
Move failed
Copy failed
No such file

The file does exist in the data out directory. The 4/archive folder structure does also exist on the SFTP site.
I have tried including a file name as well (i.e. sftp://{redacted}:***@{redacted}/4/archive/test.json) and I get the same error.

If I try to copy to the root directory of the SFTP site I get a different error (permission denied), but anywhere else I try to copy it says no such file.

I have tried including the file both explicitly by selecting it from the URL dialog and by dynamically specifying the filepath (which is how it will work eventually) but both ways it tells me no such file.

Screen Shot 2017-03-16 at 4.24.14 PM.png

The stack trace is:
org.jetel.exception.JetelRuntimeException: Component [MoveFiles:MOVE_FILES] finished with status ERROR. (In0: 1 recs, Out0: 0 recs)
at org.jetel.graph.Node.createNodeException(Node.java:636)
at org.jetel.graph.Node.run(Node.java:603)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.jetel.exception.JetelRuntimeException: Failed to move sandbox://SDX/./data-out/categories_4_2 … 00000.json to sftp://{redacted}:***@{redacted}/4/test.json
at com.opensys.cloveretl.component.MoveFiles.fail(Main:187)
at com.opensys.cloveretl.component.AbstractFileOperation.processError(Main:531)
at com.opensys.cloveretl.component.AbstractFileOperation.processResult(Main:563)
at com.opensys.cloveretl.component.AbstractFileOperation.mainExecuteOperation(Main:351)
at com.opensys.cloveretl.component.AbstractFileOperation.execute(Main:381)
at org.jetel.graph.Node.run(Node.java:563)
… 3 more
Caused by: java.io.IOException: Move failed
at org.jetel.component.fileoperation.FileManager.move(FileManager.java:448)
at org.jetel.component.fileoperation.FileManager.move(FileManager.java:374)
at com.opensys.cloveretl.component.MoveFiles.executeOperation(Main:202)
at com.opensys.cloveretl.component.MoveFiles.executeOperation(Main:40)
at com.opensys.cloveretl.component.AbstractFileOperation.mainExecuteOperation(Main:339)
… 5 more
Caused by: java.io.IOException: Copy failed
at org.jetel.component.fileoperation.DefaultOperationHandler.move(DefaultOperationHandler.java:233)
at org.jetel.component.fileoperation.FileManager.move(FileManager.java:441)
… 9 more
Caused by: java.io.IOException: Copy failed
at org.jetel.component.fileoperation.FileManager.copy(FileManager.java:354)
at org.jetel.component.fileoperation.DefaultOperationHandler.move(DefaultOperationHandler.java:231)
… 10 more
Caused by: java.io.IOException
at org.jetel.util.ExceptionUtils.getIOException(ExceptionUtils.java:581)
at org.jetel.component.fileoperation.pool.AbstractPoolableConnection.getIOException(AbstractPoolableConnection.java:46)
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.getOutputStream(PooledSFTPConnection.java:441)
at org.jetel.component.fileoperation.PooledSFTPOperationHandler$SFTPContent.write(PooledSFTPOperationHandler.java:648)
at org.jetel.component.fileoperation.FileManager$WritableContentProvider.channel(FileManager.java:573)
at org.jetel.component.fileoperation.DefaultOperationHandler.copyFile(DefaultOperationHandler.java:74)
at org.jetel.component.fileoperation.DefaultOperationHandler.copyInternal(DefaultOperationHandler.java:148)
at org.jetel.component.fileoperation.DefaultOperationHandler.copy(DefaultOperationHandler.java:189)
at org.jetel.component.fileoperation.FileManager.copy(FileManager.java:347)
… 11 more
Caused by: 2: No such file
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2846)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:753)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:700)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:697)
at org.jetel.component.fileoperation.pool.PooledSFTPConnection.getOutputStream(PooledSFTPConnection.java:428)
… 17 more

I’m a bit stumped so assistance is greatly appreciated!
The only thing that seems odd to me is that there is a . in the path in the error message that doesn’t appear in the raw path, i.e.
sandbox://SDX/./data-out/categories_4_2 … 00000.json
vs.
sandbox://SDX/data-out/categories_4_201 … 00000.json

Thanks,
Anye

2

Hi,

are you using embedded browser to select path on a remote server, or you’re copy&pasting address? Error you’re getting might relate to file system restrictions. What kind of remote system we’re talking about? Linux? Windows?

The thing is, when you’re trying to reach a directory let’s say “/my/directory/structure/is/very/deep.txt” and you don’t have “execute” permission to directory “structure” at Linux OS, you’ll get “directory does not exist” instead of expected “permission denied” error. Could not this be your case?

For example, I get following error:

--------------------------------- Error details ----------------------------------
  Component [ListFiles:LIST_FILES] finished with status ERROR. (Out0: 0 recs)
   Failed to list ftp://clover:***@griffin/opt/VBoxGuestAdditions-5.0.14/bin
    Directory listing failed
     File not found: "ftp://clover:***@griffin/opt/VBoxGuestAdditions-5.0.14/bin"
----------------------------------------------------------------------------------

When “VBoxGuestAdditions-5.0.14” has following permissions:

drwxr-xr--.  2 root   root      6 Jun 21  2016 VBoxGuestAdditions-5.0.14

If I:

chmod o+w /opt/VBoxGuestAdditions-5.0.14

my graph finishes successfully. Using vsFTP@CentOS 7.

3

Hi Pavel,
That wasn’t it but your suggestion helped me figure it out.

I added a ListFiles component beforehand to try to view the contents of the directory I was trying to write to. It turns on that when I *manually* sftp to that server and log in as the user in question it drops me in that user’s folder automatically. However, when the Clover server sftp’s there, you have to additionally add the user folder to the path or it is trying to look under the root.

So the problem was that it was actually trying to drop to serverdomain/4/archive rather than serverdomain/userfolder/4/archive
and once I manually added the username folder to the path, it worked.

So thank you very much :slight_smile: