We are trying to read a file from a SFTP server (hosted by Amazon). From our linux server we can login to sftp and upload/download files. However from clover when we are trying to read a file it says file is unreachable. In fact we can reach the file just fine from sftp prompt
File is unreachable: sftp://[username]:***@[servername]/home/oraclesftp/StatusLink.csv
Cannot obtain connection input stream for URL ‘sftp://[username]:***@[servername]/home/oraclesftp/StatusLink.csv’. Make sure the URL is valid.
Session.connect: java.io.IOException: End of IO Stream Read
17:55:49,247 ERROR FusionConnectWatchDog_0 - 83423 - Error details:
org.jetel.exception.JetelRuntimeException: Component [ReadStatusURLLink:READ_STATUS_URLLINK] finished with status ERROR. (Out0: 0 recs)
at org.jetel.graph.Node.createNodeException(Node.java:636)
at org.jetel.graph.Node.run(Node.java:603)
at java.lang.Thread.run(Thread.java:744)
Caused by: Element [READ_STATUS_URLLINK:ReadStatusURLLink]-Component pre-execute initialization failed.
at org.jetel.graph.Node.run(Node.java:555)
… 1 more
Try checking the URL for any special character that might need escaping. For example, let’s say you are connecting to the server with a user named ‘user#001’ and password ‘pA22w0rd@’. This URL will not work properly:
1. Port is default 22.
2. Password doesn’t contain any special character.
3. You mentioned that the SFTP server is hosted by Amazon. Is it an EC2 instance we are talking about?
Yes
4. Could you provide us with more details about this SFTP server software information (name, version)?
Ubuntu 16.04.3 LTS
Release 16.04
Codename: xenial
Open SSH
1:7.2p2-4ubuntu2.2
5. Can you give us some specs about your Linux server as well (sort of the Linux distribution, version)?
2014 x86_64 x86_64 x86_64 GNU/Linux
6. What is the size of the StatusLink.csv file that you are trying to transfer?
5 KB
7. Which version of CloverETL Designer and Server are you using?
4.3 community edition
We are seeing this error in the SFTP server log when Clover calls it
Nov 13 22:31:50 ip-172-31-28-244 sshd[11209]: Accepted password for oraclesftp from 35.160.99.136 port 47261 ssh2
Nov 13 22:31:50 ip-172-31-28-244 sshd[11209]: pam_unix(sshd:session): session opened for user oraclesftp by (uid=0)
Nov 13 22:31:50 ip-172-31-28-244 systemd-logind[1141]: New session 8 of user oraclesftp.
Nov 13 22:32:33 ip-172-31-28-244 sshd[11243]: fatal: mm_answer_moduli: bad parameters: 2048 2048 1024
Nov 13 22:32:33 ip-172-31-28-244 sshd[11244]: fatal: mm_answer_moduli: bad parameters: 2048 2048 1024
Nov 13 22:32:33 ip-172-31-28-244 sshd[11248]: fatal: mm_answer_moduli: bad parameters: 2048 2048 1024
Nov 13 22:32:33 ip-172-31-28-244 sshd[11247]: fatal: mm_answer_moduli: bad parameters: 2048 2048 1024
Nov 13 22:32:33 ip-172-31-28-244 sshd[11251]: fatal: mm_answer_moduli: bad parameters: 2048 2048 1024
When we call SFTP from linux command line it is working fine.
Are you receiveng the path thru metadata? if so try to put your file directly in your DataReader FileURL, if it works you are having problems with your data string remember if you are receiveing path from meta data you need to use something like this port:$0.URL:source.
Also double check your password doesnt contain an @@@@. i spent like 2 days on an error like this.
Metadata is not the problem. If we use a different SFTP server with the same path, it works fine and I have simplied the password like “abcd” and still it doesn’t work.
The issue with openssh new version on the SFTP server software. Based on the error message it seems they changed something in new version of SSH. Was wondering if clover can set some parameter to alleviate the problem
Found the root cause of the problem. OpenSSH has disabled Key Exchange Algorithm diffie-hellman-group1-sha1. It is important to notice that this algorithm is not considered safe anymore, but Open SSH supports it.
Clover somehow can’t handle the new key exchange patterns of the OpenSSH. Once we disable all other key exchange patterns and kept the diffie-hellman-group1-sha1 active, clover can read the file. However this is a security hole and sys admins are not ready to keep this pattern available.
I spoke with our developers and they strongly feel they can find a solution to this problem, but they need more details first. Could you please answer the following questions:
I’m attaching a graph with CustomJavaComponent, which will print out the supported key exchange schemes of the SFTP. When the graph runs, it will enable verbose SFTP logging at DEBUG log level on the Server, until restarted. Afterwards, you should execute your graph that reads from SFTP with DEBUG log level enabled and send us the log of the execution. To enable DEBUG log level on Designer, please go to Run - Run Configurations - Log Level - DEBUG.