Webdav credentials in cloverETL server

CloverDX Platform
1

Hi,
Newbie question:
How to provide credentials for accessing a webdav folder from within a UniversalDataReader on CloverETL Server?

Scenario:
Need a location accessible by both a graph on the server, and to users who will deliver files to a folder.
… yes a file trigger then launches the graph on file delivery.

Is this possible using CloverETL Server? The webdav is hosted by the server itself http://myserver:8080/clover/webdav/myfolder

Does the file listener function against a webdav folder?

Thanks in advance for your help.

2

Hi twaldock,

your wedav address is “http://myserver:8080/clover/webdav/…”.

So the webdav interface exposes your sandboxes (and their content) to the remote client. Then why not create “File system listener” on the local sandbox directory where you expect the added file?

Does this help?

3

Thanks kubosj!

As an alternate approach the folder was exposed as a Windows share, and a file trigger placed on it. Works fine! Next will have to solve the ‘too fast’ condition of CloverETL reaching EOF before the file has been fully written. Suggestions?

The Windows share was implemented for a newbie reason: couldn’t precisely find where Clover’s webdav root aimed at, and the permissions / authentication game was not clear.
On a linux box this might be easier, but on this windows server less so.
Is there a document for CloverETL server that spells out:
1. where the webdav config is located,
2. how to secure it under clover (Tomcat’s suggestions break CloverETL server; had to revert to unsecured and unauthenticated. Then again, my fingers could have been guilty).
3. where the webdav root really is and how to control/re-aim it to a different path.

Also, the original question on credentials for CloverETL’s webdav is still open: how or where are those set?

Thanks!

4

Hi twaldock!

Thanks kubosj!
Next will have to solve the ‘too fast’ condition of CloverETL reaching EOF before the file has been fully written. Suggestions?

“twaldock”

I am not aware of any specialized mechanism to wait for transfer end. You can one of:

  • copying large file to another directory first and then moving it to directory where you listen for file changes (SMB should support a quick move opreation) – I would recommand this

  • insert a simple waiting phase to your graph to be executed before the logic self (drawback: it may be hard to estimate the time needed for network to finish copying)

Inserting waiting phase:

  • select all your components

  • select Set phase from right click context menu

  • select Inc/Dec phase by and enter 1.

  • confirm

  • add DataGenerator, SpeedLimit (from Others) and trash, interconnect them

  • create some simple metadata (just one field), select them for the edges

  • let DataGenerator generate one record (e. g. one random number)

  • set requried delay to SpeedLimit

The Windows share was implemented for a newbie reason: couldn’t precisely find where Clover’s webdav root aimed at, and the permissions / authentication game was not clear.
On a linux box this might be easier, but on this windows server less so.
Is there a document for CloverETL server that spells out:

1. where the webdav config is located,

“twaldock”

There is no such config file, configuration is derived from Server setup.

2. how to secure it under clover (Tomcat’s suggestions break CloverETL server; had to revert to unsecured and unauthenticated. Then again, my fingers could have been guilty).

“twaldock”

Setting the tomcat’s http connector to secured mode should work… could you share your tomcat config file?

3. where the webdav root really is and how to control/re-aim it to a different path.

“twaldock”

Webdav structure is semi-virtual, specially, the root is virtual. You can see it as a directory containing all server sandboxes, no matter where they are physically located on disk.

Also, the original question on credentials for CloverETL’s webdav is still open: how or where are those set?

“twaldock”

Clover server users are able to log in to webdav – please see tab Users in Server web UI.

5

Thanks oprendekm!

So for CloverETL Server reading from a webdav url, there is no place for credentials?

Thanks for the speed-limiter idea; will have to implement it or the folder-move technique.

Cheers,

6

So for CloverETL Server reading from a webdav url, there is no place for credentials?

“twaldock”

I see three different situations here:

  • Some tool is browsing / working with CloverETL server’s sandboxes structure via webdav interface – credentials are configured in Users tab in Server Web UI (http://clover-server-host/clover/gui/users.jsf). Client application can connect to it using URL e. g.:
webdav://login:password@clover-server-host/clover/webdav/your_sandbox/your_path/your_file
  • Graph executed in CloverETL Server environment is accessing some other webdav storage – you define credentials in reader’s File URL (on tab Remote files click on Create/Edit URL and use http(s) for webdav). Resulting URL is e. g.:
http(s)://login:password@webdav-storage-host/your_path/your_file
  • Graph executed in CloverETL Server environment is accessing file from it’s own sandbox – there is no need to use webdav interface because server’s graphs can access sandbox files directly using sandbox:// URL