CloverETL is now CloverDX - Learn Why

create schedules dynamically

Support questions related to CloverDXServer

ramiro
Posts: 19
Joined: Fri Aug 08, 2014 1:13 am

create schedules dynamically

Postby ramiro » Tue Oct 16, 2018 6:51 pm

is it possible to create schedules via API calls, instead of using the manual interface at the server admin UI?

we are using clover 4.2, let us know, thanks!

bartonv
Posts: 89
Joined: Wed May 03, 2017 12:10 pm

Re: create schedules dynamically

Postby bartonv » Fri Oct 19, 2018 2:30 pm

Hi Ramiro,
CloverDX Server is currently not designed in a way that would allow for creating new schedules via an API very elegantly. In theory, you would be able to do that by taking advantage of our Simple HTTP API, namely the 'import_server_config' operation:

  1. At first, you would want to call the 'export_server_config' operation to get an XML file containing the Schedules configuration. For instance:

    Code: Select all

    wget --http-user=clover --http-password=clover http://localhost:8080/clover/simpleHttpApi/export_server_config?include=schedules

    Notice the 'include=schedules' parameter that means you are only getting the configuration of the Schedules as opposed to the entire server configuration which is the default option.
  2. You would want to grab the XML configuration file and use it as a pattern for your intended schedule imports. In other words, you can fill it with your desired properties, e.g. startTime, interval, etc. and then import it back into the Server GUI via the 'import_server_config' operation.
  3. Note that when doing this on a regular basis, you would have to implement a logic that would extract the last ID (the highest number) of the existing schedules so that the next ID that you use in your import would be a continuation of the ID list (and an existing ID is not used).
  4. Finally, in order to import your desired new schedules into the CloverDX Server, you would call the 'import_server_config' operation in a way that resembles the following:

    Code: Select all

    curl -u clover:clover -F "dryRun=true" -F "verbose=FULL" -F "xmlFile=@/tmp/clover_configuration_2018-10-19_14-03-23+0200.xml" http://localhost:8080/clover/simpleHttpApi/import_server_config?include=schedules&dryRun=false&newOnly=true

    Notice the 'dryRun=true' parameter which means the import is actually going to be committed as opposed to a dry run import. Worth noticing is also the 'newOnly' parameter which will ensure that only the new schedules will get imported and the existing schedules will stay untouched. The 'xmlFile' parameter obviously provides the path to the XML configuration file.

Although I presented the usage of the API calls with tools like wget or curl, it is certainly possible to use CloverDX instead (namely the HTTPConnector component) for the same purpose. However, the design of such a graph would be fairly complex.

Anyhow, I would still be a little curious as to what exactly you are trying to achieve and what the overall intention of your use case is. If you could describe your use case in more details I might be able to suggest a more straight-forward solution possibly by using a different tool than our Simple HTTP API.

Kind regards,
---
Vladimir Barton
CloverCARE Support
CloverDX

Visit us online at http://www.cloverdx.com

ramiro
Posts: 19
Joined: Fri Aug 08, 2014 1:13 am

Re: create schedules dynamically

Postby ramiro » Fri Oct 19, 2018 5:22 pm

Thanks for the response!

I'll try to explain our process in a nutshell, and our need for dynamic schedules.

We have a bunch of customers, and we have an ETL flow that process their data.

But we process the data per customer, not as a whole.

Each customer has custom needs, for some we run a schedule that kick start ETL every hour, most of them are at an 8hr cycle, etc.

We are in the process of automating the addition of new customers, so ETL will just run for them in a flip of a switch. Problem is: the way we setup the cloverETL flow, we need one of those schedules to kickstart the init graph, with specific params (run times, etc)

We were thinking: if the schedule job can be created via API calls, we can add that to the flow of provisioning of new customers.

Right now we have to manually still add one schedule each time we get a new customer.

We will probably refactor things, so we only have one schedule waking up every five minutes or so, check the run cycles, and feed the ones that are ready to the init cloverETL graph. We were just trying to avoid this refactor, if there was an easy way to just create the schedules via API calls.

bartonv
Posts: 89
Joined: Wed May 03, 2017 12:10 pm

Re: create schedules dynamically

Postby bartonv » Wed Oct 24, 2018 10:03 am

Hi Ramiro,
from what you described, my previously listed set of steps does not seem to be a good candidate for the optimal approach in your use case indeed. In fact, it appears that I would need to have a deeper insight into your case to suggest the optimal approach. Nevertheless, let me mention another general approach that seem to fit your basic requirements and that is the usage of event listeners, although I realize that the viability of its usage greatly depends on the details, such as how you feed the customer information to the job (that you referred to as ETL), or whether the schedule cycles are synchronized across different groups of customers, or how the automation of the new customer addition is triggered, etc.
Anyway, using event listeners would allow for:
  • setting up a job run with specific predefined parameters (similar to scheduling),
  • thanks to that, the job can be customer specific,
  • activating the event listener with a flip of a switch,
  • triggering a job run based on various events (this might be useful in case your job per customer is triggered, for instance, by adding a specific file, or upon running a certain graph as a kickstarter, etc.)
More information about the full event listener functionality can be found in our documentation.
Note: This suggestion is obviously not meant in a way that it should replace the step of the new schedule creation in your workflow as it still requires manual user action. Rather, it might be viewed as another approach you could choose when refactoring your workflow. After all, creating new event listeners via an API call would be as cumbersome as doing the same for new schedules.
If you were interested in a deeper investigation of your use case on our end I would suggest that you log a ticket via our CloverCARE Customer Portal and provide us with a detailed description of your current workflow including the graphs, schedules, and examples.
Kind regards,
---
Vladimir Barton
CloverCARE Support
CloverDX

Visit us online at http://www.cloverdx.com

ramiro
Posts: 19
Joined: Fri Aug 08, 2014 1:13 am

Re: create schedules dynamically

Postby ramiro » Wed Oct 24, 2018 6:05 pm

Thanks for the suggestions Vladimir

We will keep event listeners in mind for the refactoring

Ramiro