User avatar
Posts: 1
Joined: Thu Nov 29, 2018 7:05 pm
Location: London, United Kingdom

Data Services & Micro-Service Architecture use-cases

Postby JonathonO » Mon May 13, 2019 10:39 am

Is Data Services suitable for micro-service arch. use cases, specifically for reporting on ETL flows?

The main reason for my question is that I was thinking about an endpoint for sending notifications out to various recipients and the other for logging job stats e.g. cpu, memory usage and records processed. The payloads would - for now - only require the default data that is returned from ExecuteGraph/Jobflow components; no custom fields.

These services would not need to run sequentially/synchronously as part of the ETL flow; they only need to be invoked by the flow. I'm also conscious of the design pattern regarding Jobflows and Subgraphs with token tracking so I'd like to avoid wrapping these services into subgraphs and executing them in the ETL flow.

The API calls wouldn't post too large a payload. But we'd be looking at 10000s of calls per day.