Apache Kafka reader component

CloverDX Platform
1

Is there anyway any of the existing reader components can read from an Apache Kafka topic?

If not, has anyone developed a Kafka reader component that’s open source or otherwise available?

2

Hi,

Unfortunately, there are no official read/write components for Apache Kafka and the current readers cannot read the Kafka topics either. However, it is possible to write your own custom component based on the CustomJavaComponent. Documentation of the Apache Kafka API can be found here. Alternatively you can use a command line consumer to write the messages into a file and read this file afterwards.

However, my colleague created a new issue in our system. So if our developers decide to implement these components, you can see it there.

Best regards,

3

Thanks Lukas,

Is there a (good) way to integrate an existing groovy Kafka consumer into a generic data flow component such that the data flow can process the messages in a Kafka topic in a data flow?

4

Hi,

We are working on example components capable of writing and reading to/from the Kafka topics. We are going to release them as soon as possible.

Best regards,

5

Hi,

We have spent some time creating the example components and now they are finally here. The archive contains whole CloverETL project with everything that is needed to do a basic read and write from/to a Kafka topic. There are two subgraphs and one testing graph (among other things) in the project. These subgraphs are KafkaReader and KafkaWriter and are supposed to be used as regular components. The testing graph shows how the subgraphs can be configured.

These components use Kafka API 0.10.0.0 and were tested only in CloverETL 4.2.0.

Please test the components (subgraphs) and let us know if you run into any issues, so we can improve them.

If you have any questions regarding these components, please ask them in this thread.

kafka.zip

Best regards,

6

Hi Cholastal,

Unfortunately we haven’t had the opportunity to look into the Kafka sub-graphs that you provided in this thread primarily because we were stuck on an older version of CloverETL Designer and Server that didn’t support sub-graphs, v3.5.1.018 to be specific.

Fortunately, we’re finally sending a couple of engineers on staff to training next week in Arlington VA USA. Hopefully, they’ll come backed all energized on the current version and knowledgeable on how to upgrade. Assuming that’s the case, we’ll check out the Kafka integration shortly after upgrading. I’d estimate about 1 month from now.

Thanks!

7

Hi,

We have spent some time creating the example components and now they are finally here. The archive contains whole CloverETL project with everything that is needed to do a basic read and write from/to a Kafka topic. There are two subgraphs and one testing graph (among other things) in the project. These subgraphs are KafkaReader and KafkaWriter and are supposed to be used as regular components. The testing graph shows how the subgraphs can be configured.

These components use Kafka API 0.10.0.0 and were tested only in CloverETL 4.2.0.

Please test the components (subgraphs) and let us know if you run into any issues, so we can improve them.

If you have any questions regarding these components, please ask them in this thread.

“cholastal”

Hi this file is no longer available for download.
I’d like to test such basic functionality for a PoC.

Regards,
Lukasz

8

Hi Lukasz,

This was caused by our recent rebranding activities. It is available for download now. I’m sorry for the inconvenience.

If you want to know more about the rebranding, please visit: https://www.cloverdx.com/cloveretl

Best regards,