CloverETL is now CloverDX - Learn Why

Apache Kafka reader component.

Support/help with CloverETL (4.9) and CloverDX (5.0 or newer) implementation problems

starsailor
Posts: 31
Joined: Tue Jun 04, 2013 9:05 pm

Apache Kafka reader component.

Postby starsailor » Wed May 18, 2016 6:18 pm

Is there anyway any of the existing reader components can read from an Apache Kafka topic?

If not, has anyone developed a Kafka reader component that's open source or otherwise available?

cholastal
Posts: 114
Joined: Tue Sep 01, 2015 1:22 pm

Re: Apache Kafka reader component.

Postby cholastal » Mon May 23, 2016 3:34 pm

Hi,

Unfortunately, there are no official read/write components for Apache Kafka and the current readers cannot read the Kafka topics either. However, it is possible to write your own custom component based on the CustomJavaComponent. Documentation of the Apache Kafka API can be found here. Alternatively you can use a command line consumer to write the messages into a file and read this file afterwards.

However, my colleague created a new issue in our system. So if our developers decide to implement these components, you can see it there.

Best regards,

---
Lukas Cholasta
CloverCARE Support
CloverDX

Visit us online at http://www.cloverdx.com

starsailor
Posts: 31
Joined: Tue Jun 04, 2013 9:05 pm

Re: Apache Kafka reader component.

Postby starsailor » Tue Jun 07, 2016 7:01 pm

Thanks Lukas,

Is there a (good) way to integrate an existing groovy Kafka consumer into a generic data flow component such that the data flow can process the messages in a Kafka topic in a data flow?

cholastal
Posts: 114
Joined: Tue Sep 01, 2015 1:22 pm

Re: Apache Kafka reader component.

Postby cholastal » Tue Jun 14, 2016 11:38 am

Hi,

We are working on example components capable of writing and reading to/from the Kafka topics. We are going to release them as soon as possible.

Best regards,

---
Lukas Cholasta
CloverCARE Support
CloverDX

Visit us online at http://www.cloverdx.com

cholastal
Posts: 114
Joined: Tue Sep 01, 2015 1:22 pm

Re: Apache Kafka reader component.

Postby cholastal » Mon Jul 04, 2016 9:26 am

Hi,

We have spent some time creating the example components and now they are finally here. The archive contains whole CloverETL project with everything that is needed to do a basic read and write from/to a Kafka topic. There are two subgraphs and one testing graph (among other things) in the project. These subgraphs are KafkaReader and KafkaWriter and are supposed to be used as regular components. The testing graph shows how the subgraphs can be configured.

These components use Kafka API 0.10.0.0 and were tested only in CloverETL 4.2.0.

Please test the components (subgraphs) and let us know if you run into any issues, so we can improve them.

If you have any questions regarding these components, please ask them in this thread.


kafka.zip
(677.06 KiB) Downloaded 147 times


Best regards,

---
Lukas Cholasta
CloverCARE Support
CloverDX

Visit us online at http://www.cloverdx.com

starsailor
Posts: 31
Joined: Tue Jun 04, 2013 9:05 pm

Re: Apache Kafka reader component.

Postby starsailor » Mon Oct 31, 2016 11:37 pm

Hi Cholastal,

Unfortunately we haven't had the opportunity to look into the Kafka sub-graphs that you provided in this thread primarily because we were stuck on an older version of CloverETL Designer and Server that didn't support sub-graphs, v3.5.1.018 to be specific.

Fortunately, we're finally sending a couple of engineers on staff to training next week in Arlington VA USA. Hopefully, they'll come backed all energized on the current version and knowledgeable on how to upgrade. Assuming that's the case, we'll check out the Kafka integration shortly after upgrading. I'd estimate about 1 month from now.

Thanks!

comadot
Posts: 1
Joined: Tue Oct 09, 2018 12:13 pm

Re: Apache Kafka reader component.

Postby comadot » Tue Oct 09, 2018 12:19 pm

cholastal wrote:Hi,

We have spent some time creating the example components and now they are finally here. The archive contains whole CloverETL project with everything that is needed to do a basic read and write from/to a Kafka topic. There are two subgraphs and one testing graph (among other things) in the project. These subgraphs are KafkaReader and KafkaWriter and are supposed to be used as regular components. The testing graph shows how the subgraphs can be configured.

These components use Kafka API 0.10.0.0 and were tested only in CloverETL 4.2.0.

Please test the components (subgraphs) and let us know if you run into any issues, so we can improve them.

If you have any questions regarding these components, please ask them in this thread.



Hi this file is no longer available for download.
I'd like to test such basic functionality for a PoC.

Regards,
Lukasz

cholastal
Posts: 114
Joined: Tue Sep 01, 2015 1:22 pm

Re: Apache Kafka reader component.

Postby cholastal » Wed Oct 10, 2018 11:34 am

Hi Lukasz,

This was caused by our recent rebranding activities. It is available for download now. I'm sorry for the inconvenience.

If you want to know more about the rebranding, please visit: https://www.cloverdx.com/cloveretl

Best regards,

---
Lukas Cholasta
CloverCARE Support
CloverDX

Visit us online at http://www.cloverdx.com


cron