Kafka topic merging with Kafka Connect to HDFS

Is it possible to configure Kafka Connect’s HDFS connector to write/combine several separate topics into one file?

The topics will contain messages with the same avro schema and I want KafkaConnect to act as an intermediary between those Kafka topics and HDFS. Worst case scenario the topic contents could be combined after being written to HDFS, but I feel like a cleaner and quicker way should be possible with the HDFS connector.

Answers


Right now the HDFS connector will write each topic to its own directory. You can combine directories in HDFS after writing, or combine topics in Kafka before writing to HDFS, but the connector itself will not do it.


Need Your Help

accessing multiple objects in robot framework

python frameworks robotframework

Is there a way to explicitly access multiple objects of the same class in robot framework.

Vala, resources and localization

localization resources internationalization embedded-resource vala

What is the best way to handle resources (like images, files, ...) in vala? Is there any resource management system? And what is the best way to localize application? I have came to Vala from .NET