Kafka topic merging with Kafka Connect to HDFS
Is it possible to configure Kafka Connect’s HDFS connector to write/combine several separate topics into one file?
The topics will contain messages with the same avro schema and I want KafkaConnect to act as an intermediary between those Kafka topics and HDFS. Worst case scenario the topic contents could be combined after being written to HDFS, but I feel like a cleaner and quicker way should be possible with the HDFS connector.
Right now the HDFS connector will write each topic to its own directory. You can combine directories in HDFS after writing, or combine topics in Kafka before writing to HDFS, but the connector itself will not do it.