Make the topic name configurable in storage paths

[hariprasad-k]

Improvements to provide ability to exclude topic name in partition by configuration

[ghost]

@confluentinc It looks like @hariprasad-k just signed our Contributor License Agreement. 👍

Always at your service,

clabot

[hariprasad-k]

Fixed #126

[OneCricketeer]

Question: What happens when you have topics as a list or topics.regex capture more than one topic?

Then all files in the directory will be of multiple topics?

[OneCricketeer]

I suggest you break this out as a separate PR

[hariprasad-k]

@OneCricketeer Sure, I will break into separate PR.

[OneCricketeer]

The patch also fixes #121 too

So does #122

[hariprasad-k]

Question: What happens when you have topics as a list or topics.regex capture more than one topic?

Then all files in the directory will be of multiple topics?

@OneCricketeer Yes, all files of multiple topics will be in the same directory in this case.

But, this behavior will only happen if you use topics as a list or topics.regex in combination with path.include.topicname set as false. Further, as path.include.topicname is by default true, it will preserve the existing behavior by default.

[hariprasad-k]

@OneCricketeer This PR does not have any changes related to Timestamp scaling factor anymore, which is covered by #160

Can you please review this changes ?

[OneCricketeer]

I'm not a Confluent employee, so my reviews don't do anything

[dosvath]

Thanks @hariprasad-k for the contribution! This change will add flexibility to our connectors that use storage common to optionally exclude the topic name from the path (seems to be a popular request confluentinc/kafka-connect-hdfs#544). I'd like to get second view from @levzem before proceeding.

[hariprasad-k]

@dosvath @levzem Any thoughts, or plans to review this contribution ?

[awebneck]

Would be very much interested in having this feature integrated. The topic name being forcefully appended to the path limits us severely when sinking to already-established directory structures.