Number of sentences in dialogs

[itPiligrim]

We have the text "- Water, freeze water. - What'd I say?"
This text model converts to ONE sentence. Why?

If i delete "-" and get "Water, freeze water. What'd I say?",
that text model converts to 2 sentences.

[rachnachakraborty]

Hi @itPiligrim

You have highlighted an interesting case which is usually not found in the training datasets.

We are also not aware of any such English language specific rule to determine sentence breaks. This is what makes it tricky to tweak the model.

For now we can suggest you to use some preprocessing for which winkNLP can easily be leveraged.

In case you need any specific assistance, please do share with us.

Best,
Rachna

[itPiligrim]

Yes, I decided exactly the same thing, too. The use of "-" and similar symbols is characteristic of dialogues. There are a lot of them on youtube, etc. If you need to get subtitles (captions) from youtube channels, please contact me for future model training. Thanks for the answer.