On importing Greek into Arethusa, beginning quotation marks are not separated from the word

[nevenjovanovic]

When I paste manually this text of Plutarch into Arethusa (taken also from http://demo.fragmentarytexts.org/en/revolt-of-samos/sophocles-strategia.html):
καί ποτε τοῦ Σοφοκλέους, ὅτε συστρατηγῶν ἐξέπλευσε μετ᾽ αὐτοῦ, παῖδα καλὸν ἐπαινέσαντος, ‛οὐ μόνον,’ ἔφη, ‛τὰς χεῖρας, ὦ Σοφόκλεις, δεῖ καθαρὰς ἔχειν τὸν στρατηγόν, ἀλλὰ καὶ τὰς ὄψεις.’
-- in tokenizing, the opening quotation marks in ‛οὐ and ‛τὰς don't get separated from the words (single ' as well as double ").

[nevenjovanovic]

The same goes for em dash - the following passage from Plato's Alcibiades 104a is not tokenized correctly: ... κάλλιστός τε καὶ μέγιστος—καὶ τοῦτο μὲν δὴ παντὶ δῆλον ἰδεῖν ὅτι οὐ ψεύδῃ—ἔπειτα νεανικωτάτου...; I am offered μέγιστος—καὶ and ψεύδῃ—ἔπειτα as single words.

[LFDM]

Hi Neven,
thanks for the report! Fixed this yesterday - we'll soon deploy a next iteration of the tokenization stuff, need to add a couple of tests in the next couple of days then we're good to go!

[balmas]

This is actually a problem with the llt tokenizer not arethusa. Closing this and opened as latin-language-toolkit/llt-tokenizer#38