Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Greek tokenization errors #38

Open
balmas opened this issue Jun 23, 2015 · 0 comments
Open

Greek tokenization errors #38

balmas opened this issue Jun 23, 2015 · 0 comments

Comments

@balmas
Copy link
Contributor

balmas commented Jun 23, 2015

Reported by @nevenjovanovic in alpheios-project/arethusa#631

καί ποτε τοῦ Σοφοκλέους, ὅτε συστρατηγῶν ἐξέπλευσε μετ᾽ αὐτοῦ, παῖδα καλὸν ἐπαινέσαντος, ‛οὐ μόνον,’ ἔφη, ‛τὰς χεῖρας, ὦ Σοφόκλεις, δεῖ καθαρὰς ἔχειν τὸν στρατηγόν, ἀλλὰ καὶ τὰς ὄψεις.’
-- in tokenizing, the opening quotation marks in ‛οὐ and ‛τὰς don't get separated from the words (single ' as well as double ").

The same goes for em dash - the following passage from Plato's Alcibiades 104a is not tokenized correctly: ... κάλλιστός τε καὶ μέγιστος—καὶ τοῦτο μὲν δὴ παντὶ δῆλον ἰδεῖν ὅτι οὐ ψεύδῃ—ἔπειτα νεανικωτάτου...; I am offered μέγιστος—καὶ and ψεύδῃ—ἔπειτα as single words.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant