New, optional flex-based tokenizer

[bzz]

This change partially address #193

It follows option 3 from #193 (comment)

• a new native tokenizer implementation, hidden behind -tags flex
• existing native code in C from Linguist project, generated by Flex from tokenizer.l is added under internal/tokenizer/flex/lex.linguist_yy.{h,c}
• tokenizer_c.go is a manual transliteration to Go of the parser from linguist.c

The benchmark before:

$ go test -bench=. -run=Benchmark. ./internal/tokenizer
BenchmarkTokenizer_BaselineCopy-4   	 5000000	       304 ns/op	    1536 B/op	       1 allocs/op
BenchmarkTokenizer-4                	    2000	    672592 ns/op	   58205 B/op	     435 allocs/op

after

$ go test -bench=. -run=Benchmark. -tags flex ./internal/tokenizer
BenchmarkTokenizer_BaselineCopy-4   	 5000000	       300 ns/op	    1536 B/op	       1 allocs/op
BenchmarkTokenizer-4                	   20000	     69302 ns/op	    5064 B/op	     150 allocs/op

so it's x10 faster.

But because it's a cgo and it complicates the build I suggest to merge it, keeping behind the tag.

Right now this is only useful for debugging #194 and is ready to be merged without affecting the releases.

But if we agree on it's value for downstream uses for performance reasons, some extra work is going to be needs (in subsequent PR: documentation + updated classifier assets, based on this tokenizer that do not overwrite the default one) before allowing it's usage that goes beyond the internal dev debugging workflow. It is going to be tracked under #193

[bzz]

@creachadair all feedback addressed, ready for another round.

[creachadair]

A couple minor and optional suggestions, otherwise looks good.

[dennwc]

Suggested change

	ary := []string{}
	var ary []string

source

[bzz]

This would result in returning nil instead of empty slice sometimes and although I'm aware of https://github.com/golang/go/wiki/CodeReviewComments#declaring-empty-slices that would be inconsistent with how current impl of tokenizer API works.

Thats why I decided to do it this way and keep them the same.

[dennwc]

I'm not sure if it matters from the API point of view - it's pretty rare to test for slice nil-ness.

[dennwc]

Merge into a single var block

[bzz]

I know what you mean, but would prefer to keep it as close to original as possible as this is almost line-by-line transliteration from C

[dennwc]

The key word is "almost" :) Unless it's machine-generated, I think it makes sense to adhere to Go conventions where possible. It's not a bit change anyway. Still, it's optional, so feel free to ignore.

[dennwc]

I would also propose to include a go:generate directive that updates the C tokenizer, if it makes sense.

[bzz]

I think I might not communicate it well enough, but just to make it clear - the only code that is "generated" in this PR is the C lex.linguist_yy.{h,c}, that is vendored inside Linguist and we are using exact copy.

In the long run, if this tokeniser turns out to be useful beyond classifier accuracy debugging - it does make sense to add C code generation script from the flex grammar, but I also think it's ok to do that only after #193 is done (so it adds value, not only a maintenance burden).

[dennwc]

Sorry, I haven't meant to regenerate it. go:generate may also be used to wget the latest source, for example. Not critical, just a thought.

[bzz]

@dennwc all feedback addressed, ready for another round.