Code

New, optional flex-based tokenizer#218

Code

Merged

bzz merged 6 commits intosrc-d:mastersrc-d/enry:masterfrom bzz:tokenizer-flex-cgobzz/enry:tokenizer-flex-cgoCopy head branch name to clipboardApr 17, 2019

+2,707-15

Commits

Commits on Apr 14, 2019

• tokenizer: port flex-based C impl from linguist

  Show description for 553399e
  bzz

  committed
  553399e

• refactor to build tags

  Show description for 8756fbd
  bzz

  committed
  8756fbd

• tokenizer: cleanup & attributions

  Show description for 7929933
  bzz

  committed
  7929933

Commits on Apr 16, 2019

• address review feedback

  Show description for ada6f15
  bzz

  committed
  ada6f15

• doc: improve API doc on review feedback

  Show description for 6c7b91c
  bzz

  committed
  6c7b91c

• test: don't export tokenizer fixtures

  Show description for 7e136ba
  bzz

  committed
  7e136ba