Replaces Chunker with uber-powerful split.
authorJakub Pawlowicz <contact@jakubpawlowicz.com>
Mon, 24 Aug 2015 05:05:13 +0000 (06:05 +0100)
committerJakub Pawlowicz <contact@jakubpawlowicz.com>
Mon, 24 Aug 2015 05:51:27 +0000 (06:51 +0100)
commit87104885f3128883a523d5a65893a13df69bcaa4
treed9a3916f5047eaf43fd7a7dd03aa5e038d2d95a1
parentcd97a57871e9fa8cacce00c0a6568b3a333b3afd
Replaces Chunker with uber-powerful split.

These two were doing basically the same thing, while the latter
being smarter about a context.
lib/tokenizer/chunker.js [deleted file]
lib/tokenizer/tokenize.js
test/module-test.js
test/tokenizer/chunker-test.js [deleted file]