Fixes tokenizer edge case.
authorJakub Pawlowicz <contact@jakubpawlowicz.com>
Thu, 22 Dec 2016 07:59:41 +0000 (08:59 +0100)
committerJakub Pawlowicz <contact@jakubpawlowicz.com>
Thu, 22 Dec 2016 07:59:41 +0000 (08:59 +0100)
commita9725113de6e815594f749c736294ee6beb6c0fd
treed86d1ff8abb2306bf37e50d1de0cfded00551bf8
parent6a4c425345152205d5043a1898d6818ee18857f8
Fixes tokenizer edge case.

If there is a semicolon marker after a property then the property
ends and its reference should be discarded. Otherwise next tokens
may get appended to the property.
lib/tokenizer/tokenize.js
test/tokenizer/tokenize-test.js