summary |
shortlog | log |
commit |
commitdiff |
tree
first ⋅ prev ⋅ next
Nick Downing [Thu, 21 Dec 2023 07:25:37 +0000 (18:25 +1100)]
Fix an error recovery routine that seems to not run often
Nick Downing [Wed, 6 Dec 2023 04:17:32 +0000 (15:17 +1100)]
Make the yy_group_element() routine in the skeleton for pilex --element --groups --python not construct the element until after the text and children are known
Nick Downing [Sun, 3 Dec 2023 11:17:32 +0000 (22:17 +1100)]
Make some notes in /ndcode/pilex/nfa.py after reverse-engineering the algorithm
Nick Downing [Sat, 18 Nov 2023 23:06:17 +0000 (10:06 +1100)]
Modify text_to_python() to handle stray whitespace on otherwise blank line
Nick Downing [Mon, 13 Nov 2023 11:14:14 +0000 (22:14 +1100)]
Add /fix_bootstrap.sh and run it due to reserved ast and _ast in recent python3
Nick Downing [Tue, 17 May 2022 08:44:08 +0000 (18:44 +1000)]
Get non-groups Python skeleton working, seems it wasn't tested at all before
Nick Downing [Tue, 17 May 2022 07:29:41 +0000 (17:29 +1000)]
Pass python flag into AST.post_process(), to generate correct default actions
Nick Downing [Tue, 17 May 2022 07:22:35 +0000 (17:22 +1000)]
Fix a typo in non-groups Python skeleton (introduced with last commit), remove some prints in that skeleton, fix missing default-rule group in flex generation
Nick Downing [Sun, 15 May 2022 00:23:44 +0000 (10:23 +1000)]
Fix scanning bug of adding extra text to matched string
Nick Downing [Sat, 14 May 2022 07:00:07 +0000 (17:00 +1000)]
Option to build python lexer without groups (simpler skeleton, more like flex)
Nick Downing [Sun, 26 Jan 2020 23:56:40 +0000 (10:56 +1100)]
Move everything into ndcode/pilex/, import from ndcode.pilex, add packaging
Nick Downing [Sun, 3 Feb 2019 06:17:04 +0000 (17:17 +1100)]
Convert generate_flex.py to new element class, reinstate the C based tests, fix a bug which misnumbered the EOB action leading to improper scanner startup
Nick Downing [Wed, 30 Jan 2019 12:14:59 +0000 (23:14 +1100)]
Add YY_USER_ACTION() hook for location tracking in Python generator/skeletons
Nick Downing [Tue, 29 Jan 2019 10:24:46 +0000 (21:24 +1100)]
Freeze compiled pitree.git commit
71dd736 into bootstrap_pitree, reinstate use
Nick Downing [Tue, 29 Jan 2019 09:49:17 +0000 (20:49 +1100)]
Update to pitree.git commit
71dd736
Nick Downing [Tue, 29 Jan 2019 05:09:42 +0000 (16:09 +1100)]
Update to pitree.git commit
e4f3cdd
Nick Downing [Tue, 29 Jan 2019 04:22:05 +0000 (15:22 +1100)]
Update to pitree.git commit
350e5a9 (first cut)
Nick Downing [Tue, 29 Jan 2019 02:38:25 +0000 (13:38 +1100)]
Update to pitree.git commit
b3b2036
Nick Downing [Tue, 29 Jan 2019 01:06:42 +0000 (12:06 +1100)]
Update to pitree.git commit
5cba525 (don't use bootstrap_pitree for now)
Nick Downing [Mon, 28 Jan 2019 03:08:47 +0000 (14:08 +1100)]
Use regex.t and tests_ast/cal_py.t, get rid of old AST generator and stripper
Nick Downing [Sun, 27 Jan 2019 23:33:52 +0000 (10:33 +1100)]
Freeze compiled pilex.git commit
767c103, pitree.git commit
3daa8a6, piyacc.git commit
bc19b4a, into bootstrap_*, use pilex.t and t_def.py instead of ast.py
Nick Downing [Mon, 28 Jan 2019 02:13:28 +0000 (13:13 +1100)]
Add or update license in every file
Nick Downing [Sun, 27 Jan 2019 23:08:47 +0000 (10:08 +1100)]
Remove obsolete work.py module
Nick Downing [Sun, 27 Jan 2019 11:30:03 +0000 (22:30 +1100)]
Remove obsolete __repr__() and repr_serialize() on AST elements, change spacing
Nick Downing [Sat, 26 Jan 2019 23:28:15 +0000 (10:28 +1100)]
Remove obsolete way of initializing AST elements with serialized (string) data
Nick Downing [Sat, 26 Jan 2019 22:32:01 +0000 (09:32 +1100)]
Make element groups also behave as numbered groups (so that element constructor can access the text spanned by the group), make named and numbered groups always adopt the text, so that the nearest text (to right) is always available, and so that if the group captures multiple times, the FIRST not the LAST will be kept
Nick Downing [Mon, 21 Jan 2019 06:20:38 +0000 (17:20 +1100)]
Minor consistency fixes wrt. piyacc.git commit
6558f0b
Nick Downing [Mon, 21 Jan 2019 00:44:15 +0000 (11:44 +1100)]
Allow yylex() caller to specify factory for yy_element_space, yy_element_token
Nick Downing [Mon, 21 Jan 2019 00:08:23 +0000 (11:08 +1100)]
Streamline markup_push()/markup_pop() to call factory earlier and avoid copying
Nick Downing [Sun, 20 Jan 2019 09:46:13 +0000 (20:46 +1100)]
Make pilex.py standalone, fix accidentally deleted line of code which made the '(?E{' sequence not parse correctly, add tests_ast/cal_py.l tests to catch this
Nick Downing [Sun, 20 Jan 2019 09:30:15 +0000 (20:30 +1100)]
Add element-based Python scanner/parser in root of repository which builds AST directly, and n.sh test script to check same result as bootstrap/ scanner/parser
Nick Downing [Sun, 20 Jan 2019 02:16:55 +0000 (13:16 +1100)]
Rationalize option sense handling, fix a few bugs and typos
Nick Downing [Sun, 20 Jan 2019 00:29:12 +0000 (11:29 +1100)]
Fix some missing spots to use markup stack, move code lines, add a syntax error
Nick Downing [Sat, 19 Jan 2019 14:31:14 +0000 (01:31 +1100)]
Implement a stack of position markers to delay the insertion of an opening tag until just before the corresponding closing tag, add an optional second SECTEND
Nick Downing [Sat, 19 Jan 2019 12:55:09 +0000 (23:55 +1100)]
Remove double EOF and fix places where token injects text before the next token
Nick Downing [Sat, 19 Jan 2019 12:41:05 +0000 (23:41 +1100)]
Modify the parser/scanner interface to remove markup_action(), markup_option()
Nick Downing [Sat, 19 Jan 2019 11:17:38 +0000 (22:17 +1100)]
Add bootstrap/n.sh test script
Nick Downing [Fri, 18 Jan 2019 01:59:29 +0000 (12:59 +1100)]
Minor change to bootstrap/Makefile to simplify relative path to ../pilex.py
Nick Downing [Tue, 15 Jan 2019 02:00:26 +0000 (13:00 +1100)]
Implement state flags and fix minor trailing context/bracketing macros issue
Nick Downing [Tue, 15 Jan 2019 00:29:36 +0000 (11:29 +1100)]
Fix several bugs to get Python scanner/parser basically working, processes ../tests/cal.l correctly, minor differences for ../../boostrap_flex.git/src/scan.l
Nick Downing [Mon, 14 Jan 2019 14:47:39 +0000 (01:47 +1100)]
Add Python version of bootstrap_flex.git parser (adds markup to *.l file), doesn't work yet because Python parser is always batch, calls flexscan() too eagerly
Nick Downing [Mon, 14 Jan 2019 02:49:27 +0000 (13:49 +1100)]
Remove some compiler warnings in calculator test (needed since Linux Mint 19)
Nick Downing [Mon, 14 Jan 2019 02:47:36 +0000 (13:47 +1100)]
Add Python version of bootstrap_flex.git scanner (adds markup to *.l file)
Nick Downing [Mon, 14 Jan 2019 02:13:39 +0000 (13:13 +1100)]
Fix FlexDFA generation bug introduced in commit
166f431 with DFA.to_flex_dfa()
Nick Downing [Mon, 14 Jan 2019 01:24:19 +0000 (12:24 +1100)]
Implement BOL detection in Python skeleton (for ^ operator) and various other functions such as BEGIN() similar to the C macros, reinstate generate_flex.py flex scanner generation which was broken since user-defined groups were implemented
Nick Downing [Sat, 12 Jan 2019 01:45:59 +0000 (12:45 +1100)]
Change file reading to not re-attempt read from any stream once EOF is reached
Nick Downing [Wed, 26 Sep 2018 07:27:28 +0000 (17:27 +1000)]
Implement element groups with support from pilex (not via user code)
Nick Downing [Wed, 26 Sep 2018 07:02:34 +0000 (17:02 +1000)]
Update skel_py_element.py, implement tests_ast directory which defines AST_Num(_Mantissa|_Fraction)? elements and builds yy_element_token using action groups
Nick Downing [Wed, 26 Sep 2018 05:33:33 +0000 (15:33 +1000)]
Implement numbered and named groups with support from pilex (not via user code)
Nick Downing [Tue, 25 Sep 2018 23:51:19 +0000 (09:51 +1000)]
Rearrange so that group actions don't need any arguments, implement group names
Nick Downing [Tue, 25 Sep 2018 23:20:48 +0000 (09:20 +1000)]
Implement action groups (extension syntax), use this to make a private implementation of numbered groups in cal_py.l, extract parts of floating point by groups
Nick Downing [Sat, 22 Sep 2018 01:29:41 +0000 (11:29 +1000)]
Move Rule.post_process() code into EOFRule, FLexRule to avoid isinstance() test
Nick Downing [Thu, 13 Sep 2018 11:14:57 +0000 (21:14 +1000)]
Set tag name for yy_element_(space|token) to something sensible, we use 'root'
Nick Downing [Wed, 12 Sep 2018 13:01:53 +0000 (23:01 +1000)]
First cut at automatic AST generation with pilex.py --element switch
Nick Downing [Tue, 11 Sep 2018 11:24:31 +0000 (21:24 +1000)]
Improve the Python skeleton to implement REJECT(), unput(), yyless() and so on
Nick Downing [Tue, 11 Sep 2018 05:17:21 +0000 (15:17 +1000)]
Rename plex to pilex, rename PLex object and prefix to AST
Nick Downing [Sun, 9 Sep 2018 04:22:13 +0000 (14:22 +1000)]
Add --python switch, skel/skel_py.py template, and tests/cal_py.l test scanner
Nick Downing [Fri, 10 Aug 2018 12:28:32 +0000 (22:28 +1000)]
Rename skeleton to skel_flex.c, include skel_flex.c in repository instead of requiring bootstrap_flex, fix a bug defining 'yy' prefix in wrong place in ast.py
Nick Downing [Wed, 8 Aug 2018 05:20:23 +0000 (15:20 +1000)]
Move FlexDFA generation into DFA.to_flex_dfa() consistent with other generators, and skeleton filling code from flex_dfa.py to generate_flex.py
Nick Downing [Sat, 28 Jul 2018 02:27:39 +0000 (12:27 +1000)]
Minor fixes to yywrap and how prefixes and outfiles are handled
Nick Downing [Wed, 25 Jul 2018 15:08:38 +0000 (01:08 +1000)]
Improve handling of %option and %array/%pointer, implement %option prefix=, improve skeleton/patching (more features, removes #line directives before patching)
Nick Downing [Tue, 24 Jul 2018 01:20:42 +0000 (11:20 +1000)]
Add continued actions and %option nodefault, make acclist generation slightly more robust, reinstate test code in regex.py which is invoked when run standalone
Nick Downing [Mon, 23 Jul 2018 08:50:03 +0000 (18:50 +1000)]
Change numpy.int16 to numpy.uint16 and flex_int16_t to flex_uint16_t everywhere
Nick Downing [Mon, 23 Jul 2018 08:49:04 +0000 (18:49 +1000)]
Make packing of states generate a list of candidates for first slot via numpy
Nick Downing [Mon, 23 Jul 2018 08:38:21 +0000 (18:38 +1000)]
Use a heap to speed up the minimum spanning tree generation
Nick Downing [Mon, 23 Jul 2018 05:40:06 +0000 (15:40 +1000)]
Implement %option caseless, also accept and ignore other options used in scan.l
Nick Downing [Mon, 23 Jul 2018 03:26:24 +0000 (13:26 +1000)]
Fix a bug in start condition handling, tidy, fix skeleton, add progress reports
Nick Downing [Sun, 22 Jul 2018 09:22:35 +0000 (19:22 +1000)]
Encode states in reverse order (larger distances first)
Nick Downing [Sun, 22 Jul 2018 08:27:04 +0000 (18:27 +1000)]
Add Prim's minimal spanning tree algorithm to encode states in an optimal order
Nick Downing [Sun, 22 Jul 2018 06:30:15 +0000 (16:30 +1000)]
Tidying up
Nick Downing [Sun, 22 Jul 2018 06:18:50 +0000 (16:18 +1000)]
Compute all state-to-state distances ahead of time then refer to them
Nick Downing [Sun, 22 Jul 2018 06:13:44 +0000 (16:13 +1000)]
Make better use of numpy.nonzero in compression step, borrowing from pyacc2.git
Nick Downing [Sun, 22 Jul 2018 04:29:32 +0000 (14:29 +1000)]
Introduce PLex.Section2.Rule.FLexRule which is always a pair of (expr, trailing context) rather than the old scheme which had an optional BOLRule inserted in
Nick Downing [Sun, 22 Jul 2018 04:18:34 +0000 (14:18 +1000)]
Implement compound rules
Nick Downing [Sun, 22 Jul 2018 03:49:41 +0000 (13:49 +1000)]
Change entries from a list of nxt, chk pairs to a numpy array
Nick Downing [Sun, 22 Jul 2018 03:25:57 +0000 (13:25 +1000)]
Tidying up
Nick Downing [Sun, 22 Jul 2018 02:26:00 +0000 (12:26 +1000)]
Change states from a list of base, def pairs to a numpy array
Nick Downing [Sun, 22 Jul 2018 01:57:33 +0000 (11:57 +1000)]
Change states_accept (as list of lists) to acclist and accept (as numpy arrays)
Nick Downing [Sun, 22 Jul 2018 01:14:47 +0000 (11:14 +1000)]
Rename full_entries to transition_table, do compression step after construction
Nick Downing [Sun, 22 Jul 2018 01:12:04 +0000 (11:12 +1000)]
Remove useless state_to_flex_base_def (can't reuse base, def due to yy_chk[])
Nick Downing [Sat, 21 Jul 2018 13:07:11 +0000 (23:07 +1000)]
Change the semantic analysis pass to only index the rules applicable to each start condition instead of generating the regex per start condition immediately, allows the semantic analysis result to be serialized without reparenting regexes
Nick Downing [Sat, 21 Jul 2018 12:29:08 +0000 (22:29 +1000)]
Introduce the PLex.Text.get_text() method, and rearrange class hierarchy a bit
Nick Downing [Sat, 21 Jul 2018 12:04:59 +0000 (22:04 +1000)]
Get latest regex.py and other modules (since it has been split up) from pyacc2.git commit
740dcdb, fix some bugs where dfa/nfa required a leading underscore
Nick Downing [Sat, 21 Jul 2018 11:50:24 +0000 (21:50 +1000)]
Add https://github.com/gautamkmr/Compiler-Lab- examples
Nick Downing [Fri, 6 Jul 2018 06:41:49 +0000 (16:41 +1000)]
Minor improvements to how test programs are built
Nick Downing [Mon, 2 Jul 2018 12:02:22 +0000 (22:02 +1000)]
Change bracketing of action text so that delimiters are captured within markup
Nick Downing [Mon, 2 Jul 2018 07:48:39 +0000 (17:48 +1000)]
Move CodeBlocks class inside new Section1Or2 class (formerly SectionCodeBlocks), also implement %option noyywrap and update the cal test progarm to not use -ll
Nick Downing [Sun, 1 Jul 2018 15:04:40 +0000 (01:04 +1000)]
Add degenerate.py, to strip out generated code and thus make the logic readable
Nick Downing [Sun, 1 Jul 2018 14:56:41 +0000 (00:56 +1000)]
Move Flex DFA generation into its own module, makes bootstrap_flex.py trivial
Nick Downing [Sun, 1 Jul 2018 14:50:31 +0000 (00:50 +1000)]
Further rearrangement to put everything inside the PLex class, and move the entire source file processing into the post_process() routine, leaves a serialized copy of the regular expression recognized in each start condition in PLex object
Nick Downing [Sat, 30 Jun 2018 06:35:52 +0000 (16:35 +1000)]
Rearrange AST class structure to use inner class definitions
Nick Downing [Sat, 30 Jun 2018 03:08:23 +0000 (13:08 +1000)]
Change how start conditions are represented in the syntax tree
Nick Downing [Sat, 30 Jun 2018 00:22:16 +0000 (10:22 +1000)]
Add start conditions (AT&T style only), improve EOF rules, add default action
Nick Downing [Fri, 29 Jun 2018 12:43:08 +0000 (22:43 +1000)]
Add BOL and EOF rules, not well tested yet
Nick Downing [Thu, 28 Jun 2018 23:48:49 +0000 (09:48 +1000)]
Fix issue with transitions/threads and issue with duplicate states (but the generated tables are definitely suboptimal, need to merge indistinguishable states)
Nick Downing [Thu, 28 Jun 2018 09:51:34 +0000 (19:51 +1000)]
Minor bug fixes to get it to scan something, enhance test program to print flex state machine progression (for comparison against our generated state machine)
Nick Downing [Wed, 27 Jun 2018 14:12:22 +0000 (00:12 +1000)]
First cut at generating state machines compatible with a generic flex skeleton