1 .TH FLEX 1 "26 May 1990" "Version 2.3"
3 flex - fast lexical analyzer generator
6 .B [-bcdfinpstvFILT8 -C[efmF] -Sskeleton]
10 is a tool for generating
12 programs which recognized lexical patterns in text.
15 the given input files, or its standard input if no file names are given,
16 for a description of a scanner to generate. The description is in
18 of regular expressions and C code, called
20 generates as output a C source file,
22 which defines a routine
24 This file is compiled and linked with the
26 library to produce an executable. When the executable is run,
27 it analyzes its input for occurrences
28 of the regular expressions. Whenever it finds one, it executes
29 the corresponding C code.
31 For full documentation, see
33 This manual entry is intended for use as a quick reference.
36 has the following options:
39 Generate backtracking information to
41 This is a list of scanner states which require backtracking
42 and the input characters on which they do so. By adding rules one
43 can remove backtracking states. If all backtracking states
48 is used, the generated scanner will run faster.
51 is a do-nothing, deprecated option included for POSIX compliance.
54 in previous releases of
57 specified table-compression options. This functionality is
60 flag. To ease the the impact of this change, when
64 it currently issues a warning message and assumes that
66 was desired instead. In the future this "promotion" of
70 will go away in the name of full POSIX compliance (unless
71 the POSIX meaning is removed first).
74 makes the generated scanner run in
76 mode. Whenever a pattern is recognized and the global
78 is non-zero (which is the default), the scanner will
84 --accepting rule at line 53 ("the matched text")
87 The line number refers to the location of the rule in the file
88 defining the scanner (i.e., the file that was fed to flex). Messages
89 are also generated when the scanner backtracks, accepts the
90 default rule, reaches the end of its input buffer (or encounters
91 a NUL; the two look the same as far as the scanner's concerned),
92 or reaches an end-of-file.
95 specifies (take your pick)
99 No table compression is done. The result is large but fast.
100 This option is equivalent to
109 scanner. The case of letters given in the
112 be ignored, and tokens in the input will be matched regardless of case. The
113 matched text given in
115 will have the preserved case (i.e., it will not be folded).
118 is another do-nothing, deprecated option included only for
122 generates a performance report to stderr. The report
123 consists of comments regarding features of the
125 input file which will cause a loss of performance in the resulting scanner.
130 (that unmatched scanner input is echoed to
132 to be suppressed. If the scanner encounters input that does not
133 match any of its rules, it aborts with an error.
138 to write the scanner it generates to standard output instead
147 a summary of statistics regarding the scanner it generates.
153 scanner table representation should be used. This representation is
154 about as fast as the full table representation
157 and for some sets of patterns will be considerably smaller (and for
162 This option is equivalent to
171 scanner, that is, a scanner which stops immediately rather than
172 looking ahead if it knows
173 that the currently scanned text cannot be part of a longer rule's match.
180 cannot be used in conjunction with
197 The default is to generate such directives so error
198 messages in the actions will be correctly
199 located with respect to the original
201 input file, and not to
202 the fairly meaningless line numbers of
210 mode. It will generate a lot of messages to
213 the form of the input and the resultant non-deterministic and deterministic
214 finite automata. This option is mostly for use in maintaining
220 to generate an 8-bit scanner.
221 On some sites, this is the default. On others, the default
222 is 7-bit characters. To see which is the case, check the verbose
224 output for "equivalence classes created". If the denominator of
225 the number shown is 128, then by default
227 is generating 7-bit characters. If it is 256, then the default is
231 controls the degree of table compression.
237 .I equivalence classes,
238 i.e., sets of characters
239 which have identical lexical properties.
240 Equivalence classes usually give
241 dramatic reductions in the final table/object file sizes (typically
242 a factor of 2-5) and are pretty cheap performance-wise (one array
243 look-up per character scanned).
248 scanner tables should be generated -
250 should not compress the
251 tables by taking advantages of similar transition functions for
255 specifies that the alternate fast scanner representation (described in
263 .I meta-equivalence classes,
264 which are sets of equivalence classes (or characters, if equivalence
265 classes are not being used) that are commonly used together. Meta-equivalence
266 classes are often a big win when using compressed tables, but they
267 have a moderate performance impact (one or two "if" tests and one
268 array look-up per character scanned).
272 specifies that the scanner tables should be compressed but neither
273 equivalence classes nor meta-equivalence classes should be used.
281 do not make sense together - there is no opportunity for meta-equivalence
282 classes if the table is not being compressed. Otherwise the options
285 The default setting is
289 should generate equivalence classes
290 and meta-equivalence classes. This setting provides the highest
291 degree of table compression. You can trade off
292 faster-executing scanners at the cost of larger tables with
293 the following generally being true:
308 options are not cumulative; whenever the flag is encountered, the
309 previous -C settings are forgotten.
312 overrides the default skeleton file from which
314 constructs its scanners. You'll never need this option unless you are doing
316 maintenance or development.
317 .SH SUMMARY OF FLEX REGULAR EXPRESSIONS
318 The patterns in the input are written using an extended set of regular
319 expressions. These are:
322 x match the character 'x'
323 . any character except newline
324 [xyz] a "character class"; in this case, the pattern
325 matches either an 'x', a 'y', or a 'z'
326 [abj-oZ] a "character class" with a range in it; matches
327 an 'a', a 'b', any letter from 'j' through 'o',
329 [^A-Z] a "negated character class", i.e., any character
330 but those in the class. In this case, any
331 character EXCEPT an uppercase letter.
332 [^A-Z\\n] any character EXCEPT an uppercase letter or
334 r* zero or more r's, where r is any regular expression
336 r? zero or one r's (that is, "an optional r")
337 r{2,5} anywhere from two to five r's
338 r{2,} two or more r's
340 {name} the expansion of the "name" definition
343 the literal string: [xyz]"foo
344 \\X if X is an 'a', 'b', 'f', 'n', 'r', 't', or 'v',
345 then the ANSI-C interpretation of \\x.
346 Otherwise, a literal 'X' (used to escape
347 operators such as '*')
348 \\123 the character with octal value 123
349 \\x2a the character with hexadecimal value 2a
350 (r) match an r; parentheses are used to override
351 precedence (see below)
354 rs the regular expression r followed by the
355 regular expression s; called "concatenation"
358 r|s either an r or an s
361 r/s an r but only if it is followed by an s. The
362 s is not part of the matched text. This type
363 of pattern is called as "trailing context".
364 ^r an r, but only at the beginning of a line
365 r$ an r, but only at the end of a line. Equivalent
369 <s>r an r, but only in start condition s (see
370 below for discussion of start conditions)
372 same, but in any of start conditions s1,
376 <<EOF>> an end-of-file
378 an end-of-file when in start condition s1 or s2
381 The regular expressions listed above are grouped according to
382 precedence, from highest precedence at the top to lowest at the bottom.
383 Those grouped together have equal precedence.
385 Some notes on patterns:
387 Negated character classes
389 unless "\\n" (or an equivalent escape sequence) is one of the
390 characters explicitly present in the negated character class
393 A rule can have at most one instance of trailing context (the '/' operator
394 or the '$' operator). The start condition, '^', and "<<EOF>>" patterns
395 can only occur at the beginning of a pattern, and, as well as with '/' and '$',
396 cannot be grouped inside parentheses. The following are all illegal:
405 .SH SUMMARY OF SPECIAL ACTIONS
406 In addition to arbitrary C code, the following can appear in actions:
409 copies yytext to the scanner's output.
412 followed by the name of a start condition places the scanner in the
413 corresponding start condition.
416 directs the scanner to proceed on to the "second best" rule which matched the
417 input (or a prefix of the input).
421 are set up appropriately. Note that
423 is a particularly expensive feature in terms scanner performance;
426 of the scanner's actions it will slow down
428 of the scanner's matching. Furthermore,
430 cannot be used with the
436 Note also that unlike the other special actions,
440 code immediately following it in the action will
445 tells the scanner that the next time it matches a rule, the corresponding
448 onto the current value of
450 rather than replacing it.
453 returns all but the first
455 characters of the current token back to the input stream, where they
456 will be rescanned when the scanner looks for the next match.
460 are adjusted appropriately (e.g.,
469 back onto the input stream. It will be the next character scanned.
472 reads the next character from the input stream (this routine is called
474 if the scanner is compiled using
478 can be used in lieu of a return statement in an action. It terminates
479 the scanner and returns a 0 to the scanner's caller, indicating "all done".
483 is also called when an end-of-file is encountered. It is a macro and
487 is an action available only in <<EOF>> rules. It means "Okay, I've
488 set up a new input file, continue scanning".
490 .B yy_create_buffer( file, size )
493 pointer and an integer
495 It returns a YY_BUFFER_STATE
496 handle to a new input buffer large enough to accomodate
498 characters and associated with the given file. When in doubt, use
502 .B yy_switch_to_buffer( new_buffer )
503 switches the scanner's processing to scan for tokens from
504 the given buffer, which must be a YY_BUFFER_STATE.
506 .B yy_delete_buffer( buffer )
507 deletes the given buffer.
508 .SH VALUES AVAILABLE TO THE USER
511 holds the text of the current token. It may not be modified.
514 holds the length of the current token. It may not be modified.
517 is the file which by default
519 reads from. It may be redefined but doing so only makes sense before
520 scanning begins. Changing it in the middle of scanning will have
521 unexpected results since
523 buffers its input. Once scanning terminates because an end-of-file
526 void yyrestart( FILE *new_file )
527 may be called to point
529 at the new input file.
534 actions are done. It can be reassigned by the user.
539 handle to the current buffer.
540 .SH MACROS THE USER CAN REDEFINE
543 controls how the scanning routine is declared.
544 By default, it is "int yylex()", or, if prototypes are being
545 used, "int yylex(void)". This definition may be changed by redefining
546 the "YY_DECL" macro. Note that
547 if you give arguments to the scanning routine using a
548 K&R-style/non-prototyped function declaration, you must terminate
549 the definition with a semi-colon (;).
551 The nature of how the scanner
552 gets its input can be controlled by redefining the
555 YY_INPUT's calling sequence is "YY_INPUT(buf,result,max_size)". Its
556 action is to place up to
558 characters in the character array
560 and return in the integer variable
563 number of characters read or the constant YY_NULL (0 on Unix systems)
564 to indicate EOF. The default YY_INPUT reads from the
565 global file-pointer "yyin".
566 A sample redefinition of YY_INPUT (in the definitions
567 section of the input file):
572 #define YY_INPUT(buf,result,max_size) \\
574 int c = getchar(); \\
575 result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \\
581 When the scanner receives an end-of-file indication from YY_INPUT,
586 returns false (zero), then it is assumed that the
587 function has gone ahead and set up
589 to point to another input file, and scanning continues. If it returns
590 true (non-zero), then the scanner terminates, returning 0 to its
595 always returns 1. Presently, to redefine it you must first
596 "#undef yywrap", as it is currently implemented as a macro. It is
599 will soon be defined to be a function rather than a macro.
602 can be redefined to provide an action
603 which is always executed prior to the matched rule's action.
607 may be redefined to provide an action which is always executed before
610 In the generated scanner, the actions are all gathered in one large
611 switch statement and separated using
613 which may be redefined. By default, it is simply a "break", to separate
614 each rule's action from the following rule's.
621 generated scanner (called
626 backtracking information for
633 library with which to link the scanners.
636 flexdoc(1), lex(1), yacc(1), sed(1), awk(1).
638 M. E. Lesk and E. Schmidt,
639 .I LEX - Lexical Analyzer Generator
641 .I reject_used_but_not_detected undefined
644 .I yymore_used_but_not_detected undefined -
645 These errors can occur at compile time. They indicate that the
652 failed to notice the fact, meaning that
654 scanned the first two sections looking for occurrences of these actions
655 and failed to find any, but somehow you snuck some in (via a #include
656 file, for example). Make an explicit reference to the action in your
658 input file. (Note that previously
662 mechanism for dealing with this problem; this feature is still supported
663 but now deprecated, and will go away soon unless the author hears from
664 people who can argue compellingly that they need it.)
666 .I flex scanner jammed -
667 a scanner compiled with
669 has encountered an input string which wasn't matched by
672 .I flex input buffer overflowed -
673 a scanner rule matched a string long enough to overflow the
674 scanner's internal input buffer (16K bytes - controlled by
678 .I scanner requires -8 flag -
679 Your scanner specification includes recognizing 8-bit characters and
680 you did not specify the -8 flag (and your site has not installed flex
681 with -8 as the default).
684 fatal flex scanner internal error--end of buffer missed -
685 This can occur in an scanner which is reentered after a long-jump
686 has jumped out (or over) the scanner's activation frame. Before
687 reentering the scanner, use:
694 .I too many %t classes! -
695 You managed to put every single character into its own %t class.
697 requires that at least one of the classes share characters.
699 Vern Paxson, with the help of many ideas and much inspiration from
700 Van Jacobson. Original version by Jef Poskanzer.
702 See flexdoc(1) for additional credits and the address to send comments to.
703 .SH DEFICIENCIES / BUGS
705 Some trailing context
706 patterns cannot be properly matched and generate
707 warning messages ("Dangerous trailing context"). These are
708 patterns where the ending of the
709 first part of the rule matches the beginning of the second
710 part, such as "zx*/xy*", where the 'x*' matches the 'x' at
711 the beginning of the trailing context. (Note that the POSIX draft
712 states that the text matched by such patterns is undefined.)
714 For some trailing context rules, parts which are actually fixed-length are
715 not recognized as such, leading to the abovementioned performance loss.
716 In particular, parts using '|' or {n} (such as "foo{3}") are always
717 considered variable-length.
719 Combining trailing context with the special '|' action can result in
721 trailing context being turned into the more expensive
723 trailing context. For example, this happens in the following example:
732 Use of unput() invalidates yytext and yyleng.
734 Use of unput() to push back more text than was matched can
735 result in the pushed-back text matching a beginning-of-line ('^')
736 rule even though it didn't come at the beginning of the line
737 (though this is rare!).
739 Pattern-matching of NUL's is substantially slower than matching other
743 does not generate correct #line directives for code internal
744 to the scanner; thus, bugs in
746 yield bogus line numbers.
748 Due to both buffering of input and read-ahead, you cannot intermix
749 calls to <stdio.h> routines, such as, for example,
753 rules and expect it to work. Call
757 The total table entries listed by the
759 flag excludes the number of table entries needed to determine
760 what rule has been matched. The number of entries is equal
761 to the number of DFA states if the scanner does not use
763 and somewhat greater than the number of states if it does.
766 cannot be used with the
772 Some of the macros, such as
774 may in the future become functions which live in the
776 library. This will doubtless break a lot of code, but may be
777 required for POSIX-compliance.
781 internal algorithms need documentation.