PROLOG | NAME | SYNOPSIS | DESCRIPTION | OPTIONS | OPERANDS | STDIN | INPUT FILES | ENVIRONMENT VARIABLES | ASYNCHRONOUS EVENTS | STDOUT | STDERR | OUTPUT FILES | EXTENDED DESCRIPTION | EXIT STATUS | CONSEQUENCES OF ERRORS | APPLICATION USAGE | EXAMPLES | RATIONALE | FUTURE DIRECTIONS | SEE ALSO | COPYRIGHT

LEX(1P)                   POSIX Programmer's Manual                  LEX(1P)

PROLOG         top

       This manual page is part of the POSIX Programmer's Manual.  The Linux
       implementation of this interface may differ (consult the
       corresponding Linux manual page for details of Linux behavior), or
       the interface may not be implemented on Linux.

NAME         top

       lex — generate programs for lexical tasks (DEVELOPMENT)

SYNOPSIS         top

       lex [−t] [−n|−v] [file...]

DESCRIPTION         top

       The lex utility shall generate C programs to be used in lexical
       processing of character input, and that can be used as an interface
       to yacc.  The C programs shall be generated from lex source code and
       conform to the ISO C standard, without depending on any undefined,
       unspecified, or implementation-defined behavior, except in cases
       where the code is copied directly from the supplied source, or in
       cases that are documented by the implementation. Usually, the lex
       utility shall write the program it generates to the file lex.yy.c;
       the state of this file is unspecified if lex exits with a non-zero
       exit status. See the EXTENDED DESCRIPTION section for a complete
       description of the lex input language.

OPTIONS         top

       The lex utility shall conform to the Base Definitions volume of
       POSIX.1‐2008, Section 12.2, Utility Syntax Guidelines, except for
       Guideline 9.
       The following options shall be supported:
       −n        Suppress the summary of statistics usually written with the
                 −v option. If no table sizes are specified in the lex
                 source code and the −v option is not specified, then −n is
                 implied.
       −t        Write the resulting program to standard output instead of
                 lex.yy.c.
       −v        Write a summary of lex statistics to the standard output.
                 (See the discussion of lex table sizes in Definitions in
                 lex.)  If the −t option is specified and −n is not
                 specified, this report shall be written to standard error.
                 If table sizes are specified in the lex source code, and if
                 the −n option is not specified, the −v option may be
                 enabled.

OPERANDS         top

       The following operand shall be supported:
       file      A pathname of an input file. If more than one such file is
                 specified, all files shall be concatenated to produce a
                 single lex program. If no file operands are specified, or
                 if a file operand is '−', the standard input shall be used.

STDIN         top

       The standard input shall be used if no file operands are specified,
       or if a file operand is '−'.  See INPUT FILES.

INPUT FILES         top

       The input files shall be text files containing lex source code, as
       described in the EXTENDED DESCRIPTION section.

ENVIRONMENT VARIABLES         top

       The following environment variables shall affect the execution of
       lex:
       LANG      Provide a default value for the internationalization
                 variables that are unset or null. (See the Base Definitions
                 volume of POSIX.1‐2008, Section 8.2, Internationalization
                 Variables for the precedence of internationalization
                 variables used to determine the values of locale
                 categories.)
       LC_ALL    If set to a non-empty string value, override the values of
                 all the other internationalization variables.
       LC_COLLATE
                 Determine the locale for the behavior of ranges,
                 equivalence classes, and multi-character collating elements
                 within regular expressions. If this variable is not set to
                 the POSIX locale, the results are unspecified.
       LC_CTYPE  Determine the locale for the interpretation of sequences of
                 bytes of text data as characters (for example, single-byte
                 as opposed to multi-byte characters in arguments and input
                 files), and the behavior of character classes within
                 regular expressions. If this variable is not set to the
                 POSIX locale, the results are unspecified.
       LC_MESSAGES
                 Determine the locale that should be used to affect the
                 format and contents of diagnostic messages written to
                 standard error.
       NLSPATH   Determine the location of message catalogs for the
                 processing of LC_MESSAGES.

ASYNCHRONOUS EVENTS         top

       Default.

STDOUT         top

       If the −t option is specified, the text file of C source code output
       of lex shall be written to standard output.
       If the −t option is not specified:
        *  Implementation-defined informational, error, and warning messages
           concerning the contents of lex source code input shall be written
           to either the standard output or standard error.
        *  If the −v option is specified and the −n option is not specified,
           lex statistics shall also be written to either the standard
           output or standard error, in an implementation-defined format.
           These statistics may also be generated if table sizes are
           specified with a '%' operator in the Definitions section, as long
           as the −n option is not specified.

STDERR         top

       If the −t option is specified, implementation-defined informational,
       error, and warning messages concerning the contents of lex source
       code input shall be written to the standard error.
       If the −t option is not specified:
        1. Implementation-defined informational, error, and warning messages
           concerning the contents of lex source code input shall be written
           to either the standard output or standard error.
        2. If the −v option is specified and the −n option is not specified,
           lex statistics shall also be written to either the standard
           output or standard error, in an implementation-defined format.
           These statistics may also be generated if table sizes are
           specified with a '%' operator in the Definitions section, as long
           as the −n option is not specified.

OUTPUT FILES         top

       A text file containing C source code shall be written to lex.yy.c, or
       to the standard output if the −t option is present.

EXTENDED DESCRIPTION         top

       Each input file shall contain lex source code, which is a table of
       regular expressions with corresponding actions in the form of C
       program fragments.
       When lex.yy.c is compiled and linked with the lex library (using the
       −l l operand with c99), the resulting program shall read character
       input from the standard input and shall partition it into strings
       that match the given expressions.
       When an expression is matched, these actions shall occur:
        *  The input string that was matched shall be left in yytext as a
           null-terminated string; yytext shall either be an external
           character array or a pointer to a character string. As explained
           in Definitions in lex, the type can be explicitly selected using
           the %array or %pointer declarations, but the default is
           implementation-defined.
        *  The external int yyleng shall be set to the length of the
           matching string.
        *  The expression's corresponding program fragment, or action, shall
           be executed.
       During pattern matching, lex shall search the set of patterns for the
       single longest possible match. Among rules that match the same number
       of characters, the rule given first shall be chosen.
       The general format of lex source shall be:
              Definitions %% Rules %% UserSubroutines
       The first "%%" is required to mark the beginning of the rules
       (regular expressions and actions); the second "%%" is required only
       if user subroutines follow.
       Any line in the Definitions section beginning with a <blank> shall be
       assumed to be a C program fragment and shall be copied to the
       external definition area of the lex.yy.c file. Similarly, anything in
       the Definitions section included between delimiter lines containing
       only "%{" and "%}" shall also be copied unchanged to the external
       definition area of the lex.yy.c file.
       Any such input (beginning with a <blank> or within "%{" and "%}"
       delimiter lines) appearing at the beginning of the Rules section
       before any rules are specified shall be written to lex.yy.c after the
       declarations of variables for the yylex() function and before the
       first line of code in yylex().  Thus, user variables local to yylex()
       can be declared here, as well as application code to execute upon
       entry to yylex().
       The action taken by lex when encountering any input beginning with a
       <blank> or within "%{" and "%}" delimiter lines appearing in the
       Rules section but coming after one or more rules is undefined. The
       presence of such input may result in an erroneous definition of the
       yylex() function.
       C-language code in the input shall not contain C-language trigraphs.
       The C-language code within "%{" and "%}" delimiter lines shall not
       contain any lines consisting only of "%}", or only of "%%".
   Definitions in lex
       Definitions appear before the first "%%" delimiter. Any line in this
       section not contained between "%{" and "%}" lines and not beginning
       with a <blank> shall be assumed to define a lex substitution string.
       The format of these lines shall be:
           name substitute
       If a name does not meet the requirements for identifiers in the ISO C
       standard, the result is undefined. The string substitute shall
       replace the string {name} when it is used in a rule. The name string
       shall be recognized in this context only when the braces are provided
       and when it does not appear within a bracket expression or within
       double-quotes.
       In the Definitions section, any line beginning with a <percent-sign>
       ('%') character and followed by an alphanumeric word beginning with
       either 's' or 'S' shall define a set of start conditions. Any line
       beginning with a '%' followed by a word beginning with either 'x' or
       'X' shall define a set of exclusive start conditions. When the
       generated scanner is in a %s state, patterns with no state specified
       shall be also active; in a %x state, such patterns shall not be
       active. The rest of the line, after the first word, shall be
       considered to be one or more <blank>-separated names of start
       conditions. Start condition names shall be constructed in the same
       way as definition names. Start conditions can be used to restrict the
       matching of regular expressions to one or more states as described in
       Regular Expressions in lex.
       Implementations shall accept either of the following two mutually-
       exclusive declarations in the Definitions section:
       %array    Declare the type of yytext to be a null-terminated
                 character array.
       %pointer  Declare the type of yytext to be a pointer to a null-
                 terminated character string.
       The default type of yytext is implementation-defined. If an
       application refers to yytext outside of the scanner source file (that
       is, via an extern), the application shall include the appropriate
       %array or %pointer declaration in the scanner source file.
       Implementations shall accept declarations in the Definitions section
       for setting certain internal table sizes. The declarations are shown
       in the following table.
                       Table: Table Size Declarations in lex
        ┌────────────┬────────────────────────────────────┬───────────────┐
        │Declaration Description             Minimum Value │
        ├────────────┼────────────────────────────────────┼───────────────┤
        │%p n        │ Number of positions                │     2500      │
        │%n n        │ Number of states                   │      500      │
        │%a n        │ Number of transitions              │     2000      │
        │%e n        │ Number of parse tree nodes         │     1000      │
        │%k n        │ Number of packed character classes │     1000      │
        │%o n        │ Size of the output array           │     3000      │
        └────────────┴────────────────────────────────────┴───────────────┘
       In the table, n represents a positive decimal integer, preceded by
       one or more <blank> characters. The exact meaning of these table size
       numbers is implementation-defined. The implementation shall document
       how these numbers affect the lex utility and how they are related to
       any output that may be generated by the implementation should
       limitations be encountered during the execution of lex.  It shall be
       possible to determine from this output which of the table size values
       needs to be modified to permit lex to successfully generate tables
       for the input language. The values in the column Minimum Value
       represent the lowest values conforming implementations shall provide.
   Rules in lex
       The rules in lex source files are a table in which the left column
       contains regular expressions and the right column contains actions (C
       program fragments) to be executed when the expressions are
       recognized.
           ERE action
           ERE action
           ...
       The extended regular expression (ERE) portion of a row shall be
       separated from action by one or more <blank> characters. A regular
       expression containing <blank> characters shall be recognized under
       one of the following conditions:
        *  The entire expression appears within double-quotes.
        *  The <blank> characters appear within double-quotes or square
           brackets.
        *  Each <blank> is preceded by a <backslash> character.
   User Subroutines in lex
       Anything in the user subroutines section shall be copied to lex.yy.c
       following yylex().
   Regular Expressions in lex
       The lex utility shall support the set of extended regular expressions
       (see the Base Definitions volume of POSIX.1‐2008, Section 9.4,
       Extended Regular Expressions), with the following additions and
       exceptions to the syntax:
       "..."     Any string enclosed in double-quotes shall represent the
                 characters within the double-quotes as themselves, except
                 that <backslash>-escapes (which appear in the following
                 table) shall be recognized. Any <backslash>-escape sequence
                 shall be terminated by the closing quote. For example,
                 "\01""1" represents a single string: the octal value 1
                 followed by the character '1'.
       <state>r, <state1,state2,...>r
                 The regular expression r shall be matched only when the
                 program is in one of the start conditions indicated by
                 state, state1, and so on; see Actions in lex.  (As an
                 exception to the typographical conventions of the rest of
                 this volume of POSIX.1‐2008, in this case <state> does not
                 represent a metavariable, but the literal angle-bracket
                 characters surrounding a symbol.) The start condition shall
                 be recognized as such only at the beginning of a regular
                 expression.
       r/x       The regular expression r shall be matched only if it is
                 followed by an occurrence of regular expression x (x is the
                 instance of trailing context, further defined below). The
                 token returned in yytext shall only match r.  If the
                 trailing portion of r matches the beginning of x, the
                 result is unspecified. The r expression cannot include
                 further trailing context or the '$' (match-end-of-line)
                 operator; x cannot include the '^' (match-beginning-of-
                 line) operator, nor trailing context, nor the '$' operator.
                 That is, only one occurrence of trailing context is allowed
                 in a lex regular expression, and the '^' operator only can
                 be used at the beginning of such an expression.
       {name}    When name is one of the substitution symbols from the
                 Definitions section, the string, including the enclosing
                 braces, shall be replaced by the substitute value. The
                 substitute value shall be treated in the extended regular
                 expression as if it were enclosed in parentheses. No
                 substitution shall occur if {name} occurs within a bracket
                 expression or within double-quotes.
       Within an ERE, a <backslash> character shall be considered to begin
       an escape sequence as specified in the table in the Base Definitions
       volume of POSIX.1‐2008, Chapter 5, File Format Notation ('\\', '\a',
       '\b', '\f', '\n', '\r', '\t', '\v').  In addition, the escape
       sequences in the following table shall be recognized.
       A literal <newline> cannot occur within an ERE; the escape sequence
       '\n' can be used to represent a <newline>.  A <newline> shall not be
       matched by a period operator.
                          Table: Escape Sequences in lex
         ┌─────────┬──────────────────────────┬──────────────────────────┐
         │ Escape  │                          │                          │
         │Sequence Description        Meaning          │
         ├─────────┼──────────────────────────┼──────────────────────────┤
         │\digits  │ A <backslash> character  │ The character whose      │
         │         │ followed by the longest  │ encoding is represented  │
         │         │ sequence of one, two, or │ by the one, two, or      │
         │         │ three octal-digit        │ three-digit octal        │
         │         │ characters (01234567).   │ integer. Multi-byte      │
         │         │ If all of the digits are │ characters require       │
         │         │ 0 (that is,              │ multiple, concatenated   │
         │         │ representation of the    │ escape sequences of this │
         │         │ NUL character), the      │ type, including the      │
         │         │ behavior is undefined.   │ leading <backslash> for  │
         │         │                          │ each byte.               │
         ├─────────┼──────────────────────────┼──────────────────────────┤
         │\xdigits │ A <backslash> character  │ The character whose      │
         │         │ followed by the longest  │ encoding is represented  │
         │         │ sequence of hexadecimal- │ by the hexadecimal       │
         │         │ digit characters         │ integer.                 │
         │         │ (01234567abcdefABCDEF).  │                          │
         │         │ If all of the digits are │                          │
         │         │ 0 (that is,              │                          │
         │         │ representation of the    │                          │
         │         │ NUL character), the      │                          │
         │         │ behavior is undefined.   │                          │
         ├─────────┼──────────────────────────┼──────────────────────────┤
         │\c       │ A <backslash> character  │ The character 'c',       │
         │         │ followed by any          │ unchanged.               │
         │         │ character not described  │                          │
         │         │ in this table or in the  │                          │
         │         │ table in the Base        │                          │
         │         │ Definitions volume of    │                          │
         │         │ POSIX.1‐2008, Chapter 5, │                          │
         │         │ File Format Notation     │                          │
         │         │ ('\\', '\a', '\b', '\f', │                          │
         │         │ '\n', '\r', '\t', '\v'). │                          │
         └─────────┴──────────────────────────┴──────────────────────────┘
       Note:     If a '\x' sequence needs to be immediately  followed  by  a
                 hexadecimal  digit  character,  a sequence such as "\x1""1"
                 can be used, which represents a  character  containing  the
                 value 1, followed by the character '1'.
       The order of precedence given to extended regular expressions for lex
       differs from  that  specified  in  the  Base  Definitions  volume  of
       POSIX.1‐2008,  Section  9.4, Extended Regular Expressions.  The order
       of precedence for lex shall be as shown in the following table,  from
       high to low.
       Note:     The  escaped  characters  entry  is not meant to imply that
                 these are operators, but they are included in the table  to
                 show  their  relationships to the true operators. The start
                 condition, trailing context, and anchoring  notations  have
                 been  omitted  from  the  table  because  of  the placement
                 restrictions described  in  this  section;  they  can  only
                 appear at the beginning or ending of an ERE.
                           Table: ERE Precedence in lex
            ┌──────────────────────────────────┬──────────────────────┐
            │   Extended Regular Expression    Precedence      │
            ├──────────────────────────────────┼──────────────────────┤
            │collation-related bracket symbols │ [= =]  [: :]  [. .]  │
            │escaped characters                │ \<special character> │
            │bracket expression                │ [ ]                  │
            │quoting                           │ "..."                │
            │grouping                          │ ( )                  │
            │definition                        │ {name}               │
            │single-character RE duplication   │ * + ?                │
            │concatenation                     │                      │
            │interval expression               │ {m,n}                │
            │alternation                       │ |                    │
            └──────────────────────────────────┴──────────────────────┘
       The  ERE  anchoring operators '^' and '$' do not appear in the table.
       With lex regular expressions, these operators are restricted in their
       use:  the '^' operator can only be used at the beginning of an entire
       regular expression, and  the  '$'  operator  only  at  the  end.  The
       operators  apply to the entire regular expression. Thus, for example,
       the pattern "(^abc)|(def$)" is undefined; it can instead  be  written
       as two separate rules, one with the regular expression "^abc" and one
       with "def$", which share a common action via the special  '|'  action
       (see  below). If the pattern were written "^abc|def$", it would match
       either "abc" or "def" on a line by itself.
       Unlike the general ERE rules, embedded anchoring is  not  allowed  by
       most historical lex implementations. An example of embedded anchoring
       would be for patterns such as "(^| )foo( |$)" to match "foo" when  it
       exists  as  a complete word. This functionality can be obtained using
       existing lex features:
           ^foo/[ \n]      |
           " foo"/[ \n]    /* Found foo as a separate word. */
       Note also that '$' is a form of trailing context (it is equivalent to
       "/\n") and as such cannot be used with regular expressions containing
       another instance of the operator (see  the  preceding  discussion  of
       trailing context).
       The  additional regular expressions trailing-context operator '/' can
       be used as an ordinary character if presented  within  double-quotes,
       "/"; preceded by a <backslash>, "\/"; or within a bracket expression,
       "[/]".  The start-condition '<' and '>' operators  shall  be  special
       only  in  a start condition at the beginning of a regular expression;
       elsewhere in the regular expression they shall be treated as ordinary
       characters.
   Actions in lex
       The  action  to  be  taken  when an ERE is matched can be a C program
       fragment or the special actions described below; the program fragment
       can  contain  one  or more C statements, and can also include special
       actions. The empty C statement ';'  shall  be  a  valid  action;  any
       string in the lex.yy.c input that matches the pattern portion of such
       a rule is effectively ignored or skipped. However, the absence of  an
       action  shall  not  be  valid,  and  the  action  lex takes in such a
       condition is undefined.
       The specification for an action, including C statements  and  special
       actions, can extend across several lines if enclosed in braces:
           ERE <one or more blanks> { program statement
                                      program statement }
       The  program  statements  shall  not  contain  unbalanced curly brace
       preprocessing tokens.
       The default action when a string in the input to a  lex.yy.c  program
       is  not  matched by any expression shall be to copy the string to the
       output. Because the default behavior of a program generated by lex is
       to  read  the  input  and copy it to the output, a minimal lex source
       program that has just "%%" shall generate a  C  program  that  simply
       copies the input to the output unchanged.
       Four special actions shall be available:
           |   ECHO;   REJECT;   BEGIN
       |         The  action  '|' means that the action for the next rule is
                 the action for this rule.  Unlike the other three  actions,
                 '|'    cannot    be    enclosed    in    braces    or    be
                 <semicolon>-terminated; the application shall  ensure  that
                 it is specified alone, with no other actions.
       ECHO;     Write the contents of the string yytext on the output.
       REJECT;   Usually  only  a  single  expression  is matched by a given
                 string in the input.  REJECT means ``continue to  the  next
                 expression  that  matches  the  current  input'', and shall
                 cause whatever rule was the second choice after the current
                 rule  to  be  executed  for  the same input. Thus, multiple
                 rules can be matched and executed for one input  string  or
                 overlapping  input  strings. For example, given the regular
                 expressions "xyz" and "xy" and  the  input  "xyz",  usually
                 only  the  regular  expression  "xyz" would match. The next
                 attempted match would start after z.  If the last action in
                 the  "xyz" rule is REJECT, both this rule and the "xy" rule
                 would be executed. The REJECT action may be implemented  in
                 such a fashion that flow of control does not continue after
                 it, as if it were equivalent to a goto to another  part  of
                 yylex().   The  use of REJECT may result in somewhat larger
                 and slower scanners.
       BEGIN     The action:
                     BEGIN newstate;
                 switches the state (start condition) to newstate.   If  the
                 string newstate has not been declared previously as a start
                 condition in  the  Definitions  section,  the  results  are
                 unspecified.  The  initial  state is indicated by the digit
                 '0' or the token INITIAL.
       The functions or macros described below are accessible to  user  code
       included  in  the lex input. It is unspecified whether they appear in
       the C code output of lex, or are accessible  only  through  the  −l l
       operand to c99 (the lex library).
       int yylex(void)
             Performs  lexical  analysis  on  the input; this is the primary
             function generated by  the  lex  utility.  The  function  shall
             return  zero  when  the  end of input is reached; otherwise, it
             shall return non-zero values (tokens) determined by the actions
             that are selected.
       int yymore(void)
             When  called,  indicates  that  when  the  next input string is
             recognized, it is to be appended to the current value of yytext
             rather than replacing it; the value in yyleng shall be adjusted
             accordingly.
       int yyless(int n)
             Retains n initial characters  in  yytext,  NUL-terminated,  and
             treats  the  remaining characters as if they had not been read;
             the value in yyleng shall be adjusted accordingly.
       int input(void)
             Returns the next character from the input, or zero  on  end-of-
             file.  It  shall  obtain  input  from  the stream pointer yyin,
             although  possibly  via  an  intermediate  buffer.  Thus,  once
             scanning has begun, the effect of altering the value of yyin is
             undefined. The character read shall be removed from  the  input
             stream of the scanner without any processing by the scanner.
       int unput(int c)
             Returns  the  character 'c' to the input; yytext and yyleng are
             undefined until the next expression is matched. The  result  of
             using  unput()  for  more  characters  than  have been input is
             unspecified.
       The  following  functions  shall  appear  only  in  the  lex  library
       accessible  through the −l l operand; they can therefore be redefined
       by a conforming application:
       int yywrap(void)
             Called by yylex() at end-of-file; the  default  yywrap()  shall
             always  return  1.  If  the  application  requires  yylex()  to
             continue processing with another  source  of  input,  then  the
             application  can  include a function yywrap(), which associates
             another file with the external variable FILE * yyin  and  shall
             return a value of zero.
       int main(int argc, char *argv[])
             Calls yylex() to perform lexical analysis, then exits. The user
             code  can  contain  main()  to   perform   application-specific
             operations, calling yylex() as applicable.
       Except  for  input(),  unput(),  and  main(), all external and static
       names generated by lex shall begin with the prefix yy or YY.

EXIT STATUS         top

       The following exit values shall be returned:
        0    Successful completion.
       >0    An error occurred.

CONSEQUENCES OF ERRORS         top

       Default.
       The following sections are informative.

APPLICATION USAGE         top

       Conforming applications are warned that in the Rules section, an ERE
       without an action is not acceptable, but need not be detected as
       erroneous by lex.  This may result in compilation or runtime errors.
       The purpose of input() is to take characters off the input stream and
       discard them as far as the lexical analysis is concerned. A common
       use is to discard the body of a comment once the beginning of a
       comment is recognized.
       The lex utility is not fully internationalized in its treatment of
       regular expressions in the lex source code or generated lexical
       analyzer. It would seem desirable to have the lexical analyzer
       interpret the regular expressions given in the lex source according
       to the environment specified when the lexical analyzer is executed,
       but this is not possible with the current lex technology.
       Furthermore, the very nature of the lexical analyzers produced by lex
       must be closely tied to the lexical requirements of the input
       language being described, which is frequently locale-specific anyway.
       (For example, writing an analyzer that is used for French text is not
       automatically useful for processing other languages.)

EXAMPLES         top

       The following is an example of a lex program that implements a
       rudimentary scanner for a Pascal-like syntax:
           %{
           /* Need this for the call to atof() below. */
           #include <math.h>
           /* Need this for printf(), fopen(), and stdin below. */
           #include <stdio.h>
           %}
           DIGIT    [0−9]
           ID       [a−z][a−z0−9]*
           %%
           {DIGIT}+ {
               printf("An integer: %s (%d)\n", yytext,
                   atoi(yytext));
               }
           {DIGIT}+"."{DIGIT}*        {
               printf("A float: %s (%g)\n", yytext,
                   atof(yytext));
               }
           if|then|begin|end|procedure|function        {
               printf("A keyword: %s\n", yytext);
               }
           {ID}    printf("An identifier: %s\n", yytext);
           "+"|"−"|"*"|"/"        printf("An operator: %s\n", yytext);
           "{"[^}\n]*"}"    /* Eat up one-line comments. */
           [ \t\n]+        /* Eat up white space. */
           .  printf("Unrecognized character: %s\n", yytext);
           %%
           int main(int argc, char *argv[])
           {
               ++argv, −−argc;  /* Skip over program name. */
               if (argc > 0)
                   yyin = fopen(argv[0], "r");
               else
                   yyin = stdin;
               yylex();
           }

RATIONALE         top

       Even though the −c option and references to the C language are
       retained in this description, lex may be generalized to other
       languages, as was done at one time for EFL, the Extended FORTRAN
       Language. Since the lex input specification is essentially language-
       independent, versions of this utility could be written to produce
       Ada, Modula-2, or Pascal code, and there are known historical
       implementations that do so.
       The current description of lex bypasses the issue of dealing with
       internationalized EREs in the lex source code or generated lexical
       analyzer. If it follows the model used by awk (the source code is
       assumed to be presented in the POSIX locale, but input and output are
       in the locale specified by the environment variables), then the
       tables in the lexical analyzer produced by lex would interpret EREs
       specified in the lex source in terms of the environment variables
       specified when lex was executed. The desired effect would be to have
       the lexical analyzer interpret the EREs given in the lex source
       according to the environment specified when the lexical analyzer is
       executed, but this is not possible with the current lex technology.
       The description of octal and hexadecimal-digit escape sequences
       agrees with the ISO C standard use of escape sequences.
       Earlier versions of this standard allowed for implementations with
       bytes other than eight bits, but this has been modified in this
       version.
       There is no detailed output format specification. The observed
       behavior of lex under four different historical implementations was
       that none of these implementations consistently reported the line
       numbers for error and warning messages. Furthermore, there was a
       desire that lex be allowed to output additional diagnostic messages.
       Leaving message formats unspecified avoids these formatting questions
       and problems with internationalization.
       Although the %x specifier for exclusive start conditions is not
       historical practice, it is believed to be a minor change to
       historical implementations and greatly enhances the usability of lex
       programs since it permits an application to obtain the expected
       functionality with fewer statements.
       The %array and %pointer declarations were added as a compromise
       between historical systems.  The System V-based lex copies the
       matched text to a yytext array. The flex program, supported in BSD
       and GNU systems, uses a pointer. In the latter case, significant
       performance improvements are available for some scanners. Most
       historical programs should require no change in porting from one
       system to another because the string being referenced is null-
       terminated in both cases. (The method used by flex in its case is to
       null-terminate the token in place by remembering the character that
       used to come right after the token and replacing it before continuing
       on to the next scan.) Multi-file programs with external references to
       yytext outside the scanner source file should continue to operate on
       their historical systems, but would require one of the new
       declarations to be considered strictly portable.
       The description of EREs avoids unnecessary duplication of ERE details
       because their meanings within a lex ERE are the same as that for the
       ERE in this volume of POSIX.1‐2008.
       The reason for the undefined condition associated with text beginning
       with a <blank> or within "%{" and "%}" delimiter lines appearing in
       the Rules section is historical practice. Both the BSD and System V
       lex copy the indented (or enclosed) input in the Rules section
       (except at the beginning) to unreachable areas of the yylex()
       function (the code is written directly after a break statement). In
       some cases, the System V lex generates an error message or a syntax
       error, depending on the form of indented input.
       The intention in breaking the list of functions into those that may
       appear in lex.yy.c versus those that only appear in libl.a is that
       only those functions in libl.a can be reliably redefined by a
       conforming application.
       The descriptions of standard output and standard error are somewhat
       complicated because historical lex implementations chose to issue
       diagnostic messages to standard output (unless −t was given).
       POSIX.1‐2008 allows this behavior, but leaves an opening for the more
       expected behavior of using standard error for diagnostics.  Also, the
       System V behavior of writing the statistics when any table sizes are
       given is allowed, while BSD-derived systems can avoid it. The
       programmer can always precisely obtain the desired results by using
       either the −t or −n options.
       The OPERANDS section does not mention the use of as a synonym for
       standard input; not all historical implementations support such usage
       for any of the file operands.
       A description of the translation table was deleted from early
       proposals because of its relatively low usage in historical
       applications.
       The change to the definition of the input() function that allows
       buffering of input presents the opportunity for major performance
       gains in some applications.
       The following examples clarify the differences between lex regular
       expressions and regular expressions appearing elsewhere in this
       volume of POSIX.1‐2008. For regular expressions of the form "r/x",
       the string matching r is always returned; confusion may arise when
       the beginning of x matches the trailing portion of r.  For example,
       given the regular expression "a*b/cc" and the input "aaabcc", yytext
       would contain the string "aaab" on this match. But given the regular
       expression "x*/xy" and the input "xxxy", the token xxx, not xx, is
       returned by some implementations because xxx matches "x*".
       In the rule "ab*/bc", the "b*" at the end of r extends r's match into
       the beginning of the trailing context, so the result is unspecified.
       If this rule were "ab/bc", however, the rule matches the text "ab"
       when it is followed by the text "bc".  In this latter case, the
       matching of r cannot extend into the beginning of x, so the result is
       specified.

FUTURE DIRECTIONS         top

       None.

SEE ALSO         top

       c99(1p), ed(1p), yacc(1p)
       The Base Definitions volume of POSIX.1‐2008, Chapter 5, File Format
       Notation, Chapter 8, Environment Variables, Chapter 9, Regular
       Expressions, Section 12.2, Utility Syntax Guidelines

COPYRIGHT         top

       Portions of this text are reprinted and reproduced in electronic form
       from IEEE Std 1003.1, 2013 Edition, Standard for Information
       Technology -- Portable Operating System Interface (POSIX), The Open
       Group Base Specifications Issue 7, Copyright (C) 2013 by the
       Institute of Electrical and Electronics Engineers, Inc and The Open
       Group.  (This is POSIX.1-2008 with the 2013 Technical Corrigendum 1
       applied.) In the event of any discrepancy between this version and
       the original IEEE and The Open Group Standard, the original IEEE and
       The Open Group Standard is the referee document. The original
       Standard can be obtained online at http://www.unix.org/online.html .
       Any typographical or formatting errors that appear in this page are
       most likely to have been introduced during the conversion of the
       source files to man page format. To report such errors, see
       https://www.kernel.org/doc/man-pages/reporting_bugs.html .
IEEE/The Open Group                 2013                             LEX(1P)

Pages that refer to this page: awk(1p)cflow(1p)make(1p)yacc(1p)