Lexer

class Lexer {

OutBuffer stringbuffer;

Loc scanloc;

const(char)* base;

const(char)* end;

const(char)* p;

const(char)* line;

Token token;

bool doDocComment;

bool anyToken;

bool commentToken;

bool errors;

this(const(char)* filename, const(char)* base, size_t begoffset, size_t endoffset, bool doDocComment, bool commentToken);

TOK peekNext();

TOK peekNext2();

void scan(Token* t);

Token* peekPastParen(Token* tk);

uint escapeSequence();

TOK wysiwygStringConstant(Token* t, int tc);

TOK hexStringConstant(Token* t);

TOK delimitedStringConstant(Token* t);

TOK tokenStringConstant(Token* t);

TOK escapeStringConstant(Token* t, int wide);

TOK charConstant(Token* t);

void stringPostfix(Token* t);

TOK number(Token* t);

TOK inreal(Token* t);

void poundLine();

uint decodeUTF();

void getDocComment(Token* t, uint lineComment);

static const(char)* combineComments(const(char)* c1, const(char)* c2);

}

Constructors

this this(const(char)* filename, const(char)* base, size_t begoffset, size_t endoffset, bool doDocComment, bool commentToken): Creates a Lexer.

Members

Functions

charConstant TOK charConstant(Token* t)
decodeUTF uint decodeUTF(): Decode UTF character. Issue error messages for invalid sequences. Return decoded character, advance p to last character in UTF sequence.
delimitedStringConstant TOK delimitedStringConstant(Token* t): Lex delimited strings: q"(foo(xxx))" // "foo(xxx)" q"[foo(]" // "foo(" q"/foo]/" // "foo]" q"HERE foo HERE" // "foo\n" Input: p is on the "
escapeSequence uint escapeSequence(): Parse escape sequence.
escapeStringConstant TOK escapeStringConstant(Token* t, int wide)
getDocComment void getDocComment(Token* t, uint lineComment): Parse doc comment embedded between t->ptr and p. Remove trailing blanks and tabs from lines. Replace all newlines with \n. Remove leading comment character from each line. Decide if it's a lineComment or a blockComment. Append to previous one for this token.
hexStringConstant TOK hexStringConstant(Token* t): Lex hex strings: x"0A ae 34FE BD"
inreal TOK inreal(Token* t): Read in characters, converting them to real.
number TOK number(Token* t): Read in a number. If it's an integer, store it in tok.TKutok.Vlong. integers can be decimal, octal or hex Handle the suffixes U, UL, LU, L, etc. If it's double, store it in tok.TKutok.Vdouble.
peekNext TOK peekNext(): Look ahead at next token's value.
peekNext2 TOK peekNext2(): Look 2 tokens ahead at value.
peekPastParen Token* peekPastParen(Token* tk): tk is on the opening (. Look ahead and return token that is past the closing ).
poundLine void poundLine(): parse: #line linnum filespec also allow __LINE__ for linnum, and __FILE__ for filespec
scan void scan(Token* t): Turn next token in buffer into a token.
stringPostfix void stringPostfix(Token* t): Get postfix of string literal.
tokenStringConstant TOK tokenStringConstant(Token* t): Lex delimited strings: q{ foo(xxx) } // " foo(xxx) " q{foo(} // "foo(" q{{foo}"}"} // "{foo}"}"" Input: p is on the q
wysiwygStringConstant TOK wysiwygStringConstant(Token* t, int tc)

Static functions

combineComments const(char)* combineComments(const(char)* c1, const(char)* c2): Combine two document comments into one, separated by a newline.

Meta

Source

See Implementation

ddmd lexer classes

Lexer

variables

cmtable