Tokens vs Lexeme

Question

What is the difference between lexeme and tokens?

Answer 1

Lexeme pg. 111

A lexeme is a sequence of characters in the source program that matches the pattern for a token and is identified by the lexical analyzer as an instance of that token.

Token pg. 111

A token is a pair consisting of a token name and an optional attribute value. The token name is an abstract symbol representing a kind of lexical unit, e.g., a particular keyword, or sequence of input characters denoting an identifier. The token names are the input symbols that the parser processes.

Pattern pg. 111

A pattern is a description of the form that the lexemes of a token may take. In the case of a keyword as a token, the pattern is just the sequence of characters that form the keyword. For identifiers and some other tokens, the pattern is more complex structure that is matched by many strings.

Source: https://stackoverflow.com/questions/14954721/what-is-the-difference-between-a-token-and-a-lexeme

Comments

yes

and pattern is just the rules

So, pattern is not in phase of compiler

" A lexeme is a sequence of characters in the source program that is matched by the pattern for a token. "

this line clears all

http://cnuinfotech-cd.blogspot.in/2012/06/tokens-patterns-and-lexemes.html

another good link https://www.tutorialspoint.com/compiler_design/compiler_design_lexical_analysis.htm

---

Every time number of token would be same as number of lexem right?

---

@smsubham

why in the last mapping it is <id,3>?

didn't get it.