4,009 views
2 votes
2 votes

which one of the following string can definitely said to be a token without looking at the next input:

  1. +(++,+=)
  2. return(return a)
  3. *(*=)
  4. =(==)
  5. ++ , ( ) ‘ ;    option e is: plusplus, comma, bracket open, bracket close, single quote ;

2 Answers

3 votes
3 votes
For option a, lexical analyzer will see the next character after + , because it will choose the longest possible group as a lexeme. If after + no valid character is seen then only the lexical analyzer will generate token for + operator. Thus option a is not the answer.

For option b, on seeing the 'return' character sequence, lexical analyzer will generate token i.e. keyword for lexeme return only if there is no character except a delimiter, otherwise if the next character follows the pattern of identifier, then token id will be generated instead of keyword. Thus option b is no answer.

Similar explanation for option c and option d.

For option e, lexical analyzer, on seeing first + operator , will see the next character which is also + operator. After seeing the second + operator, lexical analyzer will group the ++ operator into a lexeme without seeing the next character. Because after ++ operator , there is no character which will together form a single lexeme. Thus lexical analyzer will generate token for lexeme ++(increment operator).

Why token is generated for ++ and not for +?

This is because lexical analyzer generated a token for longest sequence that can form a token.

Similar explanation for , ; ( ) ' operators in option e. Thus option e is the answer.
0 votes
0 votes
Why are you confusing yourself with hypothetical questions?

 

The longest lexeme needs to be matched by the lexer. You need to scan as much as possible, there is no question of not knowing the next character.

Also, do mention which lexical analyser you are referrin to: is it C or C++ or Java or Python or your own. If its your own, do provide all the regex you are using, only then one can answer such hypothetical question. Lets not confuse ourselves in this simple topic, please.

Related questions

1 votes
1 votes
1 answer
1
ankit_thawal asked Feb 1, 2018
714 views
1. macros are not considered as tokens in tokenizing.2.#include considered as single token2.stdio.h considered as single token. Are the above statements true?
1 votes
1 votes
1 answer
2
0 votes
0 votes
1 answer
3
jeewan0011 asked Aug 22, 2016
455 views
What is token and how use it?