1,693 views
2 votes
2 votes
How many tokens in this
a>>=1;
and a! , Will >>= and a! be treated as a single token ?

3 Answers

Best answer
21 votes
21 votes

A group of lexemes are declared as tokens if they match the pattern of a token.(tokens can be identifiers , numbers etc)

comments , white space and preprocessor directives are not declared as tokens in C.

a token is declared after seeing the next input string. why ?

Since “&&” is also a token, so we can’t declare “&” as a token unless we see next input string.

Since “==” is also a token, so we can’t declare “=” as a token unless we see next input string.

%=” is also a token (for ex: a% = y; equivalent to a = a%y;) so “%” can’t be declare as token without looking at next symbol.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------

a>>=1;

a

we see 'a' first and make it as token (variable) but we don't declare it as token and see the next symbol first before declaring it as token. the next lexeme is '>' and since 'a>' does not match any pattern we declare 'a' as a token.

>>=

 we see '>' (greater than) then we see next symbol '>' and so '>>' (right shift) it becomes a token but we don't declare it as token and see the next symbol first before declaring it as token

then we see '='  and check '>>='  ( this is shorthand operator ) and make it  as a token but we don't declare it as token and see the next symbol first before declaring it as token.

then we see '1' so  '>>=1' is not matching the pattern of any token so we finally declare '>>=' as a token.

1

we see '1' and make it as token (digit) but we don't declare it as token and see the next symbol first before declaring it as token. the next lexeme is ';' and since '1;' does not match any pattern we declare '1' a as a token.

';' is a token.

so answer is 4.

-------------------------------------------------------------------------------------------------------------------------------------------------------------------

similarly a! are two tokens.

selected by
0 votes
0 votes
Here's how you break down the counting of tokens:
"a",  ">>" , "=" , "1", ";" totally 5 tokens.

"a", "!" totally 2 tokens.

PS: If your query was resolved, you might consider upvoting!
0 votes
0 votes
Lexical analyzer is like DFA able to identify compound operators.

a>>=1; here >>= is considered as a single token. hence in this line #tokens= 4

a! -> here a and '!' are individual tokens.

Related questions

0 votes
0 votes
1 answer
2
Vipin Rai asked Dec 12, 2018
565 views
int main(){ int a,b; a = 10; b = 11; printf(“%d %d”, a++,b );}The number of tokens is
0 votes
0 votes
2 answers
4
Jithin Jayan asked Oct 16, 2016
658 views
Which Phase(s) of compiler produce error(s) in the following program?#include<stdio.h>int main(){ nit a b ; a = 10; b = 20; printf("%d%d",a,b); ret...