The Gateway to Computer Science Excellence
0 votes
204 views
How many tokens in this
a>>=1;
and a! , Will >>= and a! be treated as a single token ?
in Compiler Design by Junior (665 points) | 204 views
0
0
5 ??

>> Take it as a single token

a! === Two tokens
+2
0

@Satbir

thanks for correcting me

3 Answers

+10 votes
Best answer

A group of lexemes are declared as tokens if they match the pattern of a token.(tokens can be identifiers , numbers etc)

comments , white space and preprocessor directives are not declared as tokens in C.

a token is declared after seeing the next input string. why ?

Since “&&” is also a token, so we can’t declare “&” as a token unless we see next input string.

Since “==” is also a token, so we can’t declare “=” as a token unless we see next input string.

%=” is also a token (for ex: a% = y; equivalent to a = a%y;) so “%” can’t be declare as token without looking at next symbol.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------

a>>=1;

a

we see 'a' first and make it as token (variable) but we don't declare it as token and see the next symbol first before declaring it as token. the next lexeme is '>' and since 'a>' does not match any pattern we declare 'a' as a token.

>>=

 we see '>' (greater than) then we see next symbol '>' and so '>>' (right shift) it becomes a token but we don't declare it as token and see the next symbol first before declaring it as token

then we see '='  and check '>>='  ( this is shorthand operator ) and make it  as a token but we don't declare it as token and see the next symbol first before declaring it as token.

then we see '1' so  '>>=1' is not matching the pattern of any token so we finally declare '>>=' as a token.

1

we see '1' and make it as token (digit) but we don't declare it as token and see the next symbol first before declaring it as token. the next lexeme is ';' and since '1;' does not match any pattern we declare '1' a as a token.

';' is a token.

so answer is 4.

-------------------------------------------------------------------------------------------------------------------------------------------------------------------

similarly a! are two tokens.

by Boss (21.8k points)
selected by
0

@Satbir

awesome (Y)

0 votes
Here's how you break down the counting of tokens:
"a",  ">>" , "=" , "1", ";" totally 5 tokens.

"a", "!" totally 2 tokens.

PS: If your query was resolved, you might consider upvoting!
by Active (1.7k points)
0 votes
5 and 2
by (141 points)

Related questions

Quick search syntax
tags tag:apple
author user:martin
title title:apple
content content:apple
exclude -tag:apple
force match +apple
views views:100
score score:10
answers answers:2
is accepted isaccepted:true
is closed isclosed:true
50,645 questions
56,601 answers
195,850 comments
102,208 users