# Self doubt

388 views
How many tokens in this
a>>=1;
and a! , Will >>= and a! be treated as a single token ?
0
0
5 ??

>> Take it as a single token

a! === Two tokens
0

@Satbir

thanks for correcting me

A group of lexemes are declared as tokens if they match the pattern of a token.(tokens can be identifiers , numbers etc)

comments , white space and preprocessor directives are not declared as tokens in C.

a token is declared after seeing the next input string. why ?

Since “&&” is also a token, so we can’t declare “&” as a token unless we see next input string.

Since “==” is also a token, so we can’t declare “=” as a token unless we see next input string.

%=” is also a token (for ex: a% = y; equivalent to a = a%y;) so “%” can’t be declare as token without looking at next symbol.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------

a>>=1;

a

we see 'a' first and make it as token (variable) but we don't declare it as token and see the next symbol first before declaring it as token. the next lexeme is '>' and since 'a>' does not match any pattern we declare 'a' as a token.

>>=

we see '>' (greater than) then we see next symbol '>' and so '>>' (right shift) it becomes a token but we don't declare it as token and see the next symbol first before declaring it as token

then we see '='  and check '>>='  ( this is shorthand operator ) and make it  as a token but we don't declare it as token and see the next symbol first before declaring it as token.

then we see '1' so  '>>=1' is not matching the pattern of any token so we finally declare '>>=' as a token.

1

we see '1' and make it as token (digit) but we don't declare it as token and see the next symbol first before declaring it as token. the next lexeme is ';' and since '1;' does not match any pattern we declare '1' a as a token.

';' is a token.

-------------------------------------------------------------------------------------------------------------------------------------------------------------------

similarly a! are two tokens.

selected by
0

@Satbir

awesome (Y)

Here's how you break down the counting of tokens:
"a",  ">>" , "=" , "1", ";" totally 5 tokens.

"a", "!" totally 2 tokens.

PS: If your query was resolved, you might consider upvoting!
5 and 2
Lexical analyzer is like DFA able to identify compound operators.

a>>=1; here >>= is considered as a single token. hence in this line #tokens= 4

a! -> here a and '!' are individual tokens.

## Related questions

1
461 views
The above diagram is Transition Diagrams for identifiers. As we can see that the identifier is said to be accepted if it starts with a letter and ends with a valid delimiter, which includes blank symbol, arithmetic, logical operator, left parenthesis, right parenthesis, +, ... ends with a delimiter and + is a valid delimiter and the error in declaration will not be detected at this stage...
I came across few Compiler Design Doubts, Please provide your cent. $Q_1 :$ Can lexical analyser detect some/any type of errors? I think yes, because while scanning to identify tokens, it may be the cases that a string pattern doesn't match with any keyword (or) entry in ... (Left linear) $\color{navy}{A \rightarrow aA}$(Right Linear). What can be said about $\color{navy}{A \rightarrow aAb}$