search
Log In
0 votes
388 views
How many tokens in this
a>>=1;
and a! , Will >>= and a! be treated as a single token ?
in Compiler Design 388 views
0
0
5 ??

>> Take it as a single token

a! === Two tokens
0

@Satbir

thanks for correcting me

4 Answers

16 votes
 
Best answer

A group of lexemes are declared as tokens if they match the pattern of a token.(tokens can be identifiers , numbers etc)

comments , white space and preprocessor directives are not declared as tokens in C.

a token is declared after seeing the next input string. why ?

Since “&&” is also a token, so we can’t declare “&” as a token unless we see next input string.

Since “==” is also a token, so we can’t declare “=” as a token unless we see next input string.

%=” is also a token (for ex: a% = y; equivalent to a = a%y;) so “%” can’t be declare as token without looking at next symbol.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------

a>>=1;

a

we see 'a' first and make it as token (variable) but we don't declare it as token and see the next symbol first before declaring it as token. the next lexeme is '>' and since 'a>' does not match any pattern we declare 'a' as a token.

>>=

 we see '>' (greater than) then we see next symbol '>' and so '>>' (right shift) it becomes a token but we don't declare it as token and see the next symbol first before declaring it as token

then we see '='  and check '>>='  ( this is shorthand operator ) and make it  as a token but we don't declare it as token and see the next symbol first before declaring it as token.

then we see '1' so  '>>=1' is not matching the pattern of any token so we finally declare '>>=' as a token.

1

we see '1' and make it as token (digit) but we don't declare it as token and see the next symbol first before declaring it as token. the next lexeme is ';' and since '1;' does not match any pattern we declare '1' a as a token.

';' is a token.

so answer is 4.

-------------------------------------------------------------------------------------------------------------------------------------------------------------------

similarly a! are two tokens.


selected by
0

@Satbir

awesome (Y)

0 votes
Here's how you break down the counting of tokens:
"a",  ">>" , "=" , "1", ";" totally 5 tokens.

"a", "!" totally 2 tokens.

PS: If your query was resolved, you might consider upvoting!
0 votes
5 and 2
0 votes
Lexical analyzer is like DFA able to identify compound operators.

a>>=1; here >>= is considered as a single token. hence in this line #tokens= 4

a! -> here a and '!' are individual tokens.

Related questions

2 votes
1 answer
1
461 views
The above diagram is Transition Diagrams for identifiers. As we can see that the identifier is said to be accepted if it starts with a letter and ends with a valid delimiter, which includes blank symbol, arithmetic, logical operator, left parenthesis, right parenthesis, +, ... ends with a delimiter and + is a valid delimiter and the error in declaration will not be detected at this stage...
asked Jun 9, 2019 in Compiler Design Hirak 461 views
2 votes
1 answer
2
471 views
I came across few Compiler Design Doubts, Please provide your cent. $Q_1 :$ Can lexical analyser detect some/any type of errors? I think yes, because while scanning to identify tokens, it may be the cases that a string pattern doesn't match with any keyword (or) entry in ... (Left linear) $\color{navy}{A \rightarrow aA}$(Right Linear). What can be said about $\color{navy}{A \rightarrow aAb}$
asked Nov 1, 2016 in Compiler Design mcjoshi 471 views
5 votes
1 answer
3
1.2k views
MISSPELLING OF KEYWORDS CAUSE WHICH TYPE OF ERROR- SYNTAX ERROR OR LEXICAL ERROR?? printf("%d") without any integer argument will cause which type of error??
asked Aug 27, 2017 in Compiler Design sushmita 1.2k views
0 votes
4 answers
4
245 views
Consider the following statements related to compiler construction: Lexical Analysis is specified by context-free grammars and implemented by pushdown automata. Syntax Analysis is specified by regular expressions and implemented by finite-state machine. Which of the above statement(s) is/are correct? Only I Only II Both I and II Neither I nor II
asked Mar 24 in Compiler Design jothee 245 views
...