4,209 views

2 Answers

Best answer
4 votes
4 votes

While doing tokenization we look for longer pattern that is available in language.
For Example i = k++;
Like when we see + there are chances that next term is also +, and ++ is allowed in language, so if we end up getting ++ we consider it as single token.

Now your question  ' should we consider the ternary operator "? :" as 1 token or two separate tokens "?" and ":" '
We consider ? and : as two different token.

For example a ? b : c;
when we look how it work, it tells us "b if a is true, else c". You can observe here, that ? and : act as two independent entity. Parser will read it as, [a]{what is the condition} [?]{is it true or false} [b]{if true do this} [:]{And} c{if false do this}. So you can see here they both have their different use. Even we write like this "a ? : c;" parser will think it as "a ? <do nothing> : c; "

If "?:" combinedly define something meaningful, then we consider it as single token. But as we have seen they both have a different role when we use it in some expression, so we consider as two different token. 

[result] [=] [x] [>] [y] [?] ["x is greater than y"] [:] ["x is less than or equal to y"][;]

So total token here is 10.

selected by
1 votes
1 votes
Unlike Fortran, in C tokens are not context-dependent, for example when ? mark occurs lexical analyzer doesn't know, what is going to occur next. ? matches a pattern of operator, hence there will be an entry in symbol table for it.

The conclusion is ? and : will be recorded as two tokens in symbol table.

Related questions

2 votes
2 votes
2 answers
1
Na462 asked Nov 7, 2018
3,600 views
0 votes
0 votes
1 answer
2
syncronizing asked Sep 21, 2018
5,276 views
Question:Find the number of tokens in the following C code using lexical analyzer of the compiler.
0 votes
0 votes
2 answers
3
saumya mishra asked Jun 12, 2018
2,173 views
In this question we will take n-=n1 as n=n-n1 as 5 tokens or n-=n1 as 4 tokens ????
3 votes
3 votes
4 answers
4
Deepak Singh 7 asked Mar 15, 2018
1,229 views
Number of tokens in $\text{int a[5];}$