Lexical analyser, Tokens

Question

When do we count ++(post increment or pre increment) as 1 or 2 token ?

Comments

All operators are tokens, hence ++ is taken as a single token.

Correct if wrong.

---

This will be consider as single token, because compiler apply greedy approach for performing tokenization, it will try to include more character in token if possible.

https://gcc.gnu.org/onlinedocs/cpp/Tokenization.html

---

@ Shubhanshu Thanks it helps :) /\

Answer 1

Preprocessor follows greedy approach to tokenize,  This can be done in TWO steps

1 => Separated by space (A+ ++B) => Have 4 tokens(Space is not considered as token), here tokens are A, +, ++, B(In order of expression). If any operator doesn't have any space ,

then ,

2 => Greedy approach (A+++B) for longest possible valid operator in the Grammer => Like we have '++' > '+' so here the tokens are A,++, +, B(In order of expression).

Comments

a will be 1 token

+= will be another

b be will another

so total will be 3 tokens.

---

Pre processor or Lexical Analyzer?

---

4 tokens a,+=,b,;.

 

Answer 2

Always when we count the token complier always see next symbol

Eg-

++= 1token

a+++++b =5tokens

&&= 1token

Answer 3

I think 2 tokens because count is an identifier and ++ is an operator.