Tokens, Lexeme and Pattern

Question

There are 40 tokens in this question.

But I have doubt that what should be the wording of the question 

"The number of LEXEMES generated by the Lexical Analyzer in the above program is _______ "

OR

"The number of TOKENS generated by the Lexical Analyzer in the above program is _______ "

In most of the question, it is asked about calculating numbers of tokens. Is the answer remains same if they asked about the number of lexemes instead of tokens?

I am confused between these two terms, and I know there are 6 different types of tokens we have namely:-

Keyword, Operators, Identifiers, Constant, Punctuation Symbol and String Constants.

How token and lexeme are different from each other?

Comments

Thanks guys !!!

---

https://gateoverflow.in/297978/self-doubt

---

total there are 41 tokens.

Important points to consider- 

1. in the code 01 is to be taken together
2. ++ is to be taken together
3. >>= is to be taken together

so when these points are considered, there will be 41 tokens.

Answer 1

It should be 40..Key things to note here :

a) Lexical analyser is a DFA .So it has capability to identify compound operators ; so they will be considered as 1 lexeme only under the token class "operator"..

b) Numbers are treated as  constants irrespective of the base used whether it is octal or decimal..

c) compiler follows greedy approach and will try to make longest pattern available. like x++ will be 2 token instead of x , + ,+ (3)

So here "01" , "!=" and ">>=" will be treated as 1 token only..

Hence the correct answer should be 40..

Ref : https://en.wikipedia.org/wiki/Operators_in_C_and_C%2B%2B#Compound_assignment_operators

Comments

Yes

---

Does -= is also considered as one token ??

---

@shubham yes "-=" is one token

Answer 2

unsigned int counted once as Compiler generates one token for that, 01 counted once as it is constant. As >> and =, they are different operators so counted twice.

Comments

Is there any way we can count these tokens programatically?

---

@rahul sharma 5 find the list of all keywords and write your own scanner in Lex :D something like this http://ideone.com/nfqZU5 (wont run online)

---

Like that ! and = are different operators so even should that be counted separately?

Answer 3

i hope you will get it. 

Answer 4

List of tokens final answer is 40

1. int

2. foo

3. (

4. unsigned

5. int

6. n

7. )

8.{

9. int

10. c

11. ,

12. x

13. =

14. 0

15. ;

16 while

17 (

18 n

19 != (Greedy method mentioned by Habib Sir compiler combines operators and considers it as one token)

20 0

21 )

22 {

23 if

24  (

25 n

26 &

27 01 Same greedy approach

28 )

29 x

30 ++

31 ;

32 n

33 >>= Greedy Approach

34 1

35 ;

36 }

37 return

38 c

39 ;

40 }

Hope this rests all the confusion