Tokenization in lexical analysis

Question

Consider the following C program:
 

int main (void)
{
    in/*this is an example*/z;
    
    double/*is it an error?*/y;
    
    print( “This is simple” );
    
    return 0;
}

- How many Different tokens are there in the above code.

Comments

please categorize the tokens.

Answer 1

int main ( void )
{
    in z ;
    
    double y ;
    
    print( “This is simple” );
    
    return 0 ;
}

All Colored ones are different Tokens = 16

Types of tokens-  

keywords - int, void , double, return.

Identifiers - main , in, z, y, print.

Strings     - “This is simple”.

Constant  - 0.

Special- symbols / Delimiter / Punctuator /  - ( , ), ; , { , } , comma.

NOTE:

 ( 6 types of  c tokens )

Comments

i have run this on a compiler it takes in and z as seperate tokens ..they will be identifiers

---

I am getting 17 as the answer by counting in the best answer given above.

Can someone help where I am going wrong??

I counted beginning from keyword and my counts are

1. -> 4

2 -> 5

3 -> 1

4 -> 1

5 -> 6

---

the answer is right you added up wrong it's 17 you counted only 16

Answer 2

"int","main","(","void",")"-5 tokens

"{"-1 token

"inz",";"-2 tokens

"double","y",";"-3 tokens

"printf","(","String literal",")",";"-5 tokens

"return","0",";"- 3 tokens

"}"- 1 token

Total 20 tokens

int,void-keyword

"inz","y"-identifier

";"-delimeter

Comments

please mention the type of the tokens also--like delimiter,,identifier etc etc

---

@srestha why u counted inz as one single token??
after removing comments in will be one and z will be one token...

---

ok means it will give lexical error, rt?

---

no it will not given any lexical error...in will be treated as one token and z as another...lexical analyser follows longest match rule...it will absolutely work fine here..

---

@Srestha

It throws a syntax error.

---

but "in" which type of token?

---

tokens are alays be separated by white spaces. Is it contain white space?

---

I think shrestha is right.. But one more confusion if comment is replaced with no space the doubley should be treated as one token isn't??

---

no token need not be seperated by white space like () = 2 token but no space , white space need when we define by own i.e. not known to computer

---

Anu.. Yes I am getting your point but the example you are giving is two different separators which is considered to be two tokens definitely.. But what about above example.. Plz Cross check it would be very  helpful.. If any reference From book or anywhere..

---

21 token but distinct are 16 as selected answer explain.