GO Classes DPP | ML |DECISION TREE

Answer 1

The information gain because of this split is equal to the decrease in impurity. Here, $\left|L_1\right|$ and $\left|L_2\right|$ denote the cardinality of the leaves. $N$ is the total number of points before the split at node $Q$.

$$
I G=E(Q)-\left(\frac{\left|L_1\right|}{N} E\left(L_1\right)+\frac{\left|L_2\right|}{N} E\left(L_2\right)\right)
$$

For this problem, the variables take on the following values:

• $N=200$, there are 200 points in the node $Q$.
   
• $\left|L_1\right|=80$, there are 80 points in the node $L_1$.
   
• $\left|L_2\right|=120$, there are 120 points in the node $L_2$.

To calculate the entropy of the three nodes, we need the proportion of points that belong to class-1 in each of the three nodes. Let us call them $p$ for node $Q, p_1$ for node $L_1$ and $p_2$ for node $L_2$ :

• $p=\frac{100}{100+100}=\frac{1}{2}$
   
• $p_1=\frac{30}{30+50}=\frac{3}{8}$
   
• $p_2=\frac{70}{70+50}=\frac{7}{12}$

Now, we have all the data that we need to compute $E(Q), E\left(L_1\right)$ and $E\left(L_2\right)$ :

• $E(Q)=-\frac{1}{2} \log \left(\frac{1}{2}\right)-\frac{1}{2} \log \left(\frac{1}{2}\right)=1$
   
• $E\left(L_1\right)=-\frac{3}{8} \log \left(\frac{3}{8}\right)-\frac{5}{8} \log \left(\frac{5}{8}\right) \approx 0.954$
   
• $E\left(L_2\right)=-\frac{7}{12} \log \left(\frac{7}{12}\right)-\frac{5}{12} \log \left(\frac{5}{12}\right) \approx 0.980$

Now, we have all the values to compute the information gain:

$$
I G=1-\left(\frac{80}{200} 0.954+\frac{120}{200} 0.980\right) \approx 0.030
$$

Comments

The entropy calculated here is wrong I guess. for example entropy for l1 is coming 0.287 and not 0.954

---

If we use log to base 2 then we get correct entropy for L1 which is 0.9543.

---

Yup , thanks now I got it !!