GATE Data Science and Artificial Intelligence 2024 | Sample Paper | Question: 6

Question

Which among the following may help to reduce overfitting demonstrated by a model

1. Change the loss function. 
2. Reduce model complexity.
3. Increase the training data.
4. Increase the number of optimization routine steps.

 

• A. $\text{ii and i}$
• B. $\text{ii and iii}$
• C. $\text{i, ii, and iii}$
• D. $\text{i, ii, iii, and iv}$

Answer 1

Ans: Choice B

Option 2 is an obvious one and hence included in every choice 😁

Option 3 Train with more data is one of the techniques to reduce overfitting.

Reference: https://www.ibm.com/topics/overfitting

Comments

i.Change the loss function.

and,

iv. Increase the number of optimization routine step.

doesn’t really reduce the overfitting problem.

---

^ Edited. Thank you

---

Regularization involves changing the loss function and is a way to avoid overfitting. Shouldn’t i) then be an answer?

---

@Aditya1205 In regularization, the cost function is adjusted by adding a penalty term to the sum of losses (obtained from loss function). Loss function is not changed. So, cost function = sum of losses + term that puts penalty on input features with too large weights.

---

Even if you are adjusting a cost function by adding a penalty term then also you are ultimately changing the cost function.

Writing $J(w)_{Least \ square} =  \sum_{i=1}^{n}(y^{(i)} – \hat{y}^{(i)})^2$ to $J(w)_{Ridge} =  \sum_{i=1}^{n}(y^{(i)} – \hat{y}^{(i)})^2+\lambda||w||_2^2$ or $J(w)_{Lasso} =  \sum_{i=1}^{n}(y^{(i)} – \hat{y}^{(i)})^2+\lambda||w||_1$ is still a change of cost function.

---

Answer: C

Statement 1 is also true
Modifying the loss function can indeed help reduce overfitting. For example, using regularization terms in the loss function (like L1 or L2 regularization) can penalize overly complex models

---

here A, B, C will be the correct answer

Answer 2

1. Change the loss function:

   • Choosing a loss function that penalizes the model for overfitting tendencies or introduces regularization terms might guide the model towards better generalization.

2. Reduce model complexity:

   • Simplifying the model architecture, like using fewer layers or parameters, can make it less prone to memorizing the training data and more likely to generalize well to new, unseen data.

3. Increase the training data:

   • More data can provide the model with a broader view of the underlying patterns in the data, making it less likely to overfit specific examples. It's like giving the model a richer experience to learn from.

4. Increase the number of optimization routine steps:

   • increasing the number of optimization steps might make the model more prone to overfitting the training data, especially if the model is already complex. It's generally better to focus on the other strategies mentioned.

So, the effective ways to combat overfitting from your list would be:

• Change the loss function.
• Reduce model complexity.
• Increase the training data.

 

correct answer is C

Answer 3

• A. ii and iii –  Answer

Overfitting means the training error is low but the testing error is very high. That means the variance of the trained model is high but bias of the model is low for the given training data.

To reduce the variance of the model the following steps can be taken – 

1. Reduce the model complexity(Increase regularization).
2. Train on larger dataset.
3. Early stopping the training.
4. Bagging.

Reference – https://www.youtube.com/watch?v=zUJbRO0Wavo&list=PLl8OlHZGYOQ7bkVbuRthEsaLr7bONzbXS&index=19&pp=iAQB

https://www.youtube.com/watch?v=65UJPA10dW8&list=PLl8OlHZGYOQ7bkVbuRthEsaLr7bONzbXS&index=20&pp=iAQB

Answer 4

Answer: C

Statement 1 is also true
Modifying the loss function can indeed help reduce overfitting. For example, using regularization terms in the loss function (like L1 or L2 regularization) can penalize overly complex models