2024 Final Examination (27th Batch)

Notation & Assumptions: For all of the below questions, assume, where applicable, the features are $x_1, x_2, \dots, x_n$ and the examples are $x^{(1)}, x^{(2)}, \dots, x^{(m)}$, so $x_j$ and $x^{(i)}$ are $n$ and $m$-dimensional vectors, respectively. Accordingly, $x_j^{(i)}$ is the feature value of the $i$-th example on the $j$-th feature. If a supervised setting is considered, $y^{(i)}$ is the ground truth label for example $x^{(i)}$.

---

Question 1

1 (a)

In regularized version linear regression, write down the cost functions for:

1. Training set
2. Validation set

Justify your cost functions. [2+2]

Regularization and Cost Functions · Linear Regression with Multiple Variables

---

1 (b)

Consider a learning algorithm whose training cost function is convex and that uses gradient descent for optimizing the training cost; assume that a proper learning rate is being used.

Draw a hypothetical learning curve for this algorithm where both training and validation error curves are in the y-axis and the number of iterations of the optimization in x-axis.

• Will the validation set's error curve (as measured by its cost function) be always convex? Why or why not?
• What may happen if the learning rate is not appropriate, i.e., too large and what if too small? [4+2]

Advice for Applying Machine Learning · Gradient Descent Fundamentals

---

1 (c)

Suppose you are working with a dataset of a regression task having three features $x_1, x_2, x_3$ where you know that the data is not linear, and suppose someone told you that the pattern contains monomials of degree 2. So naturally you will need to fit a polynomial regression model here.

However, you are lazy today, so you want to use a linear regression algorithm already implemented in your code, you do not want to implement a polynomial regression for today. What can you do with your dataset to make it eligible for applying a linear regression algorithm? [4]

Polynomial Feature Expansion

---

Question 2

2 (a)

Suppose you are evaluating two different binary classification models, namely A and B, for a business application. The application requires the models to have the following criteria:

1. The recall rate should be at least 80%
2. False-positive should be 10% or less (False-Positive Rate $\text{FPR} = \frac{\text{FP}}{\text{TN} + \text{FP}} \le 10\%$)

For each of the models, you calculate the confusion matrix as follows:

Which of the above models satisfies the requirements? [4]

Machine Learning System Design & Metrics

---

2 (b)

Calculate the accuracy, specificity, and sensitivity of model A of question 2(a). [4]

---

2 (c)

You are tasked with evaluating the performance of two different machine learning classifiers for a binary classification problem. Both classifiers provide probability scores for each instance, and you are asked to plot the ROC curve and calculate the AUC for each model.

1. Explain the concept of the ROC curve. How does it illustrate the performance of a classifier, and what do the axes represent?
2. Describe the significance of the AUC score. What does a higher AUC indicate about a model's performance?
3. If both classifiers have similar AUC values, but Classifier A has a much steeper curve on the ROC plot, what can be inferred about the behavior of the two models to different thresholds? [6]

---

Question 3

3 (a)

Explain two activation functions of neural networks. Why should a neural network always use non-linear activation functions? [4+2]

Neural Networks Representation · Neural Networks Learning

---

3 (b)

As compared to logistic regression, why is it difficult to use gradient descent in a neural network? That is, why do we need a backpropagation algorithm? [3]

---

3 (c)

Suppose you have a 5-class classification problem that you want to solve with a neural network.

• How many output nodes will you have, and why?
• If you use softmax regression activation function, write down and explain the output formulae. [1+4]

---

Question 4

4 (a)

What do you understand about large margin classifiers? Explain how support vector machines (SVMs) achieve this large margin; you should do this just by deriving SVM's primal optimization problem using the concept of functional margin. [2+4]

Support Vector Machines

---

4 (b)

The dual formulation of SVM optimization problem is as follows:

$\max_{\alpha} \sum_{i=1}^m \alpha_i - \frac{1}{2}\sum_{i=1}^m\sum_{j=1}^m \alpha_i \alpha_j y^{(i)} y^{(j)} x^{(i)} \cdot x^{(j)}$ $\text{Subject to } \alpha_i \ge 0, \quad \sum_{i=1}^m \alpha_i y^{(i)} = 0$

where $\alpha_i$ are Lagrange multipliers. How does this formulation pave the way to use the kernel trick? [4]

---

4 (c)

“The kernel trick is a general technique, and it can be applied to any learning algorithm other than SVM” — do you agree or disagree with this statement, and why? [4]

---

Question 5

5 (a)

Suppose you are using a decision tree for a classification problem without pruning. Comment on its training error and validation error. [3]

---

5 (b)

In a decision tree for multiclass classification task, state three choices of measuring the node impurity/uncertainty for the class distribution at that node.

Justify the rationale behind each measure, i.e., how each of them captures the impurity/uncertainty. For binary classification task, plot the patterns of these three metrics on the same graph. [6+2]

---

5 (c)

Why assessing human-level performance is important in practical machine learning projects? How can we define it for a medical diagnosis task? [3]

Machine Learning Diagnostics & Bias-Variance

---

Question 6

6 (a)

A company wants to segment its customers based on their annual income ($x_1$) and spending score ($x_2$) into $k=2$ clusters using the K-means clustering algorithm. The dataset contains the following points:

$P_1(15, 39), \quad P_2(16, 81), \quad P_3(17, 6), \quad P_4(18, 77), \quad P_5(19, 40), \quad P_6(20, 76)$

Write the whole mathematical process until completion of Iteration 1. [5]

Clustering & K-Means

---

6 (b)

The anomaly detection problem can be solved using a univariate gaussian model. Describe the training and prediction phases of this technique. Mention the benefits and drawbacks. [4+2]

Anomaly Detection

---

6 (c)

Anomaly detection is performed usually with unlabeled dataset. However, if a few labeled data are available, how can you best use this labeled data for the anomaly detection task? [3]

---

Question 7

7 (a)

How is a Gaussian Mixture Model (GMM) different from a Gaussian Discriminant Analysis (GDA) model? Briefly explain. [4]

---

7 (b)

In what scenarios do we use Expectation-Maximization (EM) algorithm? State its two main steps. [1+2]

---

7 (c)

What are the two main goals of Principal Component Analysis (PCA)? Explain its main steps. [2+5]

Dimensionality Reduction & PCA