212 views
1 1 vote
Consider the function $f: \mathbb{R}^d \mapsto \mathbb{R}^d$ whose components are given by

$$
f_i(x)=\frac{x_i^2}{\sum_{k=1}^d x_k^2}
$$

for $i=1, \ldots, d$ and $x \in \mathbb{R}^d$.

We can see that $0 \leq f_i \leq 1$ for each $i=1, \ldots, d$ and $\sum_{i=1}^d f_i=1$. Thus, $f$ normalizes the input vector $x$ to a probability distribution, just like Softmax does! That said, what is one way in which our function $f$ differs from the Softmax function?

1 Answer

0 0 votes
Let $\sigma: \mathbb{R}^d \mapsto \mathbb{R}^d$ be the softmax function. We can see that softmax preserves order, i.e., $x_j>x_i \Leftrightarrow \sigma_j(x)>\sigma_i(x)$ since $\exp (x)$ is monotonically increasing. On the other hand, our new function $f$ is only monotonic in magnitude, i.e., $|x|_j>|x|_i \Leftrightarrow f_j(x)>f_i(x)$ (it's possible for $x_j<x_i$ but $\left.f_j(x)>f_i(x)\right)$. This is because $x^2$ is an even function that is monotonically decreasing for $x<0$ and (symmetrically) monotonically increasing for $x>0$.
Position:
Show:

Related questions

1 1 vote
0 0 answers
238
238 views
GO Classes asked Mar 18, 2025
238 views
One of the objectives of cross-entropy loss is to both increase the probability of the true class and decrease the probabilities of the other classes. The formula for cro...
1 1 vote
1 1 answer
209
209 views
GO Classes asked Mar 18, 2025
209 views
Consider the following image data point, where the model's predicted probabilities for the classes (Dog, Cat, Mountain) are:\begin{array}{ll}\text { Class } & \text { Pro...
1 1 vote
0 0 answers
195
195 views
GO Classes asked Mar 18, 2025
195 views
A binary classification problem could be solved with the two approaches described below:Approach 1 : Simple Logistic Regression (one neuron)Your output will be $\hat y = ...
1 1 vote
0 0 answers
141
141 views
GO Classes asked Mar 18, 2025
141 views
Given input data $x \in \mathbb{R}^3$ , how many parameters do the following models have? Assume that linear components of all models have bias terms.Binary logistic regr...