Let
$
f(x)=
\begin{bmatrix}
\cos x & -\sin x & 0\\
\sin x & \cos x & 0\\
0 & 0 & 1
\end{bmatrix}.
$
Then
$(f(x))^2=f(x)\,f(x)
=
\begin{bmatrix}
\cos x & -\sin x & 0\\
\sin x & \cos x & 0\\
0 & 0 & 1
\end{bmatrix}
\begin{bmatrix}
\cos x & -\sin x & 0\\
\sin x & \cos x & 0\\
0 & 0 & 1
\end{bmatrix}.
$
Multiplying,
$
(f(x))^2=
\begin{bmatrix}
\cos^2 x-\sin^2 x & -2\sin x\cos x & 0\\
2\sin x\cos x & \cos^2 x-\sin^2 x & 0\\
0 & 0 & 1
\end{bmatrix}.
$
Using, $\cos 2x=\cos^2 x-\sin^2 x \quad \text{and} \quad \sin 2x=2\sin x\cos x,$
we get
$
(f(x))^2=
\begin{bmatrix}
\cos 2x & -\sin 2x & 0\\
\sin 2x & \cos 2x & 0\\
0 & 0 & 1
\end{bmatrix}
=f(2x).
$
Therefore, $\boxed{(f(x))^2=f(2x)}$
So the correct option is B.