But here I mean by matrix multiplication of two matrices both containing the non-negative integers. So, result after binary operation has to be a matrix containing non-negative integer.(closure satisfied)

and matrix multiplication we already know associative and not commutative..

The most important associative operation that's not commutative is function composition.

If you have two functions f and g, their composition, usually denoted f∘g, is defined by

(f∘g)(x)=f(g(x)).
It is associative, (f∘g)∘h=f∘(g∘h),

but it's usually not commutative. f∘g is usually not equal to g∘f.

For our case suppose $\forall$x $\in$ N of non-negative integers, if f(x)=x^{2} and g(x)=x+1, then (f∘g)(x)=(x+1)^{2} while (g∘f)(x)=x^{2}+1, and they're different functions.

"Composition" operation can only be the answer when set consists of functions (on suitably defined set of functions). For this question, this won't be a viable answer as the given base set is set of all natural numbers.

this is fine as an example of an operation that is associative but not commutative. But this does not satisfy what the question demands. Suppose $a,b\in \mathbb{N}$ and $\star$ is the binary operation. You need to define something for: $a\star b$. Using this example of function, how to define $a\star b$? Your example of $f \circ g$ takes only one parameter as input from $\mathbb{N}$ set, but $a\star b$ requires us to take two values $a$ and $b$.