$abc\neq0$
The product of diagonal elements in a triangular matrix is the determinant.
=> determinant of such matrices $\neq0$
=> Matrices are non-singular
=> Matrices are invertible. -----------> #1
Closure holds.
Associativity holds. Matrix Chain Multiplication in Dynamic Programming is an example of this.
Identity holds. (The identity matrix)
Inverse holds. // From #1
Commutativity doesn't hold. As A.B $\neq$ B.A for matrices.
So, this is a group. Option A