Hexadecimal number system represents numbers using base $16$, using $0-9$ for decimal numbers $0-9$ and $A-F$ for decimal numbers from $10-15.$
Here,
$(A)_H=(10)_{10}=(1010)_2$
similarly
$(B)_H=(11)_{10}=(1011)_2$
To obtain octal equivalent for a given number first convert into binary representation and make a group of $\log_28=3$ bits.
$(A.B)_H=(\ 1010 .\ 1011)_2=(\ 001 \ 010. \ 101 \ 100)_2$
$\therefore (A.B)_H=(12.54)_8$
So correct answer is $(12.54)_8$
Note: none of the options are correct here.