$\begin{align*} T(n) &= 64\ T(n/8) - n^2 \log n\\ &= 64\ T(n/8) + n^2 \log \frac{1}{n}\\ \end{align*}$
this does not satisfy the generic form of master method. It is close to the case of Master Method, where :
but fails so do other cases too. So, one of the method to do this is Akra–Bazzi Method.
here,
we need to choose $p$ such that $\sum_{i=1}^{k} {a_ib_i^p}=1$
$64 \left( \frac{1}{8} \right )^p = 1 \\ \text{satisfies for p = 2}$ and also, $\left| \mathrm{g'(x)} \right| \leq \mathrm{x}^c \qquad \text{for some } c \in \mathbb{R}$
so,
$\begin{align*} T(x) &= \Theta\left( x^p + x^p \int_{1}^{x} \frac{u^2 \log \frac{1}{u}}{u^{p+1}}\ du \right )\\ &= \Theta\left( x^2 - x^2 \int_{1}^{x} \frac{u^2 \log {u}}{u^{3}}\ du \right )\\ &= \Theta\left( x^2 - x^2 \int_{0}^{\log x} t\ dt \right )\\ &= \Theta\left( x^2 - \frac{x^2}{2} \{\log x\}^2 \right )\\ &= \Theta\left( x^2 + \frac{x^2}{2} \log x\ .\ \log \frac{1}{x} \right )\\ &= \Theta\left( \frac{x^2}{2}\ \log \frac{1}{x}\ \log x \right )\\ \\ T(n) &= \Theta\left( \frac{n^2}{2}\ \log \frac{1}{n}\ \log n \right )\\ \end{align*}$