Binary search takes $O(nlogn)$ time. You can't just substitute 11 in place of n because $O$ actually represents some constant.
Let the array be {1,2,3,4,5,6,7,8,9,10,11}
Searching 6 would take 1 comparison.
Searching 3 or 9 would take 2 comparisons.
Searching 1 or 4 or 10 or 7 would take 3 comparisons.
Searching 2 or 5 or 8 or 11 would take 4 comparisons.
Average = $\frac{1*1+2*2+4*3+4*4}{11} = \frac{33}{11}=3$
Option A
Alternate method:
You can construct a BST of 11 nodes. No matter how you create it you'll end up with:-
- 1 node in Level 0
- 2 nodes in Level 1
- 4 nodes in Level 2
- 4 nodes in Level 3
For this question, let's consider the numbering of levels from 1. So:-
- 1 node in Level 1
- 2 nodes in Level 2
- 4 nodes in Level 3
- 4 nodes in Level 4
Average = $\frac{1*1+2*2+4*3+4*4}{11} = \frac{33}{11}=3$
PS: In case of unsuccessful search, we'd have added 1 comparison, so that we'd know the tree is over and search is unsuccessful.
It'd look like:-
- 1 node in Level 1
- 2 nodes in Level 2
- 4 nodes in Level 3
- 4 nodes in Level 4
- +1 futile comparison.
Average = $\frac{(1*1+2*2+4*3+4*4)+1}{11} = \frac{34}{11}=3.09$