As @aambazinga mentions, the answer is 7! / 2! = 2520
There are 5 books (nothing has been mentioned about whether the books are different or identical, so we always assume different). There are 3 shelves.
You can think of arranging the books into 3 shelves as having 5 markers for books, and 2 markers for the dividers of the shelves.
example: 12 | 345 | 67
So, you have 7 objects in total, 2 of which are identical (the dividers)
The number of ways of arranging these is $7! / 2!$
Why is $3^5$ incorrect? Understanding this is as important as being able to correctly answer the question.
$3^5$ says that each of the 5 books can go in any of the 3 shelves. This is correct, but it counts the ways of "putting" the books in the shelves, and not "arranging" them. For "arranging" them, we also need to count the number of ways the books can be arranged (permutations), once they're placed in the shelves.
That would be a longer calculation, as @balchandar reddy san did, and will finally lead you to the correct answer.