Both are grammatically correct. Though "Mike's brother's book" is kind of clunky, you can layer as many of those as you want without breaking any rules. Personally, if I was writing this into a paragraph, I'd try to avoid the situation altogether by mentioning the name of Mike's brother beforehand, so when I bring up the book, I can just say "Dave's book" or whatever his brother's name is.