Well, you're right, I do not have an explicit verse that a 4th temple will be built (Solomon's, Zerubbubel's, Herod's). But, i have an explicit passage that, in the future, there will be an entity blaspheming in the Temple of God. And that, because the verse that states this was didactic and explicit in context, i do not give the passage a metaphorical or figurative sense.
I mean, whether by the will of God or not, you must admit that there is a large demograph that has this hope i.e. current day Jews, Zionists, certain schools of Christianity, etc. Thus, the impetus is there.