It's actually mentioned very clearly.
He was not on Mt of Olives when he ascended.
Luk 24:50
And he led them out as far as to Bethany, and he lifted up his hands, and blessed them.
Luk 24:51 And it came to pass, while he blessed them,
he was parted from them, and carried up into heaven.
Luk 24:52 And they worshipped him, and returned to Jerusalem with great
Here we see that they went to Bethany and he ascended there, which is a mile from the Mount of Olives and about 2 miles from Jerusalem. They left Bethany and returned to Jerusalem having to cross the Mt of Olives on their way back. See the map provided.
Christ ascended in Bethany, they went back to Jerusalem via the Mt of Olives which is a Sabbath's journey from the Mt to Jerusalem. They traveled further than that because Bethany (it's one and only location) is on the other side of the Mt and past it's base just as Jerusalem is on that other side.
Acts only makes it seem like the ascension was on the Mount but it doesn't say that. It only mentions their return journey from the Mount to Jerusalem but doesn't bother to mention Bethany.
Mar 11:1 And when they came nigh to Jerusalem, unto Bethphage and Bethany, at the mount of Olives, he sendeth forth two of his disciples,
Bethany is "at the Mount" just as it can be said that Jerusalem is at the Mount. That simply means near not on the Mount. Neither Jerusalem nor Bethany are actually on any part of the Mount. Both are about a mile away from the it.
Joh_11:18 Now Bethany was nigh unto Jerusalem, about fifteen furlongs off:
15 furlongs is two miles just as the map shows with the Mount of Olives in the middle between cities. It's just one of those mistaken traditions that are so common that says Jesus ascended from the Mount.