The Bundesliga side looked to have pulled off a sensational comeback after being 2-0 down after the first leg in Amsterdam. They took a 3-0 lead on the night, only for two late extra-time goals to swing the tie in Ajax's favour.
Nick Viergever and Amin Younes scored the crucial goals for the Eredivise side in the 111th and 120th minutes, after Daniel Caligiuri's headed goal in the first half of extra-time appeared to have won it for Schalke.
Ajax had claimed a dominant 2-0 win over Schalke in the first leg of their last-eight clash, with captain Davy Klaassen scoring both goals.
Leon Goretzka scored for Schalke early in the second half at the Veltins-Arena and Guido Burgstaller levelled the tie less than three minutes later.
Ajax were reduced to 10 men with 80 minutes on the clock after Joel Veltman was shown his second yellow card.
That proved to be a key moment, as Schalke made the most of their numerical advantage in extra-time, with Caligiuri scoring in the 101st minute.
However, the drama was not over as Viergever bundled the ball into the net in the second half of extra-time, before Younes made sure of the result at the end.
Schalke had the best of the early chances in the return, with Goretzka shooting wide in the first minute before Max Meyer struck the post moments later.
Bertrand Traore got the ball into the back of Schalke's net just after the half-hour mark after a period of pressure by the visitors - but it was correctly ruled out for offside.
Meyer then missed the target after a powerful run down the right by Goretzka, who cut it back for Meyer, but he got underneath the ball and lifted it over the bar.
The sides went in locked at 0-0 at the break but with the Eredivisie side well in control on aggregate, but things rapidly changed in the second half.
Goretzka made the breakthrough when he burst into the box in the 53rd minute and drove the ball low under Ajax goalkeeper Andre Onana to get the Bundesliga side back in the tie.
Less than three minutes later Burgstaller levelled the tie when he drove home, setting up a tense final half-hour.
Onana made a crucial save, keeping out a low shot after spilling it briefly before regathering, as neither side could find the decisive goal in normal time.
Goalscorer Goretzka left the pitch in stoppage-time with a leg injury while Ajax faced the prospect of an extra half hour with 10 men.
Caligiuri's scrappy extra-time goal came late in the first period of extra-time, with Sead Kolasinac deserving just as much credit for setting up the chance.
But it proved not to be enough, as Viergever tipped the tie in Ajax's favour on the away goals rule, before Younes sealed the win.
