The Gunners had been looking for a response to the weekend's lacklustre 1-0 loss at Norwich City but failed to produce one as manager Arsene Wenger watched on from the stands, serving out his European touchline ban.
Schalke now overtakes Arsenal at the top of Group B, and was good value for the victory, secured by second-half goals from Klaas-Jan Huntelaar and Ibrahim Afellay.
The Gunners must now regroup ahead of Sunday's (AEDT) visit of Barclays Premier League struggler Queens Park Rangers and next week's Capital One Cup tie at Reading, but the midweek's poor showing will have done little to lift the mood ahead of Friday's annual general meeting.
Schalke beat Bundesliga rival Borussia Dortmund away last time out, and was looking to recover from the disappointment of seeing 10-man Montpellier snatch a late 2-2 draw from its previous European tie.
The visitor started brightly and had a penalty shout on 13 minutes after a quick break.
Jefferson Farfan played in Dutch winger Afellay, who fell over the outstretched arm of Vito Mannone.
There looked contact from the Italian keeper, whose handling error gifted Norwich the winning goal. However, Swedish referee Jonas Eriksson waved away the appeals and instead brandished a yellow card to the Schalke player.
German Lukas Podolski was Arsenal's main outlet down the left and his low centre flew back across the six-yard box, just behind Aaron Ramsey.
Gervinho had been preferred in the central striker's role to Olivier Giroud, and was sent clear - but lacked support and rather than trying to go it alone for goal, was eventually hounded off possession as he tried to hold the ball up.
On 32 minutes, German centre-half Per Mertesacker made an important block to charge down Farfan's fierce drive.
There was another scare for the host just before the break when Japan fullback Atsuto Uchida got clear down the right and crossed for Huntelaar at the near post, but the usually reliable Dutchman swept a poor effort wide.
Schalke made a change for the second half as American midfielder Jermaine Jones, who had a loan spell with Blackburn in 2011, replaced Marco Hoger.
The German team should have taken the lead when Farfan charged past Andre Santo down the right and crossed back to the penalty spot, but captain Benedikt Howedes blasted over.
Slowly Arsenal started to find its passing game and Gervinho was unfortunate to lack support in the middle as he pushed on down the left to cut the ball across an empty six-yard box.
The host, however, still had plenty of defending to do as Schalke remained dangerous down both flanks.
On 70 minutes, Gervinho went into the referee's book for going down under a challenge from Jones in the box. The Ivory Coast forward was soon replaced by Giroud.
It was Schalke, though, which eventually got the breakthrough on 76 minutes.
Arsenal failed to clear its lines, with Afellay heading the ball back through the centre of the defence and into the path of Huntelaar, who made no mistake from 10 metres.
The Gunners made a double change when Podolski was replaced by Andrey Arshavin and full-back Carl Jenkinson came off for 17-year-old Serge Gnabry.
It was, however, game over with four minutes to go when Schalke broke quickly down the right through Jones and his low centre was swept home by Afellay.
The inquest started among the disappointed Arsenal players - a debate which no doubt will be continued by disgruntled shareholders, many of whom will have jeered the home side off at the final whistle.
UEFA Champions League
Borussia Dortmund centre-back