Robin van Persie's hat-trick helped Manchester United stage an incredible late comeback at Southampton and ensure Sir Alex Ferguson's 1,000th league game at the helm would end in victory.
Rickie Lambert put the newly-promoted host in front with a far-post header after 16 minutes, although Van Persie levelled seven minutes later with a fine strike.
Morgan Schneiderlin then headed Southampton back in front in the 55th minute after a slip by United defender Patrice Evra and that looked set to be the winner after Van Persie saw his cheeky penalty attempt saved by Kelvin Davis in the 68th minute.
However, Netherlands international Van Persie went from villain to hero with a double salvo inside the final three minutes that turned the game on its head and saw him rack up his 100th Premier League goal.
The 3-2 victory lifted United onto six points from its three games, while Arsenal improved to five points after beating struggling Liverpool 2-0 at Anfield to record its first victory of the campaign.
Two of the club's northern summer signings scored the goals for Arsene Wenger's men, with Lukas Podolski opening the scoring in the 31st minute before Santi Cazorla added a second in the 68th minute with a low shot that his fellow Spaniard Jose Reina will feel he should have saved.
The result saw Liverpool condemned to its worst start to a season for half a century after collecting just one point from its opening three games.
Hatem Ben Arfa got Newcastle United out of jail with a stunning equaliser as Aston Villa saw victory snatched from its grasp in a 1-1 draw.
The France international blasted an unstoppable long-range shot past goalkeeper Brad Guzan with 59 minutes gone to cancel out Ciaran Clark's first-half opener on a day when the Magpies were all at sea for long periods.