Striker Mark Bridge's close-range header two minutes after the hour mark had put Sydney, who topped the table at the end of the regular season, ahead only for defender Adrian Leijer to equalise for defending champion Melbourne with nine minutes remaining.
In the shootout, Melbourne captain Kevin Muscat had a rare miss only for Victory keeper Mitchell Langerak to save from Shannon Cole and tie things up.
But Sydney keeper Clint Bolton made the ultimately decisive save from Marvin Angulo as the Sky Blues emerged victorious in front of a season best crowd of 44,560 football fans.
After veteran midfielder Tom Pondeljak failed to recover from a hamstring injury and was ruled out for Melbourne before kick-off, Victory suffered a further injury blow after just 10 minutes as striker Archie Thompson suffered a serious knee injury following a collision with Simon Colosimo and was replaced by Angulo.
After a cagey opening, Langerak parried away Chris Payne's shot four minutes before half-time before Surat Sukha headed over for Melbourne from Nick Ward's cross.
Bridge had an excellent chance to score early in the second half but his volley was pushed away by Langerak before Bolton was called into action for the first time after 52 minutes to deny Ward.
Roddy Vargas had the ball in the net for Melbourne just after the hour mark but the goal was ruled out for offside and Sydney took full advantage to instantly open the scoring.
Bolton played the free-kick quickly and Sydney swept forward and, after Surat could only deflect Alex Brosque's cross over a stranded Langerak, Bridge headed in at the back post to stun the home crowd.
Payne had an opportunity to all but seal it for Sydney but turned the ball wide with the goal at his mercy before Carlos Hernandez went close for Melbourne.
But the home side did not have to wait long for an equaliser as Muscat delivered a free-kick into the area after 81 minutes and Leijer leapt high above Sebastian Ryall and finished superbly to Bolton's left.
Leijer had another chance to score a few minutes later but headed just over from Leigh Broxham's cross before Melbourne substitute Aziz Behich had a chance to be a hero.
But he failed to make enough contact from close range to beat Bolton as the grand final went to extra-time for the first time.
Bridge's rifled shot five minutes into the second period of extra-time brought the best out of Langerak before Melbourne's Hernandez and Sydney's Cole wasted free-kicks in the closing stages.
Terry McFlynn, standing in as captain following the career-ending injury to Steve Corica and John Aloisi's unavailability through injury, was not surprisingly ecstatic with his team's performance.
"It's a fantastic feeling amongst all the boys, I think tonight showed that us and Melbourne are the two best teams in the competition," he said.
"All season we couldn't be separated, tonight after 120 minutes we couldn't be separated, and it's come down to the dreaded penalty shootout.
"It's never an easy way to win the game but someone has to win and thankfully it was us.
"We put a belief in ourselves that we could do it, and the minor premiership and grand final win tonight proves that we could do it."
Bolton, bound for new franchise Melbourne Heart, said the triumph was all down to team unity.
"It was good to save the pen at the time but this year's been all about the team," he said.
"Vit (coach Vitezslav Lavicka) has driven that into us all year and that's what's won us the title."
Lavicka was celebrating claiming both the minor premiership and title in his first season in charge.
"It was easy for me because around me was nice people - players, coaching staff, Sydney FC fans... I enjoyed the whole season and this is like a bonus," he said.
Muscat said it was disappointing not to reward the Victory's fans with another trophy on home soil.
"It's probably the first one (penalty) I've missed and to miss it tonight on the big stage is going to haunt me for a very long time," he said.
"It was unfortunate we lost in a penalty shootout but that's life."
He refused to use the loss of Thompson as an excuse for Melbourne's inability to prevail.
"There's no doubt Archie's a quality footballer and taking him out of any team's going to alter the performance, but I thought we still created enough chances and played some good stuff at times," he said.
In the shootout the usually reliable Muscat missed from 12 metres after Colosimo had netted for the Sky Blues.
But Langerak produced a superb save from Cole, allowing Grant Brebner to pull Melbourne level.
Hayden Foxe held his nerve for Sydney before Bolton denied Angulo, allowing Kisel to push Sydney ahead.
Broxham kept Melbourne's hopes alive but that only delayed Sydney's celebrations as South Korea defender Byun Sung-Hwan netted and the Sky Blues claimed a second A-League title, having won the inaugural grand final in 2006.
Melbourne Victory line-up: 1.Mitchell LANGERAK (gk), 2.Kevin MUSCAT (c), 5.Surat SUKHA (19. Evan BERGER 71’), 6.Leigh BROXHAM, 8.Grant BREBNER, 10.Archie THOMPSON (11.Marvin ANGULO 16'), 12.Rodrigo VARGAS, 16.Carlos HERNANDEZ, 21.Robbie KRUSE, 22.Nick WARD (25.Aziz BEHICH 76’) , 23.Adrian LEIJER.
Sydney FC line-up: 1.Clint BOLTON (gk), 2.Sebastian RYALL, 3.Stephan KELLER, 4.Simon COLOSIMO, 6.Karol KISEL, 8.Stuart MUSIALIK (5.Hayden FOXE 105’), 14. Alex BROSQUE (31. Joe GIBBS 116’), 15. Terry MCFLYNN (c), 16.Chris PAYNE (12.Shannon COLE ),19.Mark BRIDGE, 22. Sung-Hwan BYUN.