Talisca scored the opening goal of a pulsating match after 27 minutes before Alexandre Lacazette ensured Lyon were back in charge of the tie with his fine finish moments later.
But Talisca added his second shortly before the hour mark to make it 2-1 to the hosts on the night and 3-3 on aggregate and take the game to extra-time.
There was no further scoring and the French side prevailed in the shoot-out, with captain Maxime Gonalons converting the decisive spot-kick after Matej Mitrovic had seen his penalty saved.
Lyon held a slender one-goal advantage heading into the second leg after securing a 2-1 victory in an opening fixture which was marred by crowd disturbances.
After both clubs were handed suspended bans from European competition and hefty fines after the trouble in France it was not long before Besiktas were back in the tie.
Atiba Hutchinson's effort was fumbled behind for a corner by Lyon goalkeeper Anthony Lopes, and when the ball fell to Talisca from the resulting set piece, he made no mistake. Talisca's effort flew through the legs of Christophe Jallet and Lopes, despite diving low to his right, had no chance.
Seven minutes later however, Lyon were in control once more. Lacazette's exquisite touch - after fine interplay with Gonalons - allowed him to execute a perfect side-foot volley over Besiktas goalkeeper Fabricio.
It was Lacazette's 29th goal of the season, and moments later it looked as though he had taken his tally to 30, only for his sumptuous curling effort to rebound off the woodwork.
Lyon, fourth in Ligue 1, were in charge at this stage - but shortly before the hour-mark the home side hauled themselves back into the game through Talisca's second goal of the night.
Adriano cut back on his right foot before delivering the inch-perfect cross for the on-loan Fenerbahce striker, who jumped highest to net his 11th of the season.
Lyon applied the pressure and looked the more likely of the two sides to win the tie - but the best chance of normal time fell to the home side's Ryan Babel.
Ricardo Quaresma's cross-shot hit the post, and it looked easier for Babel to score - only for the former Liverpool forward to mishit his shot from a matter of yards.
In the final minute of normal time Lacazette hit the crossbar for the visitors after a fine throughball from Corentin Tolisso.
There were chances for both sides in extra-time of a thrilling encounter - but the contest went to a shootout.
Babel, Cenk Tosun, Hutchinson, Tolgay Arslan, Talisca and Necip Uysal all scored for the hosts before Dusko Tosic saw his penalty saved by Lopes.
Nabil Fekir, Tolisso, Rachid Ghezzal, Maciej Rybus, Mathieu Valbuena and Mouctar Diakhaby were on target for Lyon - then Jallet shot over.
Mitrovic then saw his penalty saved by Lopes - and when Gonalons scored it meant Lyon would be in the hat for the semi-final draw.
