An 89th-minute free-kick from Ulsan star Junior Negrao completed the comeback and returned Davidson's side to the top of the league after two rounds.
Socceroos striker Taggart was made to wait until the hour-mark for his introduction, though it seemed Suwon were doing just fine without him after goals from Ko Seung-Beom and Sulejman Krpic set up a surprise 2-0 lead.
Long a thorn in the side of K League opposition, Junior soon sparked hopes of a comeback, before Kim In-Sung restored parity some seven minutes later.
A stunned Suwon did little to arrest the away side's momentum as the second half wore on, save for Taggart's golden opportunity with 15 minutes remaining.
Played clean through on goal, the 26-year-old failed to fire his effort past the outstretched leg of Korea Republic international Jo Su-huk.
That miss would prove costly in the end, as Junior then stepped up to bury a last-gasp free-kick past the wrong-footed No Dong-Geon.
Suwon are now now without a point to start the new season and will next face Rashid Mahazi's Incheon United on Saturday.
As for Ulsan, Busan IPark await on the same day.
