England won 1-0 to qualify for the finals 2-0 on aggregate.
Serbian supporters, players and other officials appeared to get involved while England defender
Carlisle watched the chaos and violence unfold in the ESPN studio alongside former England midfielder
And the 33-year-old agreed a hefty punishment was the only answer.
"You should not have to experience those scenes in this day and age. It was utterly deplorable behaviour," Carlisle told BBC Radio Five.
"It's quite farcical. A guy is racially abused and then he gets punished for containing his reaction to that.
"We saw how the whole debacle progressed into punching, kicking. We saw photos of headbutts going on in the staff. But
"UEFA need to make a standpoint here and more importantly FIFA, I feel, because it was a FIFA qualification tournament. The message has to come from the top down that this behaviour will not be tolerated."
England Under-21 manager
A match between the two countries at the 2007 Under-21 European Championship in
Serbia were fined by European governing body UEFA after that incident.
Carlisle believes a lengthy ban would force Serbia to address the issue.
"It's immaterial where it is, who the perpetrators are," he said. "What's material is that the governing bodies lay down extremely harsh sanctions so that they deter countries and people from acting in this way.
"You can draw parity with what happened to English football when we were kicked out of
"I believe Serbia should be banned because it's a repeat offence.
"Banning them for a start, from any tournament, would be progress but I think if it's significant - if it's a couple of tournaments - then that would cause that nation to address the issue that has deprived them of international competition."