Alli appeared to make the offensive signal at the referee during Tuesday's 2-1 FIFA World Cup qualifying win against Slovakia at Wembley.
But the Tottenham star later insisted via social media that the gesture was aimed at his former Spurs team-mate, Kyle Walker, and made in jest.
Vardy, who was an unused substitute for Gareth Southgate's side in that match, has defended Alli over suggestions he could face an investigation.
"The middle finger? Come on, it's harmless, isn't it? It's just something that has been blown out of proportion," Vardy said.
"He was doing it to Walks and he's replied on his Twitter. It's just harmless banter. They're always having a laugh and a giggle. That's just how they are.
"The two of them were team-mates for a few years. They're going to like to mess around, especially now that Walks has moved on himself.
"It's probably the only chance they do get to mess around, when they're with England. It will definitely be harsh if FIFA punish him."
Vardy suggested moments such as Alli's gesture towards his former team-mate should not be taken out of football.
"If he gets reprimanded for that, then we might as well take emotion and enjoyment out of the game," he said.
"Everyone has got to have a bit of banter, otherwise I don't think you would get through the day of being a footballer."