The A-League club stormed off the ground two hours before the scheduled kick-off claiming it did not have an opportunity to warm up on the pitch due to an under-16 game being played at the time.
The trial match, which was sanctioned by Football Federation Australia, had to be called off.
"Unfortunately when we arrived at Belmore we were unable to prepare the players for the game," Perth chief executive Jason Brewer said.
"We played there at the weekend and we were satisfied with everything but last night things changed.
"I heard somebody say that an under-16 game which was in progress was always scheduled to serve as a curtain-raiser.
"But neither us nor Sydney Olympic were aware such a game was to be played.
"We expected a pitch on which our players could warm up.
"It was all about a lack of communication and having the correct environment in which to prepare the players.
"We have obligations in terms of player safety and welfare.
"We did not want to put anybody at risk in that respect.
"Mitch Nichols and Youssouf Hersi have had hamstring injuries and were both due to start and a lack of a proper warm-up could have had major implications."
Brewer said he recognised that the sensational incident did not look good on Perth but he was sure that the club will have no case to answer to FFA.
"Unfortunately the whole affair does not look good on us because we are the A-League team, the bigger club and the professional club," he said.
"Clearly there is the perception that this is all our fault for pulling out of the game.
"But quite clearly that is not the case.
"We have discussed the matter with the FFA and they are aware of why the decision was made.
"It was made on player safety and I do not expect any further discussion on the matter.
"The decision was not made lightly.
"Coach Kenny Lowe was in contact with myself and the football department.
"We spent considerable capital to have a camp in Sydney to play two games and this fixture was to be a highlight of our tour.
"We were keen to push ahead with because it would have been a great match for us from a competition point of view a few days before we take on Malaga of Spain.
"So the decision was not made on the whim of a coach."
Brewer made it a point to express the club's genuine disappointment at the turn of events.
"It is a very disappointing situation both clubs find themselves in," he said.
"It was also disappointing for the fans who were hoping to watch two clubs that have such a great rivalry.
"It's a rivalry that stretches several years ago and we want to re-ignite."
Olympic coach Grant Lee said he and his club could not understand Perth's attitude.
"They played at Belmore at the weekend and two hours before the kick-off on Wednesday night we were told that they were pulling out of the game," Lee said.
"They said the conditions of the pitch and the fact that the under-16s were playing were unsatisfactory which is ridiculous.
"We told them we were happy to move the under-16s to another field but Perth claimed that the players were not in the best state of mind to play the game.
"It was very disappointing. At the end of the day we are talking about an A-League club pulling out of an FFA-sanctioned game.
"A coach cannot cancel a game. Only the referee or the two clubs together can do that if the pitch or the conditions were unsafe for the players.
"Perth had a duty to our sponsors and our supporters and to the game of football."