Brazil striker Rosana lashed the ball home with her left foot in the 54th minute after the Matildas failed to clear the ball on the edge of the area.
Australia had earlier been unable to capitalise on a strong start and a possession advantage in the first half.
The Matildas had a few late chances, the best of which striker Lisa De Vanna hoisted over the bar after she was put clear in the 87th minute.
The loss leaves Australia desperately needing to pick up at least a point from its second match on Sunday against African nation Equatorial Guinea, which lost 1-0 to Norway in the other Group D game.
For the most part, the Australian defence contained Brazil's celebrated striking duo of Marta and Cristiane.
Australia enjoyed more possession in the first half as Brazil surprisingly opted not to pressure the Matildas in their own half.
It was a different story after the break, though, with Brazil showing greater purpose and applying more pressure, especially after it took the lead.
"I think for long periods of the game we more than held our own," Matildas coach Tom Sermanni said afterwards.
"Very proud of the effort, very proud of the performance and a little bit disappointed we didn't get anything out of the game."
The Matildas looked the more consistently dangerous team in the early stages.
Midfielder Collette McCallum curled a free-kick over the bar and Kyah Simon had a chip acrobatically saved at the near post.
Rosana put a couple of attempts over the bar from close range, but De Vanna squandered a good chance on the half-hour.
She took possession after Brazilian skipper and defender Aline lost control on the edge of her area, but the Australian forward miscued her shot wide.
Brazil almost capitalised on a couple of Australian defensive lapses in the last 15 minutes of the half.
Marta's first-time shot flew just wide after 16-year-old right back Caitlin Foord lost control on the edge of the area and Cristiane blazed over from close range.
Sermanni's side started the second half well but Simon miscued a close-range header from a corner, just a minute before Rosana scored.
Brazil's midfield then took control and the Matildas struggled to create anything for the next 20 minutes until De Vanna blasted a shot well over the bar.
Veteran midfielder Heather Garriock and Tameka Butt had attempts comfortably saved by Brazilian goalkeeper Andreia as the Matildas pressed for an equaliser.
Butt got behind the defence but couldn't find a team-mate with her cross, while a corner in the closing seconds caused concern for the Brazil defence.