I've never read that Olympic Arms were 'worse' than other brands, just that other brands were 'better.' Olympic makes mil-spec stuff just like everyone else, but their finish may not be as pretty (not as yummy-smooth and seams may still be visible).
The only negative that I have read about Oly is that one guy had a Plinker Plus, ($595 MSRP, complete), dropped it, and the barrel snapped off. He wasn't happy, and I wouldn't be either. That was it. I've read a lot of good reports on the Plinker Plus, as well as other OA models. There is an OA-specific forum out there, too, if you really want to dig into Oly stuff.
I'd recommend checking at local gun shops. Not big box stores like Gander or what not, but actual real-life gun shops. There are a few guys in my area that do AR business out of their home shops. I found them on Google Maps. Anyhoo, local shops can usually do better than the MSRP. Online prices may be attractive, but keep in mind that the lower receiver will need to be shipped to a FFL dealer. The shop I frequent charges $45 to do a transfer (it may be $35 for a LR).
Be advised that the best deals aren't necessarily at gun shows. Although deals can still be found there.
Finally, a co-worker of mine built a flat-top M4 for about $650, minus the optic. I believe he acquired a complete Bushmaster lower and an upper from Model1Sales.
FWIW