A review found that the exchange’s compliance program violated state laws, making it “vulnerable to serious criminal conduct.” It will pay a $50 million fine and spend the rest beefing up oversight.