Anglo American rebuffed a roughly $50 billion takeover proposal from BHP but said it would enter talks with its mining rival, opening the door to a potential deal.