He can't tell if that's metaphorical or not, but decides to take it literally. "You weren't being held as a gladiator on Sakaar, were you?" That's be an awful big coincidence if she was. And it'd be a shame if he and Banner had completely missed Rogers' presence. (How would he even have gotten there...?)