A crucial step in the proof of Fermat's last theorem was Frey's insight that a nontrivial solution would yield an elliptic curve with modular p-torsion but which was itself not modular. The connection between an elliptic curve and its p-torsion is very deep: a conjecture of Frey and Mazur, stating that the p-torsion group scheme actually determines the elliptic curve up to isogeny (at least when p>13), implies an asymptotic generalization of Fermat's last theorem. We study a geometric analog of this conjecture, and show that the map from isogeny classes of "fake elliptic curves"---abelian surfaces with quaternionic multiplication---to their p-torsion is at most two-to-one, and one-to-one in special cases. Our proof involves understanding curves on a certain Shimura surface, and fundamentally uses the interaction between its hyperbolic and algebraic properties.