Maybe I just haven't done enough research yet, but I don't see why it's puzzling that we could learn new things about the numbers by learning things about the sets, and then applying them, given that we know perfectly well how facts about the numbers relate to facts about the sets (some people even identify the numbers with certain sets).
I mean: Is it puzzling that adding to a theory of shapes on a computer monitor (e.g. trangle, square etc) a theory of individual pixels that make up the shapes should let you derive new consequences about what shapes the monitor can display? I don't think this is puzzling - we see phenomena like this all the time e.g. new facts about chemistry can teach us new facts about how DNA will behave, hence about biology.
Or what about the way that reasoning about sets (with ur-elements) could teach you things about ordinary objects: If there's no non-empty subset S of the people you invited to the party such that each person is in that subset was formerly married to some other person in S, then if anyone shows up to the party (and only invited people come), there will be at least one person who fails to meet an ex-spouse there.
I am tempted to suspect that this whole thing is not a problem if you are as much of a realist about math as about computer displays or chemestry or biology or party-goes, and if you face problems about how we can *ever* know *anything* about mathematical facts, head on. (what my thesis claims to do). I mean, maybe if you thought that all mathematical knowledge was just a matter of stipulative definition it would puzzle you how we could learn things about the numbers from reasoning about the sets which was (presumably) not part of the stipulative definition of the numbers (or the sets?). But even then, the mere fact that we can *ever* know bridge laws relating the numbers to the sets should be puzzling, not the fact that these bridge laws are fruitful...
Does anyone have ideas for a more charitable understanding of the concern here?