Intuitively, many people feel that epistemic worries about moral facts (if there are moral facts, how to explain why our moral intuitions should be even even remotely correct about them?) are WAY more serious than epistemic worries about mathematical facts (if there are mathematical facts, how to explain why our mathematical intuitions should be even even remotely correct about them?). But is there really a difference here?
Well, here's one thing that I think does make a difference: mathematical claims about number theory have direct and specific consequences for stuff that we can check by logic and/or scientific observation.
-what will happens whenever a person or a computer to successfully applies a certain syntactic alogorithm
-how many apples-or-oranges do you have when you have n apples and m oranges (cf Frege for why this is a logical fact)
This matters because, plausibly, the need to get these concrete applications right likely prevents our beliefs about number theory from getting too off the wall - whereas, our moral intuitions have no such multitude of consequences which are directly checkable by logic and observation.