Everyone likes to talk about Neurath's boat, but I think common discussion leaves out something critical. Not only do we all start with some beliefs, but we also start out accepting certain methods of revising those beliefs, in response to new experience or in the course of further reflection. This is crucial because it brings out a deep symmetry between all believers:
At a certain level of description, there's no difference between the atheist philosopher who finds it immediately plausible that bread won't nourish us for a while and then suddenly poison us, and the religious person who finds it immediately plausible that god exists, or the madman who finds it immediately plausible that he's the victim of a massive conspiracy. Everyone involved is (just) starting with whatever they feel is initially plausible, and revising this in whatever ways they find immediately compelling.
Thinking about things this way, can make one feel uncomfortable in deploying normative notions of justification. Being justified is supposed to be a matter of (something like) doing the best you can, epistemically, whether or not you are lucky enough to be right. But there's no difference in effort (or even, perhaps, in care) between the philosopher and the madman. It's just that the philosopher is lucky enough to find immediately compelling principles *that happen to be mostly true*, and inference methods *that happen to be mostly truth-preserving/reliable*. So how can we say that one of them is justified?
One reaction to this is to deny that there is such a thing as epistemic normativity. There are facts about which people have true beliefs, and which of them are on course to form more true beliefs, which belief forming mechanisms are reliable (in various senses) etc. But there are no epistemically normative facts e.g. facts about which reliably true propositions are OK to to assume, or which reliable inference methods are OK to employ without any external testing.
Another possible reaction is to say that even though "ultimately" there's no difference between believing finding it obvious that bread will nourish you if it always has in the past vs. believing you are the center of a conspiracy, there still are facts about justification. We can pick out certain broad methods of reasoning (logical, empirical, analytic(??), initially trusting the results of putative senses) which are both popular and generally truth preserving, and what it means to be justified is just to have arrived at a belief via one of those.
In either case, the result will give an answer to philosophical skepticism. The skeptic asks: "how can you be justified in believing that you have a hand, given that it depends on your just assuming without proof that you aren't a BIV?" Someone who has the first reaction can simply deny the contentious facts about justification. Someone who has the second reaction will be unimpressed by the point that they are "just assuming" that ~BIV. All possible belief is a matter of starting out "just assuming" some propositions and inference methods, and then applying the one to the other.