We use the publicly available subhalo catalogs from the Via Lactea simulation along with a Gpc-scale N-body simulation to understand the impact of inhomogeneous reionization on the satellite galaxy population of the Milky Way. The large-volume simulation is combined with a model for reionization that allows us to predict the distribution of reionization times for Milky Way mass halos. Motivated by this distribution, we identify candidate satellite galaxies in the simulation by requiring that any subhalo must grow above a specified mass threshold before it is reionized; after this time the photoionizing background will suppress both the formation of stars and the accretion of gas. We show that varying the reionization time over the range expected for Milky Way mass halos can change the number of satellite galaxies by roughly two orders of magnitude. This conclusion is in contradiction with a number of studies in the literature, and we conclude that this is a result of inconsistent application of the results of Gnedin (2000). We compare our satellite galaxies to observations using both abundance matching and stellar population synthesis methods to assign luminosities to our subhalos and account for observational completeness effects. Additionally, if we assume that the mass threshold is set by the virial temperature Tvir = 8e3K we find that our model accurately matches the vmax distribution, radial distribution, and luminosity function of observed Milky Way satellites for a reionization time zreion = 9.6^{1.0}_{-2.1}, assuming that the Via Lacteasubhalo distribution is representative of the Milky Way. This results in the presence of 119^{+202}_{-50} satellite galaxies. Show Partial Abstract