We study the mass spectrum of destroyed dwarfs that contribute to the accreted stellar mass of Milky Way (MW) mass M_vir ~ 10^12.1 M_sun) halos using a suite of 45 zoom-in, dissipationless simulations. Empirical models are employed to relate (peak) subhalo mass to dwarf stellar mass, and we use constraints from z=0 observations and hydrodynamical simulations to estimate the metallicity distribution of the accreted stellar material. The dominant contributors to the accreted stellar mass are relatively massive dwarfs with M_star ~ 10^8-10^10 M_sun. Halos with more quiescent accretion histories tend to have lower mass progenitors (10^8-10^9 M_sun), and lower overall accreted stellar masses. Ultra-faint mass (M_star < 10^5 M_sun) dwarfs contribute a negligible amount (<< 1%) to the accreted stellar mass and, despite having low average metallicities, supply a small fraction (~2-5 %) of the very metal-poor stars with [Fe/H] < -2. Dwarfs with masses 10^5 < M_star/M_sun < 10^8 provide a substantial amount of the very metal-poor stellar material (~40-80 %), and even relatively metal-rich dwarfs with M_star > 10^8 M_sun can contribute a considerable fraction (~20-60 %) of metal-poor stars if their metallicity distributions have significant metal-poor tails. Finally, we find that the generic assumption of a quiescent assembly history for the MW halo seems to be in tension with the mass spectrum of its surviving dwarfs. We suggest that the MW could be a "transient fossil"; a quiescent halo with a recent accretion event(s) that disguises the preceding formation history of the halo. Show Partial Abstract