We measure the cross-correlation between the galaxy density in the Dark Energy Survey (DES) Science Verification data and the lensing of the cosmic microwave background (CMB) as reconstructed with the Planck satellite and the South Pole Telescope (SPT). When using the DES main galaxy sample over the full redshift range 0.2 < z < 1.2, a cross-correlation signal is detected at 6 \sigma and 4\sigma with SPT and Planck respectively. We then divide the DES galaxies into five photometric redshift bins, finding significant (>2 \sigma) detections in all bins. Comparing to the fiducial Planck cosmology, we find the redshift evolution of the signal matches expectations, although the amplitude is consistently lower than predicted across redshift bins. We test for possible systematics that could affect our result and find no evidence for significant contamination. Finally, we demonstrate how these measurements can be used to constrain the growth of structure across cosmic time. We find the data are fit by a model in which the amplitude of structure in the z<1.2 universe is 0.73 \pm 0.16 times as large as predicted in the LCDM Planck cosmology, a 1.7\sigma deviation. Show Partial Abstract