The accelerating gradient in a laser-driven dielectric accelerating structure is often limited by the laser damage threshold of the structure. For a given laser-driven dielectric accelerator design, we can maximize the accelerating gradient by choosing the best combination of the accelerator's constituent material and operating wavelength. We present here a model of the damage mechanism from ultrafast infrared pulses and compare that model with experimental measurements of the damage threshold of bulk silicon. Additionally, we present experimental measurements of a variety of candidate materials, thin films, and nanofabricated accelerating structures. Show Partial Abstract