The SELEX Collaboration has reported a very large isospin splitting of doubly charmed baryons. We show that this effect would imply that the doubly charmed baryons are very compact. One intriguing possibility is that such baryons have a linear geometry Q-q-Q where the light quark q oscillates between the two heavy quarks Q, analogous to a linear molecule such as carbon dioxide. However, using conventional arguments, the size of a heavy-light hadron is expected to be around 0.5 fm, much larger than the size needed to explain the observed large isospin splitting. Assuming the distance between two heavy quarks is much smaller than that between the light quark and a heavy one, the doubly heavy baryons are related to the heavy mesons via heavy quark-diquark symmetry. Based on this symmetry, we predict the isospin splittings for doubly heavy baryons including Xi_{cc}, Xi_{bb} and Xi_{bc}. The prediction for the Xi_{cc} is much smaller than the SELEX value. On the other hand, the Xi_{bb} baryons are predicted to have an isospin splitting as large as (6.3\pm1.7) MeV. An experimental study of doubly bottomed baryons is therefore very important to better understand the structure of baryons with heavy quarks. Show Partial Abstract