We develop an Effective Field Theory (EFT) formalism to solve for the conservative dynamics of binary systems in gravity via Post-Minkowskian (PM) scattering data. Our framework combines a systematic EFT approach to compute the deflection angle in the PM expansion, together with the 'Boundary-to-Bound' (B2B) dictionary introduced in [1910.03008, 1911.09130]. Due to the nature of scattering processes, a remarkable reduction of complexity occurs both in the number of Feynman diagrams and type of integrals, compared to a direct EFT computation of the potential in a PM scheme. We provide two illustrative examples. Firstly, we compute all the conservative gravitational observables for bound orbits to 2PM, which follow from only one topology beyond leading order. The results agree with those in [1910.03008, 1911.09130], obtained through the 'impetus formula' applied to the classical limit of the one loop amplitude in Cheung et al. [1808.02489]. For the sake of comparison we reconstruct the conservative Hamiltonian to 2PM order, which is equivalent to the one derived in [1808.02489] from a matching calculation. Secondly, we compute the scattering angle due to tidal effects from the electric- and magnetic-type Love numbers at leading PM order. Using the B2B dictionary we then obtain the tidal contribution to the periastron advance. We also construct a Hamiltonian including tidal effects at leading PM order. Although relying on (relativistic) Feynman diagrams, the EFT formalism developed here does not involve taking the classical limit of a quantum amplitude, neither integrals with internal massive fields, nor additional matching calculations, nor spurious ('super-classical') infrared singularities. By construction, the EFT approach can be automatized to all PM orders. Show Partial Abstract