We report on the application of Machine Learning (ML) based methods for predicting the Longitudinal Phase Space (LPS) distribution of particle accelerators. Our approach is based on training a ML-based virtual diagnostic to predict the LPS using only non-destructive linac and e-beam measurements as inputs. We validate this approach with a simulation study for the FACET-II linac and with an experimental demonstration conducted at LCLS. At LCLS, the e-beam LPS images are obtained with a transverse deflecting cavity and used as training data for our ML model. In both the FACET-II and LCLS cases we find good agreement between the predicted and simulated/measured LPS profiles, an important step towards showing the feasibility of implementing such a virtual diagnostic on particle accelerators in the future. Show Partial Abstract