Longitudinal Phase Spaces (LPS) of electron bunches provide critical information for tuning and optimizing of FEL facilities. However, high-resolution physics-based start-to-end simulation is extremely time-consuming and oftentimes does not agree with the measurement very well. Moreover, some LPS diagnostic methods, e.g. the time-domain measurement using a transverse deflecting structure (TDS), interfere with delivering photons to user experiments and thus cannot be used for online diagnostics. Here, we leverage the power of artificial intelligence to build neural network models using a data-driven approach, in order to bring those destructive LPS diagnostics online and create a digital twin for the real machine. We present both the experimental results at the European XFEL photoinjector and the end of the FLASH1 linac. The results demonstrate that neural network models can make high-fidelity predictions of megapixel LPS images and signals from a THz spectrometer simutaneously, for electron bunches with a wide range of bunch lengths. In addition, we propose a mixed diagnostic approach which combines the prediction and the online measurement so as to provide more comprehensive LPS information in real time.