Today, high brightness and highly relativistic electron beams are generated by circular or linear accelerators (LINAC) typically operating with 1-3 GHz accelerating frequencies and approaches towards X-band frequencies in the 10 GHz range are maturing. The achievable accelerating gradients are limited by field emission from cavity walls or pulsed heating to several tens of MV/m in the case of low frequencies and up to 100 MV/m in the case of X-band frequencies. Moving up in frequency to the Terahertz (THz) or even Infrared frequency range, here hundreds of GHz or 2 um, respectively, experimentally confirmed scaling laws predict the realization of few-hundred MV/m to 1 GV/m accelerating fields in THz devices and multi-GV/m gradients in the infrared range. At DESY an ERC Synergy Project called Attosecond X-ray Science: Imaging and Spectroscopy (AXSIS) is underway to demonstrate THz-based accelerators and X-ray sources. Several of its team members and the DESY Laser Group are also heavily involved in an international research program towards direct laser acceleration using 2 m light, called Accelerator on a Chip Program (ACHIP), and is funded through the Betty and Gordon Moore foundation. The high field strength and field gradients enable direct generation of very short bunches in compact devices. The talk will highlight the promises, progress and challenges of these projects.