Experiments demonstrated that plasma-based accelerators can produce high energy electrons (e.g., 8 and 42GeV) in short distances (20 and 85cm, respectively). They are very high gradient accelerators (i.e., 50GeV/m) and therefore hold the promise to be more compact and affordable than RF-based linear accelerators. An alternative path investigates the use of dielectric accelerators with high accelerating fields. The challenge for these new accelerators driven by a laser pulse or a relativistic particle bunch is to produce beams with parameters that would make them competitive in terms of luminosity and efficiency. In follow-up to the European strategy of particle physics, an Expert Panel for High Gradient Plasma and Laser accelerators has recently been set up. We present specificities of these accelerators, and discuss various inputs received on how to meet the challenges posed by various HEP applications.