Many features of holographic dualities can be captured with discrete, finite-dimensional quantum error-correcting codes. In this talk, I review the state of the field and the previous attempts of building increasingly realistic toy models of holography with such codes, arguing that the central challenge is to reproduce two key features of AdS/CFT: First, to reproduce finite N corrections beyond the semi-classical bulk limit, and second, to associate these codes with critical lattice theories on the boundary, i.e., discretized versions of CFTs. Following this, I introduce our recent proposal of hyperinvariant codes (arXiv:2304.02732v2) that seeks to address both issues, and discuss the general challenges and limitations of stabilizer codes to reproduce features of holography.