Configuration is volatile and FPGAs lose it whenever you turn power off.
If you don't want to load configuration every time with USB programmer (and usually you can't do it when the system has been deployed on the field) you need a non-volatile memory connected to fpga. This can be a serial or parallel flash memory.
The common solution is using epcs serial flash devices supplied by Altera itself