The problem with putting everything in SDRAM is not its size, but bandwidth. It becomes a bottleneck if used too intensively (i.e. in something like reverb with lots of delay lines… and bats!)
So being able to read from QSPI may work better in some cases, I would expect it to work faster if you need random IO that has lots of cache misses. Ideally you would want to avoid both and preload only data that is going to be used for each audio block to internal SRAM in advance.