Caching into SRAM sounds good, is there DMA controller access doing that? It sounds a bit like the good old PS3 Cell processors SPUs, creating a software cache.
For my granular synthesizer on Seed, I load up to 64MB worth of WAV samples in SDRAM, and stream up to 20 parallel grains (playing any of the wav files), which seems sufficient. I had to optimize a bit, avoiding memory copies etc. Running out of CV’s / controls is an issue, will use the screen to enable ‘banks’ of controls (or use MIDI instead).