Increased sample rate and background calculations

There is a thing I don’t fully have intuition for.
When changing SR from 48kHz to 96kHz, will things happening in the background (not in audio interrupt) take longer to calculate?

In my case, I’m doing some large FFT and some extra things outside of audio callback. When I changed SR to 96kHz, I also reduced the FFT size, so memory consumption and calculations are also reduced, but it still takes longer to calculate everything compared with 48kHz (but with more heavy calculations).

My assumption is that switching to the higher SR makes some high-priority things take more resources so low-priority things take longer. Is that true?

Yes, your assumption is correct - doubling the work of sampling lowers the number of CPU cycles available for background tasks. + more switching overhead.