Pitchshifting algorithm uses 2 interpolated delay lines, so it introduces delay by design. You can change its amount here:
I would expect shorter delay buffer sizes to add more artifacts, but current value is surprisingly big - I think it would give you 340ms latency.
That PitchShifter class would be more usable if it was a template class with delay size specified as parameter - that wouldn’t require editing library sources to change it in your patch.