I guess I would put it down to plugins, then.
I doubt it's the TBM causing your fps issues. It's been heavily optimized to make use of all your CPU cores.
I'm assuming it's just the VR headset putting a load on the CPU. Take note of the CPU time figure. The lower the figure, the better the performance. If the figure creeps up, it means there's extra load on the CPU. It could be plugins, other Operating System processes running in the background, or any other possibilities.
I don't own a VR headset, so I can't run any tests on that, but from what has been told to me, VR headsets can have a significant impact on performance. One thing I can say for certain is that performance has been further optimized in this update over the last version.
A sure way to tell for certain is to remove all 3rd party plugins and test again. Plugins can put a massive strain on the CPU.