Just wanted to give an update.
My issue returned almost immediately after I last posted, so I gave up for a few months, but lately I've been trying every single potential remedy I could find on this forum. I've removed all plugins, scenery, cleared the shadercache folder, tried with a fresh install of X-plane, going through the log every time, but nothing stood out and the issue persisted. Turns out that Rivatuner Statistics Server was the culprit. Turning it off completely or setting detection level to none on the x-plane executable fixed the problem. I still have no idea why it only affected the TBM, but I'm very relieved that I finally get to properly enjoy this airplane.
Hope this helps anybody with weird performance issues and thanks again for this amazing plane.