I Had the same problem coming off A/P using both my trim wheel and Saitek A/P board. After awhile I gave up even using the A/P, the fix was
A trim wheel .Lua that centers the trim so it doesn't reset back to it's last known setting , will have to see if I can find it I haven't had any problems since.
The TBM is by far the Best A/C created Period !!!!!! I wasted a Ton of money on other Platforms and Aircraft. 1 sim 1 Plane now so Thank you Hot Start !!