This has always been an issue for me also, I am not a pilot in RL so I really don’t have anything to compare it to but with that said….
Hearing everyone complain about it does give it some merit if only a little as some of these are real pilots.
I do find it hard to believe that everything I have read over the last year and 1/2, forums, magazines, youtube about real life pilots and students, I never hear them complain about keeping the plane on center for takeoffs or landings. I never see them struggle like they do on x plane I never really hear anyone in real life talk about it. If I didn’t know better coming away from x plane I would guess that staying on center line for takeoffs and landings is the single hardest thing to do when becoming a pilot, Not sure. it might be, I just never read, heard or seen videos of it.
I have learned a lot in the last year ½, and it doesn’t matter which plane in x plane I use I still struggle with this today. Lastly I will say that this, I have used a joystick, Logitech rudder pedals and red bird pedals which I use now so this is across multiple pieces of hardware. In the end I’m not sure if it’s me or x plane but I am not buying that this is something that I can’t do. Nothing I can do about it so I just deal with it.
quick edit, I am not blaming the TBM I experience this with all the planes with xwind landings and takeoffs.