Hi,
with so limited VRAM of 1GB and all your addon setup, the graphic card has to decide which textures to keep in VRAM and which to transfer to RAM temporarily . Once you change your view, the textures actually stored in RAM needs to be transfered into VRAM, and this eats a lot of fps for a short moment, especially if there are no textures available to move to RAM, as all are in use.
50mb free VRAM is totally normal, as the graphic cards needs some space to juggle with textures. If you use something like GPUZ, then you can see that if the VRAM hits the limit, it will instantly drop some content to free a minimum of VRAM space.
You will have loaded a lot more then 1GB of textures (inclusive processing while rendering - Anti aliasing and others), but till some point you are lucky, that only 1GB are in use at the same time. If this is no longer the case, and more then 1GB is needed, fps drops rapidly.
Best,
Michael