Always after the engines start x plane crash to Desktop

[cwjohan]

I removed SilverLining from the plugins folder and added xjet back to the plugins folder -- resulted in no CTD -- flight from Yakima to Spokane went fine.

Log.txt

GizmoLog.txt

[Ben Russell]

I've added some code to gizmo to report more info about memory usage when it encounters this sort of error. 

Hopefully be able to release a new beta update soon. 

This is a relatively new bug and I'm not sure what the cause is yet. 

[Goran_M]

Disregard

Edited February 3, 2020 by Goran_M

[JGregory]

What you are experiencing is the X-Plane shared Lua memory pool running out.  The logs are reporting which program was running at the time, but it does NOT mean that program is the culprit.  Gizmo has been tested extensively and it has never used (or abused) the Lua memory pool.  So, even though the last two (2) logs you posted show Gizmo reporting the error, it is not likely that Gizmo is causing the problem. 

You previously indicated the following...

On 11/17/2019 at 2:53 AM, cwjohan said:

After much testing of my stripped down copy of X-Plane with the LES Saab 340A and just Western US scenery, no crashes occurred and I could have all plugins installed -- no problem.

That indicates that Gizmo was not abusing the Lua memory pool and the problem is most likely another plugin.  We also have extensive experience with SkyMaxx so I doubt that is the problem either. 

The only way to narrow this down is go back to your stripped down version of X-Plane (all plugins and custom scenery removed except Gizmo) and then put back the plugins and custom scenery one-by-one and run the sim again until you find the offending plugin.

[cwjohan]

Here are the basic facts and how I see the problem based on more than 40 years software development experience (at EA, among others):
1.) The Saab 340A is one of my favorites but is the ONLY aircraft with which I experience the CTD in question.  This tells me that somehow the Saab 340A does something that results in a crash (not necessarily anything wrong).  And it tells me that the Saab 340A does something that none of my other aircraft do.  We have to ask "What is that something?"
2.) With the basic install of X-Plane and no ortho and no custom scenery, I can have any plugins as I like -- none of them by itself or in combination with others brings down the Saab 340A.  This tells me that the CTD is not entirely the fault of any plugin nor is it entirely the fault of the Saab 340A.
3.) The CTD only happens with  my FULL X-Plane setup with a large amount of aircraft, scenery, ortho, and an extensive set of plugins.  This tells me that the problem likely is memory related.  X-Plane is memory constrained when fully loaded like my full setup.  Each aircraft, item of scenery, and plugin uses some additional memory just by being present in the configuration, even if it is not directly being used.  Loaded ortho tiles also take up additional memory.  A few memory error messages in the logs confirm memory usage likely is part of the problem.
4.)  The CTD reliably happens without doing anything with the Saab 340A.  With my full setup, I can load it cold & dark at PGUM in Guam, where there is no ortho within thousands of miles, and just leave X-Plane running and take no further action and within 12 to 15 minutes it will crash.  This is typical of a memory problem.
5.) I have 32 GB of RAM and the Task Manager doesn't show anywhere near all the memory used up before the crash and doesn't show memory gradually being used up more and more the longer X-Plane runs.  This indicates that the problem is not a typical memory leak of the sort caused by not freeing up some frequent memory allocation.

CONCLUSIONS:
1.)  I see no evidence that any plugin is doing anything wrong other than using up some valuable memory space, pushing X-Plane or one of its drivers ever closer to running up against some limitation (a memory pool size?).
2.)  I see no evidence that the Saab 340A has an ordinary memory leak.  However, it (or possibly Gizmo) does something after 12 to 15 minutes that results in a crash and it something contingent only on the elapsed time -- the user need do nothing specific to cause the crash.  That "something" likely is grabbing a big chunk of memory and it likely is a bigger chunk than any of my other aircraft ever grab.  The fact that some part of X-Plane or the Saab 340A runs out of memory after a relatively constant amount of time suggests that the alleged memory grab is driven by a timer or possibly some sort of countdown.
3.)  I can conclude that no plugin is responsible for this alleged memory grab since, when starting up or flying a different aircraft, I never experience the CTD.  If a plugin were responsible, we would see the crash with one or more other aircraft, but we don't. 
4.)  I suspect that removing SilverLining helps only because it uses a lot more memory than most other plugins (speculation on my part), not because SilverLining has any particular fault.  I feel that experimenting with adding and removing various plugins is just a game of "wack a mole" since the critical issue is how much memory is being used, not some fault with any particular plugin.
5.)  It is not likely that the Saab 340A simply requires more memory than any of my other aircraft.  I have several others that are big and quite complex (A319, A330, B777, TBM 900).

So, if there's no timer-driven process in the Saab 340A that grabs a very big chunk of memory around 12 to 15 minutes after the aircraft has loaded, I don't know what to think.  Possibly, one of the plugins does that and the Saab 340A is just really big?

If you see a flaw in my reasoning, please point it out.  There could be some possibility I'm overlooking -- but what?

Cheers,
Craig

[cwjohan]

Here's an example of the Saab 340A loaded in PGUM Guam and just left to run in cold and dark state.  There is no ortho within thousands of miles of this place.  My only action was to click on the X-Aviation licensing window to close it.  After about 17 minutes, it hung.  Three log messages indicate an out of memory situation.  The Task Manager never showed RAM or VRAM exceeding about 30%.  I have 32 GB of RAM and 8 GB of VRAM.

G64: 611.008: Memory Allocation Error: Run(gfx): draw_SaabOnDrawBeforeGauges3D: not enough memory
G64: 611.008: Memory Allocation Error: Run(gfx): draw_SAAB_PLII_COM1_Panel: not enough memory
G64: 611.008: Memory Allocation Error: Run(gfx): draw_SAAB_PLII_COM2_Panel: not enough memory

 

Log.txt

GizmoLog.txt

Edited February 5, 2020 by cwjohan

[Ben Russell]

Given your stated computing experience, please consider the following:

X-Plane plugins all share the same memory address space. There is no modern memory protection applied to them. Any single plugin can override or clash with any other plugin, similar to the joy that was Windows 3.x or Mac OS Classic.

Because of this, failures often are only present when under "resource pressure" and it may be that a relatively rare plugin or bug causes something else to crash, as it was back in the 90s too.

 

Your last post above containing Memory Allocation Error messages is specific to the Lua JIT ram pool that X-Plane allocates.

The Saab typically uses 10-20 megabytes of this depending on how you have the Garbage Collector tuned via Gizmo.

The total pool available should be* somewhere around 1.6 gigabytes.

Both Skymaxx and the Lua memory pool usage have been stable issues for many years prior to these recent events.

 

You can find more information regarding the Lua JIT memory pool here:

http://hacksoflife.blogspot.com/2012/12/integrating-luajit-with-x-plane-64-bit.html

 

You can see some usage information if you open up the Lua Garabge Collector prefs window in Gizmo itself.

You will be able to see the current Gizmo Lua VM usage as well as total X-Plane Lua pool usage.

If one of these has a leak you should be able to detect it here. If neither is leaking then some other plugin is likely abusing the address space.

There's a button available in the RH toolbar or you can look for it via the Plugins > Gizmo > Windows entry.

( I cannot remember the exact name it's listed under as I've been reworking these accesories for the new XP11 GUI style and Gizmo update to come..)

 

I hope to get an update to Gizmo out in the first quarter of this year that will add additional debug data to the Logs if and when Lua Memory Allocation error occurs.

It's been nearly 2 years since a public update to Gizmo so there's also the potential that the code maintenance over that time may resolve this mystery issue.

 

 

 

 

 

[cwjohan]

18 hours ago, Ben Russell said:

Given your stated computing experience, please consider the following:

X-Plane plugins all share the same memory address space. There is no modern memory protection applied to them. Any single plugin can override or clash with any other plugin, similar to the joy that was Windows 3.x or Mac OS Classic.

Because of this, failures often are only present when under "resource pressure" and it may be that a relatively rare plugin or bug causes something else to crash, as it was back in the 90s too.

 

Your last post above containing Memory Allocation Error messages is specific to the Lua JIT ram pool that X-Plane allocates.

The Saab typically uses 10-20 megabytes of this depending on how you have the Garbage Collector tuned via Gizmo.

The total pool available should be* somewhere around 1.6 gigabytes.

Both Skymaxx and the Lua memory pool usage have been stable issues for many years prior to these recent events.

 

You can find more information regarding the Lua JIT memory pool here:

http://hacksoflife.blogspot.com/2012/12/integrating-luajit-with-x-plane-64-bit.html

 

You can see some usage information if you open up the Lua Garabge Collector prefs window in Gizmo itself.

You will be able to see the current Gizmo Lua VM usage as well as total X-Plane Lua pool usage.

If one of these has a leak you should be able to detect it here. If neither is leaking then some other plugin is likely abusing the address space.

There's a button available in the RH toolbar or you can look for it via the Plugins > Gizmo > Windows entry.

( I cannot remember the exact name it's listed under as I've been reworking these accesories for the new XP11 GUI style and Gizmo update to come..)

 

I hope to get an update to Gizmo out in the first quarter of this year that will add additional debug data to the Logs if and when Lua Memory Allocation error occurs.

It's been nearly 2 years since a public update to Gizmo so there's also the potential that the code maintenance over that time may resolve this mystery issue.

 

 

 

 

 

Thanks for the input, Ben.  Very informative.  Pretty esoteric stuff, especially with the use of offset addresses that have to be in the first 2 GB of address space!  Someone probably thought no one ever would need more than 2 GB for this type of memory pool, and they thought they were being very clever by using just 32-bit addresses to save space in the memory pool block chains.

Is it possible that, in my case with the Saab 340A, the Garbage Collector is running out of memory because the instrument panel drawing code is doing nothing to de-reference some objects no longer needed (e.g., re-creating them on each drawing cycle and never setting to null the reference pointers to the old ones)?  The code might get away with it when memory usage is not under stress, but fail when it is stressed?

Looking forward to your coming update of Gizmo to provide more diagnostic info. 

[cwjohan]

Another experiment.

From my full setup, I removed:
12 airports
AutoGate
AviTab
Little Xpconnect
SAM
ToLissData
ToLissFlightPlans
ToLissTerrainData
Xchecklist
xjet
XPUIPC
NavigraphSimlink_64.xpl

Previously, I had removed:
124thATC64
DataRefEditor
EZPushback
FlyWithLua core edition
HeadShake
librain
SkunkCraftsUpdater
TerrainRadar
XFMC
xreloaded
XSquawkBox

Plugins remaining in the plugins folder:
Gizmo64.plugin
PluginAdmin
RealWeatherConnector
SilverLining
XPLM.framework
XPWidgets.framework

Result:
Started up X-Plane and loaded the Saab 340A at CYLW airport and let it run cold & dark with no further action.  After 12 minutes, I got a CTD.

Log.txt

[JGregory]

1 hour ago, cwjohan said:

Another experiment.

From my full setup, I removed:
12 airports
AutoGate
AviTab
Little Xpconnect
SAM
ToLissData
ToLissFlightPlans
ToLissTerrainData
Xchecklist
xjet
XPUIPC
NavigraphSimlink_64.xpl

Previously, I had removed:
124thATC64
DataRefEditor
EZPushback
FlyWithLua core edition
HeadShake
librain
SkunkCraftsUpdater
TerrainRadar
XFMC
xreloaded
XSquawkBox

Plugins remaining in the plugins folder:
Gizmo64.plugin
PluginAdmin
RealWeatherConnector
SilverLining
XPLM.framework
XPWidgets.framework

Result:
Started up X-Plane and loaded the Saab 340A at CYLW airport and let it run cold & dark with no further action.  After 12 minutes, I got a CTD.

Log.txt

You still have a LOT of scenery files that are loading their own plugins.

[cwjohan]

Followup to previous experiment removing almost all 3d party plugins from the plugins folder:

Removed:
RealWeatherConnector
SilverLining

Now all 3d party plugins have been removed from the plugins folder.

Started up X-Plane and loaded the Saab 340A at CYLW airport and let it run cold & dark with no further action.  After letting it run several hours, I had gotten no CTD.  I then went through the standard manual start up procedure and flew IFR from CYLW Kelowna to KGEG Spokane with no problems other than that, throughout the flight prep and the flight, a hydraulics gauge wasn't working, though the hydraulic system seemed otherwise OK.  I think I have seen hydraulic gauge malfunctions before.

J. Gregory rightly points out that quite a bit of scenery still is loading plugins.  These plugins are:
Marginal.GroundTraffic (57 instances)
sandybarbour.projects.customsbdatarefs004 (1 instance)
Note:  We can see by this experiment that the Saab 340A (like all my other aircraft) has no issues with these latter two plugins -- at least at this level of memory stress.

Cheers,
Craig
 

Log.txt

GizmoLog.txt

Edited February 6, 2020 by cwjohan

[Georg L]

Did you find any solution to this? I'm having pretty much exactly the same issues flying IXEG 737 with SkyMaxx.

[cwjohan]

For my setup, the key to de-stressing the memory seemed to be removal of SilverLining (SkyMaxx).  speedbird1229, do you also have a lot of airports and aircraft installed in X-Plane?  You might also try removing all AI aircraft, since each one uses some memory and generates extra ATC activity.

[josephsevern]

my 2 cents :

definitively not working vulcan

and you MUST absolutely get rid of SkunkCraftUpdate

 

[josephsevern]

and SAM plugging

and ActiveSkyXP too....

This is unaceptable for an addon of that price level, no matter how the bird is simulated accurately

[JGregory]

1 hour ago, josephsevern said:

my 2 cents :

definitively not working vulcan

and you MUST absolutely get rid of SkunkCraftUpdate

 

 

55 minutes ago, josephsevern said:

and SAM plugging

and ActiveSkyXP too....

This is unaceptable for an addon of that price level, no matter how the bird is simulated accurately

If you are going to report bugs you MUST attach your Log.txt and GizmoLog.txt files.  

In addition, Vulkan is in beta so we do not make any guarantees of compatibility with the Saab.

[Cameron]

59 minutes ago, josephsevern said:

This is unaceptable for an addon of that price level, no matter how the bird is simulated accurately

Cool it. We aren't going to play arguing games with you here.

[josephsevern]

not working on vulcan is not an issue to me, my PC is good enough to run OpenGL quite well and this is not the point

we all know 11.50 is still beta, all my payware aircrafts run it under OpenGL(FF, toolis, ... what ever) but SAAB  340 is not

I m ready to get ride of some pluging like SAM or SkunkCraftUpDater, no problem with that

But not beeing abble to fly the thing without ActiveskyXP, witch is not beta, is a concerne

You MUST warn futur customers before taking their money.

 

GizmoLog.txt

Log.txt

[Cameron]

4 minutes ago, josephsevern said:

You MUST warn futur customers before taking their money.

You're really going to hate this discussion if you keep it up. You need to STOP running to absolutely dumb conclusions.

Let me break this down for you REALLY quick: We have sold thousands of this product. It is very highly regarded in the community. If this problem were real or even close to dire as you state, you would NOT be essentially alone in complaining such words in here.

There is something wrong on your end, somewhere. This product has been in the wild and actively updated since 2013.

Stop the drama. NOW.

Allow those to help you who are willing and leave the assumptions and attitude at the door.

[JGregory]

9 hours ago, josephsevern said:

not working on vulcan is not an issue to me, my PC is good enough to run OpenGL quite well and this is not the point

we all know 11.50 is still beta, all my payware aircrafts run it under OpenGL(FF, toolis, ... what ever) but SAAB  340 is not

I m ready to get ride of some pluging like SAM or SkunkCraftUpDater, no problem with that

But not beeing abble to fly the thing without ActiveskyXP, witch is not beta, is a concerne

You MUST warn futur customers before taking their money.

 

GizmoLog.txt

Log.txt

Run the Saab in 11.41 and then report back.  We will not diagnose bugs with the Saab running under a beta version of X-Plane.

[bdeniaud]

I have the same problem with out of memory in the last beta of xplane.

I know it's beta but Vulkan is mentioned on the store page in x-aviation so it might be supported, no ?

 

Nothing really relevent in my log files...

 

GizmoLog.txt

Log.txt

[Cameron]

15 minutes ago, bdeniaud said:

I know it's beta but Vulkan is mentioned on the store page in x-aviation so it might be supported, no ?

We have tested and ran Vulkan and Metal and indicated no issue. I personally did extensive testing.

Things change when people have so many plugins and add-ons added to it, and with Vulkan/Metal being SO new Laminar is making rapid changes that will undoubtedly affect harmonious environments. Waiting till after 11.50 is final is definitely the only way to go from our side when it comes to any potential (if needed) changes.

[cwjohan]

4 hours ago, bdeniaud said:

I have the same problem with out of memory in the last beta of xplane.

I know it's beta but Vulkan is mentioned on the store page in x-aviation so it might be supported, no ?

 

Nothing really relevent in my log files...

 

GizmoLog.txt

Log.txt

I see you have Silver Lining and Skunkcrafts Updater.  Try removing those two plugins and see if the problem goes away.

[bdeniaud]

14 hours ago, Cameron said:

We have tested and ran Vulkan and Metal and indicated no issue. I personally did extensive testing.

Things change when people have so many plugins and add-ons added to it, and with Vulkan/Metal being SO new Laminar is making rapid changes that will undoubtedly affect harmonious environments. Waiting till after 11.50 is final is definitely the only way to go from our side when it comes to any potential (if needed) changes.

Of course... I will wait for final version and in the meantime try without skymaxx and skunkcraft.

BTW, do you know what these error means in the gizmolog ?

Quote

error:   90.911: Run(gfx): draw_SaabOnDrawBeforeGauges3D: [string "JGX.1.Saab.00.lua.ra1"]:32806: font.drawString(..): Error: Glyph atlas not ready. Try moving your code to OnBoot() or later.
error:   90.918: Run(gfx): draw_SaabOnDrawGauges3D: [string "JGX.1.Saab.00.lua.ra1"]:35554: font.drawString(..): Error: Glyph atlas not ready. Try moving your code to OnBoot() or later.

edit : tested a 30min flight with skymaxx and skunkcraft disabled without a crash

Edited May 11, 2020 by bdeniaud

[cwjohan]

11 hours ago, bdeniaud said:

edit : tested a 30min flight with skymaxx and skunkcraft disabled without a crash

Cool. Glad to hear that not running those plugins worked for you.  But did you actually remove them from the plugins folder or just disable them in the Plugins menu in X-Plane?