XP Scenery Generator [WIP]

[tonywob]

This will make generating orthophoto scenery so cool... imagine this plus RealScenery Oahu....

 

All we need now is all those Hawaiian styled buildings creating :-) 

[tonywob]

A couple more screenshots of the collision detection in action:

 

 

 

 

 

 

The collision detection surprisingly is pretty quick, and has managed to place trees inside courtyards etc. Although at the moment it's placing real object trees, and  it would probably be more efficient to place individual trees using .for files, as I'm sure performance is going to suffer. I believe .for files can be placed using individual points, so this maybe worth looking at. I'm sure Andras will correct me, but I'm pretty sure it's possible to ask X-Plane to place a single tree from a .for file using just a single point.

 

Additionally, the rule can be used to place random objects in a park such as benches (if you wanted to go to such extreme levels of detail).

 

Also kudos to the new Skymaxx Pro update, which looks fantastic with this scenery :-)

Edited February 9, 2014 by tonywob

[alpilotx]

Hi Tony,

 

Well, now you are going a bit too far with FOR forests (and I never told, that trees should ONLY be placed as forests  ) ... if you only need a single object (or two), then there will be no difference if you place it as an object (as long as the object is a similarly simple one like the billboarded forest stuff). For a single (or two) tress, it might be even a tiny bit more expensive to use the forest mechanism!

 

So, as a rule of thumb:

• if you need an area filled with lots of trees (well, at least more than lets say 5-10), or need a line of trees, or have a long line string (with many segments) where you want trees at the vertices: then use the FOR approach
• if you want single (or very few) trees placed at a given location, then use objects ....

I am doing the same in my NZ Pro scenery. For real forests and tree lines, i use the FOR approach. For the single trees (the raw data has a few million such data points in New Zealand!) I use a randomized set of objects placed at the right location. ==> always using the most reasonable approach.

[tonywob]

Thank you Andras for you advice as always 

 

Here's the issue, when placing in residential areas, there are lots of trees. For example in area which is 100m2, there maybe upto 40 trees. In my initial tests with the scenery, it's not too bad, and really adds to the look of residential areas. But for larger residential areas, I can see a potential problem.

 

There are some pretty huge residential areas which I'm worried will kill DSFTool again, or kill performance.

 

Someone has suggested I create one big "residential" forest and create inner polygons for groups of houses, but I'm not sure whether this will make things worse or not.  I'm trying to mimic ORBX's scenery, where they place a couple of trees around a house, and although I've produced a similar effect, I don't want to make the simulator unusuable :-)

Edited February 9, 2014 by tonywob

[alpilotx]

Well, which ever way you choose, you will have to "pay a price" ... at some point it will be irrelevant, if you have 1.000.000 tiny forest polygons or 1.000.000 single objects ... OR, maybe in this case, the objects might even win ... as it might be more expensive to have those million polygons (while each of them still only contributes 1-2 trees!) than a million single objects ... At least the DSF size would suffer, and the DSF laod time too (while X-plane extrudes the trees from all forest polys). But in the end, when everything is loaded, the overall amount of objects will decide (and then its almost irrelevant where they come from) ... so, with "too many" objects, you will always be able to bring FPS to its knees ... Thats why we (and you here) already tell potential 3D building developers to make their stuff as easy as possible (because the simpler, the cheaper ... when it comes to FPS).

[Leporello]

Yes, maybe the key should be to decrease object level the more you go outside heavy residential area. No clue though if such a tag exist in OSM. For example in large cities you would set lot of single trees to "too many", in small cities you can set the major part to "tons". Maybe you could resort to an external database to set coordinates of large city. Bear with me if it sounds silly or not doable, the fact is that cities with trees are much more realistic for what I've seen above.

 

What can look a FPS killer now might prove irrelevant in one year.

Edited February 10, 2014 by Leporello

[tonywob]

Yes, maybe the key should be to decrease object level the more you go outside heavy residential area. No clue though if such a tag exist in OSM. For example in large cities you would set lot of single trees to "too many", in small cities you can set the major part to "tons". Maybe you could resort to an external database to set coordinates of large city. Bear with me if it sounds silly or not doable, the fact is that cities with trees are much more realistic for what I've seen above.

 

Actually, you don't need to do anything, it already does this based on density. If there are more buildings inside a small area, then there will be less trees, and vice versa. This is simply a matter of there being no space to put the trees :-)

[Leporello]

I've started some house building, how many tris have your houses average Tonywob?

I was thinking about retexturing JB houses but some have 400+ tris and I wonder how much should be the maximum here?

[tonywob]

I started with doing too many triangles and high res textures, I've now reduced the count below 100, the last one I did had about 80, which still seems awfully high for a simple box house with a roof. I'm slowing getting to grips with how to make simple houses quickly without killing performance. I've added another 6 houses today, each a slight variation on the other.

 

Also, if possible, put as many as you can onto one texture sheet, you can really squeeze a lot of textures onto a 1024x1024 sheet, and they don't need to be high resolution.

 

Thanks for you help

[tonywob]

I've added some regional UK houses, 4 in all, it's amazing how far just 4 models will go, and I'm actually surprised terraced houses worked as well as they did.

 

 

 

 

Also, the zones are working well, as it has quite clearly separated out residential and industrial zones, and knowing this area very well, it also looks very realistic

 

 

 

[PilotBalu]

That is amazing... With few objects you can change the overall look. That's what I also recognized with the 'Europe Library'.

 

I'm doing just an update of my OSM sceneries at simheaven.com, as OSM data grew by 20 percent. Hopefully it's the last update doing this way.

[tonywob]

Hi guys

 

I've created a git repository for regional models

 

https://github.com/tonywob/world-models

 

Anyone wishing to contribute, please let me know and I'll add editor rights. Additionally, work to be done can be allocated here via the ticketing system. I will open some tickets in the next few days.

[tonywob]

I've added a very basic GUI, so that the generator is easier to use:

 

 

So running the application is simply a matter of double-clicking on a file. All configuration/rules (and there are a lot of them) are edited via the config file, and I'm in the process of writing instructions and also commenting the file so hopefully it's understandable. The log will show you any problems with the data and the IDs, so that the relevant items can be fixed in OSM.

 

Additionally, I've added some extra features which have cleaned up things a bit and helped fix some problems:

 

1. If you specify your X-Plane folder and ask it to validate, it will check that all the models and facades you've included in your config file actually exist (or will be copied over). This saves so many headaches when X-Plane crashes out because something is missing.
   
2. I've updated the object rule so it will take a CSV file of models. The rule will use a best-fit algorithm, which is much better suited for placing object footprints. Previously, you had to have a rule for every object, which was becoming a pain. The best-fit also is doing a better job, and is matching up objects into their footprints better. The CSV files can be viewed in repository I added earlier.
   
3. I've added persistence. What this means is that the application can be configured to continue where it left of. e.g. If you have started generating a large country, and stop it halfway through, you will be given the option to resume upon starting the app back up again. Additionally, I've added database support, so that it will be possible to use the application on systems with less RAM (It's slower though).
   

Edited February 15, 2014 by tonywob

[Tom Knudsen]

Excellent progress sir..

[Daikan]

Hi guys

 

I've created a git repository for regional models

 

https://github.com/tonywob/world-models

 

Anyone wishing to contribute, please let me know and I'll add editor rights. Additionally, work to be done can be allocated here via the ticketing system. I will open some tickets in the next few days.

 

 I just committed a first batch of swiss buildings to the repository all of which are residential houses except for one which is a barn. I used the prefix "BAR_" for it, as there was no suggested naming convention for this type of building. I hope that's okay.

 

Curiously, how is regionalization supposed to work? And is there a fallback mechanism in the sense that regional objects are always prioritized over global/generic objets unless there is a better match?

[tonywob]

Thank you Daikan for the first batch of bulidings, it's great to see some new models appearing from other developers

 

If you have a look in the directory, there are currently 3 CSV files containing agriculture, industrial, and residential buildings. So, the name isn't too important, as long as it can be identified. As we will use the CSV files inside the best-fit rule to place the buildings.

 

In regards to regionalization, I'm not 100% sure which method is best, and I'm open to ideas on what people prefer. There are currently a few options:

 

1. In the config file, it's possible to specify a lon/lat range along with a rule, e.g. For your Swiss buildings, we could add a rule to only show these items in a specific box. The problem here is that it's a box, and countries aren't boxes, so you'd have overspill into France, Italy, etc.
2. In the CSV file, you can add a region code (or whatever code you like), and when you run the generator, you can tell it (similar to step 1), where the region code is active. Currently this is again limited to a box.

I think we need to able to define a region with a polygon, possibly using a really basic ESRI shape file, or a simple txt file with a list of points. So, for example, we could use a low resolution shape file for Switzerland (The simpler the better), and tell it that it has the region code of CH, and then the rule will check if the building is inside that region and show the building. This is a fairly trivial thing to implement, but I'm not sure where to grab the regions from (At a push I could grab them from OSM, but I'm sure there are ready made ones about).

 

Let me know your thoughts on this.

 

Also, the system will use a default batch of buildings without regional codes (Currently I use CE for Central Europe) as a fallback. But this is all configurable

Edited February 16, 2014 by tonywob

[Daikan]

It would be great to have a regional hierarchy structure, for example Switzerland -> Central Europe -> World. This would allow objects from a parent region to fill in should they provide a better match than the regional ones.

 

As for the region polygons, you can easily get them as .poly files from Geofabrik, for example, here's the one for Switzerland. They're not exactly simple, but since you're using the JTS library (which I used for the MultiOSM2XP tool as well btw) I'm quite sure there is a simplify() function in there...

[tonywob]

The structure would be hierarchichal, in a sense that the the generator would detect all of the defined regions a particular object is part of, e.g. It would pick up that a house was inside Europe and inside Switzerland, and would use the best fitting building from both. So if a building isn't inside Switzerland, it would use one of from Europe. You could further divide countries up into counties (If one day we have so many buildings :-))

 

 

They're not exactly simple, but since you're using the JTS library (which I used for the MultiOSM2XP tool as well btw) I'm quite sure there is a simplify() function in there...

 

 

I've found this on might meet the purpose just fine, and I have used these before in a project I did for iOS.

 

 

The entire map consists of just countries and their names/codes and includes low resolution boundries. There are higher resolution versions available as well if these prove to be problematic, and of course there are the ones on geofabrik.

 

There is a Douglas-Peucker simplifier algorithm inside JTS, and also a shape preserving simplifier, I found both didn't work too well for buildings, but would work fine for large polygons. 

 

I'm really hoping to get an alpha out soon to a few people for testing (Armin and yourself included). There are still lots of things to do before it can be used to create larger sceneries, e.g:

 

1. Smart Exclusions and boundary exclusions. i.e. Being able to create exclusions that only wrap around areas populated in OSM and also on country borders. I believe Benny added something similar into OSM2XP for the scenery on Simheaven
2. I don't know how it will handle entire continent sceneries. The biggest areas I've tried is the UK and Poland, and it generates fine. When using local storage mode, it takes a while to populate the initial database, but afterwards it runs quite well. Also, the ability to stop the generation and restart it where it left off is very useful. With all the filter options and geometry tests enabled, it runs slower than OSM2XP, but at least an application or computer crash won't mean restarting again. I really want to work on speed, but for now I'm just interested if it will even work
3. Documentation. There is a document I'm updating as I do it with some basic instructions and examples of how to use the config file. But I'm hoping, for the alpha at least, that the config file is pretty much self-documenting. Anyone who can handle a Unix service config file should have no problem working it out
4. Using the polygons discussed area for regions (I'll hopefully get this done today or tomorrow).

I also intend to branch out from OSM once it's generating acceptable sceneries. OSM is great where there is data, but in other areas, e.g. In the UK and Spain, there are large empty areas. The application can read most standard GIS data and formats, so I think using multiple sources is the way to go and make even better scenery for X-Plane (which is the idea

 

 

 Edit: The polygon I've included is too low resolution, as some Islands are missing. I've updated to the 1:50m version which is much better.

Edited February 16, 2014 by tonywob

[alpilotx]

Just a few thoughts:

• First, I don't believe that a "simple" 1 degree spill over at borders is too critical when it comes to building styles. If you travel trough Europe, you will often notice, that specific building styles don't stop at national borders, but that there is always - in any direction (often quite random) - some spill over. This is something you can barely model ... neither do I think its that important.
• So second: why don't stay with 1x1 degree resolution regionalization? And then! Then you don't need to do anything in your scenery generator, but have the freedom to spcify it afterwars! How? Well, with the built in regionalization in the X-Plane Library system. That feature is there since quite a few years, and we are even using it for some of the ground textures.

So, the Library system can indeed use small 360x180 bitmap masks, which will define, ho some art elements will be overridden in a given region (yes, its cool, you don't need to override all ... you can selectively tell which art items have a different representation in a given "region").

 

In the "/Resources/default scenery/1000 world terrain/" folder you can see the examples. In the library.txt there are some "REGION"commands, and they all get their bitmaps from the "maps" subfolder. Then there is a list of TER files, which in that region are different from the "default" set (and every other TER- which is not mentioned - will stay the same). And of course this works with ALL other art stuff which you can define via the library too (its not limited to TERs!).

 

Here is the official description of this:

http://wiki.x-plane.com/Library_File_Format_Specification#REGION_BITMAP_.3Cfile_name.3

 

So, here my recommendation would be, that maybe don't do much of the regionalization in the DSFs! That "hard codes" this information in the scenery, and you can't change it afterwards. Instead try the library regionalization .... This way, arbitrary regions can be introduced later on. You can start with a basic set, and as new art items (for a region) become available, users can start to "overwrite" via region masks the artwork there.

[Daikan]

Of course it would make sense to use the built-in regionalization of the X-Plane library system whenever possible  but I think the 1x1 degree limitation is quite a bit harsh, even considering the spill-overs at country borders. I'm not sure if it would look good (nor how noticable it would be) to see the types of buildings suddenly change along some artificial straight line at a DSF border.... In fact, for a small and diverse country like Switzerland a quite noticable distinction in building architecture is mainly due to a mix of topographical (flatland / mountains) and climate properties (mainly North / South of the alps) and it doesn't even depend much on administrative borders. All in all I fear this would collide with the 1x1 degree limitation  but maybe I just don't understand the advanced possibilities that it provides in order to consider more fine-grained "natural" (topographical / climate) properties for art asset replacement rules...

[alpilotx]

Still ... I would like to remind everybody, that at the current level, we would be all happy if we could at least have a few regions. Thinking about small scale, topography based differentiation (which in reality is absolutely valid and correct) is maybe a bit too early. Thats why I recommended the "easier" and more flexible approach. Which allows some more generic scenery generation, and regionalization via "post-customization" (trough the library system).

 

Just my two cents ... but I like to point out, that maybe we need to learn to walk before we can run.

[tonywob]

If using the regionalisation in the X-Plane library files, then I'd imagine (and excuse my ignorance, since I've only briefly looked at it), that we would have to create equal area buildings for each region, e.g. We'd need a list of 10x10 buildings for France and another set for Switzerland. It wouldn't be possible AFAIK, to have a particular size building appear for one region without specifically adding it. 

 

The advantage of doing this would be autogen, since we could regionalise the autogen to these regions, which once smart exclusions are added, will become very important. Having a smattering of regional buildings in OSM areas and not in autogen areas IMO would look odd.

 

I also agree with Daikan in that a 1x1 limit for countries is a bit harsh. e.g. Belgium would be swallowed up by France, etc.. However, I've made the region identification pluggable, so you can use whatever you like to define what a region is. You could use a simple text file with a list of square boxes, or use an image/shape file to get the regions. However, I'd love to be in a siutation where I'm worried that Belgium is being swallowed up by France

Edited February 16, 2014 by tonywob

[tonywob]

Daikan has kindly added some regional buildings in for Switzerland, so I've generated Switzerland, and enabled the regionalisation from the shape file earlier:

 

 

 

 

 

There are still lots of Eastern European buildings about, as we have lots of them, but it's nice to see initial results with a few houses has worked and the regions using country codes seems to work. The buildings disappear once going over the border into France.

 

Switzerland looks really nice, especially the residential buildings mixed with the trees, and the treelines along the edges of the farms. 

[Daikan]

Very nice! Interesting mix of architecture :-)

 

Given that the "hardcoded" way of regionalization via DSF already works reasonably well I am tempted to say that we should prefer it to the built-in library system at this point, even if that means losing some flexibility with post-processing. I really like the possibility to have fine control over region borders.

 

To be honest, I doubt that there will be high enough demand for the possibility to tweak the region appearance via changes in library assets after the OSM custom scenery has been released. For instance it's probably not something an average user would like or be able to do, so in that sense it will remain part of the job that experienced custom scenery providers like Simheaven do and as such I don't think it would make much of a difference if the regionalization is inherent to the DSF or not. It would perhaps be interesting to hear PilotBalu's opinion on this...

Edited February 16, 2014 by Daikan

[tonywob]

Having had a look at the regional buildings, I may need to tweak the generation a little bit, so that the best-fit algorithm gives a higher score to a building which is in the region than to a generic regional building, so for example, let's say we have the following 2 buildings:

 

- Residential Central Europe 10.0 x 10.0

- Residential Switzerland 10.20 x 10.10

 

Let's presume in OSM that there is a house that is 10.05 by 10.05,   The building using the default (but configurable) tolerance rule of ~0.50 metres, which would match both buildings, however the Central European one would be used as it has the tightest fit. The generator uses a scoring system, as often several buildings will match if the tolerance is around half a meter,  and uses the one that matches the closest (or random if several all match and have the same score). If I increase the score for the regional building, then this would be used instead, and might look better.