XP Scenery Generator [WIP]

[Tom Knudsen]

 

but I like to point out, that maybe we need to learn to walk before we can run.

 

Always thought the run cycle was a tad more fun to animate yes.

[PilotBalu]

Given that the "hardcoded" way of regionalization via DSF already works reasonably well I am tempted to say that we should prefer it to the built-in library system at this point, even if that means losing some flexibility with post-processing. I really like the possibility to have fine control over region borders.

It would perhaps be interesting to hear PilotBalu's opinion on this...

 

Sorry for the late answer, I follow the topic when and where I can...

 

For the regionalisation, I prefer as you wrote a library based solution instead of a hardcoded solution. The fact is, that the library solution offers more flexibility to change regions, houses after generation. Even when objects are added later.

 

Let's say we have a standard set of objects for houses, defined by length x width x height and type. This standard set is used during generation of the osm scenery. After finishing you define a region and within this region you define that some houses of the standard set are substituted with typical houses for this region.

 

I've done this with the Europe library, Uwespeed with his European vehicles, it work's quite well and you need just an update of the library.txt to change everything.

[Daikan]

Maybe a stupid question but wouldn't it be possible to achieve the same (or similar) by just using a different set of objects which are referenced by the DSF? Of course leaving any filenames unchanged...

Edited February 17, 2014 by Daikan

[PilotBalu]

That is possible!

[mroe]

Maybe a stupid question but wouldn't it be possible to achieve the same (or similar) by just using a different set of objects which are referenced by the DSF? Of course leaving any filenames unchanged...

+ 1 for lib.

 

1.To make it possible for other packages to use the buildings , it needs the library anyway.  Or you have to copy all objs  and textures to all places  .

(Think about simheavens package-structure , autogen-mods  like Europe-Lib, etc )

2.Objects that have not been created yet  can first obtain a virtual name with a temporary object  assigned .

The real obj  can be added later  without the dsf to recalculate.

IMO , it's impossible to build all necessary types for all footprints all over the world in no-time.

3.The library also allows creating variants of the same types later  .

You can set a ratio for the variants.

4.That not breaks the intention to see a Swiss alpin hut only in the Switz , if it has a specific virtual path .

[tonywob]

The generator could be just told to place "virtual" objects, and then it's possible to just use the X-Plane lib files to manage regions. It's up to the person creating the scenery.

 

The buildings I've currently used are all inside CSV files for each type, i.e. Residential, Industrial, etc, which could very easily been turned into a lib, so other applications could use it. Here are my thoughts on the advantages/disadvantages:

 

Advantages

 

• New objects of existing sizes can be added into regions without rebuilding the area in the generator. 
• The library can be picked up and used in other projects.
• Easier on the generator, as it doesn't need to be aware or regions and will generate scenery faster and is easier to manage (maybe!!)

Disadvantages

 

• You'd need a matching sized object for each region, and different countries have different average building sizes. Just on my small amount of work on UK and PL houses, there is a big difference between the average house size. So if I wanted to add a new house just for the UK and it didn't have a suitable sized "placeholder" in the library file, I'd have to rebuild the DSF files anyway. Maybe this won't be a problem when we have lots of houses.
• Having regions in the generator lets you be very specific, e.g. You could define Berlin City Center as a region, and make all the buildings bigger, use different facades, etc. I'm not sure if/how this is possible with the library files (maybe it is). You can also use third-party regions (e.g. goverment sources) to do some nice stuff in the generator. 

 

Remember though, you can generate the scenery and link to it however you like. The generator just does what you tell it to.

[alpilotx]

I think, in the end Tony is right ... Having his tool being already able to make some - more detailed but "hardcoded" (in the DSF) - rationalization is a nice option which can always be used if someone needs it (whether its needed for a special city center, or an accurate country border matching ... thats up to how the user uses it). But it doesn't stops anybody either from just leaving this option alone, and do the regionalization via the Library system (as described above) ... or for the really mad among us, why not combine both (I will tell you a "secret": we use some kind of mixture like that in the assignment of  some more special TERs to the base mesh too in the Global Scenery / HD Mesh Scenery v2 )

[mroe]

+1  for Tony is right . 

Absolutely nothing to say against this amount of flexibility . An impressive piece of work .

[tonywob]

Just a small update.

 

I've added in smart exclusions, but have only tested them on small areas, such as my favourite Brandenburg. It seems to work fine, and doesn't slow the generation down too much. It works presently like this:

 

1. You define a distance between two buildings, the default is 50m (configurable). If the two buildings are within 50m of each other, an exclusion zone is created around them, and this continues on for all buildings. So that eventually, any buildings which are close together, e.g. Towns/Cities end up in one or more exclusion zones, but the surrounding "empty" areas don't have any exclusions. Exclusion zones for buildings exclude everything, i.e. Buildings/Objects and Forests.
2. Similar to buildings, forests are combined with a higher range of 1km (configurable). The exclusion zone only effects forests, so, it's possible to use this with the HD-Mesh and base scenery, and the HD-Mesh's forests will show through where there are no forests in OSM. This won't effect buildings. I tried this out in Germany, and the transition between the HD mesh and the OSM forest scenery wasn't noticable, accept the OSM forests followed the photo scenery below more precisely.

 

I personally don't like smart exclusions as they slow my machine to a halt and luckily they can be turned off. However, because the default autogen scenery and this scenery use some of the same 3D objects (i.e. Buildings from the Europe Library), the transition is very smooth. Once we have a library of objects for each region, we can update the Europe Library to use these, so it should look good everywhere.

 

I'm gearing up to get an alpha release out to a few people, hopefully this weekend. The application itself will handle smaller files ~500MB without problems that I can see, however larger files take longer to process and require local storage (which is slower than memory). e.g. Without using local storage, Germany which is presently 2GB, causes it to run out of memory quite quickly. This is quite different than OSM2XP, which could handle this as it didn't need to prestore relations, etc. Using local storage/databases is a compromise I have to make, so that larger sceneries can be generated, however I don't see this a major issue, since I'd imagine most simmers will just download scenery made by this application and distruted by others, instead of trying to generating entire countries/continents theirselves.

 

I also intend to enable exclusions just for boundary tiles, so it becomes possible to generate individual countries together. I've no idea how long generating an entire continent like Europe would take, and how much storage space would be needed, but it probably isn't for the faint-of-heart :-D. I tried this once in OSM2XP and it took 6 days, I can only imagine how long it takes Pilotbalu at simheaven.

[PilotBalu]

You're right Tony! Last year, it took quite a long time to generate the osm continents, 3 - 4 weeks of calculation and lot of errors to correct as file are really huge. I made continents to avoid the problem with overlapping tiles at the border of countries. Now with 12 GB of osm data for Europe -> I think it's nearly impossible to generate in an adequate time.

 

Then I spend some weeks to speed up the process and generate all continents automatically - I do this by a pre-processing of osm data and parallel tasks. Now it's a 3 days calculation in total and only 4 errors at the multipolygon part to correct manually.

 

So, I would appreciate if your new osm tool could run in a batch file.

[tonywob]

So, I would appreciate if your new osm tool could run in a batch file.

 

This application works a little differently, in that when running in local storage mode, it will precreate a DB it can use containing the relation and node information. This process can take a while, it took 20 minutes to create just the initial DB for Germany.

 

Once the initial DB is done. you can then resume the application, or run it on multiple computers or processes with the same DB. You'd have to pass each process a list of tile coordinates and sort this one out for yourself ;-)

 

So you can get an idea of how long it will take in its current state. For Germany which is 1.9GB, and with all options, i.e. Smart Exclusions, Area Calculations, Random Trees, Buildings, etc, it takes

• 20 minutes to build the initial database, after this point you have resume support.
• Depending on how much is in each tile, it takes about 6-7 minutes per tile, and there are 70 of them. So it would probably take 8 hours to do Germany, with all the bells and whistles enabled.

The bottleneck is the actual PBF file. The bigger the file is, the longer it takes per tile. I'm sure you're aware that using osmfilter to filter out all the stuff you don't want, e.g. Addresses, and other tags can speed it up somewhat.

 

Although the application will probably simplify all the work you need to do to get it generating, it will require patience and disk space if you want to generate entire continents in one go ;-)

 

I'll create a very simple command-line interface for you, so you can run it from batch files, and enable it so that you pass it a simple text file containing coordinates you wish to generate. This way you could split the load between different computers, or if you have a beast of a computer, different processes.

 

In other tests, it took me around 2 hours to generate the UK, and about 20 minutes to generate Switzerland. So I think parallel processing of countries is the way to go.

 

Edit: My preference would be to use a prepared database, e.g. inside PostGIS using osm2pgsql (http://wiki.openstreetmap.org/wiki/Osm2pgsql). This would save lots of hassle of managing/creating databases using the application, and would also mean that the application wouldn't have to run through the file each time it processes a tile, but instead would grab everything from a database. Spatial queries are pretty quick, so I'd imagine it would be faster than parsing the file on each tile, and would enable you to generate the entire planet if you had the disk-space and time . I'll be adding this in after initial release, as I will be expanding the application to use more than OSM as a datasource. I'd keep the old file based inside as well, but anyone who can setup a Linux machine should have no problems install PostGIS and converting the databases theirselves.

Edited February 19, 2014 by tonywob

[AngeloM]

I'm gearing up to get an alpha release out to a few people, hopefully this weekend. The application itself will handle smaller files ~500MB without problems that I can see, however larger files take longer to process and require local storage (which is slower than memory). e.g. Without using local storage, Germany which is presently 2GB, causes it to run out of memory quite quickly. 

 

Tony, without repeating what others said before about your stunning job (my hat is off to you!), I'm wondering about what you said regarding memory: working on Germany on a system with, let's say, 16Gb ram, do you think there will be still need of disk storage for temporary files (assuming that temp files would be used during the generation)?

The use of local storage is a choice done by using one (or more) parameter(s) or it will be managed by the generator in an automatic way?

Many thanks in advance!

 

Angelo

 

[tonywob]

The use of local storage is a choice done by using one (or more) parameter(s) or it will be managed by the generator in an automatic way?

 

At the moment it's a tick box in the GUI which says "Use Local Storage". By default it's unticked, which means it will use memory. If you receive an out of memory exception and you can't give the application any more RAM, then the only option is to tick this box and start the generator again. I assigned my application 4GB of RAM, and Germany easily used this up, although I was able to generate the UK, which is about 600MB, using just 2GB of RAM. If you have a 64-Bit computer and 16GB of RAM, and you give the application, let's say 8GB, it will probably handle it fine.

 

Just to summarise, there are two run modes

 

Use Local Storage.

 

This will preprocess the OSM file into a local database. Once the initial processing has finished, it runs just a little slower than memory mode for each tile. In this mode, if you stop and restart the generator you don't need to recreate the database. So you could use this mode to generate some tiles, test them and then continue generating.

 

If you add or a change a new rule, you will need to start all over again. However you can add a new building into a building set (which is just a CSV file) and resume from there. You could, e.g. Add some new buildings for Switzerland, and then just regenerate Switzerland from the entire Europe continent PBF file.

 

Memory

 

This doesn't require any local storage and stores the information needed in RAM. It's quicker than local storage, but because the database isn't persisted to the disk, if you stop/restart you will need to rerun the initial scan of the file. You can still use resume like in local disk storage, but starting back up will be noticabally slower.

Edited February 19, 2014 by tonywob

[alpilotx]

but anyone who can setup a Linux machine should have no problems install PostGIS and converting the databases theirselves.

... well, yes, a little bit of tuning is always necessary, otherwise you might quickly end up with a PostGIS DB running slower that a turtle ... its always necessary to tune it to hardware. And as I already told you, with large regions (or full Planet PBF) a big SSD (with around 300-400 GByte free space) is an absolute must (otherwise it will never finish importing the data).

[tonywob]

And as I already told you, with large regions (or full Planet PBF) a big SSD (with around 300-400 GByte free space) is an absolute must (otherwise it will never finish importing the data).

 

The OSM PBF files are very well compressed, they use lots of tricks like packing numbers together, lookup tables. Even my local storage option uses more storage than the original PBF file. I do a few tricks like packing coordinates, and switching datatypes in my objects based on size, but the problem I have is my indexes, which take up most of the space (but are badly needed). I don't support OSM XML files, this format is just too slow and it's easy enough to convert such a file to PBF using osmconvert.

 

I'm not sure why anyone would want to import a full planet file, as generating smaller countries is a far quicker and more sane option. The PostGIS option is something I will look at as I want to have more data than just OSM, and I don't want to handle the data storage myself. There is also H2 embedded, but it's slow and falls over when the database gets over about 2GB.

[alpilotx]

Yes, of course ... in your situation most people won't need the full planet PBFs ... I just referred to that as the extreme (which I do need to import every now or then). But my experience with even loading a full Planet export shows, that PostGIS is definitely a viable (and not too slow option). But of course, you will always end up with a much larger DB than the compressed export file ... the indexes alone need lots of space (but as you already pointed out ... you don't want to work without them :-) ). Still, I think anyone working with larger regions in PostGIS should strongly consider using an SSD (especially as they become really affordable nowadays).

[AngeloM]

At the moment it's a tick box in the GUI which says "Use Local Storage". By default it's unticked, which means it will use memory. If you receive an out of memory exception and you can't give the application any more RAM, then the only option is to tick this box and start the generator again. I assigned my application 4GB of RAM, and Germany easily used this up, although I was able to generate the UK, which is about 600MB, using just 2GB of RAM. If you have a 64-Bit computer and 16GB of RAM, and you give the application, let's say 8GB, it will probably handle it fine.

 

Just to summarise, there are two run modes...

Thank you very much Tony for the informations you provided, can't wait to have it when available!

Many thanks again!

 

Angelo

 

[tonywob]

A couple of screenshots generated in the Netherlands around Amsterdam. This is an incredibly detailed area inside OSM, and the results show. People have mapped forest types, and correctly tagged landuse, forest types and even the height of almost every building. This just shows the potential of using OSM data inside X-Plane :-)

 

 

 

 

 

 

 

 

 

[Patrick G.]

that's just awesome

Gesendet von meinem GT-I9505 mit Tapatalk

[AngeloM]

stunning, simply stunning.

Speechless.

[skiflyzander]

Holy wah. That is unreal. If we can expect the same results throughout Europe, we've got a really special tool on our hands. Time for me to get to work on my home town osm data keep up the great work!

Sent from my SCH-I545 using Tapatalk

[tonywob]

If we can expect the same results throughout Europe

 

Unfortunately not all of Europe is a good as the data in the Netherlands. Germany and France are also pretty well populated, but Spain, Italy and the UK are somewhat behind. Where there are a lack of buildings/forests, the autogen from X-Plane can used (with the regional objects from the library). Of course, this situation is improving literally by the minute http://live.openstreetmap.fr/ :-). Europe alone has grown by a few gigabytes in just under a year :-)

[AngeloM]

Here you can see some screenshots about Milan (Italy), not too much data as you can see.

Finland was generated too, I've still to deep test it then I'll post some screenshot.

 

Of course, this situation is improving literally by the minute http://live.openstreetmap.fr/

Impressive!

Edited February 24, 2014 by AngeloM

[tonywob]

Thank you Angelo

 

It looks quite bare here, lots of empty space, unfortunately such is the case with OSM, you have good areas and bad areas. We also badly need some Spanish and Italian style buildings

[tonywob]

A call out for help! . If there are any Linux users out there who can dedicate some time to testing the application on that platform, then please let me know via a PM.