XYZ file terrain modeling

@jimhami42

The different properties of the two surfaces are very interesting.

The first thing I noted is the size different, the Trimble Business Center (TBC) generated surface file size is much biggest.

The second thing I noted is the how the triangulation method look different, looking at your code it seen you are using a rasterized data structure, I chose to use the Delaunay triangulation method.

@thomas.engel

There are a number ways do this.

For very large datasets I like to put it into Microsoft Access and then use SQL programming language to query the table. This can easily reduce a dataset from 30 million down to 5 to 6 million with no problems at all.

If I need to reduce spatial resolution, I use Surfer from Golden Software and any number of interpolation method (like Kriging or Approximation / Interpolation Based on Smoothing (ABOS)) another very good interpolation method for very large LiDAR set is the Nearest Neighbour method.

Next I import the dataset into Trimble Business Center (TBC) from Trimble and use the Delaunay triangulation method to create a surface. TBC does everything automatically, the whole process takes about 20 seconds depending on the file size.

Otherwise, you can use Meshlab from Visual Computing Lab (MeshLab is free and open-source software) that can create 3D triangular meshes. The new version of Meshlab is 64-bits and GPU powered, so is very fast.

1 Like

Now that is cool.

Yes. Since the data is already rasterized, it made sense to me :wink:

Thanks everyone for excellent inputs. I have to go step by step and struggle to reduce the wil file in access.
I change the original .xyz file to .txt file and import it in access with the standard function import from .txt and standard space delimited options and get to the following error:

"The method “ExecuteTempImexSpe"for the object"WizHook” failed.

Why is that? - I tried many times, played through all importing options, no success.

How do I get the data in access - how can I solve the above problem?

If needed for reference purpose here the .txt file (but I want to achieve it myself if possible):
https://drive.google.com/file/d/0B359nqbEg_T-ZVF2M29aS3VVSGs/view

What verson of MS Access are you using?

I created a plugin that is compatible with your files. Once you select the file, you can specify the X and Y minimums as well as whether or not to skip the first record (it’s “X Y Z” in your files and should be skipped). It will selectively import the data in the range you specify and create triangles between the grid points. These are placed in a group that you can optionally name.

The plugin is here: https://sites.google.com/site/spirixcode/code/thomas_import.rbz

It should install as any other plugin (tested with 2017 Make) and will show up on the Extensions menu as Thomas Import. I believe it will handle any size of points you give it, but SketchUp itself may complain once you reach a million points or so (i.e., 1000 x 1000). At least that’s where my computer bogs down completely.

Access 2016

And thanks jimhami42 for the file. I will try after having some sleep and report back. Thanks again.

I was able to install the extension in Sktech Up, it shows under extensions > Thomas import. I processed the .txt files under discussion with the plugin, but also after a long time no results were generated. Nor was an error reported. I am sure the plugin works fine and thanks again, but I need to reduce the data as indicated by jimhami42 in access or somewhere else before I process the data with the plugin.

I have access 2016 and get the below error when importing the .txt with the standard function “import from .txt” and standard space delimited options:

The method “ExecuteTempImexSpe” for the object “WizHook” failed.

How can I solve this error and get the big data files rickenbach and wil in access in order I can filter and reduce them there before I run the plugin provided again?

@jimhami42, I would like to understand what the plugin does (can you comment this in the code?). It is an awesome approach to help me solve my problem by making a program, however I kindly ask to understand how it works. From research performed I understand that a .rbz file is a ruby script that can be run in Sketch Up. Thank you.

I was able to import the combined data from both the wil file and the rickenbach file. Until I saw them together, I wasn’t sure why you needed to “merge” them. As @NZSnowman pointed out, this is a lot of data points. The final result is about a million entities and a SketchUp file size of 170 MB. The plugin is not fast … on my computer it took a long long time to complete. Also, note that when it finishes, you will need to zoom extents to see where it was created. The coordinate data is imported as if they were in inches … the data will be somewhere northeast of X = 2,720,920 & Y = 1,256,200.

For testing purposes, you might try running the plugin with a much smaller range of data. For example, 2721335, 1256535 to 2721920,1257100. Also, if you open the Ruby Console window before running the plugin, it will display any errors that might occur.

You had originally asked about the -9999 values. These are used to indicate datum points in the data that are not valid. When imported, they “hang down” like this:

Opening the group and selecting the “tail” with the right-to-left crossing window:

You can then delete the “tail” and zoom extents on the result. Here are plan views of both files with the data imported and exploded over each other:

If you are wanting to learn a little Ruby to assist you in doing this yourself, I’ll be more than happy to help. The RBZ file is a RB text file that has been zipped. The original ZIP extension was simply changed to RBZ. You can change it back to ZIP and unzip the file or download the RB (text) file from the website (RB refers to the RuBy language).

I not sure that the error is, I never get error while importing data. This is how I do it.

  1. After starting MS Access, select ‘File’, ‘New’ and ‘Black database’.
  2. From the menu, select ‘External Data’, and ‘Text File’ import. A new window open ‘Get External Data – Text File’.
  3. Select the location of the file, and select ‘Import the source data into a new table in the current database’.
  4. Select ‘Delimited – characters such as comma or tab separate each field’.
  5. Select ‘Space’ and checked ‘First Row contains field names’.
  6. Make sure each data type is a ‘double’.
  7. Select ‘No primary key’.
  8. Hit ‘Finish’. Should take about 60 sec and then select ‘Close’.

If you get this to work I can show you how I filter.

From time to time, I come across an interesting programming problem and, once I start, I can’t seem to leave well enough alone. This is one of them.

I’ve created a new plugin for you that merges two files constrained by the x/y min/max values you specify. This is done by first creating a two-dimensional array and pre-populating it with points that all have a Z value of -9999. The first file is read into the array and fills whatever points exist within the boundaries specified. The second file is then read into the array and overwrites only those points whose Z value is -9999. The resultant mesh ends up containing the merged values. There is no geo-scaling attempted. Each point is assumed to be in inches … you can scale the X/Y to whatever size is appropriate and then scale the Z height separately to match a known elevation.

It’s worth noting that the same location in the two files may have different Z values. This is due to the fact that each dataset was produced independently and the fact that the vertical resolution may vary slightly. In my algorithm, I use the value from the second file. This could be modified to average the two differing values instead of arbitrarily picking one over the other.

Using this approach, the end result is a SketchUp file that is 76.8 MB in size and contains 1,121,203 entities. My previous approach of merging the two separate imports in SketchUp created a lot of spurious edges due to the mismatch between corresponding Z values between the two files.

The new plugin is available at: https://sites.google.com/site/spirixcode/code/thomas_merge.rbz

I’ve added two additional fields called DeltaX and DeltaY. These are the distances between two consecutive values in the files. For the files you are using, they are spaced two units in each direction. I’ve also pre-populated the dropdown with your min/max values to avoid having to type them in. You will be asked to select the first file and then select the second file. If you have the Ruby Console open, it will display some progress information (as well as any errors that might occur). I’ve also reversed the sense of the polygons created such that they display the white face up (front side) instead of the gray face (back side).

As for commenting the code, I need to know what level of programming expertise you possess. This allows me to better explain what you might need to know about the code.

How beautiful is this! I have different questions about, but am full with work stuff all day. I will get back to you Tonight or tomorrow. Thank you very much for this.

1 Like

The merge script works very fast. I believe I understand the basic approach which is excellent. Questions:

A. Somehow the measures (as already indicated by you) do not work. Although in Model Info (under Window) I see everything is set to Decimal / Meters - how can I make that the points are read in Meters and how can I achieve the Z-values are according to the provided values by the original file? (the Z-values in the original .txt file are correct indications of meters above sea level)

B. What is the best way to have a look at the code? I will not be able to understand more than the basic concept and functionality - I did some Java at university and a lot of VBA (excel) for my masters degree (development of a break even analysis tool for SME’s) - but frankly: I do not have programming expertise. Still I would like to understand what the code does (not just in the result) and dive deeper into the possibilities of Sketch Up.

C. After that I would achieve that I pull the less exact google earth alti model for the wider area (2700000,1265000 to 2755000,1235000) and embed the existing model to the wider area. At the end, I also would like to learn how to add the data of the built environment (houses etc., which I also purchased - but the parcel (only postal delivery possible) did not yet arrive) - this should also be very accurate data but is definitely a different step.

Thanks you in advance for your answers -

Thomas

@NZ Snowman - with your detailed guideline the import of the huge wil file to access works with no error, see printscreen:

What I find very interesting: The only and single difference to my procedures is that I left the Z value as integer and did not change that to double. I did some research and came to conclusion that -9999 can be integer, so I left it.

I still would like to learn how to filter and get rid of the lines not necessary?

Thanks in advance and best regards - Thomas

@thomas.engel

That great news!

The filtering is easy, we can filter, resort the data so it will work with jimhami42 code and export it in one easy step. I will post tomorrow, off to bed now :smiley:

The input data is treated as inches since this is the internal unit of choice for SketchUp. You can scale and relocate the existing import, but this is not as easy as it would normally be due to the size of the geometry and the offsets involved. To make this a little more painless, I added a choice of units to the plugin which will scale the data accordingly. You will need to download and install the new version and re-import your data for the point info to display correctly for you.

[added at 15:52 GMT] I left off a parentheses in the previously uploaded files … this has been corrected in the current file. My bad for not testing one last change …

I use Notepad++ although Windows regular Notepad will work just fine. It’s just a text file. The actual code is called “thomas_merge.rb” and the zipped version is “thomas_merge.rbz”. You can download either from my Spirix site (or find them in your AppData folder under SketchUp).

The ability to import Google Earth terrain is going to change soon. See this thread for more information:

Adding buildings to the imported terrain should be fairly straight-forward if the models are to the same scale.

@jimhami42: Thank you very much. I will study, test and get back to you. Thomas

  1. Open saved MS database.
  2. Then select ‘Create’ on the ribbon tab and the ‘Query Design’ button, this will open a new ribbon tab ‘Query Tools – Design’.
  3. In the ‘Show Table’ window, highlight the table with the raw data within it and click the ‘Add’ button’ and then the ‘Close’ button.
  4. On the ribbon tab select the ‘Make Table’ button. This will create new table with our filtered data.
  5. Enter the name of the new table. Now we can start filtering our data.
  6. At the bottom of the window full in the query table. Field = X; Sort = Ascending; Show = TRUE; Criteria = Between 2720920 And 2721920…

  1. Save the database, and enter query name, then closed query window.
  2. Right-hand click the query and ‘Open’. This will create new table with our filtered data. Select ‘Yes’.

  1. Select ‘Yes’ to paste your data into a new table.

  1. You will new see a new table with our filtered data.
  2. Now you can right-hand click the new table with our filtered data and ‘Export’, ‘Text File’…