New tools and products are emerging in the LiDAR, photogrammetry and satellite imaging fields today as 3D environments and 3D city modeling gain wider acceptance. These are impacting work related to digital surfaces, object extraction – recognition and leading to additional research and applications. Sweden based Håkan Wiman has been centrally involved in this work including the development of Metria CityModeler and WiMap BM (building modeler). V1 Magazine editor Jeff Thurston interviewed him with respect to his work related to 3D city modeling, visualisation, LiDAR and digital surface modeling.
V1 Magazine: You have several initiatives going at the same time – Revisitor for building modeling using Lidar data and the new Metria CityModeler 5.0. Let’s start at the beginning. How did you initially become interested in photogrammetry, city modeling and 3D environments?
HW: My first contact with photogrammetry was when I studied land surveying at the Royal Institute of Technology, KTH, in Stockholm. I was fascinated of viewing 3D models stereoscopically. I guess many share my fascination after viewing the Avatar movie in 3D. I soon became interested in digital image analysis and 3D modeling as well. I did my PhD 1992-97 on image segmentation and matching of aerial images for 3D description of buildings.
The thesis pathed the way for my first company, StrateGIS. I developed a software for automatic height estimation of buildings, using 2D building outlines (polygons) and aerial images as input. It was used for cell planning in telecommunications. From 1998 and the following 10 years, I was responsible for image analysis and visualization at Readsoft, a Sweden based company developing software for document understanding. That is where I really learned programming. Although interpreting documents does not include any photogrammetry, I also learned a lot about image analysis and machine learning at Readsoft.
V1 Magazine: Most of your work involves digital surface models (DSM). I’ve read about your work with stereo photogrammetry, but also LIDAR data. Can you explain where DSM fit into your work and why you select different sources for the surface data?
HW: DSM:s are used for a number of different applications, e.g. to compute aspect, slope, masses etc. in a GIS. For me, however, the primary use of DSM:s is to approximate buildings and its surroundings. Clearly, the quality of the DSM depends on the source. Buildings tend to be pretty bumpy and rough in DSM:s generated from stereo satellite scenes, but very detailed and accurate when using LiDAR data. I work on both ends of the scale and anything between. Availability and cost determines which source to use.
V1 Magazine: You are also interested in CityGML. How does that fit into your projects?
HW: CityGML is rich in both geometric and semantic description. CityGML fits perfectly in ReVisitors business idea – to create software for 3D city model generation. I think the bottle neck is how to preserve all the information in a CityGML file in existing data bases. The times when city models were used only once for visualization of some planned construction project and then thrown away are gone. The 3D city models need to be persistent just like the 2D maps. My work on CityGML was partly driven by ViSuCity, a joint reseach project for sustainable city planning.
V1 Magazine: The Metria City Modeler creates city models from very high resolution satellite imagery. Is there a preference for the kind of imagery? I’m assuming that features are extracted from the satellite imagery, is that correct? What about textures?
HW: The greatest advantage with Very High Resolution (VHR) satellite scenes is the availability. There may be problems to find cloud free scenes, but apart from that, you can quickly retrieve scenes from any part of the world. There are large amounts of archive data available today. Another advantage is that the scenes cover quite large areas. The geometric resolution does not match aerial images, but is sufficient for most mapping purposes. Stereo pairs are however rather expensive to order from the vendors.
Therefore, Metria CityModeler has the capability to create stereo pairs from different VHR satellite images. When two images from different satellites or times have been included to a project, it may be impossible to make the measuring mark in one image to coincide with the measuring mark in the other image by only changing the height of the object. An adjustment of the sensor model is required.
This model adjustment can be made in two different ways in Metria CityModeler: (i) by manually measuring the discrepancy in one or two points or (ii) by automatically adjust the rational polynomial parameters of one of the images. (i) is usually sufficient to create a stereo model. (ii) uses a more advanced adjustment and includes matching of previously measured objects as well as tie points. In addition to these two model adjustments, Metria CityModeler can also be used to translate the sensor model in object space. This is often required to make the measurements fit in an exterior system, e.g. Google Earth, because the absolute accuracy of the sensor parameters is much worse than the relative accuracy. Metria CityModeler does not extract any textures for the buildings from the satellite scenes. Only the geometry of the buildings is exported.
V1 Magazine: Is Lidar information being combined with multi-spectral data in any of your projects? Do you see this integration as a possibility and what benefits would be possible?
HW: Currently, I do not include any images when using LiDAR data for 3D building reconstruction. Clearly, the combination of LiDAR and images has advantages over either data source alone. The geometric accuracy of LiDAR point clouds is better than what can be accomplished by image matching, particularily in the Z dimension. The radiometric information of images, particularily when including Near Infrared (NIR), is on the other hand useful for classification and object reconstruction and for edge detection.
However, I do not want my software to require both LiDAR and image data to create city models, because there may be a large additional cost to acquire both sets of data. Therefore, I am currently finalizing WiMap BM (BuildingModeler) so that it automatically creates 3D buildings from LiDAR data alone. Eventually, I may integrate LiDAR and images for building modeling in WiMap BM.
V1 Magazine: One of the unique features to Metria CityModeler is the ability to generate terrain models automatically. Can you explain how this process works? Are there advantages for automatic generation over the use of independent surface models?
HW: Automatic DSM generation is a quite new, but not really unique, feature in Metria CityModeler. We want Metria CityModeler to be a quick and accurate tool for mapping in unknown terrain where maps and data bases may be old, bad quality or even non-existent. With DSM and ortho photo generation integrated in Metria CityModeler, it is less dependent on both external tools and data. We implemented an algorithm tailored for VHR scenes with respect to e.g. sensor model, radiometric resolution etc. The original idea was to adapt a DSM generator I developed at KTH in 1997 to VHR satellite scenes.
It was pretty straight forward, but turned out to be very slow and rather poor quality. So I implemented a completely different algorithm, which first segments both images into radiometrically homogeneous areas and then matches only points at borders between different segments. Some constraints are set on these borders to eliminate gross errors and favour good continuity. Metria made an evaluation and compared my DSM generator to Socet Set:s latest DSM generator NGATE.
The overall elevation standard deviation with LiDAR data as reference was about 5.1 meter using Metria CityModeler and 4.8 meter using NGATE. The DSM:s were also compared using a much smaller set of geodetically measured objects as reference. The results were to Metria CityModelers advantage in this comparison:
|CLASS||Metria CityModeler std dev [m]||NGATE std dev [m]|
|Open area (51 points)||2.75||3.99|
|Buildings and forest (152 points)||4.35||4.74|
|Unclassified (122 points)||2.71||3.29|
V1 Magazine: Revisitor WiMap is also something that you produce. I understand that it works with Lidar data, can you explain what the product is and how it it works?
HW: WiMap BM is a software for generation of 3D buildings from LiDAR data as described above. WiMap W2S (Wireframe2Surface) is a software that transforms wireframe models to surface models. WiMap W2S takes labeled 3D vectors as input and connects the vectors to closed 3D polygons. It also adds unique identities to each building and to each building part. A building part is separated from another building part if there is a vertical difference between them of at least 0.2 meter.
If the input wireframe model includes different levels of detail, they are also maintained in the exported surface model. The vectors in the input wireframe model do not have to snap perfectly, there is a user defined threshold for the largest distance between end points. The surface model is exported both to Shape files and to CityGML. It was developed for Blom Sweden AB and first used for a detailed 3D city model of Gothenburg.
V1 Magazine: What do you consider to be the current challenges facing digital city models? What is needed for them to be more widely used in your view?
HW: I think that the benefits of 3D city models are well known to most users, e.g. planning, visualisation, modeling of noise, pollution, flooding etc. The infrastructure to maintain 3D city models is however not well developed yet. Most municipalities do not want to throw away their 2D data base, but rather update it to a 3D data base and also include new attributes, such as those included in CityGML.
This may be quite a challenge. Another challenge is that customers are uncertain of how to specify the quality and content of city models. How detailed city model is needed? How large absolute errors are acceptable? If there is a specific purpose for the city model, the questions may be answered after some reasoning, but if the city model is a general purpose replacement of an existing 2D map data base it is harder.
Photorealistic 3D visualization of the existing landscape when planning new infrastructure may require not only oblique high scale aerial images, but even terrestrial images. To model which areas would be flooded for different water levels, it is probably sufficient, at least from a geometric aspect, with a DSM from satellite images. Cost and time to delivery are also challenges. ReVisitor tries to introduce an increased level of automation in the production of 3D city models and thus contributes to reduce both time to delivery and cost.
V1 Magazine: Visualisation usually is a factor in 3D city modeling. Does Metria CityModeler include provision for graphics visualisation or how is that accomplished?
HW: Metria CityModeler only includes on screen visualization of the buildings in 2D, projected to each of the two images. You digitize a building in 2D in one of the images. The software automatically determines the object coordinates in three dimensions through image matching and displays the digitized polygons projected to the second image. If the matching fails, the height can be manually adjusted. The measured polygon will then move along a line in the second image and the correct height is found when the polygon fits the same building as measured in the first image. Unlike many photogrammetric work stations, there is no need for stereo devices and trained stereo operators.
For visualization, all buildings in the project may be viewed in Google Earth by a single button click in Metria CityModeler. When this feature was released it was quite hyped – Google Earth was brand new and the larger vendors were nowhere near adopting KML/KMZ as a file format. Small software companies, such as ReVisitor, can be much quicker in adopting new technologies, since the agenda is easier to change. There are many excellent graphic engines for 3D visualization, e.g. those developed by the two Swedish companies Sightline and Agency9. ReVisitors business idea is to extract 3D information from data rather than visualize it.
V1 Magazine: Can you describe a few projects where your products are being used? What did you learn from them?
HW: WiMap BM for automatic 3D building modeling from LiDAR data is still under development, but the results so far look very promising.
VHR satellite scenes are easy to access and are therefore suitable for mapping in areas which are hard to access. Metria CityModeler is thus used for mapping of e.g. nuclear plants, military missions abroad etc. The new DSM and ortho photo generator will also be marketed as a stand alone software. You pay per square km of DSM rather than for the software, which I think makes more sense than paying high initial and annual fees for the software either it is used or not. There will also be a demo mode which generates DSMs of small randomized regions within the stereo model, so you can evaluate the quality without spending a dime.
WiMap W2S was developed for Blom Sweden AB and would probably need modifications to suit other organizations. Since Blom asked for a renewed license just the other day, it is obviously used there at least.
V1 Magazine: Change over time is becoming an interest to some people for digital city models. What is the relationship of level of detail (LOD) for different applications and the reality of using a city model for change detection? Is this a dream or could it be a reality?
HW: I am not quite sure if I understand what you are asking, but I give a try: A natural next step after creating a city model is to maintain it by updating changes rather than recreating new city models. The classical change detection algorithms in remote sensing are of course still valid to detect 2D areas with e.g. new builtup areas, but are not applicable to detect changes in height, e.g. if an additional storey has been built. 3D change detection literary adds a new dimension to the problem.
A city model is always a generalization of the reality, more so for coarser LOD:s. Thus, there is an intentional difference between the city model and the reality. For automatic change detection and city model updating, these differences should be ignored, while significant differences would need to be remodeled. This is a challenge of almost the same magnitude as creating city models from scratch.
This following is only speculation on my part. While there are examples of both parametric top-down and generic bottom-up approaches for creation of city models, I would guess that there should be a preference for top-down trial-and-error methods for change detection, since they may be more suitable for the boolean change/no-change decision, but I have neither read nor conducted research on change detection.
V1 Magazine: What challenges are you facing with your work? Where would you like to see it evolve toward in the medium term?
HW: The major challenge remain the same as when I did my PhD in late 1990´s – to automatically create 3D building models. I think the lack of practically useful results reduced the research interest, but lately it has grown again, due to a more mature market (which actually asks for city models), better data (LiDAR, high resolution and oblique aerial imagery) and the continuous improvements in computational power of computers. Large city models have been created automatically using LiDAR data alone, but they either lack in generality (i.e. they model only a small library of building types) or are too general (i.e. they do not force e.g. perpendicular corners or parallell lines where they exist).
We will see continuous improvements and I hope that ReVisitor will make significant contributions to automize the creation of city models. I recently visited the ISPRS congresses on Laserscanning and CMRT (Object extraction for 3D City Models, Road Databases and Traffic Monitoring) in Paris and already the existence of these conferences is a sure sign that this research field is once again a hot topic and I expect to see lots of progress the next few years.
V1 Magazine: What can you say to students interested in city modeling as a career choice?
HW: There is an interest in city models that we have never seen before. The applications are mainly the same as ten years ago, but the technology and usability has evolved so that city models are at anybodies fingertips in a way that no one predicted ten years ago. In Sweden, we have seen a worrying decline in interest in photogrammetry in a time when there is a great need.
It should be said that most photogrammetric production is made in countries with lower personel costs than Sweden, but what is needed is technically oriented researchers and entrepeneaurs finding new results and inventing new applications. 3D city models of real cities in computer games has just started, for example. 3D TV:s, projectors and computer displays will be commonplace in our homes. I thinks that photogrammetry and city modeling has a bright future.