Spatial data quality relates to and is connected to many of the processes involving spatial applications. However, some people may be unaware of the relationship and why it is important. Vector1 Media editor Jeff Thurston interviewed Steven Ramage (left) and Graham Stickler (right) of the UK-based company 1Spatial who are involved in many spatial data quality applications around the globe. The result was a series of probing questions designed to get at the meaning of spatial data quality, why it is important and where it makes a difference – and how.
Question: Different people think of data quality in different ways. A large number of people are still grappling to understand what data involves and what it means. What does ‘data quality’ mean?
Answer: That’s a very good question. It’s interesting that the Open Geospatial Consortium (OGC) Data Quality Working Group has taken steps to try and understand that question. They recently conducted a worldwide survey to explore what ‘data quality’ means to the geospatial industry; with almost 800 responses this is obviously a topic of concern to industry professionals. There is an expectation that it means different things to different stakeholders across the spatial data supply chain. We all understand that it includes more than just geometric accuracy; it must also include areas like logical or semantic consistency, and it must also relate to fitness for purpose and the data’s intended use. As an example, if you were to receive a road network dataset that was highly accurate from a geometric standpoint, but that was not actually connected at all nodes, then its ‘quality’ would be very poor if you intended to use it for routing purposes.
The following quote from J.M Juran, the recently deceased ‘quality guru’, sums up the situation well:
“Data are of high quality if they are fit for their intended uses in operations, decision making and planning. Alternatively, the data are deemed of high quality if they correctly represent the real-world construct to which they refer. These two views can often be in disagreement, even about the same set of data used for the same purpose.”
Question: Can data quality impact work processes? How? Have you been able to see the benefits of improved data quality and could you explain some of them please.
Answer: Very much so. Improving data quality is not a once-only task, and implementing an ongoing quality control regime can have major impact on work processes and benefit a business by cutting out ambiguity and improving overall operational efficiency, e.g. limiting duplication of effort. Data quality has been defined earlier as assessing fitness for purpose and plays a vitally important role when looking at the integration of datasets. We often talk about the ‘supply chain problem’ and data quality is pivotal to this concept. A classic manufacturing supply chain derives source materials from various places, manufactures a product from these materials and then delivers the product to distributors or end customers.
In the IT world, we are attempting to do the same thing, and spatial data is playing an increasingly large role in this process. We take data from multiple sources, combine and integrate it (often to a single source) to create a product – information. We then need to distribute this product to the end customer or decision-makers. Source data may well be of high quality in the context of what it is being collected for and operationally used for (fit for purpose). However, the source systems will be ‘doing things right’ in the context of their operational role. What we are increasingly trying to achieve, as an organisation, community or enterprise, is combining all these data so that we can undertake effective analysis, planning and business decisions – i.e. ‘doing the right thing’. This spatial data integration or conflation is what 1Spatial has being doing successfully for many years and one of the key areas of expertise that will be highlighted at the Conference.
One organisation attempting to tackle the data quality issue is MidCoast Water, in New South Wales, Australia. MidCoast Water serves an area of 7000 square kilometres and the authority is responsible for reticulated water supply and sewerage systems to communities in the Manning and Great Lakes regions. The delivery of services to such a vast area brings with it many challenges. Formed just under 10 years ago from the water and sewerage sections of three local authorities, MidCoast Water has quickly grown to be an industry leader, and sees its encouragement of innovation as crucial to this positioning. MidCoast Water introduced a programme to provide improved access to accurate geographical and asset information. The programme was developed in two stages:-
1. Improving the efficiency and accuracy with which information is gathered and recorded
2. Improving the accessibility of this information.
The authority needed to ensure its data quality in order to provide the best possible service to its end customers. They needed to be confident in the reliability of their data at the attribute level so that no manual checking was required and that time was not wasted searching for assets in the field due to errors within their data. Topological connectivity between networks also needed to be assured to prevent the duplication of editing tasks, which could provide a drain on their manpower. As well as MidCoast’s ongoing programme of internal improvement, from 1st July 2006, state government regulations required utility organisations to have the ability to accurately pinpoint their assets. MidCoast Water needed to ensure that they were fully compliant with this before the legislation came into force.
Following the implementation of a spatial data quality regime, MidCoast Water now enjoys the following advantages and benefits:
• Interoperability – centrally stored data is error-free and accessible via multiple applications across the company
• Enhanced productivity – significant time and cost savings have been achieved through increased query performance
• Enterprise-wide data management – business and spatial data has been centralised into a single database, reducing the duplication of effort in maintenance of that data
• Improvements in data gathering – it now takes just a few hours, instead of a week, to translate spatial data into maps
• Efficient processing of property information – there has been a 60% reduction in staff time for this task when combined with other processes
• Versatility of application – the return on investment has increased as a result of being able to apply the data in a variety of new ways and through data mining opportunities.
The return on investment for MidCoast Water has been substantial. Not only have they met their objectives to centralise business data and to accurately manage and pinpoint their assets, they have also achieved measurable time and cost savings. As just one example, before the implementation of the new regime it appeared that two sewer stations were needed at a particular development. Under the new system, with the resultant improvement in data quality, the information supplied clearly showed that only one sewer station was required, saving MidCoast Water up to $300,000 AUD.
Question: There has been a growing discussion relating to spatial data infrastructure (SDI). It is hard to understand how all these countries and people are going to share data and information. What initiatives are you involved in which will help to make SDI a reality? What are the challenges that SDI face in your opinion?
Answer: 1Spatial has been involved with the Digital National Framework (DNF) group in Great Britain since its inception several years ago. The vision of the DNF is “to enable and support easy and reliable integration of business and geographic information regardless of who is responsible for its maintenance and where this is undertaken, thus achieving the goal of “plug and play information”.” This vision runs parallel to the idea of building a spatial data infrastructure (SDI) since it is all about providing increased access to, and being able to share, quality information across organisations. DNF is not just an activity that is unique to ‘building SDIs’, it crosses all areas of spatial information management and therefore supports SDI efforts, as well as the wider goals of the Digital National Framework. 1Spatial is also involved in SDI research in Scandinavia and the Benelux region, for example, one project is assessing ‘quality elements within the supply chain to deliver an SDI’.
Although not called an SDI, the work carried out as part of the VISTA project could be described as exactly that. The VISTA project is examining innovative ways of integrating, presenting and making use of utility data with the aim of reducing the direct and indirect costs of streetworks. Inaccuracies in existing information, and the lack of methods to integrate, share, reuse and effectively communicate knowledge held by the owners of underground assets, means that more excavations are required to locate those assets, causing unnecessary traffic congestion and increased costs for the economy, i.e. unnecessary damage to underground assets results in increased costs, injury and or possible death of workers and loss of service to consumers, both business and domestic.
UK Water Industry Research (UKWIR) is the lead co-ordinating partner for the project, with Leeds and Nottingham Universities providing the research input, and over 20 utility and other industrial organisations also involved, who have been working on a Global Schema to provide a framework that will unify the utility data from each of the partner domains (electricity, gas, sewer, telecoms and water). This should not require a change in the source schema, as well as providing enough flexibility to articulate the visualisation and analytical requirements of each domain.
The VISTA project industry partners have provided access to 16 datasets unique to company and domain, which were used extensively in the modelling and design phases. 1Spatial has helped to define the Global Schema mapping metadata, this complex metadata being maintained within Radius Studio. It is envisaged that Radius Studio will significantly reduce the time taken to integrate these different assets and improve the mechanisms of domain validation with the project partners. This will result in decreased development time and an increase in data quality and domain knowledge.
From our experience of working with organisations such as Fujitsu and ESRI as our partners on the Ordnance Survey of Northern Ireland GeoHub NI project, we know there are some practical issues to be addressed with an SDI. GeoHub NI is a great example of an SDI since it aims to host and connect to spatial data from multiple sources, enable metadata searching and, via Web thin clients, allow multiple data layers to be accessed and shared. In addition to the cultural aspects of people buying in and using the service, there are also commercial considerations around licensing the data. The main problems that 1Spatial addressed were to do with data management issues and included the validation of critical third party datasets, ease of administration, data integration, security, standards compliance and robustness of the solution. It highlighted how encompassing an SDI can be with its many components.
While trying to provide a secure gateway to web services and web mapping we encountered some limitations in building these elements of SDIs due to the maturity of OGC Web Feature Services (WFS) tools. This project was done using a modular architecture that highlights that no single vendor can provide it all, making interoperability a key consideration when mixing and matching architecture and using a number of mainstream IT approaches, such as Simple Object Access Protocol (SOAP) and Web Services-Security for remote access via WFS and Web Mapping Services (WMS). As experienced with GeoHub NI, the practical issues of secure, reliable access to fit for purpose spatial data will be significant challenges for any SDI.
This Vector1Media article covers it well: http://vector1media.com/article/feature/source-of-truth:-is-the-it-community-prepared-for-spatial-data-infrastructures?/
Question: There is a trend to web services for many organisations. In some places it appears people are still at the desktop approach, while at others, almost all of the work is web service based. Are the data quality needs for desktop versus web service oriented approaches any different? Can you explain how data quality applies to web services and desktop environments and the connection to organisational processes for each?
Answer: These needs are no different from a corporate standpoint. Individual users may have different needs in terms of what they do to ‘their’ data (e.g. tidying up geometric accuracy on a GIS workstation) but organisations are now seeking to build on and, in some instances, move beyond this departmental approach to integrate, share and reuse data across the entire enterprise. In this scenario, and if we think back to question two and the concept of supply chains, then the benefits of web services, or component architectures, seem obvious. Having the flexibility to ‘interface’ into a data quality control regime from any access point in an information workflow is vital to creating a sustainable, enterprise-wide culture and the creation of a single source of truth for spatial data. This has now been made possible with web services and maturity of mainstream tools for areas like BPEL and transaction management.
Question: It seems that data access is a big concern for many people. When you hear the term ‘access’ what does it mean to you? If someone has to transform data, what are the quality considerations involved? Are there alternate approaches involving non-transformation?
Answer: Access can be widely interpreted and has many issues associated with it depending on the interpretation. However, the issue of security is perhaps becoming the most important concern, and goes way beyond digital rights management and IT security issues. The ‘Google effect’ has opened up access to spatial data but have we thought through the implications? As data becomes more available at ever increasing levels of detail, are we happy for it to be available to everyone? Open Source Software has been around for a while, but what are the implications of Open Source Data? You may be happy for the floor plans of your buildings to be available to the Emergency Services, but do you want other organisations to have unrestricted access to such data? Data transformation is all about the repurposing of such data for alternate uses. This may involve spatial data integration considerations or simply involve physical change to the data to enable its reuse for other purposes. The quality considerations go back to the previous answers in that the data needs to be fit for purpose. The Quality Challenge is how to define and describe a fitness for purpose measure and to report on this during the data supply stage. This is the next step for the OGC Data Quality Working Group (also referred to earlier).
Question: Increasingly we see a trend toward real-time or near real-time applications. These are not all necessarily mobile based. Many of them are field based from static sites which are continually collecting information and feeding that to other parts of networks. How does one go about ensuring data quality over a real-time network?
Answer: Our approach to data quality is generic, i.e. it is a rules-based approach for automating as much of the spatial data mining, rules discovery and conformance checking as possible. If you can provide the data specifications or the business rules associated with the spatial data coming in over a real-time network, then it is still about the data validation process and reducing the roundtrip engineering costs associated with carrying out data quality checking manually on input or once the data has been integrated into a workflow.
Real-time data validation services should provide tools to increase data accuracy at the time it is entered. By using a data validation Web Service, such as Radius Studio, when the collected information is shared or distributed to other parts of the network the service can assess conformance of the incoming data, according to minimum quality levels or service level agreements if the associated business rules are known. In the event that invalid data is submitted, a system of automatic checks can be set up to prompt the provider to enter corrected information, reject a transaction, or apply business rules to improve the performance of the data collection efforts. The end result is an improved workflow that cuts out unnecessary and additional manual data validation stages.
Automated data validation in real-time requires some thought as to the subset of rules to be applied and the source of the data. For live data streams, obvious checks would be things like “all vehicle locations must be within x metres of a road or ferry route” and could be applied in real-time. In other situations, particularly for data being input by a user or copied from another system, some rules must be deferred or ignored until enough information is present. For example, “pipes must have connectors at both ends” and “connectors must be on a pipe” should not be checked for each feature when it is input, as the data will always fail at first (a new pipe will have no connectors, but a new connector will have no pipe). Instead, a related group of inputs need to be read before the checks can take place. Other checks, which require large amounts of contextual data, need to be applied at different intervals when enough data has been captured to allow a meaningful result.
Question: The architecture industry is heavily involved in the use of spatial geometry for design, construction and operation of buildings. In some cases hundreds if not thousands of polygons, nodes and lines are involved, yet, many architects are primarily concerned with building design. It seems like data quality would play an important role in building geometry, or is this not the case?
Answer: The CAD arena does indeed develop very sophisticated models and, fundamentally, they are made up of basic geometric primitives in the same way as we in the GI space put together maps in the 2D world. In the past it has been a case of never the twain shall meet, but this is changing and we are seeing a convergence of these data ‘spaces’. From a quality standpoint, the issues are no different to those mentioned above. These data are certainly fit for purpose in their source systems, but as soon as we look to integrate the data then we have to consider a new set of quality rules. 1Spatial is currently involved in research projects in the USA and have been looking at how 3D models can be ‘stitched’ into the 2D space and how topology can then be built in line with the 2D data to give a uniform structure and internal intelligence – i.e. a uniform ‘quality’ measure. Our partner LSI will be presenting on this topic as part of the CAD to GIS Track at the Conference.
Question: In Europe the INSPIRE program aims to harmonise spatial data and enable more integration at the policy level. Why should Europeans care about data quality in INSPIRE?
Answer: Policy is good for setting the path for where we want to go; maybe we could look at data quality as one of the vehicles for getting there. If we define spatial data quality as ‘fit for purpose spatial data’ and follow the old adage, ‘garbage in, garbage out’ then how can organisations even consider providing increased access, sharing or migrating spatial data without knowing and understanding the quality of their spatial data. INSPIRE references data quality numerous times, but it is not clear who has ownership of data quality in Europe at any regional, national or European level. It is also not clear what tools exist in the market place to discover information on quality.
As mentioned, one of the key aims of setting up the OGC Data Quality Working Group was to address specifically this topic, i.e. data quality across the geospatial supply chain. Finding better ways to measure, communicate and report on spatial data quality by building on the work already done by ISO with the ISO 19000 series standards are key aims. Working with organisations like ePSIplus, EuroSDR and EuroGeographics, 1Spatial is trying to connect a number of stakeholders and raise awareness of issues around spatial data reuse and data quality to help prepare for INSPIRE.
If European geospatial professionals don’t consider data quality issues now, it will be too late once spatial data is accessible in INSPIRE and used in applications and the data is not fit for purpose. At that stage they will have to tackle it retrospectively, incur the huge costs, headaches and business process breakdowns associated with poor quality data. We should learn from painful lessons in other IT data management disciplines, such as CRM (customer details and addresses) and ERP (asset information) where there is now a multi-$billion industry addressing all their data quality problems and avoid the geospatial industry having to spend similar amounts around INSPIRE. Expertise, methodologies and tools exist now to avoid this situation with geospatial data and to help prepare for INSPIRE.
Question: The game industry is another industry that is heavily based on spatial geometry for the design of new games as well as virtual environments. Much of this work is leaning toward 3D. Can you explain how spatial data applies to games in general but also the impacts on the trends towards 3D applications.
Answer: We can learn from the gaming industry, in particular from a visualisation perspective and how individuals may interact with spatial data in the future. By combining 3D models with 2D geospatial data and providing access through gaming style platforms and interfaces, it will be possible to create authentic virtual versions of our world, rather than fictional ones, that we may explore remotely. The application possibilities for planning, emergency services and, of course, the military are also now becoming apparent. For other 3D applications it will eventually become important for analysis and decision making to have greater intelligence in the data, such as topology. Without this type of connectivity, extensive manual efforts will be necessary to try and obtain more information relating to the nature of the data and make joined-up decisions.
Question: What are some of the environmental and sustainability applications that involve your products and how have they fared?
Answer: Our products are all involved in spatial data integration or data management exercises on a worldwide basis. Some examples that will have an impact on environmental and sustainability applications are:
The Environment Agency used 1Spatial’s data re-engineering expertise to speed up spatial queries and image rendering by simplifying the data within its National Flood and Coastal Defence Database (NFCDD) system. An initial prototype exercise showed that there were significant benefits to be gained from generalising the LIDAR data using persistent topology. Rendering an image yielded a 115% increase in the speed, whilst spatial queries using address points showed a 229% increase. There was also an accompanying increase in the accuracy of the query results returned.
Rural Payments under the Common Agricultural Policy have data quality challenges associated with aligning raster plots with the national topographic vector datasets. INGA, the Portuguese Ministry of Agriculture, is implementing a solution to ensure more efficient administration of the system which co-ordinates and pays EU subsidies to over 400,000 Portuguese farmers per year. This work has been done in collaboration with Intergraph, a member of the 1Spatial Community.
CEH (Centre for Ecology and Hydrology, part of the Natural Environment Research Council) initially undertook a joint feasibility study with 1Spatial and Ordnance Survey Great Britain to generalise OS MasterMap® data as a potential foundation for the Land Cover Map (LCM) 2007. This is now at production rollout stage and will form the basis of the UK’s submission for the Coordination of information on the environment (CORINE) programme.
MidCoast Water, Australia purchased Radius Topology from 1Spatial to improve asset data quality and made substantial savings, as previously mentioned in question 2.
Question: Today – more people are interested in extracting ‘value’ from all the costs involved in their data holdings. Could you elaborate on how value could be realised more fully and provide a few examples as well?
Answer: There has been a huge amount of investment made in the collection of both 2D and 3D data over the years and this process shows no signs of slowing down. In many cases it could be argued that these data represent an organisation’s biggest and most valuable asset. Software, databases and even staff will come and go, technologies will change, but the data remains. However, very often the knowledge will disappear or be isolated and the data itself will drift and slowly deteriorate.
In order to extract the maximum use from these data (maximum value from the asset) two things are needed. Firstly, we need to store all the intelligence, knowledge and wisdom about these data alongside the relevant data and not have them locked in the logic of some software application. Secondly, we must be able to communicate how these data may be used. Metadata takes us a long way towards the first of these aims, but the ability to define the quality in terms of fitness for purpose will enable us to solve the second issue; this will allow us to fully recognise or realise the value.
As an example, Pira International Ltd noted in a report in 2000 that the spatial component of data held within the Public Sector across Europe was over €36bn, and growing rapidly. At that time, the re-collection cost of these datasets was estimated at in excess of €100bn – one can only imagine the total investment today. At that same time, the estimated costs for North America were $200bn. This was almost 10 years ago and there has been a data deluge since that period. By assessing the fitness for purpose for spatial data, organisations can take large strides towards realising the value.
Question: “We have been doing it this way for a long time and it is the procedure we use to collect our information and make our decisions.” How often do you hear this? What pieces of wisdom would you add to this line of thinking?
Answer: Unfortunately it is human nature to respond in this way, either because change threatens the status quo in terms of business processes and could be seen to entail additional work, or because it can be perceived as threatening job security or highlighting problems that may have been well hidden until procedures were reviewed and adapted. So the answer is that we hear it very often.
It’s a very open statement, but change is the only constant in business life, so we would encourage all geospatial professionals to consider this fact when thinking about their spatial data management. It’s only by challenging the status quo and reviewing existing processes and procedures that you can improve them and the overall effectiveness of your operation.
We believe that quality underpins analysis, planning and decision making based on spatial data, so we would encourage you to step back from the way you have been doing it and think about how you could be doing it better, starting with an assessment of your data’s fitness for use.
Question: What is unique about your upcoming 1Spatial Conference in the UK and what might I learn by attending? What are some of the topics that will be discussed and how can I find out more about it?
Answer: The 1Spatial Conference brings together a wide range of organisations to consider industry-relevant topics, as opposed to simply being a user group or an event limited to one specific area of the market. The experiences shared and the problems addressed by the conference are global in nature and cross into all sectors of the geospatial community, hence the marketing messages that it will highlight the topics of Spatial Data Quality, Spatial Data Infrastructures and CAD to GIS convergence.
The Conference aims to provide practical advice and share experiences, for example in trying to achieve INSPIRE goals. There will be a FREE INSPIRE seminar and metadata workshop, speakers from the European Commission Joint Research Centre, EuroGeographics and the Chief Executive of Ordnance Survey of Northern Ireland will also provide insights on this topic.
Two useful opportunities in this area concern INSPIRE and DNF activities; delegates can come and learn about tools and processes that can be used for the discovery of metadata that can subsequently be published in catalogues for other users to assess its fitness for use. Published metadata consists of data quality items (qualitative and quantitative metrics) and is encoded in a standard form (ISO 19139 Metadata – XML schema implementation). The final results of the conformance tests using Radius Studio are obtained in the form of metadata, which is compliant to the conceptual model of ISO 19115 Metadata and encoded in the form recommended in ISO 19139. There will also be a DNF demonstration that will highlight the benefits of Unique IDs and how users can profit from data providers supplying stable identifiers with lifecycles. It will also highlight how the spatial data supply chain can use the lifecycle information once it has been generated.
The speaker line-up is exceptional and provides a comprehensive insight into a range of geospatial areas, including CAD/GIS convergence, database technologies, industry standards and best practice, Open Source and SDIs.
You can find out more, and register for the Conference, by visiting the conference web pages at www.1spatial.com/conference
