Village names in southern Laos

Following a discussion about an old changeset on OpenStreetMap, I’m posting a few photos here that I took during mapping trips. All pictures are taken back in Oct 2011.

Overview map with geotagged pictures: 

The Nam Ngiep Hydropower Scheme

Just recently I continued mapping the Nam Ngiep hydropower scheme, see also my previous post about Nam Ngiep 1.

So far my findings:

Nam Ngiep1

Yet Another Hydropower Dam: Nam Ngiep 1

Meanwhile Nam Ngiep 1 dam has been completed and the reservoir has been filled during rainy season in 2018. I.e., it's high time to put in on the map!

Nam Ngiep 1 reservoir area in January 2018

New hydropower dam: Nam Phay

Just recently I've put the Nam Phay hydropower reservoir on the map!

During Lao New Year 2014 I had the chance to visite this area, see pictures below:

The Nam Phay reservoir area in Januar 2015

Another Hydropower dam: Xekaman 1

Just recently I've found by chance the Xekaman 1 hydropower reservoir on Landsat 8 imagery. Time to put it on the map!

The Xekaman river area recorded on 24th January 2015, just before water storage started

Raster math in GRASS GIS using numpy

Houay Lamphan Gnai, yet another hydropower plant just started commercial operations recently with a reservoir area of seven square kilometers as reported by local media. Time to put it on OpenStreetMap using up-to-date Landsat 8 imagery and the new GRASS GIS 7 plugin for QGIS 2.12. To get familiar with GRASS GIS' interface to numpy, I've decided to extract the reservoir using numpy.

Houay Lamphan Gnai on Landsat 8 image taken on the 23rd Oct, 2015

New downloads available

I'm happy to announce additional Shapefile downloads on openstreetmap.la:
those include buildings, national parks, provincial and national boundaries for Laos and Cambodia.
Please find all files at openstreetmap.la/downloads.

Two villages close to Pakxan with buildings displayed in QGIS.
Data © OpenStreetMap contributors

Nam Ngum 2 dam three year ago

Yesterday I found by chance a picture that is exactly three years old. I took it when I flew from Phonsavan back to Vientiane and it shows the Nam Ngum 2 dam under construction.

Nam Ngum 2 dam under construction on 21. November 2009

Line extraction with GRASS GIS

This post presents another way how to use Landsat imagery to map features for OpenStreetMap in an easy way with free and open source software only. After remapping the Nam Ngum 1, Nam Leuk reservoir and other lakes using a similar approach, I wanted to improve the Nam Ngum river. The second largest river in Laos and an important tributary to the Mekong was in OpenStreetMap only mapped as a single line instead of an area, although it's more than hundred metre wide downstream of the Nam Ngum 1 reservoir.

Armchair mapping using GRASS GIS

After reading my previous post a friend asked me why I didn't use just GRASS GIS. That's really a valid question. Since the GDAL/OGR utilities are some of my daily tools, I'm more familiar with them than with GRASS. Out of interest I developed a workflow how to extract and vectorize waterbodies from Landsat images using (almost solely) GRASS GIS. The resulting features I wanted to upload to OpenStreetMap.
This time the Nam Ngum 1 hydro power reservoir in Laos was my test area. Like Nam Leuk last time it was only roughly digitized and some of its islands needed to be remapped.

Armchair mapping with Landsat imagery

In this post I want to show step by step how I extracted hydro power reservoirs and lakes in Laos from up-to-date Landsat imagery for further use in OpenStreetMap. The extraction is done straightforward with common GIS tools but without sophisticated and/or proprietary remote-sensing software.
As an example I take the Nam Leuk hydro power reservoir in Laos. This reservoir needs to be remapped, since the origin user declined the upcoming license change. Currently the reservoir is rather roughly generalized.

Two years of mapping in Laos

After more than two years of OpenStreetMapping in Laos I want to look back and give a short review. Two years ago the OpenStreetMap in Vientiane but also in whole Laos was quite empty, as I illustrated in a previous post.
I started mapping Vientiane downtown rather unsystematic till I met another mapper and we decided to initiate the probably first mapping party in Vientiane end of October 2009!

Vientiane downtown and Lonely Planet update

When I arrived in Vientiane in August 2009 the OpenStreetMap was quite empty, there were only some unconnected roads traced from Yahoo imagery and a few points of interest. It was my second or third weekend when I took my GPS, a pencil and a sketchbook and started mapping the downtown.

Find your way with openstreetmap.la

I'm really happy to announce the latest update on openstreetmap.la: Routing has been integrated to the main page.

As illustrated in a previous post I evaluated the capabilities of the Open Source Routing Machine amongst other routing engines. The speed and the web-oriented architecture of the Open Source Routing Machine convinced me to use it as a backend to implement routing on openstreetmap.la.

Rivers and lakes downloads available on openstreetmap.la

During my two-week trip through Laos I collected some tracks and points of interest, of course ;). Last week I started to add them to OpenStreetMap when I realized that there are still a lot of rivers missing. I traced (rather randomly) some parts of rivers from Landsat or where available Bing imagery.

Garmin maps with contour lines

Some days ago I read in the OpenStreetMap wiki about maps for Garmin devices featuring contour lines. Interested in topographic Garmin maps for openstreetmap.la, I started to create maps with contour lines for Laos. As often happens it took me longer than I expected to figure out a suitable work flow, mainly because I ran in several memory issues.

I started with the generation of contour lines from the SRTM (and CGIAR improved) terrain model using gdal_contour. After about two hours calculating on a more capable work station (than my laptop), I got a 900MB heavy Shapefile. Based on this GDAL output, my plan was to convert the lines to the OSM format to use finally mkgmap.

To be able to deal reasonably with this amount of data I put the lines with shp2pgsql into a PostGIS database. Storing data in a PostGIS database facilitates faster spatial and attribute queries.

My first idea was to convert each contour level with ogr2osm to the OSM format. ogr2osm is a very flexible Python script I like to use, but in this case it refused to work for unknown reason. I had to use the two step approach with ogr2ogr and gpsbabel. For each contour level between 0 and 3000 I scripted the following steps and subdivided the levels in minor, medium and major contours according to the Garmin features land-contour-thin, land-contour-medium and land-contour-thick.

Query the database and write the result as tracks to a GPX file.

# ${level} means the currently processed contour level
ogr2ogr -f GPX -lco FORCE_GPX_TRACK=YES -sql "SELECT wkb_geometry, name FROM srtm_contours WHERE name=${level} AND ST_INTERSECTS(wkb_geometry, ST_SetSRID( ST_MakeBox2D( ST_POINT(100.1, 13.9),ST_POINT(107.7, 22.5)), 4326))" -nlt MULTILINESTRING ${outdir}/${prefix}_${level}.gpx PG:"dbname=openstreet user=openstreet"

Use gpsbabel to convert the GPX file to an OSM file and add corresponding tags.

gpsbabel -i gpx -f ${outdir}/${prefix}_${level}.gpx -o osm,tag="contour:elevation;contour_ext:${contour_ext}",created_by= -F ${outdir}/${prefix}_${level}_tmp.osm

Since it is not possible to add an elevation to a GPX track (only to points), I wrote the elevation to the name attribute. Now I had to replace the name tags with ele.

sed 's/name/ele/g'

The versatile osmosis program, I wanted to use to compress the data to PBF format, still didn't accept the file since it did not comply with the API v0.6. I had to add a version and timestamp attribute to each node and way and replace the negative identifiers with positive ones.

cat ${outdir}/${prefix}_${level}.osm | sed "s/node\ /node\ version='1'\ timestamp='2010-08-01T12:00:00+02:00' /g" | sed "s/way\ /way\ version='1'\ timestamp='2010-08-01T12:00:00+02:00' /g" | sed "s/'-/'/g" | osmosis --read-xml file=- --write-pbf omitmetadata=true ${outdir}/${prefix}_${level}.osm.pbf

Applying mkgmap to this file, mkgmap ran inevitable out of memory. It was necessary to split first the data (with splitter) to tiles manageable by mkgmap. I used the following tiles with dynamically generated map names for each tile and contour level:

KML split-file for splitter on Google Earth

Using this approach I finally managed to create map tiles in the Garmin IMG format.
Last step was to merge all contour lines maps with the OpenStreetMap data, but this could easily be done by mkgmap.

The result looks as follows:

Highest mountain in Laos

Cave near Vang Vieng

The map is can be downloaded as gmapsupp.img from openstreetmap.la or directly from here. There are detailed instructions on the OpenStreetMap wiki how to store a gmapsupp.img file in your Garmin device.

Update 9.9.2011: Due to problems with the routing, I've removed again the contours. To be continued ...

Search bar on openstreetmap.la

I'm pleased to announce a new feature on openstreetmap.la: a search bar to find places and points of interests like hotels, hospitals, restaurants or convenience shops.

To facilitate the search, the search bar implements auto-completion: Start typing the name of any place or point of interest, and a drop-down menu suggests available places or points containing this name. A small icon in front of the name indicates if it's a village, restaurant, hotel etc. The icons corresponds in most (!) cases to the map symbols. Villages, towns and cities are the exception, since places have a label but no icon on the map. The icons for places used in the drop-down menu look as follows:

Village

Town

City

See the following screenshots:

Search bar with autocompletion

Works also in Lao

A map marker highlights the found place:

Map marker

As always: feedback is highly appreciated!

Open Source Routing Machine: Yet another routing engine

Initially I only wanted to some routing on OpenStreetMap data with GRASS GIS. Then I also tried pgRouting and later SpatiaLite. Using pgRouting as backend, I set up a simple web application following the well explained pgRouting workshop. But I wasn't really satisfied since the routing starts only at the closest node, that's basically less a problem in the city than on the countryside with less junctions (and thus less nodes).

Finally I turned away from GIS to specialized routing software for OpenStreetMap, namely to Routino and to the Open Source Routing Machine, short OSRM. I was that impressed by the speed of OSRM, that I decided to have a deeper look at this engine:

Pros

• Speed! OSRM is about speed and it claims to be capable to handle continental sized networks using memory efficient third party libraries like google-sparsehash. It was a matter of seconds to preprocess the 6.2 MB Laos PBF file.
• Web-oriented: includes a server that runs on a configurable port and thus it is very easy to integrate in any web application.
• Since it's developed for OpenStreetMap, it considers out of the box OpenStreetMap tags like oneways etc.

Cons

• Configuration is not trivial, since speed and highway categories are hard-coded.
• Installation with Scons was very laborious on the server, since I had some dependencies in my home directory and it was necessary to fiddle the SConstruct file.
• Missing GeoJSON support. But since it needs anyway a proxy script to use OSRM in Ajax requests, it was a quick hack to reformat the homebrew JSON output to proper GeoJSON.

I adapted the simple web frontend I developed for pgRouting to work with OSRM as backend and put everything online. I'm pleased to present the current proof-of-concept prototype:

http://www.openstreetmap.la/dev/routing/

As soon as time allows I'll integrate the routing on the openstreetmap.la main page.

Power lines from Landsat imagery

Hydropower is the big deal in Laos at the moment and therefore power lines are required. Some weeks ago on an offroad ride we crossed the relatively new Nam Ngum 2 power transmission line. Realizing the long straight and wide forest aisle, I knew that it must be possible to identify the line on Landsat imagery to map it on OpenStreetMap. Landsat imagery is in the public domain and thus suitable for mapping.

I downloaded the Landsat imagery from January and February 2011 for this region already earlier. Therefore I could start mapping the power line immediately, starting from the point where the power line crossed our way and following the forest aisle on Landsat. It's probably the first mapped power line on OpenStreetMap in Laos.

Nam Ngum 2 dam and power lines on Landsat

There is the Nam Ngum 2 reservoir and its dam visible in the middle at the top border. In the lower left corner is the older Nam Ngum 1 reservoir. The power line going south is clearly identifiable in the middle section. Furthermore two access roads to the dam are identifiable, one on the north shore of Nam Ngum 1 and one leading east from the dam. Just downstream of the new dam there is the confluence of Nam Bak.

Just for the visual appearance I tweaked the image to get rid of the stripes using GDAL, GIMP and OSSIM. OSSIM is as fas as I know still the only FOSS that implements histogram stretch based on the standard deviation.

The power lines in natura

Set up pgRouting with OpenStreetMap data

Another rainy season Saturday got me time for some GIS exercises. Inspired by Carson Farmer's excellent introduction to pgRouting I decided to set up a routing enabled PostGIS database with OpenStreetMap data and compare it with my GRASS GIS routing setup.
Contrary to Carson Farmer I didn't compile pgRouting from source but installed it from the ubuntugis-unstable repository

sudo apt-get install postgres-8.4-pgrouting postgres-8.4-pgrouting-dd postgres-8.4-pgrouting-tsp

and followed the instruction from the mentioned blog.

The osm2pgrouting script creates all necessary tables including table "public.ways", which is probably the most important one. The table schema without indexes and constraints looks like:

Table "public.ways"
    Column    |       Type       | Modifiers 
--------------+------------------+-----------
 gid          | integer          | not null
 class_id     | integer          | not null
 length       | double precision | 
 name         | character(200)   | 
 x1           | double precision | 
 y1           | double precision | 
 x2           | double precision | 
 y2           | double precision | 
 reverse_cost | double precision | 
 rule         | text             | 
 to_cost      | double precision | 
 osm_id       | integer          | 
 the_geom     | geometry         | 
 source       | integer          | 
 target       | integer          | 

The relevant columns for the routing are "length","reverse_cost" and "to_cost". The latter two are the forward and backward costs to use a way and are used in directed shortest path calculations. It's necessary to use an algorithm that can handle direction when one-ways have to be taken in account.

Since OpenStreetMap data are in geographic coordinates the length is in degree, but I prefer to have the length of a way in meter. Thankfully PostGIS provides ST_Distance_Spheroid to calculate lengths on the (earth) spheorid:

UPDATE ways
SET length = ST_Length_Spheroid(the_geom,'SPHEROID["WGS 84",6378137,298.257223563]')::DOUBLE PRECISION;

Next step is to filter the one-ways paths and set the "reverse_cost" to -1 to force pgRouting to choose these ways only in forward direction. Unfortunately only the name and the osm_id are imported from the OpenStreetMap raw data to the database. Thus I needed a list of osm_ids that represents one-ways. Since I also used a PostgreSQL database to store the GRASS attribute tables while working with OpenStreetMap data, I could easily get a comma-separated list of osm_ids with the following query:

psql -A -t -R ',' -d grass -U grass -c "SELECT osm_id FROM osm_net WHERE oneway = 'yes';" > oneways.sql

Then I updated the "reverse_cost" column:

UPDATE ways SET reverse_cost = -1
WHERE osm_id IN(55909972,55909973,28977387, ... ,102306365,27158132)

Finally I could start with undirected and directed queries and compare the results.

-- Undirected shortest path query with astar
SELECT * FROM astar_sp( 'ways',3365,3412)
-- Directed shortest path query with astar
SELECT * FROM astar_sp_directed( 'ways',3365,3412,true,true)

Verify both calculated shortest paths from node 3365 to node 3412 in QGIS:

The red path is the undirected query and ignores apparently one-ways, contrary to the directed green path that follows one-way directions.

Until now I was considering only length and direction of a way. To get a more life-like model I wanted to take the road classes into account, too. I updated table "public.classes" and estimated for each road class an average speed in m/s in column "cost":

psql -A -d osm_routing -U openstreet -c "SELECT id,type_id,TRIM(trailing FROM name) AS name,cost FROM classes LIMIT 8;"
id|type_id|name|cost
101|1|motorway|33.3333
102|1|motorway_link|27.7778
103|1|motorway_junction|27.7778
104|1|trunk|27.7778
105|1|trunk_link|27.7778
106|1|primary|13.8889
107|1|primary_link|13.8889
108|1|secondary|11.1111
(8 rows)

With these assumed costs for each road class I updated the costs in table "public.ways" and divided the length by the average speed and got the time in seconds that is necessary to pass a certain way.

-- Update to_cost column
UPDATE ways
SET to_cost=(length/(SELECT "cost" FROM classes WHERE id=class_id))::DOUBLE PRECISION
-- Update reverse_cost columns
UPDATE ways
SET reverse_cost=(length/(SELECT "cost" FROM classes WHERE id=class_id))::DOUBLE PRECISION
WHERE reverse_cost <> -1;

Now everything is set up and it needs nothing but another rainy weekend to compare pgRouting results with GRASS GIS results.

Last but not least I'd like to share yet another way how to display results of PostGIS queries without using an intermediate (Shape-)file, namely using the very versatile Virtual Format from the OGR library. Create a text file with the following lines and open it as a vector layer in QGIS or any other OGR backed application:

<OGRVRTDataSource>
  <OGRVRTLayer name="shortest_path">
    <SrcDataSource>
      PG:dbname='osm_routing' user='openstreet'
    </SrcDataSource> 
    <SrcSQL>
      SELECT * FROM astar_sp_directed( 'ways',3365,3414,true,true)
    </SrcSQL>
  </OGRVRTLayer>
</OGRVRTDataSource>