Create / distribute tiled map
159 Views
November 06, 25
スライド概要
JICA セミナー 2025 @ 国土地理院
Geolonia / OSGeo.JP / Japan Unix Society / OpenStreetMap Foundation Japan
関連スライド
各ページのテキスト
Create / distribute tiled map Taro Matsuzawa Geolonia Inc. JICA Seminar 2025 1
Notes This presentation use some command line tools Open following link in your browser. Copy and paste the command line to your terminal. https://smellman.github.io/jica_2025/ JICA Seminar 2025 2
Self introduction GIS Engineer at Geolonia Inc. Programming: Python, JavaScript, TypeScript, Ruby etc. UNIX and Linux guru GIS skill: Data processing, Tiled based Map Community Director of OSGeo.JP Director of OpenStreetMap Foundation Japan Sub president of Japan Unix Society UNOpenGIS/7 volunteer Contact: [email protected] / @smellman on X JICA Seminar 2025 3
Today's agenda System setup What is tiled map? Introduction of software and data in this presentation How to create your own tiled map How to design your own tiled map How to distribute your own tiled map JICA Seminar 2025 4
System setup This presentation requires Linux based OS. Use Raspberry Pi 4 in this seminar. JICA Seminar 2025 5
System setup - Connect your device https://hackmd.io/@smartmaps/26 Connect to SSID "vectortiles" Launch your terminal application. (Powershell) Access to your host. ssh niroku@<your ip> JICA Seminar 2025 6
System setup - Prepare your device niroku is installer for this seminar. https://github.com/unvt/niroku wget -qO- https://unvt.github.io/niroku/install.sh | sudo -E bash - JICA Seminar 2025 7
System setup - checkout this seminar repository git clone https://github.com/smellman/jica_scripts.git cd jica_scripts/system sudo make JICA Seminar 2025 8
What is tiled map? JICA Seminar 2025 9
Tile technology Provide map image or data over the internet. Map images are separated as tiles. Zoom Level 0 = World Each zoom level doubles in the dimensions. Too many tiles use "Web Mercator" projection. https://a.tile.openstreetmap.org/0/0/0. JICA Seminar 2025 png 10
Useful to web Structure of tile is useful for web. Enable to scroll map smoothly. Enable to zoom up and zoom down map smoothly. HTTP GET request. Tile become known for Google Maps. Tile has existed from the late 1990s. JICA Seminar 2025 11
Zoom Zoom level 0 : 1 file Zoom level 1 : 2 x 2 = 4 files Zoom level 2 : 4 x 4 = 16 files ... Zoom level 18 : 2^18 x 2^18 = 262,144 x 262,144 = 68,719,476,736 files https://maps.gsi.go.jp/help/image/tile Num.png JICA Seminar 2025 12
GET Request Many services use REST API(GET Request). https://.../Z/X/Y.Format Z: Zoom Level X: X coordinate Y: Y coordinate Format: Raster image format(png, jpg, webp) Vector data format(pbf, mvt) JICA Seminar 2025 13
GET Request example https://a.tile.openstreetmap.org/3 /2/4.png Zoom = 3, X = 2, Y = 4, format = png X and Y coordinates start with 0. JICA Seminar 2025 14
Specification Two tile service specifications are popular. Tile Map Service(TMS) Web Map Tile Service(WMTS) TMS is simpler than WMTS. TMS's X Y coordinate is started from bottom left. Same as Cartesian coordinate system. WMTS's X Y coordinate is started from top left. Same as Coordinate system of 2D computer graphics. JICA Seminar 2025 15
JICA Seminar 2025 16
The Y coordinate flipped OpenStreetMap use TMS like protocol but Y coordinate is numbered from top. OpenStreetMap call "Slippy Map". We call xyz tile. {z}/{x}/{y}.png Also we call zxy tile. https://maps.gsi.go.jp/help/image/tile JICA Seminar 2025 17
XYZ tile De facto standard of tiled map. Web Mercator projection Y coordinate flipped TMS Provide REST API {z}/{x}/{y}.{format} Anyone provide "Specification" Too many libraries support XYZ tile. Leaflet, OpenLayers, Maplibre GL JS, Google Maps API etc. JICA Seminar 2025 18
Raster tile (1/3) Provides "rendered image" The image doesn't have any "data". Focus to visualization. JICA Seminar 2025 19
Raster tile (2/3) Provides "Satellite images" or "Aerial photograph" Focus to photography. The image doesn't have any "data" too. JICA Seminar 2025 20
Raster tile (3/3) Provides "data" as image. Focus to data. Population, Temperature, Rainfall, Elevation, etc. The image has "data" as color. Sample raster tiles contain the elevation value obtainable by JICA Seminar 2025 calculating with RGB values. 21
Vector tile (1/2) Provides "Vector data" Each tile contains "Vector data". The tile like a data container. JICA Seminar 2025 22
Vector tile (2/2) Vector tile doesn't have a style. The client renders images with style settings. Easy to rotation and bearing. Supports 3D rendering. Programmable. The client can change the style dynamically. Interactive demo: https://smellman.github.io/osm-sound-demo/ JICA Seminar 2025 23
Vector tile example Multilingual https://openmaptiles.org/languag es/ Enable to change main language dynamically. JICA Seminar 2025 24
Vector tile example Geospatial Information Authority of Japan https://maps.gsi.go.jp/vector/ GSI provides vector tile. Enable to change style dynamically. JICA Seminar 2025 25
Mapbox Vector Tile De facto standard of vector tile. Vector tile specification by Mapbox Inc. Specification A tile encoded by Protocol Buffer format. Desinged for Web Mercator projection. Supports Layers and Features. https://docs.mapbox.com/data/tilesets/guides/vector-tiles-standards/ JICA Seminar 2025 26
Mapbox GL ecosystem and Style Specification Mapbox provides Mapbox GL JS(Web), Mapbox GL Native(Smartphone and Desktop application). Mapbox provides specification of styling. https://docs.mapbox.com/m apbox-gl-js/style-spec/ JICA Seminar 2025 27
Note: Mapbox GL is proprietary software Mapbox GL became proprietary software from end of 2020. Mapbox GL JS is OpenSource software until v1.5. Mapbox GL JS over v2 must require mapbox service’s token. MapLibre GL ecosystems are fork of mapbox OpenSource versions. https://maplibre.org/ Highly recommend to use MapLibre GL JS now. JICA Seminar 2025 28
Tile support libraries - Javascript (1) Leaflet https://leafletjs.com/ Lightweight and easy to use. Supports Mapbox Vector Tile with plugin. OpenLayers https://openlayers.org/ Difficult to use but powerful. Supports Mapbox Vector Tile. JICA Seminar 2025 29
Tile support libraries - Javascript (2) MapLibre GL JS https://maplibre.org/ Easy to use for Mapbox Vector Tile. Supports raster xyz tile too. JICA Seminar 2025 30
Tile support libraries - Android MapLibre GL Native https://maplibre.org/ Easy to use for Mapbox Vector Tile. Supports raster xyz tile too. Google Maps SDK https://developers.google.com/maps/documentation/android-sdk/overview Easy to use for raster xyz tile. JICA Seminar 2025 31
Tile support libraries - iOS MapLibre GL Native https://maplibre.org/ Easy to use for Mapbox Vector Tile. Supports raster xyz tile too. Mapkit https://developer.apple.com/documentation/mapkit Easy to use for raster xyz tile. JICA Seminar 2025 32
Desktop application QGIS https://qgis.org/ Supports raster xyz tile. Supports Mapbox Vector Tile. JICA Seminar 2025 33
Introduction of software and data in this presentation JICA Seminar 2025 34
Requirements This presentation requires Linux based OS. Also, you can use Raspberry Pi 4. Raspberry Pi 4 is cheap and powerful. Raspberry Pi 4 is ARM64/aarch64 architecture. Raspberry Pi 4 is easy to use for GIS. My repository for this presentation supports only ARM64/aarch64 architecture. JICA Seminar 2025 35
Software - GDAL/OGR https://gdal.org/ GDAL/OGR is the most popular GIS library and provides command line tools. QGIS based on GDAL/OGR. GDAL/OGR supports many GIS data formats. GDAL/OGR supports raster xyz tile. JICA Seminar 2025 36
Note: GDAL interface changes GDAL 3.11 or later use gdal command instead of gdalinfo , gdal_translate , ogr2ogr etc. Old commands are still available as symlink to gdal command. This presentation use GDAL 3.10 so old commands are used. JICA Seminar 2025 37
Software - Tippecanoe https://github.com/felt/tippecanoe/ Build vector tilesets from large (or small) collections of GeoJSON, FlatGeoBuf or CSV features. Tippecanoe is the most popular vector tile builder. JICA Seminar 2025 38
Software - Charites Command line tool for writing Mapbox/MapLibre Vector Style Specification in YAML. Organized by The United Nation Vector Tile Toolkit(UNVT). Charites convert Style Specification(JSON) to YAML. YAML is easy to read and write for human. YAML is easy to edit for beginners. Charites enable to dynamic serving style. JICA Seminar 2025 39
Software - editor nano is a simple text editor. nano is easy to use for both beginners. vim is a powerful text editor. vim is difficult to use for beginners. vim is easy to use for experts. JICA Seminar 2025 40
Software - make make is a build automation tool. make is easy to use for both beginners and experts. make is a standard tool of UNIX and Linux. This presentation use make for build and deploy. JICA Seminar 2025 41
Software - caddy caddy is a web server. caddy is easy to use for beginners. This presentation use caddy for serving tiles. JICA Seminar 2025 42
Software - tileserver-gl-light tileserver-gl-light is a vector tile server. useful for inspecting vector tile. JICA Seminar 2025 43
Data - Global Map Digital geographic information Provided by International Steering Committee for Global Mapping(ISCGM). Composed of 8 Data Sets Vector Data (Transportation, Boundaries, Darainage, Population Centre) Raster Data (Elevation, Vegetation, Land Cover, Land Use) Free for non-commercial use. JICA Seminar 2025 44
Global Map - archive Archives and website were moved into github by GSI. https://github.com/globalmaps https://globalmaps.github.io/ Old website was closed Some countries provides global map archives at the national site. All links: https://github.com/globalmaps/projectmanagement/blob/master/REPOS.md Some links are dead now. JICA Seminar 2025 45
Global map – format Vector data provide as Shapefile. It provided as Geography Markup Language (GML) format. Raster data provide as GeoTiff file. It provided as Band interleaved by line (BIL) format. JICA Seminar 2025 46
Data – Aerial photograph https://www.mlit.go.jp/plateau/ In Japan, Plateau Project release too many Aerial photograph data. Plateau released PointCloud, 3D data, and Aerial photograph. Aerial photograph is released as GeoTiff data. It is good sample to create raster tile. JICA Seminar 2025 47
Data - OpenStreetMap https://www.openstreetmap.org/ OpenStreetMap is the most popular OpenData. OpenStreetMap provides planet data as PBF format. Today's presentation use OpenStreetMap data as sample data. Use small area data for easy to understand. JICA Seminar 2025 48
Data for this presentation Global Map Sri Lanka 1.0 https://github.com/globalmaps/gmlk10 Global Map Sri Lanka 2.0 https://github.com/globalmaps/gmlk20 Plateau Higashimurayama City in Tokyo GeoTIFF https://www.geospatial.jp/ckan/dataset/plateau-13213-higashimurayama-shi2020 OpenStreetMap data https://tile.opensteetmap.jp/static/planet.pmtiles JICA Seminar 2025 49
How to create your own tiled map JICA Seminar 2025 50
Raster tile processing pattern 1: Global map (One GeoTIFF file) Download GeoTIFF file from Global Map archive. Enable transparency. Convert GeoTIFF to XYZ tile using gdal2tiles. JICA Seminar 2025 51
How to process cd ~/jica_scripts/raster_tile_gm make fetch # Download GeoTIFF file from Global Map archive. make transparent # Enable transparency. make generate_tile # Convert GeoTIFF to XYZ tile using gdal2tiles. make deploy # copy to caddy server directory. JICA Seminar 2025 52
How to read Makefile (1) fetch: git clone https://github.com/globalmaps/gmlk10.git transparent: gdalbuildvrt -srcnodata "0 0 99" el.vrt gmlk10/el.tif generate_tile: gdal_translate -of vrt -expand rgba el.vrt temp.vrt gdal2tiles.py --xyz -s EPSG:4326 -z 0-11 temp.vrt deploy: sudo cp -r temp/ /opt/niroku/data/gmlk10/ JICA Seminar 2025 53
How to read Makefile (2) Makefile is simple to run tasks. task_name: command command ... JICA Seminar 2025 54
Result Access to http://<your ip>/gmlk10/leaflet.html JICA Seminar 2025 55
Raster tile processing pattern 2: Plateau (Many GeoTIFF files) Generate VRT file from GeoTIFF files. Convert VRT file to XYZ tile using gdal2tiles. JICA Seminar 2025 56
How to process cd ~/jica_scripts/raster_tile_plateau make fetch # Download GeoTIFF file from Plateau archive and unarchive make buildvrt # Generate VRT file from GeoTIFF files. make generate_tile # Convert VRT file to XYZ tile using gdal2tiles. make deploy # copy to caddy server directory. JICA Seminar 2025 57
Result Access to http://<your ip>/plateau/leaflet.html JICA Seminar 2025 58
Vector tile processing pattern: Global map Download Shapefile file from Global Map archive. Convert Shapefile to GeoJSON using ogr2ogr. Convert GeoJSON to Mapbox Vector Tile using tippecanoe. JICA Seminar 2025 59
How to process cd ~/jica_scripts/vector_tile make fetch # Download Shapefile file from Global Map archive. make convert # Convert Shapefile to GeoJSON using ogr2ogr. make generate # Convert GeoJSON to Mapbox Vector Tile using tippecanoe. make tileserver-gl # run tileserver-gl-light JICA Seminar 2025 60
Result Access to http://<your ip>:8000/ JICA Seminar 2025 61
Makefile (1/3) fetch: git clone https://github.com/globalmaps/gmlk20.git convert: cd gmlk20; \ ogr2ogr airp_lka.geojson -s_srs EPSG:4326 -t_srs EPSG:4326 airp_lka.shp; \ ogr2ogr builtupp_lka.geojson -s_srs EPSG:4326 -t_srs EPSG:4326 builtupp_lka.shp; \ ogr2ogr coastl_lka.geojson -s_srs EPSG:4326 -t_srs EPSG:4326 coastl_lka.shp; \ ogr2ogr inwatera_lka.geojson -s_srs EPSG:4326 -t_srs EPSG:4326 inwatera_lka.shp; \ ogr2ogr polbnda_lka.geojson -s_srs EPSG:4326 -t_srs EPSG:4326 polbnda_lka.shp; \ ogr2ogr polbndl_lka.geojson -s_srs EPSG:4326 -t_srs EPSG:4326 polbndl_lka.shp; \ ogr2ogr raill_lka.geojson -s_srs EPSG:4326 -t_srs EPSG:4326 raill_lka.shp; \ ogr2ogr riverl_lka.geojson -s_srs EPSG:4326 -t_srs EPSG:4326 riverl_lka.shp; \ ogr2ogr roadl_lka.geojson -s_srs EPSG:4326 -t_srs EPSG:4326 roadl_lka.shp JICA Seminar 2025 62
9 outputs ogr2ogr convert Shapefile to GeoJSON. Notes: Those Shapefiles are not included .prj file. JICA Seminar 2025 63
Makefile (2/3) generate: tippecanoe -o lka.pmtiles \ -L airp:gmlk20/airp_lka.geojson \ -L builtupp:gmlk20/builtupp_lka.geojson \ -L coastl:gmlk20/coastl_lka.geojson \ -L inwatera:gmlk20/inwatera_lka.geojson \ -L polbnda:gmlk20/polbnda_lka.geojson \ -L polbndl:gmlk20/polbndl_lka.geojson \ -L raill:gmlk20/raill_lka.geojson \ -L riverl:gmlk20/riverl_lka.geojson \ -L roadl:gmlk20/roadl_lka.geojson tippecanoe -o lka.mbtiles \ -L airp:gmlk20/airp_lka.geojson \ -L builtupp:gmlk20/builtupp_lka.geojson \ -L coastl:gmlk20/coastl_lka.geojson \ -L inwatera:gmlk20/inwatera_lka.geojson \ -L polbnda:gmlk20/polbnda_lka.geojson \ -L polbndl:gmlk20/polbndl_lka.geojson \ -L raill:gmlk20/raill_lka.geojson \ -L riverl:gmlk20/riverl_lka.geojson \ JICA Seminar 2025 -L roadl:gmlk20/roadl_lka.geojson 64
2 outputs tippecanoe runs 2 times and generate 2 outputs. .mbtiles file SQLite database file. Contains vector tile. .pmtiles file "Cloud Native" format. You can host .pmtiles as static file. JICA Seminar 2025 65
MBTiles - SQLite database MBTiles is container of tile. MBTiles is single file database(SQLite). TMS schema. JICA Seminar 2025 66
Makefile (3/3) tileserver-gl: tileserver-gl-light --port 8000 --bind 0.0.0.0 --file lka.mbtiles JICA Seminar 2025 67
PMTiles - Cloud Native format PMTiles is similar to MBTiles. "Cloud Native" format. You can easy to convert mbtiles to pmtiles using pmtiles command. https://smellman.github.io/pmtiles-example/ JICA Seminar 2025 68
How to design your own tiled map JICA Seminar 2025 69
Design tiled map Vector tile doesn't have a style. The client renders images with style settings. In this presentation, we use charites to design tiled map. Charites convert Style Specification(JSON) to YAML. YAML is easy to read and write for human. YAML is easy to edit for beginners. JICA Seminar 2025 70
Try to edit style cd ~/jica_scripts/vector_tile sudo make practice Open http://<your ip>:8000/ in your browser. Open other terminal and run following command. cd ~/jica_scripts/vector_tile nano style-practice.yml JICA Seminar 2025 71
nano nano is a simple text editor. nano is easy to use for begineer. Ctrl + O: Save file Ctrl + X: Exit nano JICA Seminar 2025 72
Remove comments on layers Remove comments in style-practice.yml. layers: - !!inc/file layers/background.yml # - !!inc/file layers/polbnda.yml # - !!inc/file layers/riverl.yml # - !!inc/file layers/inwatera.yml # - !!inc/file layers/roadl-primary.yml # - !!inc/file layers/roadl-secondary.yml # - !!inc/file layers/raill-base.yml # - !!inc/file layers/raill-dot.yml # - !!inc/file layers/airp.yml JICA Seminar 2025 73
Layers in Maplibre Style Specification Background Fill Line Symbol Circle Raster Hillshade Fill Extrusion It used for 3D rendering. https://maplibre.org/maplibre-style-spec/ JICA Seminar 2025 74
Background layer id: background type: background paint: background-color: rgb(0,0,0) JICA Seminar 2025 75
Fill layer id: polbnda type: fill source: global_map source-layer: polbnda paint: fill-color: '#f2efe9' source: global_map means "global_map" source in sources section. source-layer: polbnda means "polbnda" layer in global_map source. JICA Seminar 2025 76
Line layer id: riverl type: line source: global_map source-layer: riverl paint: line-color: rgb(0,0,255) line-width: base: 1 stops: - - 6 - 0.5 - - 10 - 2 JICA Seminar 2025 77
Line layer Draw line with polyline features. Normal line. Dash-array line. Following example is Normal line + Dash-array line. JICA Seminar 2025 78
Filter roadl-primary.yml and roadl-secondary.yml use filter. roadl-primary.yml use following filter. filter: - all - - '==' - rtt - '14' 'rtt' is a field name in roadl layer and '14' is primary route. JICA Seminar 2025 79
Zoom function Zoom function is useful to change style dynamically. Set Zoom 6 to 1 and Zoom 10 to 6, the value will increase between Zoom 6 and 10. "base" property will use to control the rate which the function output increases. "base = 1" will be increased linearly. https://maplibre.org/maplibre-style-spec/expressions/ JICA Seminar 2025 80
Symbol layer id: airp type: symbol source: global_map source-layer: airp layout: icon-image: airport_11 text-field: '{nam}' text-offset: - 0 - 0.6 JICA Seminar 2025 81
Symbol layer Draw symbol with point features. Icon Text Text with icon Allow Point, Polygon and Polyline features. JICA Seminar 2025 82
icon-image and text-field icon-image is a property of symbol layer. icon-image is a name of icon. icon-image is defined in sprite. text-field is a property of symbol layer. text-field is a name of field in source data. It can use Feature Properties using {field_name}. JICA Seminar 2025 83
text-offset
text-offset is a property of symbol layer.
text-offset is a offset of text.
It can use array of [x, y].
x and y are offset from center of point.
layout:
icon-image: airport_11
text-field: '{nam}'
text-offset:
- 0
- 0.6
JICA Seminar 2025
84
Convert your style via charites Stop make practice command and run following command. make build Makefile is simple to run tasks. build: charites convert style-practice.yml style-practice.json JICA Seminar 2025 85
Result make deploy Open http://<your ip>/vector/ in your browser. JICA Seminar 2025 86
How to distribute your own tiled map JICA Seminar 2025 87
Raster tile hosting (1) If you use small number of data only, hosting as static image is easy. Use nginx or Apache HTTP Server. Use AWS S3 or Google Cloud Storage. Github Pages is good for small data and free. Be careful to license of tile images. https://docs.github.com/en/pages/getting-started-with-github-pages/about-github-pages JICA Seminar 2025 88
Raster tile hosting (2) If you use large number of data, be careful to hosting. File system limitation: max number of files. Ext4 on Linux: 4,294,967,295 files(specified at filesystem creation time) File copy will take a long time. MBTiles is a solution to hosting large number of data. MBUtil is useful to create .mbtiles from tile images. mb-util temp/ el.mbtiles JICA Seminar 2025 89
Vector tile hosting - tileserver-gl Tileserver GL is useful. But vector tiles needs SSL access in internet. Let's encrypt is useful to get SSL certificate. https://letsencrypt.org/ Setup frontend server(Apache/nginx/etc) and connect from server with reverse proxy. JICA Seminar 2025 90
Overview - nginx JICA Seminar 2025 91
Reverse proxy setting Nginx is easy to setup reverse proxy. location / { proxy_set_header X-Forwarded-Proto https; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_pass http://localhost:8080; } JICA Seminar 2025 92
Server-side rendering Tileserver GL can deliver raster tile if you host style. But rendering is slow. If you use this function, you should use cache server. JICA Seminar 2025 93
Tileserver GL - cache Use varnish cache for raster rendering. https://varnish-cache.org/ https://tile.openstreetmap.jp/ use varnish cache. 20 processs run. 20Core CPU and 64GB RAM. JICA Seminar 2025 94
Vector tile hosting - pmtiles PMTiles is useful to hosting vector tile. PMTiles is "Cloud Native" format. PMTiles is easy to host as static file. Nginx / Apache / AWS S3 / Google Cloud Storage etc. https://github.com/protomaps/PMTiles JICA Seminar 2025 95
PMTiles nginx setting - CORS setting
add_header 'Access-Control-Allow-Origin' "$http_origin" always;
add_header 'Access-Control-Allow-Credentials' 'true' always;
add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, OPTIONS' always;
add_header 'Access-Control-Allow-Headers' 'Accept,Authorization,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,Range,User-Agent,X-Requested-With' always;
if ($request_method = 'OPTIONS') {
# Tell client that this pre-flight info is valid for 20 days
add_header 'Access-Control-Allow-Origin' "$http_origin" always;
add_header 'Access-Control-Allow-Headers' 'Accept,Authorization,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,Range,User-Agent,X-Requested-With' always;
add_header 'Access-Control-Max-Age' 1728000;
add_header 'Content-Type' 'text/plain charset=UTF-8';
add_header 'Content-Length' 0;
return 204;
}
see: https://github.com/smellman/pmtiles-example
JICA Seminar 2025
96
copyright This presentation is licensed under CC BY 4.0. All pictures with OpenStreetMap images are licensed under CC BY-SA 2.0. © OpenStreetMap contributors JICA Seminar 2025 97