3
votes

Summary

I have .geojson files and want to convert them (with tippecanoe / geobuf / other) to .mbtiles or .pbf files to serve them as vector tiles from a server (TileServer-GL / OpenMapTiles / other vector tile server) into Google Maps with Deck.GL's MVTLayer.

Expected results:

  1. To be able to serve from TileServer-GL .pbf file instead of .mbtiles file.
  2. To be able to serve from TileServer-GL multiple .mbtiles (or .pbf) files from a folder without explicitly starting it with a specific .mbtiles file.

Actual results:

  1. As presented below, I'm using Google Maps with Deck.GL MVTLayer integration with vector tiling to serve specific .mbtiles file from TileServer-GL, as collection of shapes or points.
  2. Didn't manage anything.

My attempts

const map = new google.maps.Map(document.getElementById('container'), {
        center: { lat: 51.47, lng: 0.45 },
        zoom: 10
    });
const deckOverlay = new deck.GoogleMapsOverlay({
        layers: [
            new deck.MVTLayer({
                //working               
                data: `http://localhost:8080/data/SA1_2016-AU-tippecanoe/{z}/{x}/{y}.pbf`,
                //expected, but server NOT starting
                //data: `http://localhost:8080/data/SA1_2016-AU-geobuf/{z}/{x}/{y}.pbf`,

                minZoom: 0,
                maxZoom: 23,
                getLineColor: [1, 1, 1],
                getFillColor: [0, 153, 76],
                pickable: true,
                autoHighlight: true,
                onClick: info => info.object && console.log('onClick', info.object)
            })
        ]
    });
deckOverlay.setMap(map);

I'm running TileServer-GL from Docker, from the the folder that contains data files:

docker run --rm -it -v ${pwd}:/data -p 8080:80 maptiler/tileserver-gl --verbose --mbtiles SA1_2016-AU-tippecanoe.mbtiles

From Docker, I'm converting .geojson files to .mbtiles files through tippecanoe. However, it seems the conversion takes some time for bigger files. I'm obtaining a 890 MB .mbtiles file in aprox. 60 minutes from a 45 MB .geojson file with 57k features.

docker run -it --rm -v ${pwd}:/data tippecanoe:latest tippecanoe --output=/data/SA1_2016-AU-tippecanoe.mbtiles /data/SA1_2016-AU.geojson

I managed to faster convert a .geojson file directly to a .pbf file through geobuf (json2geobuf). I'm obtaining a 32 MB .pbf file in aprox. 37 seconds from a 45 MB .geojson file with 57k features.

json2geobuf SA1_2016-AU.geojson > SA1_2016-AU-geobuf.pbf

However, it seems i'm not able to directly serve .pbf files from TileServer-GL, nor from OpenMapTiles-Server.


I tried

docker run --rm -it -v ${pwd}:/data -p 8080:80 maptiler/tileserver-gl --verbose --mbtiles SA1_2016-AU-geobuf.pbf

but TileServer-GL not starting & i'm getting

ERROR: Metadata missing in the MBTiles.
       Make sure SA1_2016-AU-geobuf.pbf is valid MBTiles

I also tried re-runing after creating locally a config.json file

docker run --rm -it -v ${pwd}:/data -p 8080:80 maptiler/tileserver-gl --verbose --mbtiles SA1_2016-AU-geobuf.pbf
docker run --rm -it -v ${pwd}:/data -p 8080:80 maptiler/tileserver-gl --verbose SA1_2016-AU-geobuf.pbf

config.json

{
  "options": {
    "paths": {
      "root": "/usr/src/app/node_modules/tileserver-gl-styles",
      "fonts": "fonts",
      "styles": "styles",
      "mbtiles": "/data"
    }
  },
  "styles": {},
  "data": {
    "SA1_2016-AU-geobuf": {
      "mbtiles": "SA1_2016-AU-geobuf.pbf"
    }
  }
}

but TileServer-GL not starting & i'm getting

SQLITE_NOTADB: file is not a database

On OpenMapTiles server I don't even find how to specify the input file (reference):

docker run --rm -it -v ${pwd}:/data -p 8080:80 klokantech/openmaptiles-server

My questions

  1. How can I directly serve .pbf files instead of .mbtiles files with TileServer-GL or OpenMapTiles servers?
  2. How can I use TileServer-GL to serve all the files (.mbtiles) form a folder, without explicitly starting it with a specific .mbtiles file docker ... maptiler/tileserver-gl --mbtiles some-file.mbtiles?

1

1 Answers

0
votes

Geobuf pbf is not a tiled dataset, but a smaller more compact binary form vs geojson (https://github.com/mapbox/geobuf/blob/master/README.md#geobuf). You would still need to run that geobuf through tippecanoe to create an mbtiles or directory of pbf in a z/x/y tiled scheme (https://github.com/mapbox/tippecanoe#input-files-and-layer-names).

You would see a much faster runtime with tippecanoe if you converted that geojson to ndjson using tippecanoe-json-tool input.geojson > output_nd.geojson, the utilize the -P flag for parallel processing the output_nd.geojson to mbtiles. tippecanoe parallel processing is also enabled by default if the input for tippecanoe is a geobuf file (https://github.com/mapbox/tippecanoe#parallel-processing-of-input). A simple tippecanoe cmd using the ndjson file would be tippecanoe -t /dev/shm -o output.mbtiles -P -Z 1 -z 12 --drop-smallest-as-needed output_nd.geojson.

As for tileserver-gl you can server as many mbtiles files as you want, but you need to provide your own config.json, mount it to the container -v ${pwd}/config:/config, and explicitly call on it with the --config flag in your docker run cmd.

A simple shell script like below would allow finer grain control over the styles if using custom styles, fonts if using custom fonts, sprites if using custom sprites, along with your data and config files.

#!/usr/bin/env bash

TILESERVER_HOME=/tileserver-gl

DATA_DIR=${TILESERVER_HOME}/mbtiles/
CONFIG_DIR=${TILESERVER_HOME}/config/
STYLE_DIR=${TILESERVER_HOME}/styles/
SPRITES_DIR=${TILESERVER_HOME}/sprites/
FONT_DIR=${TILESERVER_HOME}/fonts/

docker stop tileserver-gl && docker rm tileserver-gl

docker run --log-driver json-file --log-opt compress=true --log-opt max-size=100m --log-opt max-file=3 -d --name tileserver-gl --restart=always -v ${SPRITES_DIR}:/sprites -v ${DATA_DIR}:/data -v ${STYLE_DIR}:/styles -v ${FONT_DIR}:/fonts -v ${CONFIG_DIR}:/config -p 0.0.0.0:8080:80 maptiler/tileserver-gl:latest --verbose --config /config/config.json

An example config.json to go with the above docker cmd listing multiple mbtiles files.

{
    "options": {
        "paths": {
            "root": "",
            "fonts": "../fonts",
            "mbtiles": "../data",
            "styles": "../styles",
            "sprites": "../sprites"
        }
    },
    "data": {
        "STREETS": {
            "mbtiles": "STREETS.mbtiles"
        },
        "LANDMASS": {
            "mbtiles": "LANDMASS.mbtiles"
        },
        "CONTOUR": {
            "mbtiles": "CONTOUR.mbtiles"
        },
        "HILLSHADE": {
            "mbtiles": "HILLSHADE.mbtiles"
        },
        "OSM": {
            "mbtiles": "OSM.mbtiles"
        }
    },
    "styles": {
        "STREETS": {
            "style": "STREETS/style.json",
            "serve_data": true,
            "serve_rendered": true,
            "tilejson": {
                "format": "png",
                "bounds": [
                    -120,
                    -50,
                    160,
                    80
                ]
            }
        },
        "OSM-POSITRON": {
            "style": "OSM-POSITRON/style.json",
            "serve_data": true,
            "serve_rendered": true,
            "tilejson": {
                "format": "png",
                "bounds": [
                    4.3026,
                    48.6085,
                    4.42742,
                    48.707
                ]
            }
        }
    }
}