Generating Tilesets

cargo build --package martin --bin martin --features unstable-generate

A long-requested Martin feature is generating tiles on the fly from a feature source (raw data) inside Martin itself, when serving tiles. Until now, a source had to either already contain tiles (MBTiles, PMTiles) or generate them itself, like PostgreSQL does with ST_AsMVT.

Martin adapted the tile generation method of Planetiler. Thank you to Mike Barry (@msbarry), the author of Planetiler, for an amazing tool, and to its contributors. Planetiler is licensed under the Apache License 2.0. Martin also adapted a similar layer configuration, which takes features from any source Martin supports that can produce a stream of features. Generation Architecture lists what this takes from Planetiler.

As part of this effort, martin cp can now also build a whole tileset at once, like Planetiler. By default it asks the source for each tile separately, so the source reads a feature again for every tile it touches, at every zoom. With --mode bulk, it instead reads each source once, renders every feature at every zoom inside Martin, and writes the tiles in order. For large tilesets this is much faster, and it puts far less load on the source. That also saves money: Martin instances are stateless and cheap to scale out, for example in Kubernetes, while scaling PostgreSQL is much harder and more expensive.

The default --mode auto still picks tile. Once bulk mode covers enough, it will pick bulk whenever it can.

It renders points, lines and polygons. Geometry collections are skipped and counted.

Usage

martin cp --mode bulk                   \
          --output-file tileset.mbtiles \
          --source roads,shops          \
          --max-zoom 14                 \
          postgres://postgres@localhost:5432/db

This renders the sources roads and shops into tileset.mbtiles, at zooms 0 to 14. Without --source, every source is rendered. Sources come from the same connection strings or configuration file as for the tile server.

Each source becomes one layer, named by its layer_id or its source id. A source with layers feeds several layers from one read instead, each with its own features, zooms, attributes and draw order.

The same data and options give the same tiles, however many threads are used.

Configuration file

The configuration file only describes the sources. The options of a run are command-line arguments, and the file has no section for martin cp.

martin cp --mode bulk --config config.yaml --output-file tileset.pmtiles --max-zoom 14

postgres:
  connection_string: postgres://postgres@localhost:5432/db
  tables:
    shops:
      schema: public
      table: shops
      geometry_column: geom
      id_column: osm_id
      minzoom: 12
      properties:
        name: text
        shop: text
    roads:
      schema: public
      table: roads
      geometry_column: geom
      properties:
        class: text
        name: text
      layers:
        roads:
          attributes: [class]
        road_labels:
          minzoom: 10
          where: "name != null"
          attributes: [name]

Options

The tile server's configuration and PostgreSQL arguments also apply, such as --config and --pool-size. A configuration file and connection strings cannot be combined.

Bulk mode refuses the martin cp options it does not read: --url-query, --mbtiles-type, --on-duplicate, --concurrency, --skip-agg-tiles-hash, --set-meta, --save-config and --max-feature-count. Tile mode refuses --memory, --temp-dir and --threads.

--bbox filters features, which tile mode does not. A low-zoom tile at the edge of the box only holds the features near the box, not everything the tile covers.

The highest zoom is 27. The tile grid can lower it further, since extent × 2^zoom must not exceed 2^30. With the default extent of 4096, that is zoom 18.

A failed or interrupted run removes the output file. Temporary files are always removed.

Sizing a run

• --memory is split evenly over the render workers. Each gets at least 16 MiB and at most 4 GiB, so with many threads the total can exceed --memory. More memory means fewer, longer sorted runs. Up to 256 runs are merged at once. With more, groups of runs are merged into longer ones first, which reads and writes the temporary data one more time.
• The temporary data holds a piece of every feature for every tile it touches, at every zoom. It grows with the zoom range and is usually several times the size of the input. Merging more than 256 runs needs room for one more copy.
• --temp-dir takes the sorted runs in turn. Directories on separate disks spread the I/O.
• --threads sets the number of render workers and encoders.

Generation Architecture explains how a run reads, sorts, encodes and writes the tiles, and what bounds its memory.