mirror of
https://github.com/gravitystorm/openstreetmap-carto.git
synced 2025-12-08 17:36:10 +00:00
Adapt Docker setup to latest changes (#5045)
* Adapt Docker setup to latest changes And use newer versions of base images. * Remove fonts from Dockerfile, as they are downloaded separately * Mention the external data download in Docker documentation * Add more log messages to track the import process * Change URL for Noto Emoji font in (deprecated) get-fonts.sh script
This commit is contained in:
Thomas Pétillon
GitHub
parent
d5522564f8
commit
b7e0691161
@ -1,3 +1,2 @@
|
||||
*
|
||||
!scripts/tune-postgis.sh
|
||||
!openstreetmap-carto.style
|
||||
|
||||
12
DOCKER.md
12
DOCKER.md
@ -11,9 +11,9 @@ PostgreSQL database to store the OpenStreetMap data and [Kosmtik](https://github
|
||||
to be able to run Docker containers. You also need Docker Compose, which should be available once you install
|
||||
Docker itself. Otherwise, you need to [install Docker Compose manually](https://docs.docker.com/compose/install/).
|
||||
|
||||
* You need sufficient disk space of _several Gigabytes_.
|
||||
* Docker creates a disk image for its virtual machine that holds the virtualised operating system and the containers.
|
||||
* The format (Docker.raw, Docker.qcow2, \*.vhdx, etc.) depends on the host system. It can be a sparse file allocating large amounts of disk space, but still the physical size starts with 2-3 GB for the virtual OS and grows to 6-7 GB when filled with the containers needed for the database, Kosmtik, and a chosen small region of OSM data.
|
||||
* You need sufficient disk space of _several Gigabytes_.
|
||||
* Docker creates a disk image for its virtual machine that holds the virtualised operating system and the containers.
|
||||
* The format (Docker.raw, Docker.qcow2, \*.vhdx, etc.) depends on the host system. It can be a sparse file allocating large amounts of disk space, but still the physical size starts with 2-3 GB for the virtual OS and grows to 6-7 GB when filled with the containers needed for the database, Kosmtik, and a chosen small region of OSM data.
|
||||
* An additional 1-2 GB are needed for shapefiles to be downloaded and stored in the openstreetmap-carto/data repository.
|
||||
|
||||
## Quick start
|
||||
@ -23,7 +23,7 @@ If you are eager to get started, here is an overview over the necessary steps:
|
||||
* `git clone https://github.com/gravitystorm/openstreetmap-carto.git` to clone openstreetmap-carto repository into a directory on your host system
|
||||
* Download OpenStreetMap data in osm.pbf format to a file `data.osm.pbf` and place it within the openstreetmap-carto directory (for example some small area from [Geofabrik](https://download.geofabrik.de/))
|
||||
* If necessary, `sudo service postgresql stop` to make sure you don't have a currently-running native PostgreSQL server which would conflict with Docker's PostgreSQL server.
|
||||
* `docker-compose up import` to import the data (only necessary the first time or when you change the data file). Additionally, you can set import options through [environment variables](#Importing-data). More on that [later](#Hands-on-approach)
|
||||
* `docker-compose up import` to import the osm.pbf data file and download additional external data (only necessary the first time or when you change the data file). Additionally, you can set import options through [environment variables](#Importing-data). More on that [later](#Hands-on-approach)
|
||||
* `docker-compose up kosmtik` to run the style preview application
|
||||
* Browse to [http://127.0.0.1:6789](http://127.0.0.1:6789) to view the output of Kosmtik
|
||||
* Ctrl+C to stop the style preview application
|
||||
@ -45,7 +45,7 @@ At startup of the container the script `scripts/docker-startup.sh` is invoked wh
|
||||
|
||||
### Supplying command line options as environment variables
|
||||
|
||||
**osm2pgsql** has a few [command line options](https://manpages.debian.org/testing/osm2pgsql/osm2pgsql.1.en.html) and the import by default uses a RAM cache of 512 MB, 1 worker, **and expects the import file to be named `data.osm.pbf`**.
|
||||
**osm2pgsql** has a few [command line options](https://manpages.debian.org/testing/osm2pgsql/osm2pgsql.1.en.html) and the import by default uses a RAM cache of 512 MB, 1 worker, **and expects the import file to be named `data.osm.pbf`**.
|
||||
If you want to customize any of these parameters, you have to set any number of the following environment variables:
|
||||
* `OSM2PGSQL_CACHE` (e.g. `export OSM2PGSQL_CACHE=1024` on Linux to set the cache to 1 GB) for the RAM cache (the value depends on the amount of RAM you have available - the more you can use here the faster the import may be)
|
||||
* `OSM2PGSQL_NUMPROC` for the number of workers (this depends on the number of processors you have and whether your hard disk is fast enough e.g. is a SSD)
|
||||
@ -74,7 +74,7 @@ EXTERNAL_DATA_SCRIPT_FLAGS="--cache --no-update" \
|
||||
docker-compose up import
|
||||
```
|
||||
|
||||
Note that on Linux you need to export those environment variables before calling docker-compose. If you are using sudo to call docker (because your user is not in the docker group (which we don't recommend)), you need to also use sudo -E option
|
||||
Note that on Linux you need to export those environment variables before calling `docker-compose`. If you are using sudo to call docker (because your user is not in the docker group (which we don't recommend)), you need to also use sudo -E option
|
||||
|
||||
Variables will be remembered in `.env` if you don't have that file, and values in the file will be applied unless you manually assign them. Keep in mind this means if you change your `.env` file, but keep your environment variables untouched (you haven't unset them or you haven't rebooted your host), they will be used instead of anything that you changed in `.env`.
|
||||
|
||||
|
||||
11
Dockerfile
11
Dockerfile
@ -1,18 +1,19 @@
|
||||
FROM ubuntu:focal
|
||||
FROM ubuntu:noble
|
||||
|
||||
# https://serverfault.com/questions/949991/how-to-install-tzdata-on-a-ubuntu-docker-image
|
||||
ARG DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
# Style dependencies
|
||||
RUN apt-get update && apt-get install --no-install-recommends -y \
|
||||
ca-certificates curl gnupg postgresql-client python3 python3-distutils \
|
||||
fonts-hanazono fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted \
|
||||
mapnik-utils nodejs npm ttf-unifont unzip git && rm -rf /var/lib/apt/lists/*
|
||||
ca-certificates gnupg postgresql-client curl unzip python3 \
|
||||
nodejs npm git fonts-unifont mapnik-utils \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Kosmtik with plugins, forcing prefix to /usr because Ubuntu sets
|
||||
# npm prefix to /usr/local, which breaks the install
|
||||
# We install kosmtik not from release channel, but directly from a specific commit on github.
|
||||
RUN npm set prefix /usr && npm install -g --unsafe-perm "git+https://git@github.com/kosmtik/kosmtik.git"
|
||||
# 5dbde8db6b5e22073951066b0646a91c10bb81a5 is master's tip as of 2024-11-17.
|
||||
RUN npm set prefix /usr && npm install -g --unsafe-perm "git+https://git@github.com/kosmtik/kosmtik.git#5dbde8db6b5e22073951066b0646a91c10bb81a5"
|
||||
|
||||
WORKDIR /usr/lib/node_modules/kosmtik/
|
||||
RUN kosmtik plugins --install kosmtik-overpass-layer \
|
||||
|
||||
@ -1,3 +1,3 @@
|
||||
FROM postgis/postgis:10-2.5-alpine
|
||||
FROM postgis/postgis:15-3.5-alpine
|
||||
|
||||
COPY ./scripts/tune-postgis.sh /docker-entrypoint-initdb.d/tune-postgis.sh
|
||||
|
||||
@ -1,21 +1,9 @@
|
||||
FROM ubuntu:bionic
|
||||
FROM ubuntu:noble
|
||||
|
||||
RUN apt-get update && apt-get install --no-install-recommends -y \
|
||||
ca-certificates curl gnupg && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN echo 'deb http://ppa.launchpad.net/osmadmins/ppa/ubuntu bionic main\n\
|
||||
deb-src http://ppa.launchpad.net/osmadmins/ppa/ubuntu bionic main' > \
|
||||
/etc/apt/sources.list.d/osmadmins-ppa.list
|
||||
|
||||
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 \
|
||||
--recv A438A16C88C6BE41CB1616B8D57F48750AC4F2CB
|
||||
|
||||
RUN apt-get update && apt-get install --no-install-recommends -y \
|
||||
osm2pgsql gdal-bin python3-psycopg2 python3-yaml unzip \
|
||||
osm2pgsql gdal-bin python3-psycopg2 python3-yaml curl unzip \
|
||||
python3-requests postgresql-client && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
ADD openstreetmap-carto-flex.lua /
|
||||
|
||||
RUN mkdir -p /openstreetmap-carto
|
||||
WORKDIR /openstreetmap-carto
|
||||
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
version: '2'
|
||||
services:
|
||||
kosmtik:
|
||||
image: kosmtik:v1
|
||||
image: openstreetmap-carto-kosmtik:v1
|
||||
platform: linux/amd64
|
||||
build:
|
||||
context: .
|
||||
@ -16,7 +16,7 @@ services:
|
||||
- PGHOST=db
|
||||
- PGUSER=postgres
|
||||
db:
|
||||
image: db:v1
|
||||
image: openstreetmap-carto-db:v1
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.db
|
||||
@ -25,7 +25,7 @@ services:
|
||||
- PG_WORK_MEM
|
||||
- PG_MAINTENANCE_WORK_MEM
|
||||
import:
|
||||
image: import:v1
|
||||
image: openstreetmap-carto-import:v1
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.import
|
||||
|
||||
@ -18,9 +18,11 @@ test $i -gt $MAXCOUNT && echo "Timeout while waiting for PostgreSQL to be runnin
|
||||
case "$1" in
|
||||
import)
|
||||
# Creating default database
|
||||
psql -c "SELECT 1 FROM pg_database WHERE datname = 'gis';" | grep -q 1 || createdb gis && \
|
||||
psql -d gis -c 'CREATE EXTENSION IF NOT EXISTS postgis;' && \
|
||||
psql -d gis -c 'CREATE EXTENSION IF NOT EXISTS hstore;' && \
|
||||
echo "Creating database"
|
||||
psql -c "SELECT 1 FROM pg_database WHERE datname = 'gis';" | grep -q 1 || createdb gis
|
||||
psql -d gis -c 'CREATE EXTENSION IF NOT EXISTS postgis;'
|
||||
psql -d gis -c 'CREATE EXTENSION IF NOT EXISTS hstore;'
|
||||
psql -d gis -c 'ALTER SYSTEM SET jit=off;' -c 'SELECT pg_reload_conf();'
|
||||
|
||||
# Creating default import settings file editable by user and passing values for osm2pgsql
|
||||
if [ ! -e ".env" ]; then
|
||||
@ -40,6 +42,7 @@ EOF
|
||||
fi
|
||||
|
||||
# Importing data to a database
|
||||
echo "Importing data from $OSM2PGSQL_DATAFILE"
|
||||
osm2pgsql \
|
||||
--cache $OSM2PGSQL_CACHE \
|
||||
--number-processes $OSM2PGSQL_NUMPROC \
|
||||
@ -50,6 +53,11 @@ EOF
|
||||
--style openstreetmap-carto-flex.lua \
|
||||
$OSM2PGSQL_DATAFILE
|
||||
|
||||
# Setting up indexes and functions
|
||||
echo "Setting up indexes and functions"
|
||||
psql -d gis -f indexes.sql
|
||||
psql -d gis -f functions.sql
|
||||
|
||||
# Downloading and importing needed shapefiles
|
||||
scripts/get-external-data.py $EXTERNAL_DATA_SCRIPT_FLAGS
|
||||
|
||||
|
||||
@ -313,7 +313,7 @@ def main():
|
||||
config["settings"]["metadata_table"])
|
||||
|
||||
for name, source in config["sources"].items():
|
||||
logging.info("Checking table {}".format(name))
|
||||
logging.info("Table {}".format(name))
|
||||
# Don't attempt to handle strange names
|
||||
# Even if there was code to escape them properly here, you don't want
|
||||
# in a style with all the quoting headaches
|
||||
@ -327,6 +327,7 @@ def main():
|
||||
config["settings"]["metadata_table"])
|
||||
this_table.clean_temp()
|
||||
|
||||
logging.info(" Downloading file {}".format(source["url"]))
|
||||
# This will fetch data needed for import
|
||||
download = d.download(source["url"], name, opts, data_dir, this_table.last_modified())
|
||||
|
||||
|
||||
@ -8,8 +8,9 @@ mkdir -p "${FONTDIR}"
|
||||
|
||||
# download filename url
|
||||
download() {
|
||||
echo "Downloading file $2"
|
||||
## Download if newer, and if curl fails, clean up and exit
|
||||
curl --fail --compressed -A "get-fonts.sh/osm-carto" -o "$1" -z "$1" -L "$2" || { echo "Failed to download $1 $2"; rm -f "$1"; exit 1; }
|
||||
curl --fail -s --compressed -A "get-fonts.sh/osm-carto" -o "$1" -z "$1" -L "$2" || { echo "Failed to download $1 $2"; rm -f "$1"; exit 1; }
|
||||
}
|
||||
|
||||
# TTF Hinted Noto Fonts
|
||||
@ -85,6 +86,8 @@ NotoSansYi"
|
||||
|
||||
# Download the fonts in the lists above
|
||||
|
||||
echo "Downloading fonts"
|
||||
|
||||
for font in $REGULAR_BOLD_ITALIC; do
|
||||
regular="$font-Regular.ttf"
|
||||
bold="$font-Bold.ttf"
|
||||
@ -122,12 +125,12 @@ TMPDIR=$(mktemp -d -t get-fonts.XXXXXXXXX)
|
||||
trap "rm -rf ${TMPDIR} ${FONTDIR}/static" EXIT
|
||||
|
||||
# Noto Emoji B&W isn't available as a separate download, so we need to download the package and unzip it
|
||||
curl --fail -A "get-fonts.sh/osm-carto" -o "${TMPDIR}/Noto_Emoji.zip" -L 'https://fonts.google.com/download?family=Noto%20Emoji'
|
||||
download "${TMPDIR}/Noto_Emoji.zip" 'https://archive.org/download/noto-emoji/Noto_Emoji.zip'
|
||||
|
||||
unzip -oqq "${TMPDIR}/Noto_Emoji.zip" static/NotoEmoji-Regular.ttf static/NotoEmoji-Bold.ttf -d "${FONTDIR}"
|
||||
mv "${FONTDIR}/static/NotoEmoji-Regular.ttf" "${FONTDIR}"
|
||||
mv "${FONTDIR}/static/NotoEmoji-Bold.ttf" "${FONTDIR}"
|
||||
|
||||
curl --fail -A "get-fonts.sh/osm-carto" -o "${TMPDIR}/hanazono.zip" -L 'https://mirrors.dotsrc.org/osdn/hanazono-font/68253/hanazono-20170904.zip'
|
||||
download "${TMPDIR}/hanazono.zip" 'https://mirrors.dotsrc.org/osdn/hanazono-font/68253/hanazono-20170904.zip'
|
||||
|
||||
unzip -oqq "${TMPDIR}/hanazono.zip" HanaMinA.ttf HanaMinB.ttf -d "${FONTDIR}"
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user