Running in Azure

[mikeasharp]

I have tried various methods to get Nominatim to work in Azure.

I have tried Kubernetes, Container Apps etc. All seem to fail, likely due to the startup import process and memory requirements.

Has anyone managed to get a redundant scalable version of this working Azure, and can they point me to any resources that would assist with this?

Thanks

[baxpick]

Coming late to the party :)
I want the same thing and managed to create POC on Azure by using Dockerfile on Azure Virtual Machine, basically just running the example "docker run" command from the virtual machine command line.

Does someone has a better idea then to do it like this? Is Azure Web Apps option?

[Dimii96]

Despite managing to get a local instance of the docker container to run locally, I have not been able to deploy it to an Azure Web App environment. Will continue to try to get it working but believe I may need to consider doing the same as @baxpick and deploy to a Azure VM instead.

[drrollergator]

I've been running 4.3 in azure for several years. I just did a whole new deployment this weekend.

What sort of issues are you running into?

If we're simply trying to run an instance of the image in a container I might be able to guide you in the right direction.

[baxpick]

@drrollergator Than you for spending time to actually give some full code example and useful tips.

However, when I tried to run your .bicep example (only docker hub username and pass are updated) I get a crash with not very usable log error (SIGSEGV)

I executed this inside my usual azure environment, a Ubuntu dev container with latest az CLI.

I have also tried to create all using terraform since this is the tool I need to use.

variable "docker_username" {
description = "Docker username"
type        = string
default     = "???"
}

variable "docker_password" {
description = "Docker password"
type        = string
sensitive   = true
default     = "???"
}

resource "azurerm_resource_group" "example" {
name     = "example-resources"
location = "West Europe"
}

resource "azurerm_storage_account" "example" {
name                     = "mynomvmsandboxsa"
resource_group_name      = azurerm_resource_group.example.name
location                 = azurerm_resource_group.example.location
account_tier             = "Standard"
account_replication_type = "LRS"
}

resource "azurerm_storage_share" "flatnode" {
name                 = "aci-test-share-flatnode"
storage_account_name = azurerm_storage_account.example.name
quota                = 200
}

resource "azurerm_storage_share" "postgresql" {
name                 = "aci-test-share-postgresql"
storage_account_name = azurerm_storage_account.example.name
quota                = 200
}

resource "azurerm_container_group" "example" {
name                = "example-container-group"
location            = azurerm_resource_group.example.location
resource_group_name = azurerm_resource_group.example.name
os_type             = "Linux"
ip_address_type     = "Public"

container {
name   = "nominatim"
image  = "mediagis/nominatim:4.4"
cpu    = 4
memory = 16
ports {
  port     = 8080
  protocol = "TCP"
}

environment_variables = {
    POSTGRES_SHARED_BUFFERS="2GB"
    POSTGRES_MAINTAINENCE_WORK_MEM="10GB"
    POSTGRES_AUTOVACUUM_WORK_MEM="2GB"
    POSTGRES_WORK_MEM="50MB"
    POSTGRES_EFFECTIVE_CACHE_SIZE="24GB"
    POSTGRES_SYNCHRONOUS_COMMIT="off"
    POSTGRES_MAX_WAL_SIZE="1GB"
    POSTGRES_CHECKPOINT_TIMEOUT="10min"
    POSTGRES_CHECKPOINT_COMPLETITION_TARGET="0.9"
    PBF_URL="https://download.geofabrik.de/europe/monaco-latest.osm.pbf"
    REPLICATION_URL="https://download.geofabrik.de/europe/monaco-updates/"
    REPLICATION_UPDATE_INTERVAL="43200"
    REPLICATION_RECHECK_INTERVAL="450"
    UPDATE_MODE="continuous"
    FREEZE="false"
    REVERSE_ONLY="false"
    IMPORT_WIKIPEDIA="false"
    IMPORT_US_POSTCODES="false"
    IMPORT_GB_POSTCODES="false"
    IMPORT_STYLE="full"
    IMPORT_TIGER_ADDRESSES="false"
    THREADS="10"
    NOMINATIM_PASSWORD="12345"
}

volume {
    name        = "flatnode" 
    mount_path  = "/nominatim/flatnode" # Where to mount inside the container
    read_only   = false

    share_name    = azurerm_storage_share.flatnode.name
    storage_account_name = azurerm_storage_account.example.name
    storage_account_key  = azurerm_storage_account.example.primary_access_key # Required for Azure File Share volumes
} 

volume {
    name        = "postgresql" 
    mount_path  = "/var/lib/postgresql/14/main" # Where to mount inside the container
    read_only   = false

    share_name    = azurerm_storage_share.postgresql.name
    storage_account_name = azurerm_storage_account.example.name
    storage_account_key  = azurerm_storage_account.example.primary_access_key # Required for Azure File Share volumes
} 

which should match (except for the --shm-size=15g parameter) my already correct example run command:

sudo docker run 
-v "${FOLDER_flatnode}":/nominatim/flatnode 
-v "${FOLDE_database}":/var/lib/postgresql/14/main 
-e POSTGRES_SHARED_BUFFERS=2GB 
-e POSTGRES_MAINTAINENCE_WORK_MEM=10GB 
-e POSTGRES_AUTOVACUUM_WORK_MEM=2GB 
-e POSTGRES_WORK_MEM=50MB 
-e POSTGRES_EFFECTIVE_CACHE_SIZE=24GB 
-e POSTGRES_SYNCHRONOUS_COMMIT=off 
-e POSTGRES_MAX_WAL_SIZE=1GB 
-e POSTGRES_CHECKPOINT_TIMEOUT=10min 
-e POSTGRES_CHECKPOINT_COMPLETITION_TARGET=0.9 
-e PBF_URL=https://download.geofabrik.de/europe/monaco-latest.osm.pbf 
-e REPLICATION_URL=https://download.geofabrik.de/europe/monaco-updates/ 
-e REPLICATION_UPDATE_INTERVAL=43200 
-e REPLICATION_RECHECK_INTERVAL=450 
-e UPDATE_MODE=continuous 
-e FREEZE=false 
-e REVERSE_ONLY=false 
-e IMPORT_WIKIPEDIA=false 
-e IMPORT_US_POSTCODES=false 
-e IMPORT_GB_POSTCODES=false 
-e IMPORT_STYLE=full 
-e IMPORT_TIGER_ADDRESSES=false 
-e THREADS=10 
--shm-size=15g 
-e NOMINATIM_PASSWORD=12345 
-p 8080:8080 
--name nominatim 
mediagis/nominatim:4.4

Result is that Azure container is constantly restarting and never able to finish.

Would you or someone know how to correctly setup this using terraform?

For me Azure Virtual Machine still looks like a best choice since I can not debug problems with containers like this - even "Logs" sometimes show and sometimes don't show on Azure portal.

[drrollergator]

While there may be more than one cause, I have spent roughly 140 hours down rabbit holes in the last month on something you are attempting to do and can explain a large portion of what you are dealing with.

resource "azurerm_storage_share" "postgresql" {
name                 = "aci-test-share-postgresql"
storage_account_name = azurerm_storage_account.example.name
quota                = 200
}

[...]

volume {
    name        = "postgresql" 
    mount_path  = "/var/lib/postgresql/14/main" # Where to mount inside the container
    read_only   = false

    share_name    = azurerm_storage_share.postgresql.name
    storage_account_name = azurerm_storage_account.example.name
    storage_account_key  = azurerm_storage_account.example.primary_access_key # Required for Azure File Share volumes
} 

This will not work. When azure deploys and runs the container instance it does a bind-mount and not a volume-mount. The Bind mount presents an empty storage (or whatever is in the storage volume) to the host OS. This causes the nominatim docker scripts to attempt to install postgres. However, postgres demands postgres:postgres be the sole owner of the path /var/lib/postgresql/14/main.

using dive you can see postgres:postgres (101:103 uid:gid) has exclusive permission on main

drwxr-xr-x     101:103      43 MB      │   ├── postgresql
drwxr-xr-x     101:103      43 MB      │   │   └── 14
drwx------     101:103      43 MB      │   │       └─⊕ main

The image is aware of this and tries to accommodate. /app/init.sh line 66

# if we use a bind mount then the PG directory is empty and we have to create it
if [ ! -f /var/lib/postgresql/14/main/PG_VERSION ]; then
  chown postgres:postgres /var/lib/postgresql/14/main
  sudo -u postgres /usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main
fi

However after the bind-mount the permission set is drwxrwxrwx and it is not owned by postgres:postgres. This is because the container is not running as privileged and can't change the ownership of a mounted volume. This is at least one reason your container is stuck in a loop. The moment the install attempts to try to chown main it fails and initdb immediately reboots. I was finally able to catch this by pushing my own image that called /app/start.sh with stdout and stderr redirected to a log file on an external volume.

Because the container instances do not run in full root, there is no way to specify in the mount options the uid and gid. In fact the lack of this capability is reflected in the ARM/Bicep schemas not including mountOptions

Once you see that you can't specify mountOptions, this particular avenue is lost (but there is an alternative approach, which I'll get to).

Now, Azure Kubernetes does allow you to specify mountOptions, as Kubernetes is running essentially inside an isolated VM already you get the privileges necessary to perform the mount as you need. However you will hit a brick wall when the container goes into re-install postgres because you will not have hard link support. Apparently in the postgres install uses hard links and it will error. I do not know of a way to overcome this.

So eventually I abandoned using volumes to attempt this persistent database route.

However, simply using an external database does work, in principle.

If you setup an Azure Database for PostgreSQL flexible server you can get the container instance to use it. You will not be able to avoid creating a new image with nominatim-docker as the base, though.

For example following directions here you can create your own entry point and get the import process started.

if [ -z ${NOMINATIM_DATABASE_DSN+x} ]; then
    echo "NOMINATIM_DATABASE_DSN is not set" 2>&1 | tee -a $STARTUP_LOG
    /app/start.sh 2>&1 | tee -a $STARTUP_LOG
else
    echo "NOMINATIM_DATABASE_DSN is set" 2>&1 | tee -a $STARTUP_LOG
    nominatim import --prepare-database 2>&1 | tee -a $STARTUP_LOG
    psql -d nominatim -c 'GRANT CREATE ON SCHEMA public TO "$PGUSER"' 2>&1 | tee -a $STARTUP_LOG
    NOMINATIM_DATABASE_DSN="pgsql:dbname=nominatim;user=$PGUSER;host=$PGHOST;port=$PGPORT" nominatim import --continue import-from-file --osm-file /mnt/osm/state-latest.osm.pbf 2>&1 | tee -a $STARTUP_LOG
fi

But you will need to do a little more work, as the current startup scripts look for the file /var/lib/postgresql/14/main/import-finished to determine if the import is done or not and I may have executed

/app/startup.sh line 28

IMPORT_FINISHED=/var/lib/postgresql/14/main/import-finished

if [ ! -f ${IMPORT_FINISHED} ]; then
  /app/init.sh
  touch ${IMPORT_FINISHED}
else
  chown -R nominatim:nominatim ${PROJECT_DIR}
fi

since we're not storing main on a volume, that will be lost on container reboot, so you will need to review all the startup scripts normally executed and come up with your own logic to determine if the import is needed (we don't want init.sh to run as-is, as we already did it ourselves with the external database) and get the Apache server up and running to server requests.

I had to tend to other things so I have not finished my full startup scripts, otherwise I'd provide them to you.

I hope this helps you get closer to solutions. You can go the VM only route but you might have more flexibility if you ultimately use the Azure postgres solution. I haven't fully field tested it yet, however.

[baxpick]

Thanks again for useful response and sharing your findings.

I was finally able to catch this by pushing my own image that called /app/start.sh with stdout and stderr redirected to a log file on an external volume.

You will not be able to avoid creating a new image with nominatim-docker as the base, though.

But you will need to do a little more work...

Exactly why I wanted to use VM and I would use it if there is no need to later maintain and monitor this VM for security reasons.
When using "azurerm_container_group" I transfer responsibility to maintain docker container host machine to Azure (at least I think this should be the case).

I successfully used Azure VM with managed disk which I partitioned and filled with database data so it can later be detached and attached to any VM like this:

data "azurerm_managed_disk" "existing_disk" {
  name = "my-random-disk-name"
  resource_group_name = "my-random-resource-group-name"
}

resource "azurerm_virtual_machine_data_disk_attachment" "attach" {
  managed_disk_id  = data.azurerm_managed_disk.existing_disk.id
  virtual_machine_id = azurerm_linux_virtual_machine.vm.id
  lun = "42"
  caching = "ReadWrite"
}

But, again, it's not secure enough when compared to "azurerm_container_group", so I will try to make it work without VM although I still have concerns if this will be possible for entire planet since there is not that much flexibility when configuring "azurerm_container_group" when compared to "azurerm_linux_virtual_machine".

[AlexOcampos]

@baxpick Have you finally been able to do it? I'm trying to set up exactly the same thing, nominatim in an Azure Container App with the entire planet.

On the one hand, with a Container App Job I have managed to download and preprocess the maps in a fileshare, and I have set up the osrm-route to calculate distances in a Container App using the same fileshare.
Now, my intention is to use the same storage with Nominatim, but I am having precisely the restart problems that you mention.

[baxpick]

@AlexOcampos No, I went back to Azure VM with managed disk option since I don't feel confident to make a lot of changes and VM does give a lot of flexibility/power.

But, if someone else has a working solution without VM, I would gladly take a look and try :)

[baxpick]

@drrollergator @AlexOcampos I found this, not sure if it helps running in container... #510 ?