a stop job is running for the rabbitmq messaging server

[ltalirz]

when rebooting QM, the rabbitmq server refuses to shut down gracefully.

Might be related
https://bbs.archlinux.org/viewtopic.php?id=230406
https://bugs.archlinux.org/task/55842

[ltalirz]

This is the (confirmed) fix for Ubuntu 18.04 (to be automated in AiiDA role):
https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/1784757

sudo nano /lib/systemd/system/rabbitmq-server.service

# fixup the [Unit] entry to look like this:
[Unit]
Description=RabbitMQ Messaging Server
After=network.target [email protected]
Wants=network.target [email protected]

# change just the ExecStop under [Service] to use "shutdown" instead of "stop":
ExecStop=/usr/sbin/rabbitmqctl shutdown

# reload and reboot:
sudo systemctl stop rabbitmq-server.service
sudo systemctl daemon-reload
sudo systemctl restart rabbitmq-server.service
sudo shutdown -r now

[ConradJohnston]

Hi @ltalirz and @chrisjsewell ,

Possibly related - I (and some students) have noticed that in the current release (20.06.01) that RabbitMQ sometimes falls over and refuses to restart. Any insight?

[ltalirz]

@ConradJohnston Is there any error message in the logs of the rabbitmq service?

[ConradJohnston]

@ConradJohnston Is there any error message in the logs of the rabbitmq service?

Hi @ltalirz,

Please, see the output below for some standard operations. The logs are fairly sparse.

$sudo rabbitmqctl start_app
Starting node rabbit@qmobile
Error: unable to connect to node rabbit@qmobile: nodedown

DIAGNOSTICS
===========

attempted to contact: [rabbit@qmobile]

rabbit@qmobile:
  * connected to epmd (port 4369) on qmobile
  * epmd reports: node 'rabbit' not running at all
                  no other nodes on qmobile
  * suggestion: start the node

current node details:
- node name: 'rabbitmq-cli-35@qmobile'
- home dir: /var/lib/rabbitmq
- cookie hash: l7p49vC5UHCoNTwmTK8y4A==

$sudo rabbitmq-server start

              RabbitMQ 3.6.10. Copyright (C) 2007-2017 Pivotal Software, Inc.
  ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/
  ##  ##
  ##########  Logs: /var/log/rabbitmq/[email protected]
  ######  ##        /var/log/rabbitmq/[email protected]
  ##########
              Starting broker...


BOOT FAILED
===========

Error description:
   {could_not_start,rabbit,
       {{badmatch,
            {error,
                {{{badmatch,
                      {error,
                          {"/var/lib/rabbitmq/mnesia/rabbit@qmobile/msg_store_transient/0.rdq",
                           eacces}}},
                  [{rabbit_msg_store,init,1,
                       [{file,"src/rabbit_msg_store.erl"},{line,726}]},
                   {gen_server2,init_it,6,
                       [{file,"src/gen_server2.erl"},{line,554}]},
                   {proc_lib,init_p_do_apply,3,
                       [{file,"proc_lib.erl"},{line,247}]}]},
                 {child,undefined,msg_store_transient,
                     {rabbit_msg_store,start_link,
                         [msg_store_transient,
                          "/var/lib/rabbitmq/mnesia/rabbit@qmobile",undefined,
                          {#Fun<rabbit_variable_queue.0.61220017>,ok}]},
                     transient,30000,worker,
                     [rabbit_msg_store]}}}},
        [{rabbit_variable_queue,start_msg_store,2,
             [{file,"src/rabbit_variable_queue.erl"},{line,482}]},
         {rabbit_variable_queue,start,1,
             [{file,"src/rabbit_variable_queue.erl"},{line,467}]},
         {rabbit_priority_queue,start,1,
             [{file,"src/rabbit_priority_queue.erl"},{line,92}]},
         {rabbit_amqqueue,recover,0,
             [{file,"src/rabbit_amqqueue.erl"},{line,238}]},
         {rabbit,recover,0,[{file,"src/rabbit.erl"},{line,799}]},
         {rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,
             [{file,"src/rabbit_boot_steps.erl"},{line,49}]},
         {rabbit_boot_steps,run_step,2,
             [{file,"src/rabbit_boot_steps.erl"},{line,49}]},
         {rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,
             [{file,"src/rabbit_boot_steps.erl"},{line,26}]}]}}

Log files (may contain more information):
   /var/log/rabbitmq/[email protected]
   /var/log/rabbitmq/[email protected]

{"init terminating in do_boot",{could_not_start,rabbit,{{badmatch,{error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@qmobile/msg_store_transient/0.rdq",eacces}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,726}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,554}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@qmobile",undefined,{#Fun<rabbit_variable_queue.0.61220017>,ok}]},transient,30000,worker,[rabbit_msg_store]}}}},[{rabbit_variable_queue,start_msg_store,2,[{file,"src/rabbit_variable_queue.erl"},{line,482}]},{rabbit_variable_queue,start,1,[{file,"src/rabbit_variable_queue.erl"},{line,467}]},{rabbit_priority_queue,start,1,[{file,"src/rabbit_priority_queue.erl"},{line,92}]},{rabbit_amqqueue,recover,0,[{file,"src/rabbit_amqqueue.erl"},{line,238}]},{rabbit,recover,0,[{file,"src/rabbit.erl"},{line,799}]},{rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,[{file,"src/rabbit_boot_steps.erl"},{line,49}]},{rabbit_boot_steps,run_step,2,[{file,"src/rabbit_boot_steps.erl"},{line,49}]},{rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,[{file,"src/rabbit_boot_steps.erl"},{line,26}]}]}}}
init terminating in do_boot ({could_not_start,rabbit,{{badmatch,{error,{_}}},[{rabbit_variable_queue,start_msg_store,2,[_]},{rabbit_variable_queue,start,1,[_]},{rabbit_priority_queue,start,1,[_]},{rab

Crash dump is being written to: erl_crash.dump...done

I've currently 'fixed' this by upgrading all the apt packages suggesting that it might be a dependancy issue (Erlang?). Trying to see if I can break it again.

[ltalirz]

Thanks for the report @ConradJohnston

@chrisjsewell Would be great if you could give a look whether anything needs to be done on our side to update the apt packages in the next release.
@ConradJohnston : would you mind listing the ones you upgraded and the versions you now have?

[ConradJohnston]

Thanks for the report @ConradJohnston

@chrisjsewell Would be great if you could give a look whether anything needs to be done on our side to update the apt packages in the next release.
@ConradJohnston : would you mind listing the ones you upgraded and the versions you now have?

There was quite a lot.

Here the log from apt:

Start-Date: 2020-09-09  14:28:13
Commandline: apt upgrade
Requested-By: max (1001)
Install: motd-news-config:amd64 (10.1ubuntu2.10, automatic), libllvm10:amd64 (1:10.0.0-4ubuntu1~18.04.2, automatic), linux-image-4.15.0-117-generic:amd64 (4.15.0-117.118, automatic), libpcre2-8-0:amd64 (10.31-2, automatic), linux-modules-4.15.0-117-generic:amd64 (4.15.0-117.118, automatic), linux-modules-extra-4.15.0-117-generic:amd64 (4.15.0-117.118, automatic)
Upgrade: libpam0g:amd64 (1.1.8-3.6ubuntu2.18.04.1, 1.1.8-3.6ubuntu2.18.04.2), evolution-data-server-common:amd64 (3.28.5-0ubuntu0.18.04.2, 3.28.5-0ubuntu0.18.04.3), libpython3.6-minimal:amd64 (3.6.9-1~18.04ubuntu1, 3.6.9-1~18.04ubuntu1.1), netplan.io:amd64 (0.99-0ubuntu3~18.04.1, 0.99-0ubuntu3~18.04.3), libseccomp2:amd64 (2.4.1-0ubuntu0.18.04.2, 2.4.3-1ubuntu3.18.04.3), libisccfg160:amd64 (1:9.11.3+dfsg-1ubuntu1.12, 1:9.11.3+dfsg-1ubuntu1.13), libedataserverui-1.2-2:amd64 (3.28.5-0ubuntu0.18.04.2, 3.28.5-0ubuntu0.18.04.3), intel-microcode:amd64 (3.20191115.1ubuntu0.18.04.2, 3.20200609.0ubuntu0.18.04.1), whoopsie:amd64 (0.2.62ubuntu0.4, 0.2.62ubuntu0.5), python-samba:amd64 (2:4.7.6+dfsg~ubuntu-0ubuntu2.17, 2:4.7.6+dfsg~ubuntu-0ubuntu2.18), linux-libc-dev:amd64 (4.15.0-101.102, 4.15.0-117.118), xserver-common:amd64 (2:1.19.6-1ubuntu4.4, 2:1.19.6-1ubuntu4.6), libcurl4:amd64 (7.58.0-2ubuntu3.8, 7.58.0-2ubuntu3.10), libldap-2.4-2:amd64 (2.4.45+dfsg-1ubuntu1.5, 2.4.45+dfsg-1ubuntu1.6), python3-software-properties:amd64 (0.96.24.32.13, 0.96.24.32.14), libpam-modules:amd64 (1.1.8-3.6ubuntu2.18.04.1, 1.1.8-3.6ubuntu2.18.04.2), openssl:amd64 (1.1.1-1ubuntu2.1~18.04.5, 1.1.1-1ubuntu2.1~18.04.6), xserver-xorg-core:amd64 (2:1.19.6-1ubuntu4.4, 2:1.19.6-1ubuntu4.6), libvncclient1:amd64 (0.9.11+dfsg-1ubuntu1.2, 0.9.11+dfsg-1ubuntu1.3), sane-utils:amd64 (1.0.27-1~experimental3ubuntu2.2, 1.0.27-1~experimental3ubuntu2.3), libssh-4:amd64 (0.8.0~20170825.94fa1e38-1ubuntu0.6, 0.8.0~20170825.94fa1e38-1ubuntu0.7), libpython3.6-dev:amd64 (3.6.9-1~18.04ubuntu1, 3.6.9-1~18.04ubuntu1.1), libegl-mesa0:amd64 (19.2.8-0ubuntu0~18.04.3, 20.0.8-0ubuntu1~18.04.1), libwbclient0:amd64 (2:4.7.6+dfsg~ubuntu-0ubuntu2.17, 2:4.7.6+dfsg~ubuntu-0ubuntu2.18), git-man:amd64 (1:2.17.1-1ubuntu0.7, 1:2.24.0-1~nd18.04+1), libsystemd0:amd64 (237-3ubuntu10.40, 237-3ubuntu10.42), libpulsedsp:amd64 (1:11.1-1ubuntu7.8, 1:11.1-1ubuntu7.10), python3.6-venv:amd64 (3.6.9-1~18.04ubuntu1, 3.6.9-1~18.04ubuntu1.1), grub-common:amd64 (2.02-2ubuntu8.15, 2.02-2ubuntu8.18), linux-image-generic:amd64 (4.15.0.101.91, 4.15.0.117.104), pulseaudio:amd64 (1:11.1-1ubuntu7.8, 1:11.1-1ubuntu7.10), libgs9:amd64 (9.26~dfsg+0-0ubuntu0.18.04.12, 9.26~dfsg+0-0ubuntu0.18.04.13), libkmod2:amd64 (24-1ubuntu3.4, 24-1ubuntu3.5), libglapi-mesa:amd64 (19.2.8-0ubuntu0~18.04.3, 20.0.8-0ubuntu1~18.04.1), python2.7-minimal:amd64 (2.7.17-1~18.04ubuntu1, 2.7.17-1~18.04ubuntu1.1), snapd:amd64 (2.42.1+18.04, 2.45.1+18.04.2), libsqlite3-0:amd64 (3.22.0-1ubuntu0.3, 3.22.0-1ubuntu0.4), librsvg2-common:amd64 (2.40.20-2, 2.40.20-2ubuntu0.2), libpython3.6-stdlib:amd64 (3.6.9-1~18.04ubuntu1, 3.6.9-1~18.04ubuntu1.1), binutils:amd64 (2.30-21ubuntu1~18.04.3, 2.30-21ubuntu1~18.04.4), libirs160:amd64 (1:9.11.3+dfsg-1ubuntu1.12, 1:9.11.3+dfsg-1ubuntu1.13), libwhoopsie0:amd64 (0.2.62ubuntu0.4, 0.2.62ubuntu0.5), libsnmp-base:amd64 (5.7.3+dfsg-1.8ubuntu3.3, 5.7.3+dfsg-1.8ubuntu3.6), bind9-host:amd64 (1:9.11.3+dfsg-1ubuntu1.12, 1:9.11.3+dfsg-1ubuntu1.13), open-iscsi:amd64 (2.0.874-5ubuntu2.9, 2.0.874-5ubuntu2.10), dnsutils:amd64 (1:9.11.3+dfsg-1ubuntu1.12, 1:9.11.3+dfsg-1ubuntu1.13), libpython2.7:amd64 (2.7.17-1~18.04ubuntu1, 2.7.17-1~18.04ubuntu1.1), python2.7:amd64 (2.7.17-1~18.04ubuntu1, 2.7.17-1~18.04ubuntu1.1), python3-pil:amd64 (5.1.0-1ubuntu0.2, 5.1.0-1ubuntu0.3), python3-pip:amd64 (9.0.1-2.3~ubuntu1.18.04.1, 9.0.1-2.3~ubuntu1.18.04.2), linux-base:amd64 (4.5ubuntu1.1, 4.5ubuntu1.2), libpython3.6:amd64 (3.6.9-1~18.04ubuntu1, 3.6.9-1~18.04ubuntu1.1), ubuntu-desktop:amd64 (1.417.4, 1.417.5), python3.6:amd64 (3.6.9-1~18.04ubuntu1, 3.6.9-1~18.04ubuntu1.1), python-apt-common:amd64 (1.6.5ubuntu0.2, 1.6.5ubuntu0.3), xserver-xorg-legacy:amd64 (2:1.19.6-1ubuntu4.4, 2:1.19.6-1ubuntu4.6), git:amd64 (1:2.17.1-1ubuntu0.7, 1:2.24.0-1~nd18.04+1), libxatracker2:amd64 (19.2.8-0ubuntu0~18.04.3, 20.0.8-0ubuntu1~18.04.1), libwinpr2-2:amd64 (2.1.1+dfsg1-0ubuntu0.18.04.1, 2.2.0+dfsg1-0ubuntu0.18.04.1), grub2-common:amd64 (2.02-2ubuntu8.15, 2.02-2ubuntu8.18), libisc169:amd64 (1:9.11.3+dfsg-1ubuntu1.12, 1:9.11.3+dfsg-1ubuntu1.13), udev:amd64 (237-3ubuntu10.40, 237-3ubuntu10.42), locales:amd64 (2.27-3ubuntu1, 2.27-3ubuntu1.2), evolution-data-server:amd64 (3.28.5-0ubuntu0.18.04.2, 3.28.5-0ubuntu0.18.04.3), ubuntu-server:amd64 (1.417.4, 1.417.5), libpam-runtime:amd64 (1.1.8-3.6ubuntu2.18.04.1, 1.1.8-3.6ubuntu2.18.04.2), libcurl4-gnutls-dev:amd64 (7.58.0-2ubuntu3.9, 7.58.0-2ubuntu3.10), libpq5:amd64 (10.12-0ubuntu0.18.04.1, 10.14-0ubuntu0.18.04.1), software-properties-gtk:amd64 (0.96.24.32.13, 0.96.24.32.14), grub-pc:amd64 (2.02-2ubuntu8.15, 2.02-2ubuntu8.18), openjdk-11-jre-headless:amd64 (11.0.7+10-2ubuntu2~18.04, 11.0.8+10-0ubuntu1~18.04.1), libx11-6:amd64 (2:1.6.4-3ubuntu0.2, 2:1.6.4-3ubuntu0.3), python3-httplib2:amd64 (0.9.2+dfsg-1ubuntu0.1, 0.9.2+dfsg-1ubuntu0.2), kmod:amd64 (24-1ubuntu3.4, 24-1ubuntu3.5), libudev1:amd64 (237-3ubuntu10.40, 237-3ubuntu10.42), cifs-utils:amd64 (2:6.8-1, 2:6.8-1ubuntu1), libgbm1:amd64 (19.2.8-0ubuntu0~18.04.3, 20.0.8-0ubuntu1~18.04.1), nplan:amd64 (0.99-0ubuntu3~18.04.1, 0.99-0ubuntu3~18.04.3), thunderbird-gnome-support:amd64 (1:68.8.0+build2-0ubuntu0.18.04.2, 1:68.10.0+build1-0ubuntu0.18.04.1), libpq-dev:amd64 (10.12-0ubuntu0.18.04.1, 10.14-0ubuntu0.18.04.1), python3.6-minimal:amd64 (3.6.9-1~18.04ubuntu1, 3.6.9-1~18.04ubuntu1.1), samba-libs:amd64 (2:4.7.6+dfsg~ubuntu-0ubuntu2.17, 2:4.7.6+dfsg~ubuntu-0ubuntu2.18), binutils-x86-64-linux-gnu:amd64 (2.30-21ubuntu1~18.04.3, 2.30-21ubuntu1~18.04.4), grub-pc-bin:amd64 (2.02-2ubuntu8.15, 2.02-2ubuntu8.18), xserver-xephyr:amd64 (2:1.19.6-1ubuntu4.4, 2:1.19.6-1ubuntu4.6), ubuntu-minimal:amd64 (1.417.4, 1.417.5), apport:amd64 (2.20.9-0ubuntu7.14, 2.20.9-0ubuntu7.17), libebackend-1.2-10:amd64 (3.28.5-0ubuntu0.18.04.2, 3.28.5-0ubuntu0.18.04.3), libsane1:amd64 (1.0.27-1~experimental3ubuntu2.2, 1.0.27-1~experimental3ubuntu2.3), ppp:amd64 (2.4.7-2+2ubuntu1.2, 2.4.7-2+2ubuntu1.3), libsnmp30:amd64 (5.7.3+dfsg-1.8ubuntu3.3, 5.7.3+dfsg-1.8ubuntu3.6), libxau6:amd64 (1:1.0.8-1, 1:1.0.8-1ubuntu1), postgresql-10:amd64 (10.12-0ubuntu0.18.04.1, 10.14-0ubuntu0.18.04.1), libfreerdp2-2:amd64 (2.1.1+dfsg1-0ubuntu0.18.04.1, 2.2.0+dfsg1-0ubuntu0.18.04.1), libisc-export169:amd64 (1:9.11.3+dfsg-1ubuntu1.12, 1:9.11.3+dfsg-1ubuntu1.13), python3-distupgrade:amd64 (1:18.04.37, 1:18.04.38), libc-bin:amd64 (2.27-3ubuntu1, 2.27-3ubuntu1.2), libebook-1.2-19:amd64 (3.28.5-0ubuntu0.18.04.2, 3.28.5-0ubuntu0.18.04.3), libpulse0:amd64 (1:11.1-1ubuntu7.8, 1:11.1-1ubuntu7.10), chromium-browser:amd64 (83.0.4103.61-0ubuntu0.18.04.1, 85.0.4183.83-0ubuntu0.18.04.2), ubuntu-release-upgrader-core:amd64 (1:18.04.37, 1:18.04.38), librsvg2-2:amd64 (2.40.20-2, 2.40.20-2ubuntu0.2), python3-apport:amd64 (2.20.9-0ubuntu7.15, 2.20.9-0ubuntu7.17), libnss-myhostname:amd64 (237-3ubuntu10.41, 237-3ubuntu10.42), libpulse-mainloop-glib0:amd64 (1:11.1-1ubuntu7.8, 1:11.1-1ubuntu7.10), samba-common:amd64 (2:4.7.6+dfsg~ubuntu-0ubuntu2.17, 2:4.7.6+dfsg~ubuntu-0ubuntu2.18), systemd-sysv:amd64 (237-3ubuntu10.40, 237-3ubuntu10.42), chromium-codecs-ffmpeg-extra:amd64 (83.0.4103.61-0ubuntu0.18.04.1, 85.0.4183.83-0ubuntu0.18.04.2), openjdk-11-jre:amd64 (11.0.7+10-2ubuntu2~18.04, 11.0.8+10-0ubuntu1~18.04.1), libldap-common:amd64 (2.4.45+dfsg-1ubuntu1.5, 2.4.45+dfsg-1ubuntu1.6), libwayland-egl1-mesa:amd64 (19.2.8-0ubuntu0~18.04.3, 20.0.8-0ubuntu1~18.04.1), liblwres160:amd64 (1:9.11.3+dfsg-1ubuntu1.12, 1:9.11.3+dfsg-1ubuntu1.13), libpam-systemd:amd64 (237-3ubuntu10.40, 237-3ubuntu10.42), xwayland:amd64 (2:1.19.6-1ubuntu4.4, 2:1.19.6-1ubuntu4.6), ghostscript:amd64 (9.26~dfsg+0-0ubuntu0.18.04.12, 9.26~dfsg+0-0ubuntu0.18.04.13), systemd:amd64 (237-3ubuntu10.40, 237-3ubuntu10.42), libwebkit2gtk-4.0-37:amd64 (2.28.2-0ubuntu0.18.04.1, 2.28.4-0ubuntu0.18.04.1), libssl-dev:amd64 (1.1.1-1ubuntu2.1~18.04.5, 1.1.1-1ubuntu2.1~18.04.6), libpam-modules-bin:amd64 (1.1.8-3.6ubuntu2.18.04.1, 1.1.8-3.6ubuntu2.18.04.2), libx11-data:amd64 (2:1.6.4-3ubuntu0.2, 2:1.6.4-3ubuntu0.3), iproute2:amd64 (4.15.0-2ubuntu1.1, 4.15.0-2ubuntu1.2), ghostscript-x:amd64 (9.26~dfsg+0-0ubuntu0.18.04.12, 9.26~dfsg+0-0ubuntu0.18.04.13), libgl1-mesa-dri:amd64 (19.2.8-0ubuntu0~18.04.3, 20.0.8-0ubuntu1~18.04.1), libsmbclient:amd64 (2:4.7.6+dfsg~ubuntu-0ubuntu2.17, 2:4.7.6+dfsg~ubuntu-0ubuntu2.18), bcache-tools:amd64 (1.0.8-2build1, 1.0.8-2ubuntu0.18.04.1), samba-common-bin:amd64 (2:4.7.6+dfsg~ubuntu-0ubuntu2.17, 2:4.7.6+dfsg~ubuntu-0ubuntu2.18), libdns-export1100:amd64 (1:9.11.3+dfsg-1ubuntu1.12, 1:9.11.3+dfsg-1ubuntu1.13), sosreport:amd64 (3.9-1ubuntu0.18.04.3, 3.9.1-1ubuntu0.18.04.2), libmysqlclient20:amd64 (5.7.30-0ubuntu0.18.04.1, 5.7.31-0ubuntu0.18.04.1), libnss-systemd:amd64 (237-3ubuntu10.40, 237-3ubuntu10.42), libgs9-common:amd64 (9.26~dfsg+0-0ubuntu0.18.04.12, 9.26~dfsg+0-0ubuntu0.18.04.13), thunderbird:amd64 (1:68.8.0+build2-0ubuntu0.18.04.2, 1:68.10.0+build1-0ubuntu0.18.04.1), binutils-common:amd64 (2.30-21ubuntu1~18.04.3, 2.30-21ubuntu1~18.04.4), libnetplan0:amd64 (0.99-0ubuntu3~18.04.1, 0.99-0ubuntu3~18.04.3), libgl1-mesa-glx:amd64 (19.2.8-0ubuntu0~18.04.3, 20.0.8-0ubuntu1~18.04.1), apport-gtk:amd64 (2.20.9-0ubuntu7.15, 2.20.9-0ubuntu7.17), ubuntu-release-upgrader-gtk:amd64 (1:18.04.37, 1:18.04.38), libsane-common:amd64 (1.0.27-1~experimental3ubuntu2.2, 1.0.27-1~experimental3ubuntu2.3), libbinutils:amd64 (2.30-21ubuntu1~18.04.3, 2.30-21ubuntu1~18.04.4), libebook-contacts-1.2-2:amd64 (3.28.5-0ubuntu0.18.04.2, 3.28.5-0ubuntu0.18.04.3), gir1.2-webkit2-4.0:amd64 (2.28.2-0ubuntu0.18.04.1, 2.28.4-0ubuntu0.18.04.1), libpython2.7-minimal:amd64 (2.7.17-1~18.04ubuntu1, 2.7.17-1~18.04ubuntu1.1), multiarch-support:amd64 (2.27-3ubuntu1, 2.27-3ubuntu1.2), libgnutls30:amd64 (3.5.18-1ubuntu1.3, 3.5.18-1ubuntu1.4), pulseaudio-module-bluetooth:amd64 (1:11.1-1ubuntu7.8, 1:11.1-1ubuntu7.10), libnss3:amd64 (2:3.35-2ubuntu2.9, 2:3.35-2ubuntu2.12), libedata-book-1.2-25:amd64 (3.28.5-0ubuntu0.18.04.2, 3.28.5-0ubuntu0.18.04.3), libpcap0.8:amd64 (1.8.1-6ubuntu1.18.04.1, 1.8.1-6ubuntu1.18.04.2), ca-certificates:amd64 (20180409, 20190110~18.04.1), firefox:amd64 (78.0.1+build1-0ubuntu0.18.04.1, 80.0.1+build1-0ubuntu0.18.04.1), libisccc160:amd64 (1:9.11.3+dfsg-1ubuntu1.12, 1:9.11.3+dfsg-1ubuntu1.13), libedata-cal-1.2-28:amd64 (3.28.5-0ubuntu0.18.04.2, 3.28.5-0ubuntu0.18.04.3), linux-firmware:amd64 (1.173.18, 1.173.19), libssl1.1:amd64 (1.1.1-1ubuntu2.1~18.04.5, 1.1.1-1ubuntu2.1~18.04.6), libbind9-160:amd64 (1:9.11.3+dfsg-1ubuntu1.12, 1:9.11.3+dfsg-1ubuntu1.13), python3.6-dev:amd64 (3.6.9-1~18.04ubuntu1, 3.6.9-1~18.04ubuntu1.1), postgresql-server-dev-10:amd64 (10.12-0ubuntu0.18.04.1, 10.14-0ubuntu0.18.04.1), libjson-c3:amd64 (0.12.1-1.3ubuntu0.2, 0.12.1-1.3ubuntu0.3), libdns1100:amd64 (1:9.11.3+dfsg-1ubuntu1.12, 1:9.11.3+dfsg-1ubuntu1.13), libcamel-1.2-61:amd64 (3.28.5-0ubuntu0.18.04.2, 3.28.5-0ubuntu0.18.04.3), chromium-browser-l10n:amd64 (83.0.4103.61-0ubuntu0.18.04.1, 85.0.4183.83-0ubuntu0.18.04.2), libecal-1.2-19:amd64 (3.28.5-0ubuntu0.18.04.2, 3.28.5-0ubuntu0.18.04.3), libpython2.7-stdlib:amd64 (2.7.17-1~18.04ubuntu1, 2.7.17-1~18.04ubuntu1.1), ansible:amd64 (2.9.10-1ppa~bionic, 2.9.13-1ppa~bionic), pulseaudio-utils:amd64 (1:11.1-1ubuntu7.8, 1:11.1-1ubuntu7.10), curl:amd64 (7.58.0-2ubuntu3.8, 7.58.0-2ubuntu3.10), postgresql-client-10:amd64 (10.12-0ubuntu0.18.04.1, 10.14-0ubuntu0.18.04.1), rpcbind:amd64 (0.2.3-0.6, 0.2.3-0.6ubuntu0.18.04.1), libjavascriptcoregtk-4.0-18:amd64 (2.28.2-0ubuntu0.18.04.1, 2.28.4-0ubuntu0.18.04.1), gir1.2-rsvg-2.0:amd64 (2.40.20-2, 2.40.20-2ubuntu0.2), python3-problem-report:amd64 (2.20.9-0ubuntu7.14, 2.20.9-0ubuntu7.17), python-pip-whl:amd64 (9.0.1-2.3~ubuntu1.18.04.1, 9.0.1-2.3~ubuntu1.18.04.2), libedataserver-1.2-23:amd64 (3.28.5-0ubuntu0.18.04.2, 3.28.5-0ubuntu0.18.04.3), libfreerdp-client2-2:amd64 (2.1.1+dfsg1-0ubuntu0.18.04.1, 2.2.0+dfsg1-0ubuntu0.18.04.1), rabbitmq-server:amd64 (3.6.10-1ubuntu0.1, 3.6.10-1ubuntu0.3), nfs-common:amd64 (1:1.3.4-2.1ubuntu5.2, 1:1.3.4-2.1ubuntu5.3), libcurl3-gnutls:amd64 (7.58.0-2ubuntu3.9, 7.58.0-2ubuntu3.10), libx11-xcb1:amd64 (2:1.6.4-3ubuntu0.2, 2:1.6.4-3ubuntu0.3), python3-apt:amd64 (1.6.5ubuntu0.2, 1.6.5ubuntu0.3), gir1.2-javascriptcoregtk-4.0:amd64 (2.28.2-0ubuntu0.18.04.1, 2.28.4-0ubuntu0.18.04.1), base-files:amd64 (10.1ubuntu2.8, 10.1ubuntu2.10), libglx-mesa0:amd64 (19.2.8-0ubuntu0~18.04.3, 20.0.8-0ubuntu1~18.04.1), python-httplib2:amd64 (0.9.2+dfsg-1ubuntu0.1, 0.9.2+dfsg-1ubuntu0.2), software-properties-common:amd64 (0.96.24.32.13, 0.96.24.32.14)
End-Date: 2020-09-09  14:33:30

And here's my current list of installed packages:
installed_packages.txt

[chrisjsewell]

This is the (confirmed) fix for Ubuntu 18.04 (to be automated in AiiDA role):

So I guess this is still the fix to make? what about the other distributions, would the fix be the same?

Amending /lib/systemd/system/rabbitmq-server.service should not be too difficult 🤞
I guess the service is anyhow reloaded/rebooted if you are launching from a pre-built image, so those steps are not quite as critical?

might be a dependancy issue (Erlang?).

That definitely looks like an erlang incompatibility, although I don't see how it would ever start if erlang was incompatible?

I actually added these docs, after a similar issue with rabbitmq installed from conda: https://aiida.readthedocs.io/projects/aiida-core/en/latest/intro/troubleshooting.html#rabbitmq-installation-unix
We could certainly check rabbitmq/erlang against the compatibility matrix: https://www.rabbitmq.com/which-erlang.html

I've currently 'fixed' this by upgrading all the apt package

Note we do run apt upgrade in marvel-nccr.simulationbase, so at least the baseline packages should be up to date (obviously only at the time of the original build)

[chrisjsewell]

although actually the packages are not upgraded, as noted here: #127 (comment)