RabbitMQ in Action (2): Running and administering Rabbit

Server management

the Erlang node and the Erlang application

Starting nodes

multiple Erlang applications can run inside the same node

an application on node asparagus can call functions in applications running on node artichoke as though those functions were local.

Also, if an application crashes for any reason (say, RabbitMQ crashing) the Erlang node will automatically attempt to restart the application

when we talk about RabbitMQ nodes, we’re referring to the RabbitMQ application and the Erlang node it runs on.

root@OpenstackIcehouse2:~# ps aux | grep rabbit
rabbitmq  2021  0.0  0.0   7488   312 ?        S    19:18   0:00 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
rabbitmq  2377  0.0  0.0   4444   648 ?        S    19:18   0:00 /bin/sh /usr/sbin/rabbitmq-server
rabbitmq  2393  1.1  2.0 1161984 81856 ?       Sl   19:18   1:40 /usr/lib/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@OpenstackIcehouse2 -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@OpenstackIcehouse2.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@OpenstackIcehouse2-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@OpenstackIcehouse2-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@OpenstackIcehouse2"
rabbitmq  2740  0.0  0.0   7456   420 ?        Ss   19:19   0:00 inet_gethost 4
rabbitmq  2741  0.0  0.0   9552   624 ?        S    19:19   0:00 inet_gethost 4

epmd:

• The Erlang endpoint mapper daemon
• Allows distributed Erlang applications to find one another
• Becomes important if we want to do things like RabbitMQ clustering

rabbitmq-server

• RabbitMQ start-up script

beam.smp

• The SMP Erlang VM
• Started by the erl command in rabbitmq-server
• erl is essentially a front-end to beam
• This is the thing that’s actually running RabbitMQ

inet_gethost

• Started by beam to do name service lookups

service rabbitmq-server start

Stopping nodes

./sbin/rabbitmqctl stop

root@OpenstackIcehouse2:/var/log/rabbitmq# rabbitmqctl status
Status of node rabbit@OpenstackIcehouse2 ...
[{pid,6398},
{running_applications,[{rabbit,"RabbitMQ","3.2.4"},
                        {os_mon,"CPO  CXC 138 46","2.2.14"},
                        {mnesia,"MNESIA  CXC 138 12","4.11"},
                        {xmerl,"XML parser","1.3.5"},
                        {sasl,"SASL  CXC 138 11","2.3.4"},
                        {stdlib,"ERTS  CXC 138 10","1.19.4"},
                        {kernel,"ERTS  CXC 138 10","2.16.4"}]},
{os,{unix,linux}},
{erlang_version,"Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:2:2] [async-threads:30] [kernel-poll:true]\n"},
{memory,[{total,97816240},
          {connection_procs,1142376},
          {queue_procs,492920},
          {plugins,0},
          {other_proc,13534496},
          {mnesia,134496},
          {mgmt_db,0},
          {msg_index,46936},
          {other_ets,807528},
          {binary,60441128},
          {code,16522377},
          {atom,594537},
          {other_system,4099446}]},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,1658186956},
{disk_free_limit,50000000},
{disk_free,194296217600},
{file_descriptors,[{total_limit,924},
                    {total_used,26},
                    {sockets_limit,829},
                    {sockets_used,24}]},
{processes,[{limit,1048576},{used,362}]},
{run_queue,0},
{uptime,359}]
...done.

Stopping and restarting the application: what’s the difference?

sometimes you just want to restart the RabbitMQ application and keep the Erlang node running.

For clustering, it’s required.

Since rabbitmq-server starts both the node and the application, it preconfigures the RabbitMQ application for standalone mode.

To add the node to an existing cluster, what you need to do is stop the application and reset the node to a pristine state so it can be prepared for clustering.

If you were to use ./rabbitmqctl stop you’d shutdown both the application and the node, forcing you to start both in standalone mode again via ./rabbitmq-server.

./rabbitmqctl stop_app

Rabbit configuration files

/etc/rabbitmq/rabbitmq.config

1) [
2) {mnesia, [{dump_log_write_threshold, 1000}]},
3) {rabbit, [{vm_memory_high_watermark, 0.4}]}
4) ].

each Erlang application gets its own hashtable for its configuration options

mnesia specifies configuration options for the Mnesia database (Mnesia is what RabbitMQ uses for storing exchange and queue metadata).

rabbit specifies RabbitMQ-specific configuration options.

{[option_name], [option_value]}

Mnesia configuration options

dump_log_write_threshold:

• Integer
• 100
• How often to flush/dump the contents of the append-only log into the actual database files. It’s specified in terms of how many entries must be in the log before a dump operation occurs. Setting this to a higher number can reduce I/O load and increase performance for persistent messages.

Rabbit configuration options:

tcp_listeners:

• Array of {"ip_address", port_number}
• [{"0.0.0.0", 5672},]
• Defines which IP addresses and ports RabbitMQ should listen on for non-SSL-encrypted traffic.

ssl_listeners:

• Array of {"ip_address", port_number}
• Defines which IP addresses and ports RabbitMQ should listen on for SSL-encrypted traffic.

ssl_options:

• Array of {"key", value}
• Specifies SSL-related options. Valid options are cacertfile (CA certificate file), certfile (server certificate file), keyfile (server key file) and fail_if_no_peer_cert (require client to have a valid certificate: True/False).

vm_memory_high_watermark:

• Decimal percentage
• 0.4
• Controls how much memory RAM RabbitMQ is allowed to consume. It’s specified in terms of a decimal number representing the percentage of installed memory Rabbit is allowed to use (0.4 = 40%).

msg_store_file_size_limit:

• Integer (bytes)
• 16777216
• The maximum size of the message store DB before RabbitMQ garbage collects the contents of the store.

queue_index_max_journal_entries:

• Integer
• 262144
• The maximum number of entries in the message store journal before they’re flushed into the message store DB and committed.

Asking permission

access control lists

a single user can be granted permissions across multiple vhosts.

Managing users

$ ./rabbitmqctl add_user cashing-tier cashMe1
Creating user "cashing-tier" ...
...done.

$ ./rabbitmqctl delete_user cashing-tier
Deleting user "cashing-tier" ...
...done.

when you delete a user, any access control entries referencing that user will be deleted automatically from the Rabbit permissions database.

$ ./rabbitmqctl list_users
Listing users ...
cashing-tier
guest
...done.

$ ./rabbitmqctl change_password cashing-tier compl3xPassword
Changing password for user "cashing-tier" ...
...done.

Rabbit’s permissions system

• Read—Any operation related to consuming messages, including “purging” an entire queue (also required for binding operations to succeed)
• Write—Publishing messages (also required for binding operations to succeed)
• Configure—Creation and deletion of queues and exchanges

An access control entry consists of four parts:

• The user being granted access.
• The vhost on which the permissions apply.
• The combination of read/write/configure permissions to grant.
• The permission scope—whether the permissions apply only to client-named queues/exchanges, server-named queues/exchanges, or both.

$ ./rabbitmqctl set_permissions -p sycamore \
cashing-tier ".*" ".*" ".*"
Setting permissions for user "cashing-tier" in vhost "sycamore" ...
...done.

• -p sycamore—This tells set_permissions which vhost the entry should apply to.
• cashing-tier—The user being granted the permissions.
• ".*" ".*" ".*"—These are the granted permissions. The values map to configure, write, and read respectively.

".*" means match any queue or exchange name

• you want to grant cashing-tier access to the oak vhost.
• You want to allow the user to be able to execute a read command against any queue or exchange, while restricting writes only to queues or exchanges whose names starts with checks-.
• Also, you want to prevent configure operations entirely.

$ ./rabbitmqctl set_permissions -p oak \
-s all cashing-tier "" "checks-.*" ".*"
Setting permissions for user "cashing-tier" in vhost "oak" ...
...done.

$ ./rabbitmqctl clear_permissions -p oak cashing-tier
Clearing permissions for user "cashing-tier" in vhost "oak" ...
...done.

Checking up

$ ./rabbitmqctl list_queues -p sycamore

$ ./rabbitmqctl list_queues name messages consumers memory
Listing queues ...
msg-inbox-logs 0 2 34632
msg-inbox-errors 0 1 34632
all-logs 3 0 43664
...done.

$ ./rabbitmqctl list_exchanges
Listing exchanges ...
logs-exchange topic
amq.rabbitmq.log topic
amq.match headers
amq.headers headers
amq.topic topic
amq.direct direct
amq.fanout fanout
direct
...done.

$ ./rabbitmqctl list_bindings
Listing bindings ...
all-logs all-logs []
msg-inbox-errors msg-inbox-errors []
msg-inbox-logs msg-inbox-logs []
logs-exchange all-logs # []
logs-exchange msg-inbox-logs *.msg-inbox []
logs-exchange msg-inbox-errors error.msg-inbox []
...done.

Understanding RabbitMQ’s logs

READING THE LOGS ON THE FILESYSTEM

LOG_BASE=/var/log/rabbitmq，Inside that folder RabbitMQ will create two log files: RABBITMQ_NODENAME-sasl.log and RABBITMQ_NODENAME.log, where RABBITMQ_NODENAME will be something like _rabbit@localhost_ or just rabbit depending on how you configured your system.

SASL (System Application Support Libraries) is a set of libraries that are part of the Erlang-OTP distribution.

They help the developer to have a set of standards when developing their Erlang apps. One of those is the logging format. So, when RabbitMQ logs Erlang-related information, it’ll go to the rabbit-sasl.log files.

ROTATING THE LOGS

$ ./rabbitmqctl rotate_logs suffix

ACCESSING THE LOGS IN REAL TIME VIA AMQP

when you were listing exchanges using rabbitmqctl you spotted an exchange called amq.rabbitmq.log whose type is topic. RabbitMQ will publish its logs to that exchange using the severity level as a routing key—you’ll get error, warning, and info.

 

Fixing a bad Rabbit: troubleshooting

ERLANG COOKIES

One common error that you can get when you start working with RabbitMQ is badrpc,nodedown. It usually happens when you try to use the rabbitmqctl command

rabbitmqctl will start up an Erlang node, and from there it’ll try to communicate with the RabbitMQ node using the Erlang distribution system. For this to work, you need two things: the proper Erlang cookie and the proper node name.

An Erlang cookie acts as a secret token that two Erlang nodes exchange in order to authenticate themselves.

rabbitmq使用下面路径下的cookie

openstack@OpenstackIcehouse2:/var/lib/rabbitmq$ ls -a
.  ..  .erlang.cookie  mnesia

In order for rabbitmqctl to communicate with the RabbitMQ node, it needs to share the same cookie.

ERLANG NODES

When you start an Erlang node, you can give it two mutually exclusive options, name and sname, which will specify the node name.

That node name can be long or short, hence the s in sname.

If you start your node with a long name, it will get something like rabbit@hostname.network.tld. If you use short names, you’ll see something similar to this: rabbit@hostname. The latter is the default way of starting RabbitMQ.

MNESIA AND HOST NAMES

RabbitMQ uses Mnesia to store information about queues, exchanges, bindings, and so on. One of the things that RabbitMQ does at startup time
is launch the Mnesia database.

Mnesia creates a database schema based on the machine hostname. If you list the contents of the MNESIA_BASE folder, you’ll see a folder called rabbit@hostname. If hostname changes due to some network reconfiguration, then Mnesia won’t be able to load the old schema.

openstack@OpenstackIcehouse2:~$ ps aux | grep rabbit
rabbitmq  1883  0.0  0.0   7488   312 ?        S    May28   0:07 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
opensta+  3775  0.0  0.0  11744   916 pts/9    R+   22:58   0:00 grep --color=auto rabbit
rabbitmq 18927  0.0  0.0   4444   652 ?        S    13:42   0:00 /bin/sh /usr/sbin/rabbitmq-server
rabbitmq 18935  1.2  1.9 1183552 80416 ?       Sl   13:42   7:13 /usr/lib/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@OpenstackIcehouse2 -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@OpenstackIcehouse2.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@OpenstackIcehouse2-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@OpenstackIcehouse2-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@OpenstackIcehouse2"
rabbitmq 19061  0.0  0.0   7456   420 ?        Ss   13:42   0:00 inet_gethost 4
rabbitmq 19062  0.0  0.0   9552   636 ?        S    13:42   0:00 inet_gethost 4

 

openstack@OpenstackIcehouse2:/var/lib/rabbitmq/mnesia/rabbit@OpenstackIcehouse2$ ls
cluster_nodes.config         rabbit_durable_route.DCD
DECISION_TAB.LOG             rabbit_runtime_parameters.DCD
LATEST.LOG                   rabbit_serial
msg_store_persistent         rabbit_user.DCD
msg_store_transient          rabbit_user_permission.DCD
nodes_running_at_shutdown    rabbit_vhost.DCD
queues                       schema.DAT
rabbit_durable_exchange.DCD  schema_version
rabbit_durable_queue.DCD

ERLANG TROUBLESHOOTING TIPS

start your Erlang node using test as node name.

openstack@OpenstackIcehouse2:~$ erl -sname test
Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:2:2] [async-threads:10] [kernel-poll:false]

Eshell V5.10.4  (abort with ^G)
(test@OpenstackIcehouse2)1>

Erlang Read Eval Print Loop

find out your node name (输入node命令的时候，前有空格后有点)：

(test@OpenstackIcehouse2)1>  node().
test@OpenstackIcehouse2

let’s check which other nodes are running on your machine:

(test@OpenstackIcehouse2)2> net_adm:names().
{ok,[{"rabbit",59939},{"test",40695}]}

There you called the names function from the net_adm module. As you can see, RabbitMQ is running on your machine, using rabbit as the node name and 59106 as the port.

that’s not the port you use to connect to RabbitMQ from an AMQP client.

the Erlang Port Mapper Daemon (epmd). Whenever you start a distributed Erlang node, it’ll register with the epmd process, giving it the address and port assigned to it by the OS kernel. Then when another Erlang node comes to life, it’ll do the same. Finally if the latter wants to connect to your first node, it’ll go through epmd to obtain the other node address.

root@OpenstackIcehouse2:~# netstat -nap | grep beam
tcp        0      0 0.0.0.0:40695           0.0.0.0:*               LISTEN      3887/beam.smp  
tcp        0      0 0.0.0.0:59939           0.0.0.0:*               LISTEN      18935/beam.smp 
tcp6       0      0 :::5672                 :::*                    LISTEN      18935/beam.smp 

前两个是epmd的port，后一个才是rabbitmq的port

Now that you know that you can see the RabbitMQ node, let’s try to establish a connection to it:

(test@OpenstackIcehouse2)2>  net_adm:ping('rabbit@OpenstackIcehouse2').
pong

If the answer is pong, then you succeeded; if you get pang as reply, that means you couldn’t connect to the other node. Keep in mind that for all this to work you have to share the same Erlang cookie.

Let’s confirm that you’re connected to the rabbit node:

(test@OpenstackIcehouse2)3> nodes().
[rabbit@OpenstackIcehouse2]

Finally, let’s execute a function on the remote rabbit node:

(test@OpenstackIcehouse2)4>  rpc:call('rabbit@OpenstackIcehouse2', erlang, system_info, [process_count]).
443

(test@OpenstackIcehouse2)7> rpc:call('rabbit@OpenstackIcehouse2', mnesia, info, []).
---> Processes holding locks <---
---> Processes waiting for locks <---
---> Participant transactions <---
---> Coordinator transactions <---
---> Uncertain transactions <---
---> Active tables <---
rabbit_user_permission: with 1        records occupying 331      words of mem
rabbit_topic_trie_edge: with 22       records occupying 1372     words of mem
rabbit_queue   : with 40       records occupying 1519     words of mem
rabbit_semi_durable_route: with 0        records occupying 89       words of mem
schema         : with 20       records occupying 2707     words of mem
rabbit_exchange_serial: with 0        records occupying 299      words of mem
rabbit_route   : with 80       records occupying 3726     words of mem
rabbit_exchange: with 29       records occupying 1190     words of mem
rabbit_vhost   : with 1        records occupying 310      words of mem
mirrored_sup_childspec: with 0        records occupying 89       words of mem
rabbit_listener: with 1        records occupying 355      words of mem
gm_group       : with 0        records occupying 299      words of mem
rabbit_topic_trie_binding: with 21       records occupying 998      words of mem
rabbit_reverse_route: with 80       records occupying 3726     words of mem
rabbit_durable_route: with 0        records occupying 299      words of mem
rabbit_user    : with 1        records occupying 318      words of mem
rabbit_topic_trie_node: with 25       records occupying 838      words of mem
rabbit_durable_queue: with 0        records occupying 299      words of mem
rabbit_durable_exchange: with 8        records occupying 538      words of mem
rabbit_runtime_parameters: with 0        records occupying 299      words of mem
===> System info in version "4.11", debug level = none <===
opt_disc. Directory "/var/lib/rabbitmq/mnesia/rabbit@OpenstackIcehouse2" is used.
use fallback at restart = false
running db nodes   = [rabbit@OpenstackIcehouse2]
stopped db nodes   = []
master node tables = []
remote             = []
ram_copies         = [gm_group,mirrored_sup_childspec,rabbit_exchange,
                      rabbit_exchange_serial,rabbit_listener,rabbit_queue,
                      rabbit_reverse_route,rabbit_route,
                      rabbit_semi_durable_route,rabbit_topic_trie_binding,
                      rabbit_topic_trie_edge,rabbit_topic_trie_node]
disc_copies        = [rabbit_durable_exchange,rabbit_durable_queue,
                      rabbit_durable_route,rabbit_runtime_parameters,
                      rabbit_user,rabbit_user_permission,rabbit_vhost,schema]
disc_only_copies   = []
[{rabbit@OpenstackIcehouse2,disc_copies}] = [rabbit_runtime_parameters,
                                             rabbit_durable_exchange,
                                             rabbit_durable_queue,rabbit_user,
                                             rabbit_durable_route,
                                             rabbit_vhost,schema,
                                             rabbit_user_permission]
[{rabbit@OpenstackIcehouse2,ram_copies}] = [rabbit_topic_trie_node,
                                            rabbit_reverse_route,
                                            rabbit_topic_trie_binding,
                                            gm_group,rabbit_listener,
                                            mirrored_sup_childspec,
                                            rabbit_exchange,rabbit_route,
                                            rabbit_exchange_serial,
                                            rabbit_semi_durable_route,
                                            rabbit_queue,
                                            rabbit_topic_trie_edge]
331 transactions committed, 0 aborted, 2 restarted, 0 logged to disc
0 held locks, 0 in queue; 0 local transactions, 0 remote
0 transactions waits for other nodes: []
ok

(test@OpenstackIcehouse2)8> q().
ok