RabbitMQ confirmed every DHL Express label request and put none of them in a queue

When an order is prepared, the order service transmits a label request to RabbitMQ's topic exchange named shipping. This request includes a routing key comprised of the event and the carrier, for example, label.requested.dpd or label.requested.royalmail. Our carrier integration connects to a queue bound to this exchange using the pattern label.requested.*.

The integration then schedules a label using the carrier indicated by the routing key. The order service only considers a request as sent once the broker confirms it. In October, we integrated DHL's express service into our system. The new carrier code was added to the routing key as dhl.express. For two days, no DHL Express parcel received a label.

Approximately 1,900 orders were packed in our warehouses, while the order service logged every request as published and confirmed. The carrier integration queue appeared normal because none of the DHL Express messages reached it. In a topic exchange, routing key words are separated by dots. The star symbol matches a single word in a binding, while the hash symbol matches zero or more words.

The key label.requested.dhl.express contains four words, and the key label.requested.* matches three. No exchange binding matched the new routing key. If RabbitMQ cannot find a queue to deliver a message, it discards it, unless the message was published as mandatory. In that case, it returns the message to the publisher or transfers it to an alternate exchange.

We did not have either of these options. The confirm feature did not help, as it is not a receipt from a queue. If a message has no destination, RabbitMQ sends the confirm immediately after determining there is no queue to deliver it to. Our publisher had been anticipating the acknowledgment indicating the message had been removed.

The carrier codes in routing keys are now single words, such as dhl_express. The code that generates the key rejects routing keys with dots. Our exchange has an alternate exchange set via policy, leading to a queue called unroutable. One message in this queue notifies the integration team on call. We modified the code to use single-word carrier codes and added an alternate exchange.

Every exchange we use has an alternate exchange set by policy, resulting in the unroutable queue. A single message in that queue alerts the integration team. We republished the 1,900 unprocessed requests from the outbox the same day. The confirm feature alerted us when RabbitMQ finished processing each message, and discarding a message is one of the ways an exchange can conclude its task.