python之RabbitMQ | 易学教程

RabbitMQ队列

基础

安装erlang: http://www.erlang.org/downloads
安装rabbitmq: http://www.rabbitmq.com
安装rabbitmq moudle: pip install pika
RabbitMQ介绍：http://www.rabbitmq.com/tutorials/tutorial-six-python.html
RabbitMQ web 登入： http://localhost:15672

RabbitMQ是一个消息代理：它接受和转发消息，类似邮局的角色。

实例一：

send端：

 1 import pika
 2 
 3 connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))  # 建立连接
 4 channel = connection.channel()   # 声明一个管道
 5 channel.queue_declare('hello')   # 声明一个队列
 6 channel.basic_publish(
 7                         exchange='',
 8                         routing_key='hello',
 9                         body= 'Hello Word')
10 # 发送消息
11 # routing_key : 队列名
12 # body: 消息内容
13 print('[x] sent "Hello World!"')
14 connection.close()   # 关闭连接，channel不需要关闭

receive 端：

 1 import pika
 2 
 3 connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))  # 建立连接
 4 channel = connection.channel()          # 定义管道
 5 channel.queue_declare(queue='hello')    
 6 # 声明队列，send端和recv端只需声明一次就好，但是不确定send端是否声明过，所以在两边可以都声明一次，这并不冲突
 7 
 8 def callback(ch, method,properties,body): 
 9     """
10     收到消息，调用函数
11     :param ch: 管道
12     :param method:  
13     :param properties: 
14     :param body: 消息
15     :return: 
16     """
17     print('reeive %r' % body)
18     
19 channel.basic_consume(
20                         callback,
21                         queue='hello',
22                         no_ack=True
23 )
24 print('wait for message.Press Ctrl+C to exit')
25 channel.start_consuming()   # 持续接收消息，如果没有消息就阻塞

执行结果：

send端发送一条消息就结束
recv端持续接受消息，一收到消息就调用回调函数，否则一直阻塞
如果send端没有打开，每次send的消息会存在rabbitmq服务器上，直到有recv端接入接受消息。

实例二：

　　RabbitMQ server采用轮询机制发送消息。开启多个recv端，send端发送消息，轮询recv端接收消息（每一次send，只会有一个recv端接收消息）

　　recv端： channel.basic_consume参数no_ack=True recv端是否在调用完成回调函数后给send端一个确认；

- False：默认值，需要接收确认
- True：执行后不确认，也就是服务端把一个消息分发出去后就不管了

情景模拟：no_ack=False，回调函数添加消息处理完毕的确认语句：ch.basic_ack(delivery_tag=method.delivery_tag)；在回调函数中添加sleep()，在recv端没处理完消息就中断，观察其他recv端的接受情况。

 1 import pika
 2 import time
 3 
 4 connection = pika.BlockingConnection(pika.ConnectionParameters(
 5     'localhost'))
 6 channel = connection.channel()
 7 
 8 
 9 def callback(ch, method, properties, body):
10     print(" [x] Received %r" % body)
11     time.sleep(20)
12     print(" [x] Done")
13     print("method.delivery_tag", method.delivery_tag)
14     ch.basic_ack(delivery_tag=method.delivery_tag)
15 
16 
17 channel.basic_consume(callback,
18                       queue='task_queue',
19                       )
20 
21 print(' [*] Waiting for messages. To exit press CTRL+C')
22 channel.start_consuming()

recv端

 1 import pika
 2 import time
 3 
 4 connection = pika.BlockingConnection(pika.ConnectionParameters(
 5     'localhost'))
 6 channel = connection.channel()
 7 
 8 channel.queue_declare(queue='task_queue')
 9 
10 import sys
11 
12 message = ' '.join(sys.argv[1:]) or "Hello World! %s" % time.time()
13 channel.basic_publish(exchange='',
14                       routing_key='task_queue',
15                       body=message,
16                       properties=pika.BasicProperties(
17                           delivery_mode=2,  # make message persistent
18                       )
19                       )
20 print(" [x] Sent %r" % message)
21 connection.close()

send端

　　默认的no_ack=False的模式，如果一个消息没有确认，但是连接断了，那么这个消息还会有别的recv端重新处理。只有在recv端确认了之后，才会从服务器的队列中清除。

消息持久化

　　pika队列与消息都存在RabbitMQ server缓存中，如果server down机了，缓存将会被清空。因此，server端的数据丢失会造成send端与recv端的失联，所以，server对于重要数据需要进行消息持久化。

　　RabbitMQ说明：

- rabbitmqctl.bat list_queue ：列举server端的queue信息
- rabbitmq-servece.bat start：启动service

　消息持久化

- 队列持久化 channel.queue_declare(queue='hello', durable=True)
- 消息持久化　 channel.basic_publish(properties=pika.BasicProperties(delivery_mode =2, )) # make message persistent

消息公平分发

　　不同配置的机器，对于消息的处理速度不同，如果rabbitmq server只是按照顺序给多个机器发送消息，那么处理慢的机器可能会造成消息堆积，处理快的机器会长时间处于闲置状态。

　　因此，可以配置让rabbitmq server在一个机器没有处理完上一个消息时，不给其发送新的消息。只需在recv端配置如下：

　　　　　　channel.basic_qos(prefetch_count=1)

send端

 1 import pika
 2 
 3 connection = pika.BlockingConnection(pika.ConnectionParameters(
 4         host='localhost'))
 5 channel = connection.channel()
 6  
 7 channel.queue_declare(queue='task_queue', durable=True)
 8  
 9 message =  "Hello World!"
10 channel.basic_publish(exchange='',
11                       routing_key='task_queue',
12                       body=message,
13                       properties=pika.BasicProperties(
14                          delivery_mode = 2, # make message persistent
15                       ))
16 print(" [x] Sent %r" % message)
17 connection.close()

send

recv端

 1 import pika
 2 import time
 3  
 4 connection = pika.BlockingConnection(pika.ConnectionParameters(
 5         host='localhost'))
 6 channel = connection.channel()
 7  
 8 channel.queue_declare(queue='task_queue', durable=True)
 9 print(' [*] Waiting for messages. To exit press CTRL+C')
10  
11 def callback(ch, method, properties, body):
12     print(" [x] Received %r" % body)
13     time.sleep(10)
14     print(" [x] Done")
15     ch.basic_ack(delivery_tag = method.delivery_tag)
16  
17 channel.basic_qos(prefetch_count=1)
18 
19 channel.basic_consume(callback,
20                       queue='task_queue')
21  
22 channel.start_consuming()

recv

消息发布\订阅

　　上面实例都是一对一的发送与接收消息，可以使用exchange实现一个发送端发送消息，多个接收端接收消息，即广播消息。

　　Exchange定义类型，用于判断哪些queue符合条件，可以接受消息。

- fanout: 所有bind到此exchange的queue都可以接收消息
- direct: 通过routingKey和exchange决定唯一的queue可以接收消息
- topic: 所有符合routingKey(此时可以是一个表达式)的routingKey所bind的queue可以接收消息，表达式符号说明：#代表一个或多个字符，*代表任何字符，（注：使用RoutingKey为#，Exchange Type为topic的时候相当于使用fanout）
- headers: 通过headers 来决定把消息发给哪些queue

发布端

 1 import pika
 2  
 3 connection = pika.BlockingConnection(pika.ConnectionParameters(
 4         host='localhost'))
 5 channel = connection.channel()
 6  
 7 channel.exchange_declare(exchange='logs',
 8                          type='fanout')
 9  
10 message ="info: Hello World!"
11 channel.basic_publish(exchange='logs',
12                       routing_key='',
13                       body=message)
14 print(" [x] Sent %r" % message)
15 connection.close()

send

订阅端

 1 import pika
 2  
 3 connection = pika.BlockingConnection(pika.ConnectionParameters(
 4         host='localhost'))
 5 channel = connection.channel()
 6  
 7 channel.exchange_declare(exchange='logs',
 8                          type='fanout')
 9  
10 result = channel.queue_declare(exclusive=True) #不指定queue名字,rabbit会随机分配一个名字,exclusive=True会在使用此queue的消费者断开后,自动将queue删除
11 queue_name = result.method.queue
12  
13 channel.queue_bind(exchange='logs',
14                    queue=queue_name)
15  
16 print(' [*] Waiting for logs. To exit press CTRL+C')
17  
18 def callback(ch, method, properties, body):
19     print(" [x] %r" % body)
20  
21 channel.basic_consume(callback,
22                       queue=queue_name,
23                       no_ack=True)
24  
25 channel.start_consuming()

recv

有选择的接受消息

　　RabbitMQ server根据关键字发送消息，即：队列绑定关键字，发送者将数据根据关键字发送到消息exchange，exchange根据关键字判定应该将数据发送至指定队列。

发布端：

 1 import pika
 2 import sys
 3  
 4 connection = pika.BlockingConnection(pika.ConnectionParameters(
 5         host='localhost'))
 6 channel = connection.channel()
 7  
 8 channel.exchange_declare(exchange='direct_logs',
 9                          type='direct')
10  
11 severity = sys.argv[1] if len(sys.argv) > 1 else 'info'
12 message = ' '.join(sys.argv[2:]) or 'Hello World!'
13 channel.basic_publish(exchange='direct_logs',
14                       routing_key=severity,
15                       body=message)
16 print(" [x] Sent %r:%r" % (severity, message))
17 connection.close()

send

订阅端：

 1 import pika
 2 import sys
 3  
 4 connection = pika.BlockingConnection(pika.ConnectionParameters(
 5         host='localhost'))
 6 channel = connection.channel()
 7  
 8 channel.exchange_declare(exchange='direct_logs',
 9                          type='direct')
10  
11 result = channel.queue_declare(exclusive=True)
12 queue_name = result.method.queue
13  
14 severities = sys.argv[1:]
15 if not severities:
16     sys.stderr.write("Usage: %s [info] [warning] [error]\n" % sys.argv[0])
17     sys.exit(1)
18  
19 for severity in severities:
20     channel.queue_bind(exchange='direct_logs',
21                        queue=queue_name,
22                        routing_key=severity)
23  
24 print(' [*] Waiting for logs. To exit press CTRL+C')
25  
26 def callback(ch, method, properties, body):
27     print(" [x] %r:%r" % (method.routing_key, body))
28  
29 channel.basic_consume(callback,
30                       queue=queue_name,
31                       no_ack=True)
32  
33 channel.start_consuming()

recv

更细致的消息过滤

对于需要过滤不同应用程序的log，这里可以通过定义exchange的类型为topic来实现。

exchange topic的routing_key通过由'*’和'#'通配符以及字母符号等构成的表达式表示。

发布端：

 1 import pika
 2 import sys
 3  
 4 connection = pika.BlockingConnection(pika.ConnectionParameters(
 5         host='localhost'))
 6 channel = connection.channel()
 7  
 8 channel.exchange_declare(exchange='topic_logs',
 9                          type='topic')
10  
11 routing_key = sys.argv[1] if len(sys.argv) > 1 else 'anonymous.info'
12 message = ' '.join(sys.argv[2:]) or 'Hello World!'
13 channel.basic_publish(exchange='topic_logs',
14                       routing_key=routing_key,
15                       body=message)
16 print(" [x] Sent %r:%r" % (routing_key, message))
17 connection.close()

View Code

订阅端：

 1 import pika
 2 import sys
 3  
 4 connection = pika.BlockingConnection(pika.ConnectionParameters(
 5         host='localhost'))
 6 channel = connection.channel()
 7  
 8 channel.exchange_declare(exchange='topic_logs',
 9                          type='topic')
10  
11 result = channel.queue_declare(exclusive=True)
12 queue_name = result.method.queue
13  
14 binding_keys = sys.argv[1:]
15 if not binding_keys:
16     sys.stderr.write("Usage: %s [binding_key]...\n" % sys.argv[0])
17     sys.exit(1)
18  
19 for binding_key in binding_keys:
20     channel.queue_bind(exchange='topic_logs',
21                        queue=queue_name,
22                        routing_key=binding_key)
23  
24 print(' [*] Waiting for logs. To exit press CTRL+C')
25  
26 def callback(ch, method, properties, body):
27     print(" [x] %r:%r" % (method.routing_key, body))
28  
29 channel.basic_consume(callback,
30                       queue=queue_name,
31                       no_ack=True)
32  
33 channel.start_consuming()

View Code

RPC：remote procedure call

　发送命令在远程机器上执行，并且返回执行结果。

RPC server:

 1 import pika
 2 
 3 
 4 connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
 5 channel = connection.channel()
 6 
 7 channel.queue_declare('rpc_queue')
 8 
 9 
10 def upper_data(data):
11     if data:
12         data = data.upper()
13     else:
14         data = None
15     return data
16 
17 
18 def on_request(ch,method,props,body):
19     print('receive data:',body)
20     response = upper_data(body)
21     print('send data:',response)
22     ch.basic_publish(
23                      exchange='',
24                      routing_key=props.reply_to,
25                      properties=pika.BasicProperties(correlation_id=props.correlation_id),
26                      body=response
27     )
28     ch.basic_ack(delivery_tag=method.delivery_tag)
29 
30 channel.basic_qos(prefetch_count=1)
31 channel.basic_consume(on_request, queue='rpc_queue')
32 print('[X] Waiting RPC request')
33 channel.start_consuming()

rpc_server

RPC client:

 1 import pika
 2 import uuid
 3 import  sys
 4 class RPCclient(object):
 5 
 6     def __init__(self):
 7         self.connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
 8         self.channel = self.connection.channel()
 9         result = self.channel.queue_declare(exclusive=True)
10         self.callback_queue = result.method.queue
11         self.channel.basic_consume(self.on_response,
12                              queue=self.callback_queue,
13                              no_ack=True)
14 
15 
16     def on_response(self,ch, method, props, body):
17         if self.corr_id == props.correlation_id:
18             self.response = body
19 
20     def call(self,data):
21         self.response = None
22         self.corr_id = str(uuid.uuid4())
23         self.channel.basic_publish(
24                                     exchange= '',
25                                     routing_key='rpc_queue',
26                                     properties=pika.BasicProperties(reply_to=self.callback_queue,
27                                                                     correlation_id=self.corr_id),
28                                     body=data
29         )
30         print('starting',data)
31         while self.response is  None:
32             self.connection.process_data_events()
33             print('waiting',self.response)
34 
35         return self.response
36 
37 
38 rpc = RPCclient()
39 data = ' '.join(sys.argv[1:]) if len(sys.argv) > 1 else 'Abctest'
40 response = rpc.call(data)
41 print('receive data:',response)

client

来源：oschina

链接：https://my.oschina.net/u/4288942/blog/4061601

标签

pika