经验总结

  1. 首页 > 生活 > >

认识 Redis client-output-buffer-limit 参数与源码分析

生活百科

概述Redis 的 client-output-buffer-limit 可以用来强制断开无法足够快从 redis 服务器端读取数据的客户端 。保护机制规则如下：

  1. [hard limit] 大小限制，当某一客户端缓冲区超过设定值后，直接关闭连接 。
  2. [soft limit] 持续时间限制，当某一客户端缓冲区持续一段时间占用过大空间时关闭连接 。
该参数一般用在以下几类客户端中：
  • 普通 client，包括 monitor
  • 主从同步时的 slave client
  • Pub/Sub 模式中的 client
配置介绍与分析该参数的配置语法：
client-output-buffer-limit <class> <hard limit> <soft limit> <soft seconds>配置实例：
# 普通client buffer限制client-output-buffer-limit normal 0 0 0# slave client buffer限制client-output-buffer-limit slave 256mb 64mb 60# pubsub client buffer限制client-output-buffer-limit pubsub 32mb 8mb 60
  • client-output-buffer-limit normal 0 0 0
将 hard limit 和 soft limit 同时设置为 0，则表示关闭该限制 。
  • client-output-buffer-limit slave 256mb 64mb 60
该配置表示，对于 slave 客户端来说，如果 output-buffer 占用内存达到 256M 或者超过 64M 的时间达到 60s，则关闭客户端连接 。
  • client-output-buffer-limit pubsub 32mb 8mb 60
该配置表示，对于 Pub/Sub 客户端来说，若 output-buffer 占用内存达到 32M 或者超过 8M 的时间达到 60s，则关闭客户端连接 。
概括说明：一般情况下，对于普通客户端，client-output-buffer 是不设限制的，因为 server 只会在 client 请求数据的时候才会发送，不会产生积压 。而在 server 主动发送，client 来处理的场景下，这种一般都是异步处理的，会划出一个缓冲区来“暂存”未处理的数据，若 server 发送数据比 client 处理数据快时，就会发生缓冲区积压 。对于用作 Pub/Sub 和 slave 的客户端，server 会主动把数据推送给他们，故需要设置 client-output-buffer 的限制 。
示例分析下面我们以主从同步时的 slave 客户端，来具体分析下 。在 redis 在主从同步时，master 会为 slave 创建一个输出缓冲区 。在 master 保存 rdb，将 rdb 文件传输给 slave，slave 加载 rdb 完成之前，master 会将接收到的所有写命令，写入到内存中的这个输出缓冲区去 。若 rdb 的保存，传输，加载耗时过长，或者在此期间的写命令过多，则可能会造成超过缓冲区限制，造成 master 和 slave 的连接断开 。此时则需要适当调整下 client-output-buffer-limit slave配置 。
源码浅析-主从同步时 output buffer 使用
基于 redis5.0 版本源码
redis server 通过 addReply 将数据发送给客户端，以下源码见 https://github.com/redis/redis/blob/5.0/src/networking.c#L190-L211
/* Add the object 'obj' string representation to the client output buffer. */void addReply(client *c, robj *obj) {if (prepareClientToWrite(c) != C_OK) return;if (sdsEncodedObject(obj)) {if (_addReplyToBuffer(c,obj->ptr,sdslen(obj->ptr)) != C_OK)_addReplyStringToList(c,obj->ptr,sdslen(obj->ptr));} else if (obj->encoding == OBJ_ENCODING_INT) {char buf[32];size_t len = ll2string(buf,sizeof(buf),(long)obj->ptr);if (_addReplyToBuffer(c,buf,len) != C_OK)_addReplyStringToList(c,buf,len);} else {serverPanic("Wrong obj->encoding in addReply()");}}在函数的开头，会通过prepareClientToWrite(c)判断是否需要将数据写入客户端的 output buffer 中 。我们看下什么条件下数据会被写入客户端的 output buffer 中，即返回 C_OK 。
/* This function is called every time we are going to transmit new data * to the client. The behavior is the following: * * If the client should receive new data (normal clients will) the function * returns C_OK, and make sure to install the write handler in our event * loop so that when the socket is writable new data gets written. * * If the client should not receive new data, because it is a fake client * (used to load AOF in memory), a master or because the setup of the write * handler failed, the function returns C_ERR. * * The function may return C_OK without actually installing the write * event handler in the following cases: * * 1) The event handler should already be installed since the output buffer *already contains something. * 2) The client is a slave but not yet online, so we want to just accumulate *writes in the buffer but not actually sending them yet. * * Typically gets called every time a reply is built, before adding more * data to the clients output buffers. If the function returns C_ERR no * data should be appended to the output buffers. */int prepareClientToWrite(client *c) {/* If it's the Lua client we always return ok without installing any* handler since there is no socket at all. */if (c->flags & (CLIENT_LUA|CLIENT_MODULE)) return C_OK;/* CLIENT REPLY OFF / SKIP handling: don't send replies. */if (c->flags & (CLIENT_REPLY_OFF|CLIENT_REPLY_SKIP)) return C_ERR;/* Masters don't receive replies, unless CLIENT_MASTER_FORCE_REPLY flag* is set. */if ((c->flags & CLIENT_MASTER) &&!(c->flags & CLIENT_MASTER_FORCE_REPLY)) return C_ERR;if (c->fd <= 0) return C_ERR; /* Fake client for AOF loading. *//* Schedule the client to write the output buffers to the socket, unless* it should already be setup to do so (it has already pending data). */if (!clientHasPendingReplies(c)) clientInstallWriteHandler(c);/* Authorize the caller to queue in the output buffer of this client. */return C_OK;}/* Return true if the specified client has pending reply buffers to write to * the socket. */int clientHasPendingReplies(client *c) {return c->bufpos || listLength(c->reply);}void clientInstallWriteHandler(client *c) {/* Schedule the client to write the output buffers to the socket only* if not already done and, for slaves, if the slave can actually receive* writes at this stage. */if (!(c->flags & CLIENT_PENDING_WRITE) &&(c->replstate == REPL_STATE_NONE ||(c->replstate == SLAVE_STATE_ONLINE && !c->repl_put_online_on_ack))){c->flags |= CLIENT_PENDING_WRITE;listAddNodeHead(server.clients_pending_write,c);}}

经验总结扩展阅读


上一篇：2023年10月7日是剪指甲的黄道吉日吗 2023年农历八月廿三剪指甲吉日

下一篇：Day3 最短路 最小生成树 拓扑排序