kafka clear_log.retention.hours

2021-03-03

Kafka使用log.retention.hours改变消息端的消息保存时间

配置Log过期时间
log.retention.hours
log.retention.minutes
log.retention.ms 86400*7 默认604800000
上面三个配置向配置其中一个即可,kafka默认是配置log.retention.hours,其值是7天
####################

log.cleanup.policy=delete启用删除策略
直接删除,删除后的消息不可恢复。可配置以下两个策略:
log.retention.hours=16
log.retention.bytes=1073741824

# The minimum age of a log file to be eligible for deletion due to age清理超过指定时间清理:
log.retention.hours=48

# The maximum size of a log segment file. When this size is reached a new log segment will be created.超过指定大小后,删除旧的消息:
log.segment.bytes=1073741824

./bin/kafka-topics.sh --describe --zookeeper 172.16.91.118:2181,172.16.91.117:2181,172.16.90.136:2181,172.16.90.145:2181,172.16.91.132:2181/kafka-group-log --topic log-group

./bin/kafka-topics.sh --alter --zookeeper 172.16.91.118:2181,172.16.91.117:2181,172.16.90.136:2181,172.16.90.145:2181,172.16.91.132:2181/kafka-group-log --topic log-group --partitions 3

动态修改kafka 日志保存时间
./bin/kafka-topics.sh --zookeeper 172.16.91.118:2181,172.16.91.117:2181,172.16.90.136:2181,172.16.90.145:2181,172.16.91.132:2181/kafka-group-log --topic log-group --alter --config retention.ms=129600000
./bin/kafka-topics.sh --zookeeper 172.16.91.118:2181,172.16.91.117:2181,172.16.90.136:2181,172.16.90.145:2181,172.16.91.132:2181/kafka-group-log --topic log-group --alter --config retention.ms=129600000

shell实例:
vi alter_log-group_add_part.sh
#!/bin/sh
IP=$(ifconfig eth0|grep inet|awk '{print $2}'|head -1)

cd /jesong/kafka_2.13-2.6.0
echo ##############
./bin/kafka-topics.sh --alter --zookeeper 172.16.91.118:2181,172.16.91.117:2181,172.16.90.136:2181,172.16.90.145:2181,172.16.91.132:2181/kafka-group-log --topic log-group --partitions 3
./bin/kafka-topics.sh --alter --zookeeper 172.16.91.118:2181,172.16.91.117:2181,172.16.90.136:2181,172.16.90.145:2181,172.16.91.132:2181/kafka-group-log --topic log-group --config retention.hours=36

将数据压缩,只保留每个key最后一个版本的数据。
首先在broker的配置中设置log.cleaner.enable=true启用cleaner,这个默认是关闭的。
在topic的配置中设置log.cleanup.policy=compact启用压缩策略。

分类:Linux | 标签: |

相关日志

评论被关闭!