kafka-php

kafka-php的github地址  https://github.com/jacky5059/kafka-php

生产者produce示例代码

<?php
set_include_path(
implode(PATH_SEPARATOR, array(
realpath(__DIR__ . '/../lib'),
get_include_path(),
))
);
require 'autoloader.php';
$host = 'localhost';
$port = 9092;
$topic = 'test';
$producer = new Kafka_Producer($host, $port, Kafka_Encoder::COMPRESSION_NONE);
$in = fopen('php://stdin', 'r');
while (true) {
echo "\nEnter comma separated messages:\n";
$messages = explode(',', fgets($in));
foreach (array_keys($messages) as $k) {
//$messages[$k] = trim($messages[$k]);
}
$bytes = $producer->send($messages, $topic);
printf("\nSuccessfully sent %d messages (%d bytes)\n\n", count($messages), $bytes);
}

简单消费者simple consumer示例代码

<?php
set_include_path(
implode(PATH_SEPARATOR, array(
realpath(__DIR__ . '/../lib'),
get_include_path(),
))
);
require 'autoloader.php';
$host = 'localhost';
$zkPort = 2181; //zookeeper
$kPort = 9092; //kafka server
$topic = 'test';
$maxSize = 10000000;
$socketTimeout = 2;
$offset = 0;
$partition = 0;
$nMessages = 0;
$consumer = new Kafka_SimpleConsumer($host, $kPort, $socketTimeout, $maxSize);
while (true) {
try {
//create a fetch request for topic "test", partition 0, current offset and fetch size of 1MB
$fetchRequest = new Kafka_FetchRequest($topic, $partition, $offset, $maxSize);
//get the message set from the consumer and print them out
$partialOffset = 0;
$messages = $consumer->fetch($fetchRequest);
foreach ($messages as $msg) {
++$nMessages;
echo "\nconsumed[$offset][$partialOffset][msg #{$nMessages}]: " . $msg->payload();
$partialOffset = $messages->validBytes();
}
//advance the offset after consuming each message
$offset += $messages->validBytes();
//echo "\n---[Advancing offset to $offset]------(".date('H:i:s').")";
unset($fetchRequest);
//sleep(2);
} catch (Exception $e) {
// probably consumed all items in the queue.
echo "\nERROR: " . get_class($e) . ': ' . $e->getMessage()."\n".$e->getTraceAsString()."\n";
sleep(2);
}
}

基于zookeeper的消费者zkconsumer示例代码

<?php
set_include_path(
implode(PATH_SEPARATOR, array(
realpath(__DIR__ . '/../lib'),
get_include_path(),
))
);
require 'autoloader.php';
// zookeeper address (one or more, separated by commas)
$zkaddress = 'localhost:8121';
// kafka topic to consume from
$topic = 'testtopic';
// kafka consumer group
$group = 'testgroup';
// socket buffer size: must be greater than the largest message in the queue
$socketBufferSize = 10485760; //10 MB
// approximate max number of bytes to get in a batch
$maxBatchSize = 20971520; //20 MB
$zookeeper = new Zookeeper($zkaddress);
$zkconsumer = new Kafka_ZookeeperConsumer(
new Kafka_Registry_Topic($zookeeper),
new Kafka_Registry_Broker($zookeeper),
new Kafka_Registry_Offset($zookeeper, $group),
$topic,
$socketBufferSize
);
$messages = array();
try {
foreach ($zkconsumer as $message) {
// either process each message one by one, or collect them and process them in batches
$messages[] = $message;
if ($zkconsumer->getReadBytes() >= $maxBatchSize) {
break;
}
}
} catch (Kafka_Exception_OffsetOutOfRange $exception) {
// if we haven't received any messages, resync the offsets for the next time, then bomb out
if ($zkconsumer->getReadBytes() == 0) {
$zkconsumer->resyncOffsets();
die($exception->getMessage());
}
// if we did receive some messages before the exception, carry on.
} catch (Kafka_Exception_Socket_Connection $exception) {
// deal with it below
} catch (Kafka_Exception $exception) {
// deal with it below
}
if (null !== $exception) {
// if we haven't received any messages, bomb out
if ($zkconsumer->getReadBytes() == 0) {
die($exception->getMessage());
}
// otherwise log the error, commit the offsets for the messages read so far and return the data
}
// process the data in batches, wait for ACK
$success = doSomethingWithTheMessages($messages);
// Once the data is processed successfully, commit the byte offsets.
if ($success) {
$zkconsumer->commitOffsets();
}
// get an approximate figure on the size of the queue
try {
echo "\nRemaining bytes in queue: " . $consumer->getRemainingSize();
} catch (Kafka_Exception_Socket_Connection $exception) {
die($exception->getMessage());
} catch (Kafka_Exception $exception) {
die($exception->getMessage());
}

kafka-php的更多相关文章

  1. Spark踩坑记——Spark Streaming+Kafka

    [TOC] 前言 在WeTest舆情项目中,需要对每天千万级的游戏评论信息进行词频统计,在生产者一端,我们将数据按照每天的拉取时间存入了Kafka当中,而在消费者一端,我们利用了spark strea ...

  2. 消息队列 Kafka 的基本知识及 .NET Core 客户端

    前言 最新项目中要用到消息队列来做消息的传输,之所以选着 Kafka 是因为要配合其他 java 项目中,所以就对 Kafka 了解了一下,也算是做个笔记吧. 本篇不谈论 Kafka 和其他的一些消息 ...

  3. kafka学习笔记:知识点整理

    一.为什么需要消息系统 1.解耦: 允许你独立的扩展或修改两边的处理过程,只要确保它们遵守同样的接口约束. 2.冗余: 消息队列把数据进行持久化直到它们已经被完全处理,通过这一方式规避了数据丢失风险. ...

  4. .net windows Kafka 安装与使用入门(入门笔记)

    完整解决方案请参考: Setting Up and Running Apache Kafka on Windows OS   在环境搭建过程中遇到两个问题,在这里先列出来,以方便查询: 1. \Jav ...

  5. kafka配置与使用实例

    kafka作为消息队列,在与netty.多线程配合使用时,可以达到高效的消息队列

  6. kafka源码分析之一server启动分析

    0. 关键概念 关键概念 Concepts Function Topic 用于划分Message的逻辑概念,一个Topic可以分布在多个Broker上. Partition 是Kafka中横向扩展和一 ...

  7. Kafka副本管理—— 为何去掉replica.lag.max.messages参数

    今天查看Kafka 0.10.0的官方文档,发现了这样一句话:Configuration parameter replica.lag.max.messages was removed. Partiti ...

  8. Kafka:主要参数详解(转)

    原文地址:http://kafka.apache.org/documentation.html ############################# System ############### ...

  9. kafka

    2016-11-13  20:48:43 简单说明什么是kafka? Apache kafka是消息中间件的一种,我发现很多人不知道消息中间件是什么,在开始学习之前,我这边就先简单的解释一下什么是消息 ...

  10. Spark Streaming+Kafka

    Spark Streaming+Kafka 前言 在WeTest舆情项目中,需要对每天千万级的游戏评论信息进行词频统计,在生产者一端,我们将数据按照每天的拉取时间存入了Kafka当中,而在消费者一端, ...

随机推荐

  1. 实体bean里面不要轻易加transient,反序列回来之后会变成null

     实体bean里面不要轻易加transient,反序列回来之后会变成null

  2. 方法引用(Method reference)和invokedynamic指令详细分析

    方法引用(Method reference)和invokedynamic指令详细分析 invokedynamic是jvm指令集里面最复杂的一条.本文将详细分析invokedynamic指令是如何实现方 ...

  3. Oracle PL/SQL学习之Hello World(0)

    1.PL/SQL是Oracle数据库的一大创举,让一些复杂繁琐的常规主流编程代码做的编码处理过程,只需要在PL/SQL中使用简短的几句代码就可以解决,并且准确高效.那么遵循惯例,我们学习PL/SQL编 ...

  4. 日志一直打印 DEBUG o.s.amqp.rabbit.listener.BlockingQueueConsumer

    <?xml version="1.0" encoding="UTF-8"?> <configuration> <logger na ...

  5. appium桌面版和命令行版的安装

    一.appium桌面版: 启动很慢,一般用于元素定位 Appium-Desktop官方下载地址:https://github.com/appium/appium-desktop/releases/ ( ...

  6. 多个tomcat一起运行

    1.默认为:8005 2.在D:\apache-tomcat-7.0.85\bin目录下,找到startup.bat,打开并修改. 3.在D:\apache-tomcat-7.0.85\bin目录下, ...

  7. Codeforces - 914C 数位DP

    题意有点难以描述,简略的就是给定一个二进制\(n\),每一步操作能使\(n\)的位为1的数的和转化为一个十进制,然后转化为该数的二进制再进行相同的操作 查询\([0,n]\)中操作数恰好为\(k\)的 ...

  8. SQL Server 保留关键字

    Microsoft SQL Server 2005 使用保留关键字来定义.操作或访问数据库.保留关键字是 SQL Server 使用的 Transact-SQL 语言语法的一部分,用于分析和理解 Tr ...

  9. MySQL prompt提示符总结

      A counter that increments for each statement you issue \D 当前日期 \d 当前数据库 \h 数据库主机 \l The current de ...

  10. MySQL默认约束DEFAULT

    当插入记录时,如果没有明确为字段赋值,则自动赋予默认值. 例如: 性别: 1. 男 2. 女 3. 保密