Home Agenda Doradztwo O mnie Artykuły Kontakt

Szkolenie Kafka od programisty dla programistów

O mnie

Agenda szkolenia

Kontakt

Paweł Szymczyk

Inżynier oprogramowania z bogatym doświadczeniem zawodowym, rozpoczynającym się w 2012 roku. Od 2014 roku aktywnie uczestniczy w realizacji projektów, które wykorzystują Apache Kafka jako kluczowe narzędzie technologiczne.

“Nie możemy rozwiązywać problemów używając takiego samego schematu myślowego, jakim posługiwaliśmy się w trakcie ich pojawienia się”

Albert Einstein

Ostatnie artykuły

Stack Overflow •

How to run multiple instances of the same app as producer/consumer in Kafka?

i have an simple rest api that have a h2 database so my plan is when i run multiple instances of the same app they will have different in memory databases.Now i want to syncronize these databases beetwen them.I thought kafka to be a good solution , so for example when i get an POST for instance with port 8080 , i should post also for all other instances. Now my app acts as a producer/consumer at the same time and i do not know why only one instance receive the message. The code:

Why is this hap
Stack Overflow •

1 million Kafka producer records a second?

So after studying the linked documentation you will see two groups of properties coupled:

Kafka producer has built in mechanism of batching and compression. The mechanism gets the best results when the producer groups and then compresses many messages into a single batch, so simply setting a high value for batch size and enabling compression sounds very good, and actually it is, but to see what the last piece of puzzle is we need to go through the internal Kafka producer implementation. Kafka p
Stack Overflow •

Publish multiple events shares some attributes in one Kafka topic

I need to publish multiple messages from the same project which represents employee journey events, and i need to use one topic only to publish these messages as they are representing the same project, but in some cases the message may contain extra fields for example:

All messages share (id, name, type, date) and may some events have more fields like (course id, course name), so I am intending to use one parent object called "Journey", contains "Event" object, and I will create multiple childr
Stack Overflow •

When Kafka Streams GlobalKTable is a good choice?

I'm new in Kafka Streams world. I'm wondering when to use Kafka Streams (with compacted topic under the hood) instead of regular database for persisting data. And what are advantages and disadvantages of both solution. I guess both ensure data persistence on the same level.

Let's say there is an simple e-commerce app having users registering and updating their data. And there are two microservices - first one ( ) is responsible for registering users and the second one ( ) is responsible for pla
Stack Overflow •

Connection management when using kafka producer in high traffic environment

I am going to use kafka in a very high traffic environment of more than a billion requests per day. Every request will make a connection to kafka cluster to send message. So there will be so many connections being made continuously every second. This could cause issues like socket timeouts. producer is making all the non-persistent connections. So in such case there could be socket timeout or port exhaustion issues.

Most ecosystem is in php, so I have to use php library for kafka. Now how to ef
Stack Overflow •

Kafka Streams - Retrying a message

As I understand Kafka Streams and the whole Confluent Platform architecture you shouldn’t communicate with any external resources directly from Kafka Streams application. One of the basic concept is Kafka Steams application inputs and outputs are just Kafka topics. Communication with every other external resources should be done by Kafka Connect. There is a lot different connectors made by Confluent and community, you can even write your own implementation if needed.

In this approach you don't
Stack Overflow •

Building a cache with Kafka Streams

I am trying to get a sense of what is possible and how to think when working with .

I want to create a cache that will hold all recent transactions (last 10 minutes).

The cache can be queried by a rest client by providing the transaction reference.
• Is Kafka streams (along with it's Materialized views) a good fit to implement such a cache?
• If yes how would you go about it? Remember it needs to keep only the last 10 minutes of transactions and discard the older ones.
• If not, why not?