Kafka

Beyond the Basics: Unraveling Kafka’s Log Compaction and Idempotence Welcome back to the DataFibers Community! Today, we’re ditching the superficial “what is Kafka” and plunging into the intricate mechanics that make it a robust and reliable distributed streaming platform. We’ll explore two powerful, yet often misunderstood, features: Log Compaction and Idempotent Producers. These aren’t just buzzwords; they are critical for building fault-tolerant and efficient data pipelines. The Heart of the Matter: Kafka’s Log Structure Before we dive into compaction and idempotence, let’s refresh our understanding of Kafka’s fundamental data structure: the log.

Unpacking Kafka's Internals: A Deep Dive into Its Core Mechanics

in architecture

June 17, 2026

Unpacking Kafka’s Internals: A Deep Dive into Its Core Mechanics Introduction Kafka isn’t just a message queue; it’s a distributed streaming platform designed for high-throughput, low-latency, and fault-tolerant data ingestion. While many understand its basic publish-subscribe model, its true power lies in its meticulously engineered “under-the-hood” mechanisms. This post will peel back the layers, exploring the core architectural components, data distribution, replication, and the guarantees it provides. The Foundation: Brokers, Topics, and Partitions At its heart, a Kafka cluster consists of one or more brokers (servers).

Hermes Agent Unveiled: Architectural Deep Dive for Robust Data Telemetry

in Observability & Monitoring

May 3, 2026

The landscape of distributed systems demands robust and efficient telemetry collection. While many agents exist, the Hermes Agent distinguishes itself with a lightweight footprint, modular design, and a strong emphasis on reliability and security. This deep dive moves beyond a generic overview, peeling back the layers to explore Hermes Agent’s “under-the-hood” architecture, configuration patterns, and practical implementation challenges within the DataFibers ecosystem. The Hermes Philosophy: Input, Process, Output At its core, Hermes Agent operates on a simple, yet powerful, pipeline: Input sources data, Processors transform and filter it, and Outputs deliver it to various destinations.

Apache Kafka Producers

in digest

November 2, 2019

Kafka producers send records to topics. The records are sometimes referred to as messages. The producer picks which partition to send a record to per topic. The producer can send records round-robin. The producer could implement priority systems based on sending records to certain partitions based on the priority of the record. Generally speaking, producers send records to a partition based on the record’s key. The default partitioner for Java uses a hash of the record’s key to choose the partition or uses a round-robin strategy if the record has no key.

Apache Kafka Consumers

in digest

August 4, 2018

Kafka consumer is what we use quite often to read data from Kafka. Here, we use this article to explain some key concepts and topics regarding to consumer architecture in Kafka. Consumer Groups We can always group consumers into a consumer group by use case or function of the group. One consumer group might be responsible for delivering records to high-speed, in-memory microservices while another consumer group is streaming those same records to Hadoop.

Apache Kafka Overview

in article

October 5, 2017

The big data processing started by focusing on the batch processing. Distributed data storage and querying tools like MapReduce, Hive, and Pig were all designed to process data in batches rather than continuously. Recently enterprises have discovered the power of analyzing and processing data and events as they happen instead of batches. Most traditional messaging systems, such as RabbitMq, neither scale up to handle big data in realtime nor use friendly with big data ecosystem.