# [[Apache Kafka]]
![[Apache Kafka.svg]]
[site](https://kafka.apache.org/)
Apache Kafka is a [[Message Broker]] that is optimized for scalability and performance. However, it expands significantly on other message brokering methods like [[Message queuing]] and [[Google Pub-Sub]] by introducing a hybrid of the two, called [[Event streaming]].
Officially, it is an "open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications." [^kafka]
Kafka was created at LinkedIn. [^elastic] However, it has since been [[Open source|open-sourced]] and is now mainly maintained by [[Confluent]].
<iframe width="560" height="315" src="https://www.youtube.com/embed/PzPXRmVHMxI" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
## Advantages of Kafka
- Kafka is open source
- It's distributed, improving resilience and fault tolerance. [^maarek]
- It's horizontally scalable: it can handle hundreds of brokers and millions of messages per second. [^maarek]
- It's highly performant, with a latency of <1 ms. [^maarek]
- It's real time.
## Kafka in use
- [[Netflix]] uses Kafka for show recommendations. [^maarek]
- [[Uber]] uses Kafka to calculate surge pricing for taxis based on real-time information of demand. [^maarek]
- [[LinkedIn]] uses Kafka to prevent spam and recommend connections. [^maarek]
- [[Grafana Loki|Loki]], [[Grafana Mimir|Mimir]], [[Grafana Tempo|Tempo]], and [[Grafana Pyroscope|Pyroscope]] are using Kafka or similar tools to reduce the [[Database]] [[Replication Factor]].
## Variants of Kafka
Kafka is highly scalable and tuneable, but has grown into a bit of a monolith. To address these issues, other tools have sprung up that use the Kafka API.
- [[WarpStream]] implements Kafka but writes to [[Object storage]]
- [[Redpanda]] is a Kafka-compatible streaming platform
### Managed Kafka services
- [[Aiven for Apache Kafka]] is a managed Kafka service by [[Aiven]]
- [[Amazon Managed Streaming for Kafka]] by [[AWS]]
- [[Azure Event Hubs]]
- [[Google Pub-Sub]] is not actually Kafka, but is similar
[^kafka]: Apache Kafka. Accessed in May 2021 from the [Kafka homepage](https://kafka.apache.org/).
[^elastic]: Rao, S. and Levy, T. (2016). _Just enough Kafka for the Elastic stack, part 1_. Accessed in May 2021 from [here](https://www.elastic.co/blog/just-enough-kafka-for-the-elastic-stack-part1).
[^maarek]: Maarek, S. (2019). _Apache Kafka in 5 minutes_. Retrieved from https://www.youtube.com/watch?v=PzPXRmVHMxI
%%
# Excalidraw Data
## Text Elements
## Drawing
```json
{
"type": "excalidraw",
"version": 2,
"source": "https://github.com/zsviczian/obsidian-excalidraw-plugin/releases/tag/2.1.4",
"elements": [
{
"id": "4y8R7iOA",
"type": "text",
"x": 118.49495565891266,
"y": -333.44393157958984,
"width": 3.8599853515625,
"height": 24,
"angle": 0,
"strokeColor": "#1e1e1e",
"backgroundColor": "transparent",
"fillStyle": "solid",
"strokeWidth": 2,
"strokeStyle": "solid",
"roughness": 1,
"opacity": 100,
"groupIds": [],
"frameId": null,
"roundness": null,
"seed": 967149026,
"version": 2,
"versionNonce": 939059582,
"isDeleted": true,
"boundElements": null,
"updated": 1713723615080,
"link": null,
"locked": false,
"text": "",
"rawText": "",
"fontSize": 20,
"fontFamily": 4,
"textAlign": "left",
"verticalAlign": "top",
"containerId": null,
"originalText": "",
"lineHeight": 1.2
}
],
"appState": {
"theme": "dark",
"viewBackgroundColor": "#ffffff",
"currentItemStrokeColor": "#1e1e1e",
"currentItemBackgroundColor": "transparent",
"currentItemFillStyle": "solid",
"currentItemStrokeWidth": 2,
"currentItemStrokeStyle": "solid",
"currentItemRoughness": 1,
"currentItemOpacity": 100,
"currentItemFontFamily": 4,
"currentItemFontSize": 20,
"currentItemTextAlign": "left",
"currentItemStartArrowhead": null,
"currentItemEndArrowhead": "arrow",
"scrollX": 583.2388916015625,
"scrollY": 573.6323852539062,
"zoom": {
"value": 1
},
"currentItemRoundness": "round",
"gridSize": null,
"gridColor": {
"Bold": "#C9C9C9FF",
"Regular": "#EDEDEDFF"
},
"currentStrokeOptions": null,
"previousGridSize": null,
"frameRendering": {
"enabled": true,
"clip": true,
"name": true,
"outline": true
}
},
"files": {}
}
```
%%