BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//CERN//INDICO//EN
BEGIN:VEVENT
SUMMARY:Virtual Log-Structured Storage for High-Performance Streaming
DTSTART;VALUE=DATE-TIME:20211202T120000Z
DTEND;VALUE=DATE-TIME:20211202T123000Z
DTSTAMP;VALUE=DATE-TIME:20260711T081402Z
UID:indico-contribution-1478@events.chpc.ac.za
DESCRIPTION:Speakers: Ovidiu-Cristian Marcu (University of Luxembourg)\nOv
 er the past decade\, given the higher number of data sources (e.g.\, Cloud
  applications\, Internet of things) and critical business demands\, Big Da
 ta transitioned from batch-oriented to real-time analytics. Stream storage
  systems\, such as Apache Kafka\, are well known for their increasing role
  in real-time Big Data analytics. For scalable stream data ingestion and p
 rocessing\, they logically split a data stream topic into multiple partiti
 ons. Stream storage systems keep multiple data stream copies to protect ag
 ainst data loss while implementing a stream partition as a replicated log.
  This architectural choice enables simplified development while trading cl
 uster size with performance and the number of streams optimally managed. T
 his paper introduces a shared virtual log-structured storage approach for 
 improving the cluster throughput when multiple producers and consumers wri
 te and consume in parallel data streams. Stream partitions are associated 
 with shared replicated virtual logs transparently to the user\, effectivel
 y separating the implementation of stream partitioning (and data ordering)
  from data replication (and durability). We implement the virtual log tech
 nique in the KerA stream storage system. When comparing with Apache Kafka\
 , KerA improves the cluster ingestion throughput (for replication factor t
 hree) by up to 4x when multiple producers write over hundreds of data stre
 ams. Furthermore\, we present the initial results of running experiments w
 ith KerA over Infiniband and Singularity in an HPC cluster.\n\nhttps://eve
 nts.chpc.ac.za/event/98/contributions/1478/
LOCATION:
URL:https://events.chpc.ac.za/event/98/contributions/1478/
END:VEVENT
END:VCALENDAR