...
Network Telemetry at AmLight, Jeronimo Bezerra (Amlight)
Abstract: Funded by the U.S. National Science Foundation (NSF), AmLight is a distributed academic exchange point connecting national and regional research and education networks in Latin America to the U.S. and Africa. AmLight is responsible for transporting science data related to most telescopes in Chile and supporting the Large Hadron Collider Tier 2 data center in Brazil, and many other science projects.
...
A typical Vera Rubin telescope data transfer will be 5-second bursts of 9Kbytes packets at 40+Gbps from Chile to the U.S. throughout the night. Each burst creates a telemetry flow of 1.4Gbps @ 487kpps and a total 900MB of telemetry data to be processed/stored/shared. The challenge is receiving 487.000 256-byte packets per second, single flow, single NIC queue, single CPU core, and process them in real-time. Without Kernel bypass, most CPU cores will operate at 100% and drop more than 80% of the packets due to the high CPU utilization. And, this is just one flow over AmLight.
Bio: the AmLight Network Architect
DPDK + Kafka: Multi-MPPS Telemetry Data Ingest and Stream Processing at ESnet, Richard Cziva (ESnet)
Abstract: : We will introduce ESnet’s per-packet telemetry collection system (uses Xilinx FPGAs).
The main focus will be on our DPDK application called fastcapa-ng that takes telemetry packets from the wire and pushed it to Kafka. It can do filtering, down-sampling (user specified 1:X ration) and histogram generation (user configurable) that we implemented in this DPDK app. We also show prometheus / Grafana integration to monitor our pipeline.
I will show how we can run stream processing application using Kafka Streams API. Simple code for SYN Flood detection example will be shown.
Ingest rate challenges will be highlighted.
Bio: Richard Cziva is a software engineer at ESnet. He has a range of technical interests including traffic and performance analysis, data-plane programmability, high-speed packet processing, software-defined networking, and network function virtualization.
Prior to joining ESnet in 2018, Richard was a Research Associate at University of Glasgow, where he looked at how advanced services (e.g., personalized firewalls, intrusion detection modules, measurement functions) can be implemented and managed inside wide area networks with programmable edge capabilities.
Richard holds a BSc in Computer Engineering (2013) from Budapest University of Technology and Economics, Hungary and a Ph.D. in Computer Science (2018) from University of Glasgow, United Kingdom.
...
The NetSage software consists of a set of open source tools that are deployed on local systems, and a managed, centralized, secure data archive. NetSage TestPoints are a collection of software and hardware components that gather active and passive data into records that are then sent to the Data Ingest Pipeline. The five-step Pipeline filters those records and adds additional tags before de-identifying the data. The records are then stored in the NetSage Archive, a centralized storage framework consisting of two different databases, a Time Series Data System (TSDS) archive and an Elasticsearch archive. Performance Dashboards, built using the open source Grafana analysis and visualization engine, access the records from the NetSage Archive to present visualizations to answer the questions identified by the stakeholders.
Bio: the Network Systems Analyst from Indiana University
A Proposal towards sFlow Monitoring Dashboards for AI-controlled NRENs, Mariam Kiran (Esnet)
...