Broadway is a concurrent and multi-stage data ingestion and data processing with Elixir

Broadway is a concurrent and multi-stage data ingestion and data processing with Elixir
Average rating: 0
(0 votes)

You can trust us when it comes to the topic of Elixir. We absolute fans of it. Let’s introduce you the new tool. Broadway is a concurrent and multi-stage data ingestion and data processing with Elixir.

Broadway was mainly designed to help developers build concurrent, multi-stage data ingestion and data processing pipelines. It takes the burden of defining concurrent GenStage topologies and provides a simple configuration API that automatically defines concurrent producers, concurrent processing, batch handling, and more, leading to both time and cost efficient ingestion and processing of data. It allows developers to consume data efficiently from different sources, such as Amazon SQS, RabbitMQ, and others.

Broadway comes with a handful of features that take the burden of defining concurrent GenStage topologies and provide a simple configuration API that automatically defines concurrent producers, concurrent processing, batch handling, and more, leading to both time and cost efficient ingestion and processing of data. Some of those features include:

  • Back-pressure
  • Automatic acknowledgments
  • Batching
  • Automatic restarts in case of failures
  • Graceful shutdown
  • Built-in testing
  • Partitioning

Other features are already on the roadmap, such as:

  • Rate-limiting
  • Statistics / Metrics
  • Back-off

There’s a lot more about Broadway here.

And don’t miss a chance to find out about a new experimental library that was just released – Mint, a new HTTP client for Elixir.

Rate this article, if you like it

Thanks! You’ve rated this material!

Got a project? Let's discuss it!

*By submitting this form you agree with our Privacy Policy.

Mailing & Legal Address

Syndicode Inc. 340 S Lemon Ave #3299, Walnut CA, 91789, USA

Visiting & Headquarters address
Kyiv Sofiivska 1/2a, 01001, Kyiv, Ukraine
Dnipro Hlinky 2, of. 1003, 49000, Dnipro, Ukraine
Email info@syndicode.com
Phone (+1) 9035021111