I have Dataflow pipeline with several blocks connected.
- Block deserialize data from RabbitMQ
- Block enrich received object with additional data from other API's
- Block Save object to local cache
- Block Produce data over SSE connection
Ordering is turned to OFF and parallelism is set 128. On the other side Chrome is used to consume this SSE connection and diplay it in DataGrid Pro. I use Metrics to log execution time and pending items of every block in Grafana. Rate of the messages go high from time to time (15K-20K per min.) and the object is very big. If the filters on the browser side is not setup correctly, CPU goes to 100%.
From time to time I received big time spikes in Produce SSE block of the Dataflow
My question is: Is it possible lack of resources on the browser side to make this time spikes in the produce SSE block. Thanks.
P.S. Can't post source code, because is forbidden by the company policies and because will be very big post.
During the last stress test on the application(15K messages per min.) I added just one listener from the browser console and everything looks fine.