ReliableConcurrent Queue in Service Fabric

0
votes

I have a service fabric application with one stateless service and multiple stateful services. The stateless service reads from RabbtitMQ(using MassTransit) and passes the message to corresponding stateful services to process it.The message passed to stateful service is queued in ConcurrentQueue and in RunAsync it dequeue and process. It was fine until there where less number of messages. Recently the number of messages increased multiple folds. Now the situation is message count in RabbitMq is in millions and the stateful service queue gets overloaded. The memory usage became very high and the cluster gets stuck.

  1. Does the queue gets locked during enqueue process that the RunAsync method needs to wait. Enqueue are very high.
  2. I'm not able to find a way to stop fetching from RabbitMq if the queue count is very high. Atleast the cluster won't hang or crash.
  3. What will be the best design follow in situations where input load is very high?

Thanks

1
rabbitmqazure-service-fabricservice-fabric-stateful
1. show your code (producing and consuming messages) 2. why do you need stateful services? 3. why do you need separate stateless service to read queues? why not read messages in stateful services? – mtkachenko
@mtkachenko The queue has different kind of data. To process each type there's one stateful service type which has Concurrent queue. So stateless service linked to RabbitMq will get message, checks the type and en-queue in corresponding stateful service. Stateful service is Int64 partitioned. The application was designed this way for individual scaling purpose since some types of data are in higher count. – Binil

1 Answers

0
votes

How are you partitioning your stateful services? If you have a range or named partitioning, all your messages should not be hitting the same replica. If you applied partitioning scheme correctly, your messages will end up in different partitions of the same stateful service and you can scale out horizontally.