NServiceBus pub-sub on azure service bus queues

2
votes

I'm looking to implement a simple pub-sub demo using NServiceBus backed by Azure Service Bus.

I am working with an undetermined number of subscribers with more being spawned or existing ones being killed depending on various conditions, so every time I create a subscriber, I create an Endpoint and an EndpointInstance with an unique name and subscribe to the event I am interested in.

When the Subscriber shuts down, I unsubscribe from the events and Stop the EndpointInstance.

However, if I check my Service Bus, the queue and topic subscriptions that get created in the process are not cleaned up. This means that after a while I will potentially have hundreds of queues and topic subscriptions that will never be used again due to the way I generate my EndpointInstance names.

How do I clean those up?

Am I missing something ? should i not be creating a new Endpoint and EndpointInstance for every subscriber (this seems to be the pattern used in the official samples and guides).

UPDATE:

An analogy that just came to mind is something like a push notification system where clients(subscribers) come online and subscribe to the bus. When an event is published I want all online clients to receive it. When a client is shut down, I don't care about messages that are sent not am I interested in getting any lost messages when a client comes online. I want to close the subscription and clean-up the underlying infrastructure (queues, subscriptions, etc).

In my scenario, each client will define its own unique Endpoint and Endpoint instance. How do the logical endpoints come into play at this point?

3
nservicebusazureservicebus
Can you tell more about the actual problem where you need this? Maybe its a design issue. Are all these instance the same endpoint? Is this for scaling out? – Ramon Smits
To what documentation are you referring with the statement "this seems to be the pattern used in the official samples and guides)." – Ramon Smits
When you unsubscribe, the subscription is removed and no more messages should arrive at that endpoint its queue. But stopping an endpoint doesn't mean the queues are removed. It could temporarily go down for maintenance and be brought up again. As Ramon mentioned, please provide what you want to demo and what you're expecting. – Dennis van der Stelt
Every subscriber uses its own endpoint and instance (unique endpoint name). I'm looking at publish/subscribe sample on the website - docs.particular.net/samples/pubsub. Each subscriber uses its own EndpointConfiguration (with a different name). I want to clean up the Service Bus once the subscribers are shut down. I only care to listen to messages when subscribers are online. – AlexDrenea

3 Answers

3
votes

The scenario you've described makes sense. You're interested in temporary endpoints that are created, serve their purpose, and then removed from the system. Those endpoints are unique and have no multiple instances. And even if they do, all instances are teared down.

This means that after a while I will potentially have hundreds of queues and topic subscriptions that will never be used again due to the way I generate my EndpointInstance names.

The issue you're experiencing with subscriptions is a bug. I've raised an issue here: link. Please chime in to provide more feedback. The behaviour you expect to see is what should be happening once the bug is addressed. I.e. once you've unsubscribed from a given event, that event should no longer be forwarded to your endpoint (in case of ForwardingTopology) or stored under endpoint`s subscription (in case of EndpointOrientedTopology).

Please note that for the endpoint input queue, there's no way for NSB to remove the queue while endpoint itself is running. This is something you'd have to custom code after endpoint was taken down.

1
votes

You mention a pub/sub demo and the ability to clean everything up after an endpoint is destroyed. You properly unsubscribe your endpoint, which means the subscription is removed from Azure Service Bus and message that are published, aren't arriving at the queue anymore.

In production scenarios you can bring your endpoint down for various reasons. One of them is deploying a new version or perform maintenance on your datastore. Another one could be as simple as that your server/service crashed, etc, etc. While it is down, in production scenarios, you'd want message to still arrive at the queue. This way you can reliably process them whenever the endpoint is back up again.

Due to this we don't support cleaning up queues. If you want to clean up everything after bringing down an endpoint, you'll have to manually remove queues and what not.

About the demo set up, you mention unique endpoints and instances. Just for the sake of clearness, let me try to explain what we mean with endpoints and instances. You can build a logical endpoint which performs some tasks. This can contain various handlers, sagas, etc. It has a unique name and a unique function. You can have various logical endpoints in a system.

Upon deployment, usually you have a single instance for every endpoint. If you want to scale out, you can deploy another instance of the same logical endpoint. When using Azure Service Bus, you'd have something called competing consumers. This basically means that both endpoint instances are trying to retrieve messages from the same queue. One always wins and starts processing the message.

It's not logical to publish messages to another instance of the same endpoint, because you can never tell which instance of a (logical) endpoint will pick up the message. You can still subscribe to your own messages, but the logical endpoint subscribes to its own message. Don't do this for instances specific, it's impossible.

It is however fine to publish messages, which are received by other logical endpoints. No matter how many instances they have.

Now what puzzles me is your next statement:

I am working with an undetermined number of subscribers with more being spawned or existing ones being killed depending on various conditions

It seems like you're creating new logical endpoints on the fly and deploy these. It is possible to give them unique names based on configuration, set at deployment. But I still don't understand why you would want to demo this.

Let me know if this helps and if not, comment below this post and I'll try to answer more questions. Perhaps inside this post as updates, since the comments are very limited on number of characters you can use.

0
votes

What kind of subscribers do you have here? Should all clients receive the same type of events or are you going to be filtering them? If its the last its more like a notifications.

I suspect that you are having clients that connect to your system for a little while and are only interested in events specifically targeted to this client.

A framework like SignalR is a good fit for such a use case where you want to send notifications back to a specific client.

However, within your system you can use NServiceBus to communicate between your own endpoints which can trigger events which should be pushed as notifications to the correct client.