Ideally, I think killing a process should be avoided by all means... many bad things can happen as result. That said, my proposal counts on 5 parts:
- Locating the fitler dll file in the file-system.
- Enumerating all processes
- Enumarating all loaded modules of each process
- identifying which process is using the filter.
- Killing the process.
Since you did not specify any language or programming framework, I will assume C#/.net just for convenience.
1- DirectShow filters are just COM objects, so they are registered in the system as such. You need to figure out the GUI of your filter, using this GUID, you can locate the registry key where this object information is stored, then you can retrive the location of the dll in the file system from there. Microsoft.Win32.Registry can be used to access the registry.
2- System.Diagnostics.Process.GetProcesses() can be used to enumerate all running process.
3- System.Diagnostics.Process.Modules can be used to enumerate all modules (dlls) loadded by the process.
The rest of the steps should be trivial.