Here are a few things you need to know when it comes to having your dissectors called on relevant data.
1) You need to register your dissector for the ports that are used for this protocol. Note that registration does not care whether a port is source or destination. You can register both ports or register ports that are specific to your protocol if possible. For instance, it does not make sense to register protocol 1 for port 10001 because 10001 is also used by protocol 2.
2) Regardless of what ports you chose to register, your protocol 2 will at least be registered for the same port as either protocol 1 or protocol 3. This could seem like a problem, however, when your dissector is called, it can stop dissecting at any time if it detects that the data it is handling is not valid for its protocol. When doing so, it can let Wireshark know that no data was dissected by returning 0. This will let any other dissector registered for the same port try to dissect the data.
Here is an example. Let's say the dissectors for your 3 protocols are registered as follows:
protocol 1:
local udp_encap_table = DissectorTable.get("udp.port")
udp_encap_table:add(10000, p_proto1)
protocol 2:
local udp_encap_table = DissectorTable.get("udp.port")
udp_encap_table:add(10001, p_proto2)
protocol 3:
local udp_encap_table = DissectorTable.get("udp.port")
udp_encap_table:add(10003, p_proto2)
In this scenario, if Wireshark handles a packet with src_port 10000 and dst_port 10001, two things can happen:
a) p_proto1 is called first (because p_proto1 is registered for port 10000): the packet is valid for that protocol, so it is properly dissected and everything is good.
b) p_proto2 is called first (because p_proto2 is registered for port 10001): this is not a valid protocol 2 packet so p_proto2 should return 0, and Wireshark will then call p_proto1, which will dissect the packet properly.