1) What kind of streaming protocols does XBOX 360 supports?
2) what kind of video formats can be play on XBOX 360?
3) Does H264 and rtsp supported by XBOX 360?
I want to develop some app that will support the video streaming on XBOX 360 platform.
Your first hurdle would be getting the app on the actual platform. Your only option, short of a direct contract with microsoft, would be the Indie/XNA platform. XNA games cannot connect to the internet. So forget that option.
If you manage to score a dev kit and contract with microsoft, you will be looking at h.264, wmv, avi, and even divx as possible options. There are probably more formats (re: containers) that will depend upon the codec, but the former should be sufficient for any need.