In order to achieve your expected behavior, you need to know the video's aspect ratio. You can do that quite easily using the API YouTube provides. First, to grab the video info in JSON format, you GET a URL of this form:
https://gdata.youtube.com/feeds/api/videos/video_id?v=2&prettyprint=true&alt=json
Where video_id is the ID of the video you're embedding (note that "prettyprint" just makes it human-readable -- remove it in the actual application to save bandwidth). The property that we care about is entry.media$group.yt$aspectRatio.$t. That property is set to "widescreen" when the video is 16:9, and it isn't defined if it's 4:3. Thus, you can easily check for that value and resize your <iframe> accordingly.
Here's an example:
HTML:
<iframe id = "vid" width="500" src="http://www.youtube.com/embed/ni9RS4pgRXA" frameborder="0" allowfullscreen></iframe>
(Width here is 500 -- it can be, of course, anything you want).
JavaScript (uses jQuery, but it isn't necessary):
$.getJSON("https://gdata.youtube.com/feeds/api/videos/ni9RS4pgRXA?v=2&alt=json",
function(data) {
var obj = $("#vid");
var asp = data.entry.media$group.yt$aspectRatio.$t;
console.log("Aspect Ratio is", asp); //Debugging line. "undefined" if video is 4:3.
if(asp == "widescreen") {
obj.height((Math.floor(obj.width() * 9 / 16)));
}
else {
obj.height((Math.floor(obj.width() * 3 / 4)));
}
});
Here (link) you can find a working demo with an example 16:9 video. The Javascript code is pretty straight-forward, but in case anything is vague, please tell me so that I can explain in more detail.