P.S. Most safe way to address New form is to use ListForm.aspx page, as follows:
/_layouts/listform.aspx?PageType=8&ListId={PUT-LIST-GUID-HERE}
(PageType value goes from PAGETYPE enumeration)
i think you should be work when use PAGE_NEWFORMDIALOG : New form for a file dialog box.Value=9
as describe on msdn. if New form is not open on model pop up than
Try this :
HyperLink
<a href='javascript:;' onclick='Opendialog()'>New Item</a>
Opendialog is Javascript Function
function Opendialog() {
var options = SP.UI.$create_DialogOptions();
options.resizable = 1;
options.scroll = 1;
options.url = SiteURLWhichExistList + "/_layouts/listform.aspx?PageType=8&ListId={PUT-LIST-GUID-HERE}";
options.dialogReturnValueCallback = Function.createDelegate(null, CloseCallback);
SP.UI.ModalDialog.showModalDialog(options);
}
Note : SiteURLWhichExistList
pass your site collection url in where your list exist.
Hope it helps!!