This answer assumes you are trying to open up the selected document from the view panel in the dialog. (In essence, open a dialog with the values from document listed in an embedded view)
forward
More than likely I would just open the document in another window and be done with it. But if you want to open this in a dialog, fine. If you define the data source in the dialog itself, be warned that I personally have had issues with such approaches in the past and I think that the datasources should be defined in the xpage view level.
answer
1. Do not allow any links in the view.
2. Have a checkbox available in the view.
3. Add a button where you will get the NoteID of the selected document, also making sure that a document was selected.
var vwpnl = getComponent("homeViewPanel");
var docIDArray = vwpnl.getSelectedIds();
if(docIDArray.length != 1){
view.postScript("alert('"+MessageController.getMessageForCurrentUser("msg_SelectOneDocument") +"')");
return;
}
var firstID = docIDArray.length > 0 ? docIDArray[0] : "";
return firstID;
(you can ignore the MessageController as that is one of my helper classes for language specific user messages.)
4 use that noteID (whether in viewScope or not) in the calculation of the data source in the dialog making certain that dialog has the correct settings, like refreshing etc.
EDIT
The good part about keeping this approach (performance aside) in your toolbox is that once you get a handle on the document itself, you can compute anything. using the noteID just to set a dialog property is just one example. You could create a document object and use the values to do anything you want, including opening another page, other actions, basically whatever you want. It might not be the best option for your current use case (you did not really specify your use case), but for future readers it will hopefully be a help.