Using MATLAB 2013
If you can upgrade to 2013 read on, otherwise go to the bottom of the page for some suggestions for 2012 version. Even if you are not planning to upgrade, still read the first section, it might give you some clues.
I have always had issues using MATLAB 2012 to compile MEX functions. Upgrading to 2013 made it very easy for me to compile CUDA MEX files without worrying about invoking NVCC inside MATLAB with all the crazy flags.
All you would need is to copy the mexopts.bat file from your MATLAB installation:
matlabroot\toolbox\distcomp\gpu\extern\src\mex\win64\mexopts.bat2 (where matlabroot is the MATLAB installation folder, something like:C:\Program Files\MATLAB\Matlab2013a.)
to your project folder containing the .cu file with mex entry point function. In addition, you would need to add two CUDA libraries: cuda.lib and cudart.lib to the same folder from the CUDA installation folder. I have included these two libs in my project, they should also be available somewhere in your CUDA Toolkit installation folder.
Then you would go to MATLAB and run mex -setup to select a compiler. Since you have Visual Studio 2010 installed you should see it in the list. Just follow the instructions on the screen and select your compiler -- more details. If you don't see the compiler or things go wrong, you can try installing the Windows SDK. Also look at this post about compilers and the version of Visual Studio.
If you want to see your MEX compiles checkout my project. Download the content and put them in a folder somewhere in your computer and open up MATLAB. Then go to that directory and in MATLAB confirm that you are indeed in that directory by using pwd in the command window; it will tell you if you are in that directory, but make sure that the folder and the subfolders are added to the file path (right click on the folder in MATLAB and select add path > folder and subfolders)
Then, run mex f.cu. This will compile the CUDA MEX file and puts the library in the same folder, and you can call f as a function in MATLAB and MATLAB won't care where it came from. To See if it actually works, you can make a gpuArray and call the f function on it. Like this:
input = gpuArray.ones(100,100); % makes an array of ones (100x100) on the gpu.
y = f(input) % will perform the operation defined in f.cu
You would get something like this:

Suggestions for compiling MEX files in MATLAB 2012:
I have never had success compiling MEX files on MATLAB 2012, 64bit Windows, even with Visual Studio 2010 Professional. But to start, you can look at this doc page for MEX files and CUDA. Also look at this page for some general information about compiling MEX files in MATLAB, it might give you some ideas of the things involved. Once you went through these pages, look at NVCC compiler and how to invoke it properly in MATLAB. I has always been difficult to get it right for me and I never had success. But don't get discouraged. The key is to see if you can compile basic C or C++ files using a compiler in MATLAB. And once that is successful, move onto compiling CUDA. If nothing works, check this post out about compiling directly in Visual Studio. Good luck !
References:
2: http://www.mathworks.com/help/distcomp/create-and-run-mex-files-containing-cuda-code.html