1
votes

I have uploaded a file to my Azure file storage account and created a SAS (shared access signature). Let's pretend the file in question is called fileA.nc

Now, with Python3, I am attempting to read fileA.nc:

from netCDF4 import Dataset

url ='https://<my-azure-resource-group>.file.core.windows.net/<some-file-share>/fileA.nc<SAS-token>';
dataset = Dataset(url)

print(dataset.variables.keys())

The above code does not work, instead giving me the following error:

Traceback (most recent call last): File "yadaYadaYada/test.py", line 8, in dataset = Dataset(url) File "netCDF4/_netCDF4.pyx", line 1848, in netCDF4._netCDF4.Dataset.init (netCDF4/_netCDF4.c:13983) OSError: NetCDF: Malformed or unexpected Constraint

This is line 8:

 dataset = Dataset(url)

I know the URL provided works. If I paste it into the browser, the file downloads...

I have checked the netCDF4 documentation, which says this:

Remote OPeNDAP-hosted datasets can be accessed for reading over http if a URL is provided to the Dataset constructor instead of a filename. However, this requires that the netCDF library be built with OPenDAP support, via the --enable-dap configure option (added in version 4.0.1).

However, I have no idea how to tell if when Pycharms installed netcdf4, it used the --enable-dap argument, but I cannot imagine why it would not. Besides, if I stick in a url which points to some HTML, I get the HTML in the error dump and so from that I would think netcdf4 is actually trying to load a remote dataset and so the problem is somewhere else.

I'd really appreciate some help here. Maybe someone knows of another Python 3 netCDF library that will allow me to load my datasets from Azure?

UPDATE

Okay, I can now confirm that the python netcdf4 library does come with --OPenDAP enabled:

Hello again, netCDF4 1.0.4 with OpenDAP support is now available in the conda respoitory on Unix. To install: $ conda install netcdf4

  • Ilan
1
Hi, I'm not sure what is the relation with OpenDAP here? I have not worked with Azure storage, but I very much doubt that it automatically gives OpenDAP support. And, if Azure allows direct file open, then you can open your NetCDF file as well, but in case it is similar to AWS-s S3, then you basically should download file to local disk first and then open.kakk11

1 Answers

1
votes

I have found a solution. It turns out that you cannot read directly from an Azure File share, even though when you paste the link to a file in the browser, the file begins to download.

What I needed to do was to mount the File Share on my OS. In my case, I was using Windows but this can be done with Linux, too. The following code should be modified accordingly and then put into Command Prompt:

net use <drive-letter>: \\<storage-account-name>.file.core.windows.net\<share-name>

example :
net use z: \\samples.file.core.windows.net\logs

Once the File Share is mounted, you can read from it as if it were an external HDD. You may need to add permission, but I didn't.

Here is the link to the documentation for mounting the File Share: Documentation