How to calculate number of missing values summed over time dimension in a netcdf file in bash

Question

I have a netcdf file with data as a function of lon,lat and time. I would like to calculate the total number of missing entries in each grid cell summed over the time dimension, preferably with CDO or NCO so I do not need to invoke R, python etc.

I know how to get the total number of missing values

ncap2 -s "nmiss=var.number_miss()" in.nc out.nc

as I answered to this related question: count number of missing values in netcdf file - R

and CDO can tell me the total summed over space with

cdo info in.nc

but I can't work out how to sum over time. Is there a way for example of specifying the dimension to sum over with number_miss in ncap2?

Chiel Chiel · Accepted Answer · 2017-05-11T14:19:13

Even though you are asking for another solution, I would like to show you that it takes only one very short line to find the answer with the help of Python. The variable m_data has exactly the same shape as a variable with missing values read using the netCDF4 package. With the execution of only one np.sum command with the correct axis specified, you have your answer.

import numpy as np
import matplotlib.pyplot as plt
import netCDF4 as nc4

# Generate random data for this experiment.
data = np.random.rand(365, 64, 128)

# Masked data, this is how the data is read from NetCDF by the netCDF4 package.
# For this example, I mask all values less than 0.1.
m_data = np.ma.masked_array(data, mask=data<0.1)

# It only takes one operation to find the answer.
n_values_missing = np.sum(m_data.mask, axis=0)

# Just a plot of the result.
plt.figure()
plt.pcolormesh(n_values_missing)
plt.colorbar()
plt.xlabel('lon')
plt.ylabel('lat')
plt.show()

# Save a netCDF file of the results.
f = nc4.Dataset('test.nc', 'w', format='NETCDF4')
f.createDimension('lon', 128)
f.createDimension('lat', 64 )
n_values_missing_nc = f.createVariable('n_values_missing', 'i4', ('lat', 'lon'))
n_values_missing_nc[:,:] = n_values_missing[:,:]
f.close()

How to calculate number of missing values summed over time dimension in a netcdf file in bash

2 Answers