Merging multiple Netcdf files according to the vertical level

4
votes

I have 34 netCDF (nc) files containing latitude, longitude, and the data in every file. Every filename containing a number which corresponds to the Pressure level in hPa (starts from 1 to 34 for respective pressure level from (1000 hPa to 0.4 hPa). I want to join all files into a single nc file with this vertical level dimension information.

I tried to read whole files using xarray open_mfdataset but I cannot con_cat with the level dimension since it isn't in the files.

import xarray as xr
ds = xr.open_mfdataset('/media/MediaCentre/Dataset/d9/data*.nc',concat_dim='level')

The files do not have any information in the global attributes regarding the pressure. They are names sequentially: data1.nc, data2.nc, ... dataN.nc and correspond to the pressure levels (hPa) of: 1000 975 950 925 900 850 800 750 700 650 600 550 500 450 400 350 300 *250 200 150 100 70 50 40 30 20 15 10 7 5 3 2 1 0.4

How can I merge these together using python xarray, or cdo/nco?

Sample data are here https://www.dropbox.com/sh/linfn0721ze3j1f/AACwxTsQVNyiE7mF_gRbpRfra?dl=0

3
pythonnetcdfpython-xarrayncocdo-climate
Can I ask, in the files themselves there is no data entry (dimension, variable or global attribute) with the pressure level recorded? i.e. one would like get the pressure from the file name and add a new dimension in the merged file? – Adrian Tompkins
@AdrianTompkins Yes, the file doesn't contain pressure level information, it just contains latitude, longitude, and the data. But it has pressure level information given in the file name (like 1 represents the surface level 1000 hPa, 2 represents 975 hPa like that). – Krishnaap
ps: from your reply to Robert Wiilson, it seems you are not tied to a python solution, but are happy to have an nco/cdo alternative. If so I may edit the question to reflect this, as some people downvote when a question is asked with a certain language specified and then people upload answers with an alternative language. – Adrian Tompkins
@AdrianTompkins I would really appreciate your suggestion and edits. Thanks. – Krishnaap

3 Answers

3
votes

It is probably easier to do this using CDO. The following will merge the two sample files you have supplied:

cdo -L -merge -setlevel,0.4 data1.nc -setlevel,1 data2.nc merged.nc

Just modify the above to be able to handle all of the files.

3
votes

A different approach with NCO would first combine the levels with ncecat, e.g.,

ncecat -u level in*.nc out1.nc

and then add the level coordinate with ncap2, e.g.,

ncap2 -O -s 'level[$level]={1000, 975, ... 0.4}' out1.nc out2.nc

and then add the attributes with ncatted as Adrian shows.

ncatted <Adrian's example> out2.nc out3.nc

Good luck, Charlie

3
votes

This is a BASH script which defines a list of pressure levels (look at the comments) and then the workhorse is ncap2, which is used to add a dimension called "level" to each file, and then define a variable "level" with the pressure value defined. I then use ncatted to add attributes for completeness such as the pressure units.

cdo is then used at the end to merge the files (nco could alternatively be used). The example here only merges two files, but you can merge all your .

#!/bin/bash

# define pressure levels here, start, inc, end or just a list
# this is important to get right, the number of files processed
# depends on this list, here I only process 2 files,
# data1.nc = 1000hPa, data2.nc=975 hPa

# this was my test merging two files, can use seq if p levs are regular
# p_levs=($(seq 1000 -25 975))

# this is the full code:
p_levs=(1000 975 950 925 900 850 800 750 700 650 600 550 500 450 400 350 300 250 200 150 100 70 50 40 30 20 15 10 7 5 3 2 1 0.4)
    
# loop over the pressure levels, data1.nc-> first pressure level, data2-> second etc
for i in ${!p_levs[@]} ; do
    infile=data$(expr $i + 1).nc # nc filename

    # add a level dimension and define variable with pressure level:
    ncap2 -O -s 'defdim("level",1);q[level,lat,lon]=q;level=array('${p_levs[$i]}',10,$level)' $infile tmp${i}_$infile

    # put in attributes for the new level variable:
    ncatted -h -a units,"level",o,c,"hPa" tmp${i}_$infile
    ncatted -h -a long_name,"level",o,c,"Pressure level" tmp${i}_$infile
done

# merge the files:
cdo merge tmp*.nc merged.nc

# clean up tmp files at end 
rm -f tmp*.nc

I tested it using p_levs="1000 975" and it merged the first two of your files data1.nc and data2.nc without problems and the resulting file opens okay in ncview and looks fine.

NOTE: For some reason ncap2 is removing the attributes from "q" as per the comments on this post , I'm not sure why, so it means you will need to readd those with ncatted. This answer here from Charlie Zender was also helpful to construct this code.