立即注册 登录
气象家园 返回首页

SkylarYoung http://bbs.06climate.com/?59257 [收藏] [复制] [分享] [RSS] 我们的征途是星辰大海!

日志

How to Concatenate the Time Dimension of netCDF Files with NCO

已有 113 次阅读2020-11-7 06:56

Reference: https://disc.gsfc.nasa.gov/information/howto?keywords=netCDF&title=How%20to%20Concatenate%20the%20Time%20Dimension%20of%20netCDF%20Files%20with%20NCO

Main: 
Overview:
For most satellite Level 3 or modeled monthly products at the Goddard Earth Sciences Data and Information Services Center (GES DISC), one file contains one month of data.  Similarly, a daily file contains one day of data. Data files downloaded through the current data services at GES DISC will keep the original file structure. Thus, downloading 10 years of monthly data, would require downloading 120 files. However, a user may use NCO commands to create a single file for simpler data processing.
Example:

This data recipe shows how to create a single file in netCDF format by concatenating the time dimension of multiple files with NCO commands.

Two sets of data are used in this recipe. One set of example data files are from the MERRA monthly product, containing a time dimension. The other data sets are from the SeaWiFS aerosol daily product, which does not contain a time dimension.    

This recipe works for gridded data only.

Time to complete the following procedures:   30 minutes

Prerequisites:

Tested with Software: NCO version "4.4.4" ; “ 4.3.7”

Procedure:

1. Installing NCO:

If NCO (netCDF Operator)  is available on your system, skip this step.

The software can be downloaded for free from "Bienvenue sur le netCDF Operator (NCO) site" (nco.sourceforge.net) .  The NCO comprises a dozen standalone, command-line programs that take netCDF files as input. 

 

2. Getting data in netCDF format:

Before accessing data at GES DISC, a  user must first register with Earthdata Login, then be authorized to access data at GES DISC by following steps at:  data-access. 

The GES DISC provides a number ways to download data in netCDF format . In this example, we use the Simple Subset Wizard (SSW).   For more information regarding data access through SSW, please read the data recipe How to Obtain a Spatio-temporal + Variable Subset of Data with the Simple Subset Wizard .

 

Example-1: Files with time dimension variable: (MERRA monthly data)

Data Set Keyword(s):  MATMNXSLV   (This is the shortname of the product.)

Date Range:  2014-01-01, 2014-3-31

Spatial Bounding Box:  Leave blank  (for global)

Variables:   Select two variables: Temperature at 2 m above the displacement height, Sea level pressure;

Then, select the download data format as netCDF.

By following the “Downloading Instructions” from the SSW data download page, you may download all of the data files in netCDF format.

In this example, three MERRA monthly files are downloaded:  for example, Jan 2014:

MERRA300.prod.assim.tavgM_2d_slv_Nx.201401.hdf.nc

 

Example-2: Files without time dimension variable: (SeaWiFS aerosol daily data)

Data Set Keyword(s):    SWDB_L310  

Date Range:  2009-01-01, 2009-01-05

Spatial Bounding Box:   Global

Variables:  Aerosol optical thickness estimated at 550 nm over land and ocean

Similar to Example-1, select download data format as netCDF, following the instruction from the SSW data download page.

In this example, five SeaWiFS aerosol daily files are downloaded:  for example, Jan 2009:

DeepBlue-SeaWiFS-1.0_L3_20090101_v004-20130604T022047Z.h5.nc

 

3) Making time aggregation for data in Example-1: files with a time dimension variable:

  • View the time dimension information:

The MERRA data file in Example 1 has a time dimension. To view the time dimension information, use the netCDF command ‘ncdump’, with option -h. For example,

ncdump –h  MERRA300.prod.assim.tavgM_2d_slv_Nx.201401.hdf.nc

dimensions: TIME = 1 ; <= Original time dimension YDim = 361 ; XDim = 540 ;

 

  • Use the following two NCO commands to aggregate the time dimension:

ncks -O --mk_rec_dmn TIME file_in.nc file_out.nc    (making TIME a record dimension)

ncrcat -O file_in1.nc file_in2.nc file_out.nc          (concatenating files)

Note: The command ‘ncrcat’ requires that the time dimension be a record dimension. The time dimension in the example data files is not a record dimension. So, in this example, the record dimension is made by the command ‘ncks’ with option ‘--mk_rec_dmn ‘. Please read the NCO User Guide for more information.

Sample Korn shell script for Example 1 data:

#!/usr/bin/ksh

# Sample script to concatenate MERRA monthly data files in netCDF

flist='MERRA300.prod.assim.tavgM_2d_slv_Nx.201401.hdf.nc MERRA300.prod.assim.tavgM_2d_slv_Nx.201402.hdf.nc MERRA300.prod.assim.tavgM_2d_slv_Nx.201403.hdf.nc'

# Make Time a record dimension

for f in $flist

do

  filein=$f

  fileout='tmp.'$f

  ncks -O --mk_rec_dmn TIME $filein $fileout

done

# Concatenate files

files=`ls tmp.MERRA300.prod.assim.tavgM_2d_slv_Nx*nc`

echo $files

ncrcat -O $files MERRA300.prod.assim.tavgM_2d_slv_Nx.201401-201403.nc

 

Note: You may receive the following warning message when running ‘ncrcat’;  simply ignore it as all input files have the same packing attributes.

ncrcat: INFO/WARNING Multi-file concatenator encountered packing attribute scale_factor for variable SLP. NCO copies the packing attributes from the first file to the output file. The packing attributes from the remaining files must match exactly those in the first file or the data from the subsequent files will not unpack correctly. Be sure all input files share the same packing attributes. If in doubt, unpack (with ncpdq -U) the input files, then concatenate them, then pack the result (with ncpdq). This message is printed only once per invocation.

 

  • View the time dimension and time variable of the concatenated file.

The length of the record dimension of the output file is the sum of the lengths of the input files.  

ncdump -v time MERRA300.prod.assim.tavgM_2d_slv_Nx.201401-201403.nc

dimensions: TIME = UNLIMITED ; // (3 currently) <= record dimension YDim = 361 ; XDim = 540 ; ….. double TIME(TIME) ; <= record variable TIME:long_name = "time" ; TIME:units = "minutes since 2014-01-01 00:30:00" ; TIME:time_increment = 60000 ; TIME:begin_date = 20140101 ; TIME:begin_time = 3000 ; TIME:fullpath = "TIME:EOSGRID" ; …… data: TIME = 0, 44640, 84960 ; <= TIME:units = "minutes since 2014-01-01 00:30:00" ;

 

4) Making time aggregation for data in Example-2: files without time dimension variable:

  • View dimension information:

The SeaWiFS aerosol monthly data files in Example-2 do not have a time dimension. For example,

ncdump -h DeepBlue-SeaWiFS-1.0_L3_20090101_v004-20130604T022047Z.h5.nc

dimensions: latitude = 180 ; longitude = 360 ; …..

 

  • Use the command ‘ncecat’ to concatenate files.  A sample Korn shell script is below:

#!/usr/bin/ksh

# Sample script to concatenate SWDB daily files in netCDF

# by using NCO commands: ncecat

flist='DeepBlue-SeaWiFS-1.0_L3_20090101_v004-20130604T022047Z.h5.nc DeepBlue-SeaWiFS-1.0_L3_20090102_v004-20130604T022057Z.h5.nc DeepBlue-SeaWiFS-1.0_L3_20090103_v004-20130604T022106Z.h5.nc'

# Concatenating files

ncecat -O -u time $flist DeepBlue-SeaWiFS-1.0_L3_20090101-20090103.nc

 

  • Add the time variable into the concatenated file
  • Create a CDL (network Common data form Description Language) file containing data for the time variable. Save the following as ‘time_agg.cdl’.
netcdf time_agg { dimensions: time = 3 ; variables: long time(time) ; time:long_name = "time" ; time:units = "days since 2009-01-01" ; data: time = 1, 2, 3 ; }

 

  • Use the netCDF command ‘ncgen’ to create a netCDF file ‘time_agg.nc’ for the time variable: 

ncgen -b -o time_agg.nc time_agg.cdl

  • Add variable “time” to the concatenated file with NCO command ‘ncks’:

 ncks -A time_agg.nc DeepBlue-SeaWiFS-1.0_L3_20090101-20090103.nc

  • View time dimension and time variable of the concatenated file:

ncdump –v time DeepBlue-SeaWiFS-1.0_L3_20090101-20090103.nc

dimensions: time = UNLIMITED ; // (3 currently) <= record dimension latitude = 180 ; longitude = 360 ; ….. long time(time) ; <= record variable time:long_name = "time" ; time:units = "days since 2009-01-01" ; …… data: time = 1, 2, 3 ; <= time:units = "days since 2009-01-01" ; }
Additional Info:

In the concatenated file created with ‘ncrcat’, the time values are calculated by using the original units. In Example-1, the unit is "minutes since 2014-01-01 00:30:00",   however, the local attribute in the time variable is simply copied. 

评论 (0 个评论)

facelist doodle 涂鸦板

您需要登录后才可以评论 登录 | 立即注册

Copyright ©2011-2014 bbs.06climate.com All Rights Reserved.  Powered by Discuz! (京ICP-10201084)

本站信息均由会员发表,不代表气象家园立场,禁止在本站发表与国家法律相抵触言论

返回顶部