This data recipe shows how to create a single file in netCDF format by concatenating the time dimension of multiple files with NCO commands.
Two sets of data are used in this recipe. One set of example data files are from the MERRA monthly product, containing a time dimension. The other data sets are from the SeaWiFS aerosol daily product, which does not contain a time dimension.
This recipe works for gridded data only.
Time to complete the following procedures: 30 minutes
Tested with Software: NCO version "4.4.4" ; “ 4.3.7”
1. Installing NCO:
If NCO (netCDF Operator) is available on your system, skip this step.
The software can be downloaded for free from "Bienvenue sur le netCDF Operator (NCO) site" (nco.sourceforge.net) . The NCO comprises a dozen standalone, command-line programs that take netCDF files as input.
2. Getting data in netCDF format:
Before accessing data at GES DISC, a user must first register with Earthdata Login, then be authorized to access data at GES DISC by following steps at: data-access.
The GES DISC provides a number ways to download data in netCDF format . In this example, we use the Simple Subset Wizard (SSW). For more information regarding data access through SSW, please read the data recipe How to Obtain a Spatio-temporal + Variable Subset of Data with the Simple Subset Wizard .
Example-1: Files with time dimension variable: (MERRA monthly data)
Data Set Keyword(s): MATMNXSLV (This is the shortname of the product.)
Date Range: 2014-01-01, 2014-3-31
Spatial Bounding Box: Leave blank (for global)
Variables: Select two variables: Temperature at 2 m above the displacement height, Sea level pressure;
Then, select the download data format as netCDF.
By following the “Downloading Instructions” from the SSW data download page, you may download all of the data files in netCDF format.
In this example, three MERRA monthly files are downloaded: for example, Jan 2014:
MERRA300.prod.assim.tavgM_2d_slv_Nx.201401.hdf.nc
Example-2: Files without time dimension variable: (SeaWiFS aerosol daily data)
Data Set Keyword(s): SWDB_L310
Date Range: 2009-01-01, 2009-01-05
Spatial Bounding Box: Global
Variables: Aerosol optical thickness estimated at 550 nm over land and ocean
Similar to Example-1, select download data format as netCDF, following the instruction from the SSW data download page.
In this example, five SeaWiFS aerosol daily files are downloaded: for example, Jan 2009:
DeepBlue-SeaWiFS-1.0_L3_20090101_v004-20130604T022047Z.h5.nc
3) Making time aggregation for data in Example-1: files with a time dimension variable:
The MERRA data file in Example 1 has a time dimension. To view the time dimension information, use the netCDF command ‘ncdump’, with option -h. For example,
ncdump –h MERRA300.prod.assim.tavgM_2d_slv_Nx.201401.hdf.nc
dimensions: TIME = 1 ; <= Original time dimension YDim = 361 ; XDim = 540 ; |
ncks -O --mk_rec_dmn TIME file_in.nc file_out.nc (making TIME a record dimension)
ncrcat -O file_in1.nc file_in2.nc file_out.nc (concatenating files)
Note: The command ‘ncrcat’ requires that the time dimension be a record dimension. The time dimension in the example data files is not a record dimension. So, in this example, the record dimension is made by the command ‘ncks’ with option ‘--mk_rec_dmn ‘. Please read the NCO User Guide for more information.
Sample Korn shell script for Example 1 data:
#!/usr/bin/ksh # Sample script to concatenate MERRA monthly data files in netCDF flist='MERRA300.prod.assim.tavgM_2d_slv_Nx.201401.hdf.nc MERRA300.prod.assim.tavgM_2d_slv_Nx.201402.hdf.nc MERRA300.prod.assim.tavgM_2d_slv_Nx.201403.hdf.nc' # Make Time a record dimension for f in $flist do filein=$f fileout='tmp.'$f ncks -O --mk_rec_dmn TIME $filein $fileout done # Concatenate files files=`ls tmp.MERRA300.prod.assim.tavgM_2d_slv_Nx*nc` echo $files ncrcat -O $files MERRA300.prod.assim.tavgM_2d_slv_Nx.201401-201403.nc |
Note: You may receive the following warning message when running ‘ncrcat’; simply ignore it as all input files have the same packing attributes.
ncrcat: INFO/WARNING Multi-file concatenator encountered packing attribute scale_factor for variable SLP. NCO copies the packing attributes from the first file to the output file. The packing attributes from the remaining files must match exactly those in the first file or the data from the subsequent files will not unpack correctly. Be sure all input files share the same packing attributes. If in doubt, unpack (with ncpdq -U) the input files, then concatenate them, then pack the result (with ncpdq). This message is printed only once per invocation.
The length of the record dimension of the output file is the sum of the lengths of the input files.
ncdump -v time MERRA300.prod.assim.tavgM_2d_slv_Nx.201401-201403.nc
dimensions: TIME = UNLIMITED ; // (3 currently) <= record dimension YDim = 361 ; XDim = 540 ; ….. double TIME(TIME) ; <= record variable TIME:long_name = "time" ; TIME:units = "minutes since 2014-01-01 00:30:00" ; TIME:time_increment = 60000 ; TIME:begin_date = 20140101 ; TIME:begin_time = 3000 ; TIME:fullpath = "TIME:EOSGRID" ; …… data: TIME = 0, 44640, 84960 ; <= TIME:units = "minutes since 2014-01-01 00:30:00" ; |
4) Making time aggregation for data in Example-2: files without time dimension variable:
The SeaWiFS aerosol monthly data files in Example-2 do not have a time dimension. For example,
ncdump -h DeepBlue-SeaWiFS-1.0_L3_20090101_v004-20130604T022047Z.h5.nc
dimensions: latitude = 180 ; longitude = 360 ; ….. |
#!/usr/bin/ksh # Sample script to concatenate SWDB daily files in netCDF # by using NCO commands: ncecat flist='DeepBlue-SeaWiFS-1.0_L3_20090101_v004-20130604T022047Z.h5.nc DeepBlue-SeaWiFS-1.0_L3_20090102_v004-20130604T022057Z.h5.nc DeepBlue-SeaWiFS-1.0_L3_20090103_v004-20130604T022106Z.h5.nc' # Concatenating files ncecat -O -u time $flist DeepBlue-SeaWiFS-1.0_L3_20090101-20090103.nc |
netcdf time_agg { dimensions: time = 3 ; variables: long time(time) ; time:long_name = "time" ; time:units = "days since 2009-01-01" ; data: time = 1, 2, 3 ; } |
ncgen -b -o time_agg.nc time_agg.cdl
ncks -A time_agg.nc DeepBlue-SeaWiFS-1.0_L3_20090101-20090103.nc
ncdump –v time DeepBlue-SeaWiFS-1.0_L3_20090101-20090103.nc
dimensions: time = UNLIMITED ; // (3 currently) <= record dimension latitude = 180 ; longitude = 360 ; ….. long time(time) ; <= record variable time:long_name = "time" ; time:units = "days since 2009-01-01" ; …… data: time = 1, 2, 3 ; <= time:units = "days since 2009-01-01" ; } |
In the concatenated file created with ‘ncrcat’, the time values are calculated by using the original units. In Example-1, the unit is "minutes since 2014-01-01 00:30:00", however, the local attribute in the time variable is simply copied.
Copyright ©2011-2014 bbs.06climate.com All Rights Reserved. Powered by Discuz! (京ICP-10201084)
本站信息均由会员发表,不代表气象家园立场,禁止在本站发表与国家法律相抵触言论