使用python批量下载ERA-Interim(成功)

小莹子

使用ERA-Interim数据，然后每次只能下载一个月的数据，所以还是用批量下载方便一些。
按照http://bbs.06climate.com/forum.php?mod=viewthread&tid=24435帖子进行尝试。其中也结合了一些其他操作。

一、windows

https://www.cnblogs.com/lifeofershisui/p/8135702.html
python环境变量设置
import syssys.path.append('/home/ershisui/')
在windows上安装ecmwf-python-api,失败。
官方错误指南：https://software.ecmwf.int/wiki/ ... API+Troubleshooting

尝试了很多次，一直失败！！换linux

二、Linux

1.查看是否已经安装python
CentOS7默认安装了python2.7.5
python -v 命令查看是否安装Python
which python 查看位置
cd /usr/bin
ls python*
要安装python的其他版本先执行mv python python.bak命令备份

2.安装pip[size=13.3333px]——博客 余-雷 “CentOS7下安装python-pip”
[size=13.3333px]            [size=13.3333px]1）首先检查linux有没有安装python-pip包，直接执行 [size=13.3333px]yum install python-pip
[root@localhost syrus]#  yum install python-pip
bash: /usr/bin/yum: /usr/bin/python: 坏的解释器: 没有那个文件或目录
想起来之前把python备份成了python.bak，可能和这个有关，改回来之后运行。
[root@localhost bin]#  yum install python-pip
已加载插件：fastestmirror, langpacks
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
base                                                     | 3.6 kB     00:00     
extras                                                   | 3.4 kB     00:00     
updates                                                  | 3.4 kB     00:00     
(1/2): extras/7/x86_64/primary_db                          | 185 kB   00:00     
(2/2): updates/7/x86_64/primary_db                         | 6.9 MB   00:06     
Determining fastest mirrors
* base: mirrors.neusoft.edu.cn
* extras: mirrors.neusoft.edu.cn
* updates: mirrors.neusoft.edu.cn
没有可用软件包 python-pip。
错误：无须任何处理
2）没有python-pip包就执行命令 yum -y install epel-release
Running transaction
  正在安装    : epel-release-7-9.noarch                                     1/1
  验证中      : epel-release-7-9.noarch                                     1/1
已安装:
  epel-release.noarch 0:7-9                                                     
完毕！
3)执行成功后，再次执行yum install python-pip
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  正在安装    : python2-pip-8.1.2-6.el7.noarch                              1/1
  验证中      : python2-pip-8.1.2-6.el7.noarch                              1/1
已安装:
  python2-pip.noarch 0:8.1.2-6.el7                                             
完毕！
4）对安装好的pip进行升级 pip install --upgrade pip
Installing collected packages: pip
  Found existing installation: pip 8.1.2
    Uninstalling pip-8.1.2:
      Successfully uninstalled pip-8.1.2
Successfully installed pip-10.0.1

pip安装好了
3.秘钥设置
新建/home/.ecmwfapirc
4.下载ecmwf-api-client-python.tgz
sudo pip install https://software.ecmwf.int/wiki/ ... i-client-python.tgz
默认安装在/usr/lib/python2.7/site-packages
5.试跑
按照官网上的例子，试跑。
cd /usr/bin env python
ecmwfapi.api.APIKeyFetchError: [Errno 2] No such file or directory: '/root/.ecmwfapirc'
NameError: name 'server' is not defined
尝试在/root/下新建.ecmwfapirc
cd /root
vi .ecmwfapirc
:wq
E502: ".ecmwfapirc" is a directory
Press ENTER or type command to continue
cd /root
vim .ecmwfapirc
拷贝秘钥信息
E21: 不能修改，因为选项 'modifiable' 是关的
失败
sudo vi .ecmwfapirc
:q
E37: No write since last change (add ! to override)
就是用vi命令编辑文本文件，没有文件写入权限的时候会报这个错。：q :wq怎么都不能退出。
ctrl+z或者在退出命令后面加！
：q!

解决方法：将/home/.ecmefapirc复制到/root下面
cp .ecmwfapirc /root

ok!试跑成功
6.批量下载数据
在代码中用ID代表变量，下载文档ERA report series.pdf，后来发现在MARS中也能找到变量对应的ID。
#!/usr/bin/env python
from ecmwfapi import ECMWFDataServer
   
server = ECMWFDataServer()
   
server.retrieve({
    'stream'    : "mnth",
    'levtype'   : "sfc",
    'param'     : "165.128",
    'dataset'   : "interim",
    'step'      : "0",
    'grid'      : "0.5/0.5",
    'time'      : "00/06/12/18",
    'date'      : "2014-07-01/to/2014-07-31",
    'type'      : "an",
    'class'     : "ei",
    'area'      : "50/50/10/120",
    'format'    : "netcdf",
    'domain'    : "G",
    'resol'     : "auto",
    'target'    : "interim_2014-07-01to2014-07-31_00061218.nc"
})

OK!!!






月数据下载：
19790101/19790201/19790301/19790401/19790501/19790601/19790701/19790801/19790901/19791001/19791101/19791201/

19800101/19800201/19800301/19800401/19800501/19800601/19800701/19800801/19800901/19801001/19801101/19801201/

19810101/19810201/19810301/19810401/19810501/19810601/19810701/19810801/19810901/19811001/19811101/19811201/

19820101/19820201/19820301/19820401/19820501/19820601/19820701/19820801/19820901/19821001/19821101/19821201/

19830101/19830201/19830301/19830401/19830501/19830601/19830701/19830801/19830901/19831001/19831101/19831201/

19840101/19840201/19840301/19840401/19840501/19840601/19840701/19840801/19840901/19841001/19841101/19841201/

19850101/19850201/19850301/19850401/19850501/19850601/19850701/19850801/19850901/19851001/19851101/19851201/

19860101/19860201/19860301/19860401/19860501/19860601/19860701/19860801/19860901/19861001/19861101/19861201/

19870101/19870201/19870301/19870401/19870501/19870601/19870701/19870801/19870901/19871001/19871101/19871201/

19880101/19880201/19880301/19880401/19880501/19880601/19880701/19880801/19880901/19881001/19881101/19881201/

19890101/19890201/19890301/19890401/19890501/19890601/19890701/19890801/19890901/19891001/19891101/19891201/

19900101/19900201/19900301/19900401/19900501/19900601/19900701/19900801/19900901/19901001/19901101/19901201/

19910101/19910201/19910301/19910401/19910501/19910601/19910701/19910801/19910901/19911001/19911101/19911201/

19920101/19920201/19920301/19920401/19920501/19920601/19920701/19920801/19920901/19921001/19921101/19921201/

19930101/19930201/19930301/19930401/19930501/19930601/19930701/19930801/19930901/19931001/19931101/19931201/

19940101/19940201/19940301/19940401/19940501/19940601/19940701/19940801/19940901/19941001/19941101/19941201/

19950101/19950201/19950301/19950401/19950501/19950601/19950701/19950801/19950901/19951001/19951101/19951201/

19960101/19960201/19960301/19960401/19960501/19960601/19960701/19960801/19960901/19961001/19961101/19961201/

19970101/19970201/19970301/19970401/19970501/19970601/19970701/19970801/19970901/19971001/19971101/19971201/

19980101/19980201/19980301/19980401/19980501/19980601/19980701/19980801/19980901/19981001/19981101/19981201/

19990101/19990201/19990301/19990401/19990501/19990601/19990701/19990801/19990901/19991001/19991101/19991201/

20000101/20000201/20000301/20000401/20000501/20000601/20000701/20000801/20000901/20001001/20001101/20001201/

20010101/20010201/20010301/20010401/20010501/20010601/20010701/20010801/20010901/20011001/20011101/20011201/

20020101/20020201/20020301/20020401/20020501/20020601/20020701/20020801/20020901/20021001/20021101/20021201/

20030101/20030201/20030301/20030401/20030501/20030601/20030701/20030801/20030901/20031001/20031101/20031201/

20040101/20040201/20040301/20040401/20040501/20040601/20040701/20040801/20040901/20041001/20041101/20041201/

20050101/20050201/20050301/20050401/20050501/20050601/20050701/20050801/20050901/20051001/20051101/20051201/

20060101/20060201/20060301/20060401/20060501/20060601/20060701/20060801/20060901/20061001/20061101/20061201/

20070101/20070201/20070301/20070401/20070501/20070601/20070701/20070801/20070901/20071001/20071101/20071201/

20080101/20080201/20080301/20080401/20080501/20080601/20080701/20080801/20080901/20081001/20081101/20081201/

20090101/20090201/20090301/20090401/20090501/20090601/20090701/20090801/20090901/20091001/20091101/20091201/

20100101/20100201/20100301/20100401/20100501/20100601/20100701/20100801/20100901/20101001/20101101/20101201/

20110101/20110201/20110301/20110401/20110501/20110601/20110701/20110801/20110901/20111001/20111101/20111201/

20120101/20120201/20120301/20120401/20120501/20120601/20120701/20120801/20120901/20121001/20121101/20121201/

20130101/20130201/20130301/20130401/20130501/20130601/20130701/20130801/20130901/20131001/20131101/20131201/

20140101/20140201/20140301/20140401/20140501/20140601/20140701/20140801/20140901/20141001/20141101/20141201/

20150101/20150201/20150301/20150401/20150501/20150601/20150701/20150801/20150901/20151001/20151101/20151201/

20160101/20160201/20160301/20160401/20160501/20160601/20160701/20160801/20160901/20161001/20161101/20161201/

20170101/20170201/20170301/20170401/20170501/20170601/20170701/20170801/20170901/20171001/20171101/20171201/

20180101/20180201/20180301/20180401/20180501/20180601/20180701/20180801/20180901/20181001/20181101/20181201/

独一吖

给你鼓掌

guihua8000

谢谢楼主分享

小莹子

guihua8000 发表于 2018-11-24 08:06
谢谢楼主分享

谢谢支持。。。。。。

yesyesnotion

请问这句是什么意思？可以解释下吗  'param'     : "165.128",

小莹子

yesyesnotion 发表于 2018-11-28 10:18
请问这句是什么意思？可以解释下吗  'param'     : "165.128",

这个是参数的编号吧，我是这样理解的。一般你在官网上下面的“View the MARS request”中可以查看，或者是数据说明文件中就有相关介绍

Zhangzh

您好，我批量下载是数据量小的话不报错，正常下载，数据量大的话就报错，您知道这是咋回事，是不是网站又反爬的机制，具体错误提示如下：

Traceback (most recent call last):
  File "F:\Zhangzh\ERAdata_update\ecmwf-api-client-python\ecmwf-api-client-python\ecmwf_quvairh.py", line 21, in <module>
    'target': "F:/Zhangzh/ERAdata_update/newERA/"+str(year)+"quv.nc"
  File "F:\Zhangzh\ERAdata_update\ecmwf-api-client-python\ecmwf-api-client-python\ecmwfapi\api.py", line 414, in retrieve
    c.execute(req, target)
  File "F:\Zhangzh\ERAdata_update\ecmwf-api-client-python\ecmwf-api-client-python\ecmwfapi\api.py", line 377, in execute
    size = self._transfer(result["href"], target, result["size"])
  File "F:\Zhangzh\ERAdata_update\ecmwf-api-client-python\ecmwf-api-client-python\ecmwfapi\api.py", line 76, in wrapped
    return func(*args,**kwargs)
  File "F:\Zhangzh\ERAdata_update\ecmwf-api-client-python\ecmwf-api-client-python\ecmwfapi\api.py", line 330, in _transfer
    chunk = http.read(block)
  File "D:\Python27\ArcGIS10.3\lib\socket.py", line 380, in read
    data = self._sock.recv(left)
  File "D:\Python27\ArcGIS10.3\lib\httplib.py", line 567, in read
    s = self.fp.read(amt)
  File "D:\Python27\ArcGIS10.3\lib\socket.py", line 380, in read
    data = self._sock.recv(left)
  File "D:\Python27\ArcGIS10.3\lib\ssl.py", line 246, in recv
    return self.read(buflen)
  File "D:\Python27\ArcGIS10.3\lib\ssl.py", line 165, in read
    return self._sslobj.read(len)
error: [Errno 10060]

小莹子

Zhangzh 发表于 2018-12-27 09:22
您好，我批量下载是数据量小的话不报错，正常下载，数据量大的话就报错，您知道这是咋回事，是不是网站又反 ...

我有在Linux CentOS7连着下37年的月数据，一般情况下出错，我都会先检查代码。有一次是下载的moda数据，然后忘了些step=3。你用的Windows我也不清楚你错在哪了，你先自检，好运。

Zhangzh

好的   谢谢您的回复   我再研究研究

小莹子

独一吖 发表于 2018-11-23 22:30
给你鼓掌

,谢谢支持，