标签：10000 image jpg savePath NASA Mars images 火星

## 前言 最近，我使用爬虫技术，爬取了美国航空航天局，也就是你电影里经常见到的 `NASA`, 火星探索的相关图片，有 `10000` 张吧。 嗯嗯，小事情，小事情。 完事儿之后，有点小激动，于是就有了这篇文章，将有以下内容： 1. 我为什么要爬取NASA的图片 2. 我是如何爬取NASA图片的（超详细） 3. 我得到了什么(高清大图) 3. 我发现了什么秘密（超劲爆） ## 我为什么要爬NASA的图片 我已经超过35了，瑟瑟发抖啥时候被开了。 天天想着万一哪天失业了干点啥，想着玩个自媒体吧，天天给大家瞎白话。白话点历史谜团，宇宙奥秘啥的，于是我就盯上了 `NASA`。 `NASA` 有各种宇宙探索任务，并且有相关的文章，访谈，图片，视频公开，这是不可多得的资源库。 ## 我是如何爬取NASA图片的（超详细） `NASA` 的网站是可以公开访问的，地址是 ```url https://www.nasa.gov/ ```  打开以后，它首页长这样，可以看到各种内容。右上角还有个搜索框，我们输入 `Mars` 也就是 `火星` 稍等片刻，会展示出与 `Mars` 相关的各种内容，其中有一项 `Mars Exploration` 也就是 `火星探索` 点开之后，就到了一个新页面，然后找到 `Images` 图片，就到了我们爬取的目标页 ```url https://www.nasa.gov/mission_pages/mars/images/index.html ```  页面下拉，你将会看到有个大大的按钮，上面写着 `MORE IMAGES`， 点击试试你就发现： 页面的内容，并不是页面直接加载的，而是通过 `api` 请求后，异步渲染的 F12, 打开浏览器开发者模式,重新执行刚才的步骤，观察请求信息，发现都会有如下的情况  看上去这个`url`地址非常重要，我们先看他的请求地址： ``` https://www.nasa.gov/api/2/ubernode/_search?size=24&from=24&sort=promo-date-time%3Adesc&q=((ubernode-type%3Aimage)%20AND%20(topics%3A3152))&_source_include=promo-date-time%2Cmaster-image%2Cnid%2Ctitle%2Ctopics%2Cmissions%2Ccollections%2Cother-tags%2Cubernode-type%2Cprimary-tag%2Csecondary-tag%2Ccardfeed-title%2Ctype%2Ccollection-asset-link%2Clink-or-attachment%2Cpr-leader-sentence%2Cimage-feature-caption%2Cattachments%2Curi ``` 注意看里面的参数 ``` size=24&from=24 ``` 很显然，`size` 就是每次请求图片的数量，`from` 经过试验，是查询初始位置，我们可以改它来获取其他内容 我们再来看看它的返回信息： ```json { "took": 3, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 659, "max_score": null, "hits": [{ "_index": "nasa-public", "_type": "ubernode", "_id": "450040", "_score": null, "_source": { "image-feature-caption": "Mars 2020 rover underwent an eye exam after several cameras were installed on the rover. ", "topics": ["3140", "3152"], "nid": "450040", "title": "NASA 'Optometrists' Verify Mars 2020 Rover's 20/20 Vision", "type": "ubernode", "uri": "/image-feature/jpl/nasa-optometrists-verify-mars-2020-rovers-2020-vision", "collections": ["4525", "5246"], "link-or-attachment": "link", "missions": ["6336"], "primary-tag": "6336", "cardfeed-title": "NASA 'Optometrists' Verify Mars 2020 Rover's 20/20 Vision", "promo-date-time": "2019-08-05T17:49:00-04:00", "secondary-tag": "3140", "master-image": { "fid": "603128", "alt": "Engineers test cameras on the top of the Mars 2020 rover’s mast and front chassis. ", "width": "1600", "id": "603128", "title": "Engineers test cameras on the top of the Mars 2020 rover’s mast and front chassis. ", "uri": "public://thumbnails/image/pia23314-16.jpg", "height": "900" }, "ubernode-type": "image" }, "sort": [1565041740000] }, { "_index": "nasa-public", "_type": "ubernode", "_id": "433172", "_score": null, "_source": { "image-feature-caption": "NASA still hasn't heard from the Opportunity rover, but at least we can see it again.", "topics": ["3152"], "nid": "433172", "title": "Opportunity Emerges in a Dusty Picture", "type": "ubernode", "uri": "/image-feature/opportunity-emerges-in-a-dusty-picture", "collections": ["7628"], "link-or-attachment": "link", "missions": ["3639"], "primary-tag": "3152", "cardfeed-title": "Opportunity Emerges in a Dusty Picture", "promo-date-time": "2018-09-26T12:39:00-04:00", "secondary-tag": "7628", "master-image": { "fid": "584263", "alt": "NASA's Opportunity rover appears as a blip in the center of this square", "width": "1400", "id": "584263", "title": "NASA's Opportunity rover appears as a blip in the center of this square", "uri": "public://thumbnails/image/pia22549-16.jpg", "height": "788" }, "ubernode-type": "image" }, "sort": [1537979940000] }] } } ``` 上面的 `json` 内容太长，我删除了一些重复的，实际上 `hits` 这个数组，也是24个，和页面上显示的图片数量是一样的。那基本可以断定，页面上的信息，就是从这个数组来的。 进而对比发现，`master-image`字段下，就是我们需要的信息，包括 `图片地址`, `图片尺寸`, `图片标题`。 下面就上代码，组装请求URL, 获取内容，下载图片 三步走 我使用了 `Dart` 语言，你们随意 ```dart import 'dart:convert'; import 'package:dio/dio.dart'; main() async { // 每页数量是固定24个,改变初始值即可 for (int from = 0; from < 24 * 100; from = from + 24) { await getPage(from); } } //获取每一页的信息并且下载 Future getPage(int from) async { String url = 'https://www.nasa.gov/api/2/ubernode/_search?size=24&from=' + from.toString() + '&sort=promo-date-time%3Adesc&q=((ubernode-type%3Aimage)%20AND%20(topics%3A3152))&_source_include=promo-date-time%2Cmaster-image%2Cnid%2Ctitle%2Ctopics%2Cmissions%2Ccollections%2Cother-tags%2Cubernode-type%2Cprimary-tag%2Csecondary-tag%2Ccardfeed-title%2Ctype%2Ccollection-asset-link%2Clink-or-attachment%2Cpr-leader-sentence%2Cimage-feature-caption%2Cattachments%2Curi'; //获取到内容 var res = await Dio().get(url); var map = jsonDecode(res.toString()); (map['hits']['hits'] as List).forEach((element) async { Uri fileUri = Uri.parse(getUri(element)); String savePath = getSavePath(element); await Dio().downloadUri(fileUri, savePath); print('已下载: ' + savePath); }); } //获取图片下载地址 String getUri(dynamic element) { String uri = element['_source']['master-image']['uri'].toString(); uri = uri.replaceAll('public://', 'https://www.nasa.gov/sites/default/files/styles/full_width_feature/public/'); return uri; } //处理信息，并且返回图片保存地址 String getSavePath(dynamic element) { String id = element['_id']; String fid = element['_source']['master-image']['fid'].toString(); String title = element['_source']['master-image']['title'].toString(); String uri = element['_source']['master-image']['uri'].toString(); String savePath = id + '_' + fid + '_' + title.trim() + '.' + uri.split('.').last; savePath = savePath.replaceAll('/', ''); savePath = savePath.replaceAll('\\', ''); savePath = savePath.replaceAll('"', ''); savePath = 'images/' + savePath; return savePath; } ``` 上面的代码，还是很简单的，有经验的同学应该一看就懂。 走起来吧。 ``` 已下载: images/470436_643588_This is the third color image taken by NASA’s Ingenuity helicopter.jpg 已下载: images/470435_643587_This is the second color image taken by NASA’s Ingenuity helicopter.jpg 已下载: images/468546_639327_This is the first high-resolution, color image to be sent back by the Hazard Cameras (Hazcams).jpg 已下载: images/452007_605784_Danielson Crater on Mars.jpg 已下载: images/458478_615132_Gullies on Mars.jpg 已下载: images/469416_641582_A field of sand dunes occupies this frosty 5-kilometer diameter crater in the high-latitudes of the northern plains of Mars..jpeg 已下载: images/458075_614251_Mars 2020 With Sample Tubes (Artist's Concept).jpg 已下载: images/470381_643473_CME.jpg 已下载: images/458813_615896_Mars.jpg 已下载: images/467026_635309_Illustration of NASA’s Perseverance rover begins its descent through the Martian atmosphere.jpg 已下载: images/470438_643591_This black and white image was taken by NASA’s Ingenuity helicopter during its third flight on April 25, 2021.jpg 已下载: images/465488_631398_Cliffs in Ancient Ice on Mars.jpg 已下载: images/463659_626874_Avalanche on Mars.jpg 已下载: images/470251_643164_This image from NASA’s Perseverance rover shows the agency’s Ingenuity Mars Helicopter right after it successfully completed a high-speed spin-up test..jpeg 已下载: images/468636_639726_Mars' Jezero Crater.jpg ``` ## 我得到了什么 这些图片  还有这些  图片，图片标题都有了, 够看一个月了我估计。 ## 我发现了什么秘密  这张图，我最喜欢了。一个如此清晰，一个如此浑浊，这是为什么？ 火星人的裂缝产生器？ 好吧，真正的秘密是： `NASA` 的网站竟然没有防采集，不信你也试试。。。

标签：10000,image,jpg,savePath,NASA,Mars,images,火星
来源： https://blog.51cto.com/u_12739514/2957861

本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享；
2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关；
3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关；
4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除；
5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。