目 录
BIG
DATA
CONTENTS
基础篇
第1章 大数据概述 ··································································003
1.1 数据和大数据 ·········································································003
1.1.1 数据的高速增长 ·······························································003
1.1.2 大数据 ···········································································004
1.1.3 科学的范式 ·····································································006
1.2 大数据从哪里来 ······································································007
1.3 大数据的应用场景 ···································································008
1.4 大数据对思维方式的影响 ··························································010
1.5 数据挖掘与机器学习 ································································011
1.6 数据科学项目的基本流程 ··························································012
1.7 数据安全和大数据伦理 ·····························································013
1.7.1 数据安全 ········································································013
1.7.2 大数据伦理 ·····································································015
1.8 国家层面的大数据问题 ·····························································016
1.8.1 数据主权 ········································································016
1.8.2 大数据与国家治理 ····························································017
1.8.3 大数据重塑世界新格局 ······················································018
1.8.4 中国国家大数据战略 ·························································019
1.9 云计算 ··················································································020
1.9.1 云计算的特征 ··································································022
1.9.2 云计算的典型服务模式 ······················································022
1.9.3 云计算服务部署的环境 ······················································023
BIG
DATA
大数据应用基础教程
IV
1.9.4 云计算和大数据的关系 ······················································023
1.10 物联网 ·················································································023
1.11 数字经济 ··············································································025
1.11.1 大数据与数字经济 ··························································026
1.11.2 进一步推动我国数字经济发展 ···········································029
本章小结 ·······················································································030
习题 ·····························································································032
第2章 Python及常用类库 ························································033
2.1 Python简介 ···········································································033
2.1.1 Python的诞生 ·································································033
2.1.2 Python社区 ····································································034
2.1.3 Python的版本 ·································································034
2.1.4 使用Python进行数据分析的原因 ······································036
2.2 Python的安装与运行 ·······························································037
2.2.1 Anaconda简介及安装 ························································037
2.2.2 Python的运行 ·································································041
2.2.3 小结 ··············································································046
2.3 Python语言基础 ·····································································046
2.3.1 数据结构 ········································································046
2.3.2 代码结构 ········································································058
2.3.3 小结 ··············································································069
2.4 Python数据分析的常用类库 ······················································069
2.4.1 NumPy简介 ····································································069
2.4.2 pandas简介 ·····································································076
2.4.3 小结 ··············································································095
本章小结 ·······················································································095
习题 ·····························································································096
数据分析篇
第3章 数据获取 ·····································································101
3.1 数据来源 ···············································································101
3.2 网络数据爬取 ·········································································103
BIG
DATA
目
录
V
3.2.1 网络爬虫概述 ··································································103
3.2.2 网页访问的基础知识 ·························································104
3.2.3 网页数据爬取 ··································································109
3.2.4 网页内容解析 ··································································111
3.2.5 常见的“爬取与反爬”攻防策略 ··········································115
3.3 网络数据采集器 ······································································118
3.3.1 常见采集器 ·····································································118
3.3.2 八爪鱼采集案例 ·······························································118
3.4 使用Selenium获取数据 ···························································122
3.4.1 安装Selenium ··································································122
3.4.2 使用Selenium获取页面元素 ···············································124
3.4.3 Selenium应用：链家二手房数据获取 ····································126
本章小结 ·······················································································130
习题 ·····························································································130
第4章 数据存储 ·····································································131
4.1 文件 ·····················································································131
4.2 传统数据库技术 ······································································133
4.2.1 数据库管理系统 ·······························································133
4.2.2 数据库的概念模型 ····························································134
4.2.3 关系型数据库 ··································································135
4.2.4 结构化查询语言SQL ························································136
4.2.5 MySQL数据库管理 ··························································137
4.2.6 基于MySQL monitor的基本数据库操作 ································141
4.2.7 基于HeidiSQL的基本数据库操作 ········································145
4.3 NoSQL数据库 ········································································148
4.3.1 NoSQL的发展背景 ···························································148
4.3.2 NoSQL数据库的类型 ························································149
本章小结 ·······················································································152
习题 ·····························································································152
第5章 数据预处理 ··································································153
5.1 数据质量问题 ·········································································153
5.1.1 现实世界的“脏”数据 ······················································153
5.1.2 数据质量问题的产生原因 ···················································155
BIG
DATA
大数据应用基础教程
5.1.3 数据质量审核 ··································································156
5.2 数据预处理技术 ······································································158
5.2.1 数据清洗 ········································································158
5.2.2 数据集成 ········································································159
5.2.3 数据变换 ········································································160
5.2.4 数据归约 ········································································161
5.3 预处理案例 ············································································162
本章小结 ·······················································································166
习题 ·····························································································166
第6章 数据可视化 ··································································167
6.1 数据可视化概述 ······································································167
6.1.1 什么是数据可视化 ····························································167
6.1.2 常用的数据可视化工具 ······················································168
6.1.3 Python可视化工具库 ························································169
6.2 Matplotlib数据可视化 ·······························································170
6.2.1 Matplotlib绘图基础 ··························································170
6.2.2 Matplotlib常用绘图 ··························································172
6.2.3 使用mplot3d绘制3D图形 ·················································180
6.3 pandas数据可视化 ··································································185
6.3.1 pandas绘图基础 ·······························································185
6.3.2 pandas常用绘图 ·······························································186
6.4 seaborn数据可视化 ·································································191
6.4.1 seaborn绘图基础 ······························································191
6.4.2 seaborn常用绘图 ······························································197
6.5 pyecharts数据可视化 ·······························································201
6.5.1 pyecharts绘图基础 ···························································201
6.5.2 pyecharts常用绘图 ···························································201
本章小结 ·······················································································208
习题 ·····························································································208
第7章 数据分析方法 ·······························································211
7.1 数据分析方法的数学基础 ··························································211
7.1.1 理解复合函数求导 ····························································211
7.1.2 理解多元函数偏导 ····························································212
BIG
DATA
目
录
7.1.3 理解最小二乘法 ·······························································212
7.1.4 理解梯度 ········································································213
7.1.5 理解概率 ········································································213
7.1.6 理解条件概率 ··································································214
7.1.7 理解贝叶斯公式 ·······························································214
7.2 回归 ·····················································································215
7.2.1 回归的基本概念及方法 ······················································215
7.2.2 回归预测的性能度量 ·························································217
7.2.3 线性回归 ········································································218
7.3 分类 ·····················································································227
7.3.1 分类的基本方法 ·······························································227
7.3.2 分类任务的性能度量 ·························································228
7.3.3 逻辑回归 ········································································229
7.3.4 支持向量机 ·····································································240
7.3.5 决策树理论 ·····································································254
7.3.6 朴素贝叶斯 ·····································································258
7.3.7 k-近邻（k-NN）算法 ·························································262
7.4 聚类 ·····················································································266
7.4.1 聚类算法 ········································································266
7.4.2 K-means聚类算法 ····························································267
7.4.3 K-means聚类案例 ····························································268
7.5 文本分析 ···············································································276
7.5.1 文本分析的基本步骤 ·························································277
7.5.2 文本分析的基本概念 ·························································277
7.5.3 文本分析案例 ··································································278
本章小结 ·······················································································286
习题 ·····························································································286
大数据平台篇
第8章 Linux操作系统基础 ······················································289
8.1 Linux操作系统简介··································································289
8.1.1 操作系统 ········································································289
8.1.2 Linux操作系统 ································································290
BIG
DATA
大数据应用基础教程
8.1.3 大数据平台基于Linux操作系统的原因 ·································293
8.2 Linux基本命令········································································293
8.2.1 目录与文件操作命令 ·························································293
8.2.2 文本过滤与处理 ·······························································298
8.2.3 Shell输入输出命令 ···························································300
8.2.4 进程管理命令 ··································································301
8.2.5 日常操作命令 ··································································303
本章小结 ·······················································································306
习题 ·····························································································306
第9章 大数据管理平台 ····························································307
9.1 应用场景 ···············································································307
9.2 发展历程 ···············································································309
9.3 技术体系 ···············································································311
9.3.1 数据收集层 ·····································································312
9.3.2 数据存储层 ·····································································313
9.3.3 资源管理层 ·····································································315
9.3.4 计算引擎层 ·····································································315
9.3.5 数据分析层 ·····································································317
9.3.6 数据可视化层 ··································································317
9.3.7 大数据管理平台技术栈 ······················································318
本章小结 ·······················································································319
习题 ·····························································································319
第10章 分布式存储 ································································321
10.1 HDFS介绍 ···········································································321
10.2 HDFS基本架构 ·····································································323
10.3 HDFS Shell访问 ···································································325
本章小结 ·······················································································328
习题 ·····························································································328
第11章 分布式处理 ································································329
11.1 分布式计算思想 ·····································································329
11.2 MapReduce ··········································································333
BIG
DATA
目
录
11.2.1 MapReduce介绍 ·····························································333
11.2.2 MapReduce编程模型 ·······················································334
11.2.3 MapReduce程序案例 ·······················································335
11.3 Spark ··················································································341
11.3.1 Spark介绍 ·····································································341
11.3.2 Spark编程模型 ·······························································342
11.3.3 Spark程序案例 ·······························································345
11.4 Spark相对于Hadoop的优势 ···················································352
本章小结 ·······················································································353
习题 ·····························································································353
参考文献 ·················································································355
附录A 基于虚拟机的Linux系统安装 ··········································359
A.1 虚拟机技术概述 ······································································359
A.2 虚拟机托管软件安装 ································································360
A.3 虚拟机Linux安装 ···································································362
附录B Hadoop及Spark安装 ····················································371
B.1 集群基础配置 ·········································································371
B.2 Hadoop安装 ··········································································375
B.3 Spark安装·············································································380
