Python 数据可视化实战：环境搭建与数据读取入门

常用命令：

• mkworkon ENV: 激活之前创建的 ENV
• deactivate: 脱离当前的虚拟环境

3. 其他平台安装

Mac OS X: 建议使用 Homebrew 安装 Python 和依赖项。

$ brew install python
$ pip install numpy scipy matplotlib

Windows: 建议安装预打包的 Python 科学发行版，如 Anaconda 或 Enthought Python Distribution (EPD)。

4. 安装图像处理库 (PIL/Pillow)

Python 图像库 (PIL) 用于图像处理。现在推荐使用其分支 Pillow。

$ pip install Pillow

5. 安装 Requests 模块

大多数数据可以通过 HTTP 获取，Requests 库使这项工作变得简单。

$ pip install requests

示例：

import requests
r = requests.get('https://api.github.com/timeline')
print(r.content)

6. 自定义 Matplotlib 参数

Matplotlib 配置可以从配置文件 .rc 文件中读取，也可以在代码中修改。

方法一：使用 rcParams

import matplotlib as mpl
mpl.rcParams['lines.linewidth'] = 2
mpl.rcParams['lines.color'] = 'r'

方法二：调用 rc() 命令

import matplotlib as mpl
mpl.rc('lines', linewidth=2, color='r')

若要恢复默认设置，可调用 matplotlib.rcdefaults()。

二、了解你的数据

本节介绍从各种格式导入和导出数据的基本知识，以及清理数据的方法。

1. 从 CSV 导入数据

Python 有 csv 模块支持读写 CSV 文件。

import csv
filename = 'ch02-data.csv'
data = []
try:
    with open(filename) as f:
        reader = csv.reader(f)
        header = next(reader)
        data = [row for row in reader]
except csv.Error as e:
    print("Error reading CSV file at line %s: %s" % (reader.line_num, e))
    import sys
    sys.exit(-1)

if header:
    print(header)
for datarow in data:
    print(dat ar ow)

对于大文件，可以使用 NumPy 的 loadtxt()：

import numpy
# 注意：Python 3 中 print 是函数
print(numpy.loadtxt('ch02-data.csv', dtype='string', delimiter=','))

2. 从 Excel 文件导入数据

使用 xlrd 模块读取 Excel 文件。

$ pip install xlrd

import xlrd
file = 'ch02-xlsxdata.xlsx'
wb = xlrd.open_workbook(filename=file)
ws = wb.sheet_by_name('Sheet1')
dataset = []
for r in range(ws.nrows):
    col = []
    for c in range(ws.ncols):
        col.append(ws.cell(r, c).value)
    dataset.append(col)

from pprint import pprint
pprint(dataset)

3. 从固定宽度数据文件导入

使用 struct 模块解析固定宽度数据。

import struct
import string
datafile = 'ch02-fixed-width.data'
mask = '9s14s5s'
with open(datafile, 'r') as f:
    for line in f:
        fields = struct.Struct(mask).unpack_from(line)
        print('fields:', [field.strip() for field in fields])

4. 从 JSON 资源导入数据

import requests
url = 'https://github.com/timeline.json'
r = requests.get(url)
json_obj = r.json()
repos = set()
for entry in json_obj:
    try:
        repos.add(entry['repository']['url'])
    except KeyError as e:
        print("No key %s. Skipping..." % e)

5. 将数据导出到 JSON、CSV、Excel

import csv
import json
import xlwt

def write_csv(data):
    import io
    f = io.StringIO()
    writer = csv.writer(f)
    for row in data:
        writer.writerow(row)
    return f.getvalue()

def write_json(data):
    return json.dumps(data)

def write_xlsx(data):
    book = xlwt.Workbook()
    sheet1 = book.add_sheet("Sheet 1")
    row = 0
    for line in data:
        col = 0
        for datum in line:
            sheet1.write(row, col, datum)
            col += 1
        row += 1
    f = io.StringIO()
    book.save(f)
    return f.getvalue()

6. 从数据库导入数据

使用 sqlite3 连接 SQLite 数据库。

import sqlite3
con = sqlite3.connect('world.db')
with con:
    cur = con.cursor()
    query = 'SELECT ID, Name, Population FROM City ORDER BY Population DESC LIMIT 1000'
    cur.execute(query)
    resultset = cur.fetchall()
    col_names = [cn[0] for cn in cur.description]
    print('%10s %30s %10s' % tuple(col_names))
    for row in resultset:
        print('%10s %30s %10s' % row)

7. 从异常值中清除数据

使用 MAD（中值绝对偏差）检测异常值。

import numpy as np
import matplotlib.pyplot as plt

def is_outlier(points, threshold=3.5):
    if len(points.shape) == 1:
        points = points[:, None]
    median = np.median(points, axis=0)
    diff = np.sum((points - median)**2, axis=-1)
    diff = np.sqrt(diff)
    med_abs_deviation = np.median(diff)
    modified_z_score = 0.6745 * diff / med_abs_deviation
    return modified_z_score > threshold

x = np.random.random(100)
x = np.r_[x, -49, 95, 100, -100]
filtered = x[~is_outlier(x)]

plt.figure()
plt.subplot(211)
plt.hist(x, buckets=50)
plt.xlabel('Raw')
plt.subplot(212)
plt.hist(filtered, buckets=50)
plt.xlabel('Cleaned')
plt.show()

8. 分块读取文件

处理大文件时，可以分块管理。

import sys
filename = sys.argv[1]
with open(filename, 'rb') as hugefile:
    chunksize = 1000
    readable = ''
    while True:
        start = hugefile.tell()
        file_block = ''
        for _ in range(chunksize):
            line = hugefile.next()
            file_block = file_block + line
        readable = readable + file_block
        stop = hugefile.tell()
        print('reading bytes from %s to %s' % (start, stop))

9. 读取流数据源

持续监控文件变化。

import time
import os
import sys

if len(sys.argv) != 2:
    print("Please specify filename to read")
    sys.exit(1)

filename = sys.argv[1]
if not os.path.isfile(filename):
    print("Given file is not a file")
    sys.exit(1)

with open(filename, 'r') as f:
    filesize = os.stat(filename)[6]
    f.seek(filesize)
    while True:
        where = f.tell()
        line = f.readline()
        if not line:
            time.sleep(1)
            f.seek(where)
        else:
            print(line, end='')

10. 将图像数据导入 NumPy 数组

import scipy.misc
import matplotlib.pyplot as plt

lena = scipy.misc.lena()
plt.gray()
plt.imshow(lena)
plt.colorbar()
plt.show()

print(lena.shape)
print(lena.max())
print(lena.dtype)

11. 生成受控随机数据集

import random
import pylab

SAMPLE_SIZE = 100
random.seed()
real_rand_vars = [random.random() for val in range(SAMPLE_SIZE)]
pylab.hist(real_rand_vars, 10)
pylab.xlabel("Number range")
pylab.ylabel("Count")
pylab.show()

12. 平滑真实数据中的噪声

使用卷积窗口平滑信号。

from pylab import *
from numpy import *

def moving_average(interval, window_size):
    window = ones(int(window_size))/float(window_size)
    return convolve(interval, window, 'same')

t = linspace(-4, 4, 100)
y = sin(t) + randn(len(t))*0.1
plot(t, y, "k.")
y_av = moving_average(y, 10)
plot(t, y_av, "r")
xlabel("Time")
ylabel("Value")
grid(True)
show()

三、绘制你的第一个图并自定义它们

Matplotlib 是一个强大的工具箱，几乎满足了我们对 2D 绘图的所有需求。

1. 定义绘图类型

from matplotlib.pyplot import *
x = [1, 2, 3, 4]
y = [5, 4, 3, 2]

figure()
subplot(231)
plot(x, y)
subplot(232)
bar(x, y)
subplot(233)
barh(x, y)
subplot(234)
bar(x, y)
y1 = [7, 8, 5, 3]
bar(x, y1, bottom=y, color='r')
subplot(235)
boxplot(x)
subplot(236)
scatter(x, y)
show()

2. 绘制正弦和余弦图

import matplotlib.pyplot as pl
import numpy as np

x = np.linspace(-np.pi, np.pi, 256, endpoint=True)
y = np.cos(x)
y1 = np.sin(x)
pl.plot(x, y)
pl.plot(x, y1)
pl.title("Functions $\sin$ and $\cos$")
pl.xlim(-3.0, 3.0)
pl.ylim(-1.0, 1.0)
pl.xticks([-np.pi, -np.pi/2, 0, np.pi/2, np.pi], 
          [r'$-\pi$', r'$-\pi/2$', r'$0$', r'$+\pi/2$', r'$+\pi$'])
pl.yticks([-1, 0, +1], [r'$-1$', r'$0$', r'$+1$'])
pl.show()

3. 定义轴长度和极限

l = [-1, 1, -10, 10]
axis(l)

4. 设置刻度、标签和网格

from pylab import *
ax = gca()
ax.locator_params(tight=True, nbins=10)
ax.plot(np.random.normal(10, .1, 100))
show()

5. 添加图例和标注

from matplotlib.pyplot import *
x1 = np.random.normal(30, 3, 100)
x2 = np.random.normal(20, 2, 100)
plot(x1, label='plot')
plot(x2, label='2nd plot')
legend(bbox_to_anchor=(0., 1.02, 1., .102), loc=3, ncol=3, mode="expand", borderaxespad=0.)
annotate("Important value", (55, 20), xycoords='data', xytext=(5, 38), 
         arrowprops=dict(arrowstyle='->'))
show()

6. 将脊椎移向中央

import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(-np.pi, np.pi, 500, endpoint=True)
y = np.sin(x)
plt.plot(x, y)
ax = plt.gca()
ax.spines['right'].set_color('none')
ax.spines['top'].set_color('none')
ax.spines['bottom'].set_position(('data', 0))
ax.spines['left'].set_position(('data', 0))
ax.xaxis.set_ticks_position('bottom')
ax.yaxis.set_ticks_position('left')
plt.show()

7. 制作直方图

import numpy as np
import matplotlib.pyplot as plt

mu = 100
sigma = 15
x = np.random.normal(mu, sigma, 10000)
ax = plt.gca()
ax.hist(x, bins=35, color='r')
ax.set_xlabel('Values')
ax.set_ylabel('Frequency')
ax.set_title(r'$\mathrm{Histogram:}\ \mu=%d,\ \sigma=%d$' % (mu, sigma))
plt.show()

8. 用误差线制作条形图

import numpy as np
import matplotlib.pyplot as plt

x = np.arange(0, 10, 1)
y = np.log(x)
xe = 0.1 * np.abs(np.random.randn(len(y)))
plt.bar(x, y, yerr=xe, width=0.4, align='center', ecolor='r', color='cyan', label='experiment #1')
plt.xlabel('# measurement')
plt.ylabel('Measured values')
plt.title('Measurements')
plt.legend(loc='upper left')
plt.show()

9. 让饼图有价值

from pylab import *
figure(1, figsize=(6, 6))
ax = axes([0.1, 0.1, 0.8, 0.8])
labels = 'Spring', 'Summer', 'Autumn', 'Winter'
x = [15, 30, 45, 10]
explode = (0.1, 0.1, 0.1, 0.1)
pie(x, explode=explode, labels=labels, autopct='%1.1f%%', startangle=67)
title('Rainy days by season')
show()

10. 绘制填充区域

from matplotlib.pyplot import figure, show, gca
import numpy as np

x = np.arange(0.0, 2, 0.01)
y1 = np.sin(2*np.pi*x)
y2 = 1.2*np.sin(4*np.pi*x)
fig = figure()
ax = gca()
ax.plot(x, y1, x, y2, color='black')
ax.fill_between(x, y1, y2, where=y2>=y1, facecolor='darkblue', interpolate=True)
ax.fill_between(x, y1, y2, where=y2<=y1, facecolor='deeppink', interpolate=True)
ax.set_title('filled between')
show()

11. 用彩色标记绘制散点图

import matplotlib.pyplot as plt
import numpy as np

x = np.random.randn(1000)
y1 = np.random.randn(len(x))
y2 = 1.2 + np.exp(x)

ax1 = plt.subplot(121)
plt.scatter(x, y1, color='indigo', alpha=0.3, edgecolors='white', label='no correl')
plt.xlabel('no correlation')
plt.grid(True)
plt.legend()

ax2 = plt.subplot(122, sharey=ax1, sharex=ax1)
plt.scatter(x, y2, color='green', alpha=0.3, edgecolors='grey', label='correl')
plt.xlabel('strong correlation')
plt.grid(True)
plt.legend()
plt.show()

通过上述步骤，你可以快速搭建 Python 数据可视化环境，掌握多种数据格式的读取与清洗方法，并利用 Matplotlib 绘制出专业且美观的图表。在实际项目中，根据具体需求调整代码细节即可。