• 出处：blktrace 分析 IO

前言

上篇博客介绍了 iostat 的一些输出，这篇介绍 blktrace 这个神器。上一节介绍 iostat 的时候，我们心心念念希望得到块设备处理 io 的 service time，而不是 service time + wait time，因为对于评估一个磁盘或者云磁盘而言，service time 才是衡量磁盘性能的核心指标和直接指标。很不幸 iostat 无法提供这个指标，但是 blktrace 可以。

blktrace 是一柄神器，很多工具都是基于该神器的：ioprof、seekwatcher、iowatcher，这个工具基本可以满足我们的对块设备请求的所有了解。

blktrace 的原理

一个 I/O 请求，从应用层到底层块设备，路径如下图所示 …

• 使用 blktrace 统计磁盘块 IO 访问频率
• debugfs — ext2/ext3/ext4 file system debugger

如果我们想知道，在一段时间内，那一个磁盘块被读写的频率最高，怎么办？我问这个问题是看到 taobao kernel wiki 上面有这么一段话，看了我十分心痒：

    通过对 blktrace 的输出结果进行分析，我们可以对特定时间段内发生的
    读、写操作的磁盘块进行 I/O 频次统计。结合通过将磁盘块 I/O 频度统计和
    page cache 命中率统计相结合，就可以比较有效的判断服务器节点的 cache
    使用效率。通过这套工具，CDN 系统修正了一个固态硬盘上 cache 管理的缺陷，
    显著提高了I/O性能（详细信息）

淘宝大神给了一个 …

• 原文 Debugging disk issues with blktrace, blkparse, btrace and btt in Linux environment

blktrace is a block layer IO tracing mechanism which provides detailed information about request queue operations up to user space. There are three major components: a kernel component, a utility to record the i/o trace information …

• blktrace.git
• man bno_plot
• /usr/share/doc/blktrace/btt.pdf

To use this utility, the gnuplot package needs to be installed.

bno_plot is a visualization tool for the block layer IO tracing tool called blktrace(8). As noted in its documentation, blktrace is a block layer IO tracing mechanism which …

• blktrace.git
• man btt
• /usr/share/doc/blktrace/btt.pdf

btt is a post-processing tool for the block layer IO tracing tool called blktrace(8). As noted in its documentation, blktrace is a block layer IO tracing mechanism which provides detailed information about request queue operations up to user space …

• blktrace.git
• man btreplay
• /usr/share/doc/blktrace/btreplay.pdf

The btrecord and btreplay tools provide the ability to record and replay IOs captured by the blktrace utility. Attempts are made to maintain ordering, CPU mappings and time-separation of IOs.

The blktrace utility provides the ability to collect detailed traces …

• blktrace.git
• man btrecord
• /usr/share/doc/blktrace/btreplay.pdf

The btrecord and btreplay tools provide the ability to record and replay IOs captured by the blktrace utility. Attempts are made to maintain ordering, CPU mappings and time-separation of IOs.

The blktrace utility provides the ability to collect detailed traces …

• blktrace.git
• man blkiomon

blkiomon is a block device I/O monitor. It periodically generates per-device request size and request latency statistics from blktrace data. It provides histograms as well as data that can be used to calculate min, max, average and variance. For this purpose, it consumes D and …

• blktrace.git
• man iowatcher

    iowatcher [options] [--] [program arguments ...]

    blktrace/
        iowatcher/
            ├── blkparse.c
            ├── blkparse.h
            ├── COPYING
            ├── fio.c
            ├── fio.h
            ├── list.h
            ├── main.c
            ├── Makefile
            ├── mpstat.c
            ├── mpstat.h
            ├── plot.c
            ├── plot.h
            ├── tracers.c
            └── tracers.h

iowatcher graphs the results of a blktrace run. It can graph the result of …

• blktrace.git
• man blkrawverify
• man verify_blkparse

blkrawverify 和 verify_blkparse 都是 verifies an output file produced by blkparse

blkrawverify

The blkrawverify utility can be used to verify data retrieved via blktrace. It will check for valid event formats, forward progressing sequence numbers and time stamps, also does reasonable checks for other …