{"id":861,"date":"2024-09-07T00:36:07","date_gmt":"2024-09-07T05:36:07","guid":{"rendered":"https:\/\/zhanxw.com\/blog\/?p=861"},"modified":"2024-09-07T00:36:08","modified_gmt":"2024-09-07T05:36:08","slug":"%e6%af%94%e8%be%83gzip%e5%8e%8b%e7%bc%a9%e5%92%8c%e8%a7%a3%e5%8e%8b%e7%bc%a9%e6%80%a7%e8%83%bd","status":"publish","type":"post","link":"https:\/\/zhanxw.com\/blog\/2024\/09\/%e6%af%94%e8%be%83gzip%e5%8e%8b%e7%bc%a9%e5%92%8c%e8%a7%a3%e5%8e%8b%e7%bc%a9%e6%80%a7%e8%83%bd\/","title":{"rendered":"\u6bd4\u8f83gzip\u538b\u7f29\u548c\u89e3\u538b\u7f29\u6027\u80fd"},"content":{"rendered":"\n

Benchmark gzip compression and decompression performance<\/p>\n\n\n\n

gzip\u662f\u5f88\u591a\u6d4b\u5e8f\u6570\u636e\u7684\u9ed8\u8ba4\u538b\u7f29\u683c\u5f0f\u3002\u5728\u5b66\u6821HPC\u73af\u5883\u4e2d\uff0c\u51e0\u4e2aGB\u7684\u6587\u4ef6\u7684\u538b\u7f29\u548c\u89e3\u538b\u7f29\u5f80\u5f80\u9700\u8981\u51e0\u5206\u949f\u5230\u51e0\u5341\u5206\u949f\u7684\u65f6\u95f4\u3002\u5982\u679c\u5904\u7406\u7684\u6570\u636e\u8f83\u5927\uff0cgzip\u7684\u538b\u7f29\u548c\u89e3\u538b\u7f29\u4f1a\u5360\u7528\u4e0d\u5c11\u65f6\u95f4\u3002\u56e0\u6b64\u6211\u60f3\u770b\u770b\u6709\u6ca1\u6709\u6bd4Linux\u9ed8\u8ba4\u7684gzip\u66f4\u6709\u6548\u7387\u7684\u8f6f\u4ef6\u3002\u8fd9\u91cc\u4e3b\u8981\u6bd4\u8f83\uff1a<\/p>\n\n\n\n

    \n
  1. libdeflate \u8fd9\u662fsamtools\u7684\u9009\u62e9<\/li>\n\n\n\n
  2. igzip \u8fd9\u5305\u542b\u5728Intel ISA-L\u8f6f\u4ef6\u4e2d<\/li>\n\n\n\n
  3. pigz \u8fd9\u662fgzip\u7684\u591a\u7ebf\u7a0b\u5b9e\u73b0<\/li>\n<\/ol>\n\n\n\n

    \u5177\u4f53\u7684\u6d4b\u8bd5\u5206\u4e3a\u538b\u7f29\u548c\u89e3\u538b\u7f29\u4e24\u90e8\u5206\u3002\u7528\u7684\u662f\u4e00\u4e2aFASTQ\u6587\u4ef6\uff0c\u538b\u7f29\u540e\u5927\u7ea62G\uff0c\u539f\u59cb\u6587\u4ef66G\u3002<\/p>\n\n\n\n

    \u6d4b\u8bd5\u7ed3\u679c\u5982\u4e0b\uff1a<\/p>\n\n\n\n

    Decompression<\/td><\/td><\/td><\/td><\/tr>
    <\/td>User Time (s)<\/td>Real Time (s)<\/td>Memory<\/td><\/tr>
    gzip<\/td>39.73u<\/td>1.47s<\/td>41.24r<\/td>876kB<\/td><\/tr>
    igzip<\/td>7.60u<\/td>1.81s<\/td>11.31r<\/td>2844kB<\/td><\/tr>
    libdeflate<\/td>20.45u<\/td>6.53s<\/td>28.05r<\/td>7909292kB<\/td><\/tr>
    pigz<\/td>30.18u<\/td>4.86s<\/td>23.49r<\/td>1000kB<\/td><\/tr><\/tbody><\/table>
    Decompression Benchmark<\/figcaption><\/figure>\n\n\n\n
    Compression<\/td><\/td><\/td><\/td><\/td><\/tr>
    <\/td>User Time (s)<\/td>Real Time (s)<\/td>Memory<\/td>Compressed File Size<\/td><\/tr>
    gzip<\/td>620.17u<\/td>2.54s<\/td>623.23r<\/td>1116kB<\/td>1.81G<\/td><\/tr>
    igzip<\/td>15.61u<\/td>1.92s<\/td>17.69r<\/td>2628kB<\/td>2.03G<\/td><\/tr>
    libdeflate<\/td>152.88u<\/td>10.29s<\/td>163.43r<\/td>7879072kB<\/td>1.78G<\/td><\/tr>
    pigz<\/td>995.38u<\/td>4.15s<\/td>18.18r<\/td>38352kB<\/td>1.82G<\/td><\/tr><\/tbody><\/table>
    Compression Benchmark<\/figcaption><\/figure>\n\n\n\n

    \u7ed3\u8bba\uff1aIntel igzip\u5728\u8017\u65f6\u6307\u6807\u4e2d\u5b8c\u80dc\u3002\u5728\u9ed8\u8ba4\u8bbe\u7f6e\u4e0b\uff0cigzip\u89e3\u538b\u7f29\u65f6\u95f4\u662f\u7cfb\u7edf\u63d0\u4f9bgzip\u76841\/5\uff0c\u538b\u7f29\u65f6\u95f4\u662f1\/40\u3002\u552f\u4e00\u4e0d\u8db3\u7684\u662f\u538b\u7f29\u540e\u7684\u6587\u4ef6\u504f\u5927\u3002<\/p>\n\n\n\n

    \u8ba8\u8bba\uff1a<\/p>\n\n\n\n

    \u8fd9\u4e2a\u7ed3\u679c\u548csamtools \u7684benchmark<\/a>\u5dee\u522b\u5f88\u5927\u3002\u6211\u53d1\u73b0samtools\u7528\u7684GitHub\u4ed3\u5e93\uff08https:\/\/github.com\/jtkukunas\/zlib<\/a>\uff09\u4e0d\u540c\u4e8eIntel Tuning Guide<\/a>\u91cc\u63d0\u4f9b\u7684\u4ed3\u5e93\uff08https:\/\/github.com\/intel\/isa-l\/blob\/master\/README.md<\/a>\uff09\u3002 \u56e0\u6b64\u6027\u80fd\u7684\u5dee\u522b\u53ef\u80fd\u6765\u6e90\u4e8esamtools\u6ca1\u6709\u7528\u5b98\u65b9\u7684\u5b9e\u73b0\uff0c\u5f53\u7136\u4e5f\u6709\u53ef\u80fd\u662f\u6d4b\u8bd5\u65b9\u6cd5\u4e0d\u540c\uff1asamtools\u6d4b\u91cf\u7684\u662fbam\u6587\u4ef6\u7684\u8bfb\u5199\u901f\u5ea6\uff0c\u800c\u6211\u53ea\u662f\u7528FASTQ\u6587\u4ef6\u4f5c\u4e3a\u6d4b\u8bd5\u3002<\/p>\n\n\n\n

    \u5176\u4ed6\u76f8\u5173\u7684gzip\u5de5\u5177\u8fd8\u6709zlib-ng <\/a>\u548ccloudflare\u4f18\u5316\u7684zlib<\/a>\u3002\u8fd9\u4e9b\u6027\u80fd\u5e94\u8be5\u5728igzip\u548c\u7cfb\u7edf\u81ea\u5e26\u7684gzip\u4e4b\u95f4\uff0c\u8fd9\u91cc\u5c31\u4e0d\u518d\u6d4b\u8bd5\u4e86\u3002<\/p>\n\n\n\n

    <\/p>\n","protected":false},"excerpt":{"rendered":"

    Benchmark gzip compression and decompression performance gzip\u662f\u5f88\u591a\u6d4b\u5e8f\u6570\u636e\u7684\u9ed8\u8ba4\u538b\u7f29\u683c\u5f0f\u3002\u5728\u5b66\u6821HPC\u73af\u5883\u4e2d\uff0c\u51e0\u4e2aGB\u7684\u6587\u4ef6\u7684\u538b\u7f29\u548c\u89e3\u538b\u7f29\u5f80\u5f80\u9700\u8981\u51e0\u5206\u949f\u5230\u51e0\u5341\u5206\u949f\u7684\u65f6\u95f4\u3002\u5982\u679c\u5904\u7406\u7684\u6570\u636e\u8f83\u5927\uff0cgzip\u7684\u538b\u7f29\u548c\u89e3\u538b\u7f29\u4f1a\u5360\u7528\u4e0d\u5c11\u65f6\u95f4\u3002\u56e0\u6b64\u6211\u60f3\u770b\u770b\u6709\u6ca1\u6709\u6bd4Linux\u9ed8\u8ba4\u7684gzip\u66f4\u6709\u6548\u7387\u7684\u8f6f\u4ef6\u3002\u8fd9\u91cc\u4e3b\u8981\u6bd4\u8f83\uff1a \u5177\u4f53\u7684\u6d4b\u8bd5\u5206\u4e3a\u538b\u7f29\u548c\u89e3\u538b\u7f29\u4e24\u90e8\u5206\u3002\u7528\u7684\u662f\u4e00\u4e2aFASTQ\u6587\u4ef6\uff0c\u538b\u7f29\u540e\u5927\u7ea62G\uff0c\u539f\u59cb\u6587\u4ef66G\u3002 \u6d4b\u8bd5\u7ed3\u679c\u5982\u4e0b\uff1a Decompression User Time (s) Real Time (s) Memory gzip 39.73u 1.47s 41.24r 876kB igzip 7.60u 1.81s 11.31r 2844kB libdeflate 20.45u 6.53s 28.05r 7909292kB pigz 30.18u 4.86s 23.49r 1000kB Decompression Benchmark Compression User Time (s) Real Time (s) Memory Compressed File Size gzip 620.17u 2.54s 623.23r 1116kB […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[155,156],"class_list":["post-861","post","type-post","status-publish","format-standard","hentry","category-sysadmin","tag-gzip","tag-optimization"],"_links":{"self":[{"href":"https:\/\/zhanxw.com\/blog\/wp-json\/wp\/v2\/posts\/861","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/zhanxw.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/zhanxw.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/zhanxw.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/zhanxw.com\/blog\/wp-json\/wp\/v2\/comments?post=861"}],"version-history":[{"count":4,"href":"https:\/\/zhanxw.com\/blog\/wp-json\/wp\/v2\/posts\/861\/revisions"}],"predecessor-version":[{"id":865,"href":"https:\/\/zhanxw.com\/blog\/wp-json\/wp\/v2\/posts\/861\/revisions\/865"}],"wp:attachment":[{"href":"https:\/\/zhanxw.com\/blog\/wp-json\/wp\/v2\/media?parent=861"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/zhanxw.com\/blog\/wp-json\/wp\/v2\/categories?post=861"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/zhanxw.com\/blog\/wp-json\/wp\/v2\/tags?post=861"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}