CARVIEW |
Select Language
HTTP/2 200
date: Wed, 30 Jul 2025 14:48:56 GMT
content-type: text/html; charset=utf-8
vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, X-Requested-With,Accept-Encoding, Accept, X-Requested-With
etag: W/"73c9dba73e036596065b7687840b8bc5"
cache-control: max-age=0, private, must-revalidate
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 0
referrer-policy: no-referrer-when-downgrade
content-security-policy: default-src 'none'; base-uri 'self'; child-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com *.rel.tunnels.api.visualstudio.com wss://*.rel.tunnels.api.visualstudio.com objects-origin.githubusercontent.com copilot-proxy.githubusercontent.com proxy.individual.githubcopilot.com proxy.business.githubcopilot.com proxy.enterprise.githubcopilot.com *.actions.githubusercontent.com wss://*.actions.githubusercontent.com productionresultssa0.blob.core.windows.net/ productionresultssa1.blob.core.windows.net/ productionresultssa2.blob.core.windows.net/ productionresultssa3.blob.core.windows.net/ productionresultssa4.blob.core.windows.net/ productionresultssa5.blob.core.windows.net/ productionresultssa6.blob.core.windows.net/ productionresultssa7.blob.core.windows.net/ productionresultssa8.blob.core.windows.net/ productionresultssa9.blob.core.windows.net/ productionresultssa10.blob.core.windows.net/ productionresultssa11.blob.core.windows.net/ productionresultssa12.blob.core.windows.net/ productionresultssa13.blob.core.windows.net/ productionresultssa14.blob.core.windows.net/ productionresultssa15.blob.core.windows.net/ productionresultssa16.blob.core.windows.net/ productionresultssa17.blob.core.windows.net/ productionresultssa18.blob.core.windows.net/ productionresultssa19.blob.core.windows.net/ github-production-repository-image-32fea6.s3.amazonaws.com github-production-release-asset-2e65be.s3.amazonaws.com insights.github.com wss://alive.github.com api.githubcopilot.com api.individual.githubcopilot.com api.business.githubcopilot.com api.enterprise.githubcopilot.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com copilot-workspace.githubnext.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: blob: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com private-avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com release-assets.githubusercontent.com secured-user-images.githubusercontent.com/ user-images.githubusercontent.com/ private-user-images.githubusercontent.com opengraph.githubassets.com copilotprodattachments.blob.core.windows.net/github-production-copilot-attachments/ github-production-user-asset-6210df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com objects-origin.githubusercontent.com *.githubusercontent.com; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/ secured-user-images.githubusercontent.com/ private-user-images.githubusercontent.com github-production-user-asset-6210df.s3.amazonaws.com gist.github.com; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; upgrade-insecure-requests; worker-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/
server: github.com
content-encoding: gzip
accept-ranges: bytes
set-cookie: _gh_sess=AOEW%2Fqaiefe3ghOT0GBGeKcP5StPlyiehtuhWjCIuEFOcVqr3Whbqv7wUxQMuQuqGNW%2Fiu41FfmkntqsH3VSkuC3U4qGNO883j%2F3ifg6Pxn7DkopQABq6IBsX83z3yUEGvNg7ZlIZ5BRAERU9%2FgGqghvxoQL61ugKvkhAe7qqV16rDJ6cc0QxLtQNQ7mv0o1O3BGLh78wLEBkUMDdv4bhJm6PvCywFMcWh9UWMiUJ7XWPUcbI7BR9U1iOtOL%2BUVWXvey3inetkt94DR3zAHJOA%3D%3D--Hun3wRzvd7pRJ0Iu--%2BSSKSujoMmzKCI5gu%2FCdOg%3D%3D; Path=/; HttpOnly; Secure; SameSite=Lax
set-cookie: _octo=GH1.1.419107398.1753886936; Path=/; Domain=github.com; Expires=Thu, 30 Jul 2026 14:48:56 GMT; Secure; SameSite=Lax
set-cookie: logged_in=no; Path=/; Domain=github.com; Expires=Thu, 30 Jul 2026 14:48:56 GMT; HttpOnly; Secure; SameSite=Lax
x-github-request-id: 8826:E01A9:1E748C:22A43F:688A30D8
Performance · alibaba/canal Wiki · GitHub
Skip to content
Navigation Menu
{{ message }}
-
Notifications
You must be signed in to change notification settings - Fork 7.7k
Performance
agapple edited this page Jul 20, 2018
·
2 revisions
类型 | 配置 |
---|---|
MySQL A | Intel(R) Xeon(R) CPU E5-2430 0 @ 2.20GHz (24core 96G) |
MySQL B | Intel(R) Xeon(R) CPU E5-2430 0 @ 2.20GHz (24core 96G) 日常业务库 |
Canal Server | Intel(R) Xeon(R) CPU E5-2430 0 @ 2.20GHz (24core 96G) |
Canal Client | Intel(R) Xeon(R) CPU E5-2430 0 @ 2.20GHz (24core 96G) |
为了更加精准的描述canal的性能,会从整个流程上进行优化和分析.
- 整个canal流程处理binlog,分为这么5步操作(4+1),整个优化和评测对这5步分别进行.
- 构造了两个测试场景,批量Insert/Update/Delete 和 普通业务DB(单条操作为主)
- 网络读取优化
- socket receiveBuffer/sendBuffer调优
- readBytes减少数据拷贝
- 引入BufferedInputStream提升读取能力
- binlog parser优化
- 时间毫秒精度解析优化
- 并发解析模型 (引入ringbuffer,拆分了几个阶段:网络接收、Event基本解析、DML解析和protobuf构造、加入memory store,针对瓶颈点protobuf构造采用了多线程的模式提升吞吐)
- 序列化和传输优化
- 提前序列化,避免在SessionHandler里主线程里串行序列化
- 嵌套protobuf对象序列化会产生多次byte[]数据拷贝,硬编码拉平到一个byte[]里处理,避免拷贝
- 客户端接收数据时,做lazy解析,避免在主线程串行处理
- 客户端ack做异步处理,避免在主线程串行处理
可参考github issue详细提交记录: https://github.com/alibaba/canal/issues/726
序号
|
阶段
|
批量操作Insert/Update/Delete (导入业务)
|
单条操作 (普通业务)
|
1
|
Binlog接收
|
200w TPS (网络 117MB/s)
|
71w TPS (网络 112MB/s)
|
2
|
Binlog Event解析
|
200w TPS (网络 117MB/s)
|
70w TPS (网络 110MB/s)
|
3
|
Insert/Update/Delete深度解析
|
200w TPS (网络 117MB/s)
|
65w TPS (网络 105MB/s)
|
4
|
生成CanalEntry (存储到memory store)
|
130w TPS (网络 75MB/s)
|
50w TPS (网络 90MB/s)
|
5
|
client接收
|
20w TPS 1.canal server机器网络 11MB/s
2.canal client机器网络 75MB/s
binlog膨胀率为 1:6.8
|
10w TPS 1.canal server网络 22MB/s
2.canal client网络 42MB/s
binlog膨胀率为 1:1.9
|
各个阶段测试代码:
- FetcherPerformanceTest.java
- MysqlBinlogEventPerformanceTest.java
- MysqlBinlogParsePerformanceTest.java
- MysqlBinlogDumpPerformanceTest.java
- SimpleCanalClientPermanceTest.java
从最开始接收(跑满网络带宽)到最后client机器收到格式化的binlog数据,binlog解析的5个阶段是一个漏斗形的性能。目前整个阶段4->阶段5,性能下降比较明显主要是因为网络传输、序列化的代价影响,binlog接收为了保序采用了串行方式,所以串行里的每个代码逻辑处理都会影响最后吞吐。 so. 如果基于canal做额外的数据扩展,比如对接到MQ系统,可以在步骤3、4阶段介入,最大化的吞吐.
结论数据:
- 1.0.26经过优化之后的性能,从业务binlog入库到canal client拿到数据,基本可以达到10~20w的TPS. 相比于canal 1.0.24的4w tps,提升了150%的吞吐性能.
- 单纯的binlog解析能力可以跑到60w ~ 200w的TPS,相当于100MB/s的解析速度
- Home
- Introduction / 简介
- Quick Start
- Client Guide
- Canal Admin
- Canal Performance
- AdminGuide
- DevGuide
- BinlogChange(Mysql5.6)
- BinlogChange(MariaDB)
- BinlogChange(MySQL8)
- TableMetaTSDB
- ReleaseNotes
- Download
- FAQ / 常见问题解答
Clone this wiki locally
You can’t perform that action at this time.