我目前正在将一些数据库从运行MySQl的物理服务器迁移到AWS RDS。当前服务器使用MySQL 5.5,而新的RDS使用MariaDB 10.1。在我尝试针对新服务器运行应用程序之前,一切都进展顺利。除非我给它添加一个大的限制,否则即使给定10分钟,以下查询也不会完成。
SELECT al.*
FROM US.Products p
JOIN US.Products_Contributors pc ON p.Product_id = pc.Product_id
JOIN US.Contributors c ON pc.Contributor_Id = c.Contributor_Id
JOIN US.Products_Category pca ON pca.Product_id = p.Product_id
JOIN US.Categories ca ON ca.Category_Id = pca.Category_Id
JOIN US.Asset_Links al ON (al.Asset_link_Id = p.Product_id) OR (al.Asset_link_Id = c.Contributor_ID)
WHERE p.Product_ISBN13 is not null
AND (
ca.Category_Code_3 in ("JNF","JUV")
)
AND al.Asset_Link_Table in ("Contributors","Products")
AND al.Asset_id != 0
GROUP BY al.Asset_Links_Id;
旧服务器上的查询大约在11秒内完成。如果我在新服务器上向查询添加“LIMIT 900000”,它将在大约7秒内完成。在GROUP by之前返回约800,000行,在group by之后返回约150,000行。如果我将LIMIT置于超过900000的任何值,那么查询将无法完成。
我尝试过的事情:
所以在我看来,这是使用MySQL版本的问题?但为什么添加LIMIT会解决问题呢?为什么它只能达到900000。
任何帮助将不胜感激!
谢谢
更新:16/19 17/12/2017
解析原始服务器MySQL 5.5:
解释RDS MariaDB 10.1(无限制):
{
"query_block": {
"select_id": 1,
"filesort": {
"temporary_table": {
"function": "buffer",
"table": {
"table_name": "p",
"access_type": "ALL",
"possible_keys": ["PRIMARY", "ISBN", "Product_Id"],
"rows": 120108,
"filtered": 99.999,
"attached_condition": "(p.Product_ISBN13 is not null)"
},
"table": {
"table_name": "pc",
"access_type": "ref",
"possible_keys": ["Products_contributor", "Contributor_Id"],
"key": "Products_contributor",
"key_length": "4",
"used_key_parts": ["Product_Id"],
"ref": ["US.p.Product_Id"],
"rows": 1,
"filtered": 100
},
"table": {
"table_name": "c",
"access_type": "eq_ref",
"possible_keys": ["PRIMARY"],
"key": "PRIMARY",
"key_length": "4",
"used_key_parts": ["Contributor_Id"],
"ref": ["US.pc.Contributor_Id"],
"rows": 1,
"filtered": 100,
"using_index": true
},
"table": {
"table_name": "pca",
"access_type": "ref",
"possible_keys": ["Products_Category", "Category_Id"],
"key": "Products_Category",
"key_length": "4",
"used_key_parts": ["Product_Id"],
"ref": ["US.p.Product_Id"],
"rows": 2,
"filtered": 100
},
"table": {
"table_name": "ca",
"access_type": "eq_ref",
"possible_keys": ["PRIMARY"],
"key": "PRIMARY",
"key_length": "4",
"used_key_parts": ["Category_Id"],
"ref": ["US.pca.Category_Id"],
"rows": 1,
"filtered": 100,
"index_condition": "(ca.Category_Id = pca.Category_Id)",
"attached_condition": "(ca.Category_Code_3 in ('JNF','JUV'))"
},
"block-nl-join": {
"table": {
"table_name": "al",
"access_type": "ALL",
"possible_keys": ["Asset_Link_Id", "Asset_Id"],
"rows": 908975,
"filtered": 95.517,
"attached_condition": "((al.Asset_Link_Table in ('Contributors','Products')) and (al.Asset_Id <> 0))"
},
"buffer_type": "flat",
"buffer_size": "1024Kb",
"join_type": "BNL",
"attached_condition": "((al.Asset_Link_Id = p.Product_Id) or (al.Asset_Link_Id = pc.Contributor_Id))"
}
}
}
}
}
EXPLAIN RDS MariaDB 10.1(LIMIT 908974):
{
"query_block": {
"select_id": 1,
"filesort": {
"temporary_table": {
"function": "buffer",
"table": {
"table_name": "p",
"access_type": "ALL",
"possible_keys": ["PRIMARY", "ISBN", "Product_Id"],
"rows": 120108,
"filtered": 99.999,
"attached_condition": "(p.Product_ISBN13 is not null)"
},
"table": {
"table_name": "pc",
"access_type": "ref",
"possible_keys": ["Products_contributor", "Contributor_Id"],
"key": "Products_contributor",
"key_length": "4",
"used_key_parts": ["Product_Id"],
"ref": ["US.p.Product_Id"],
"rows": 1,
"filtered": 100
},
"table": {
"table_name": "c",
"access_type": "eq_ref",
"possible_keys": ["PRIMARY"],
"key": "PRIMARY",
"key_length": "4",
"used_key_parts": ["Contributor_Id"],
"ref": ["US.pc.Contributor_Id"],
"rows": 1,
"filtered": 100,
"using_index": true
},
"table": {
"table_name": "pca",
"access_type": "ref",
"possible_keys": ["Products_Category", "Category_Id"],
"key": "Products_Category",
"key_length": "4",
"used_key_parts": ["Product_Id"],
"ref": ["US.p.Product_Id"],
"rows": 2,
"filtered": 100
},
"table": {
"table_name": "ca",
"access_type": "eq_ref",
"possible_keys": ["PRIMARY"],
"key": "PRIMARY",
"key_length": "4",
"used_key_parts": ["Category_Id"],
"ref": ["US.pca.Category_Id"],
"rows": 1,
"filtered": 100,
"index_condition": "(ca.Category_Id = pca.Category_Id)",
"attached_condition": "(ca.Category_Code_3 in ('JNF','JUV'))"
},
"range-checked-for-each-record": {
"keys": ["Asset_Link_Id", "Asset_Id"],
"table": {
"table_name": "al",
"access_type": "ALL",
"possible_keys": ["Asset_Link_Id", "Asset_Id"],
"key": "Asset_Id",
"key_length": "4",
"used_key_parts": ["Asset_Id"],
"rows": 908975,
"filtered": 95.517
}
}
}
}
}
}
我注意到的是将限制设置为任何比报告的连接ros数少1的数字,然后查询使用“检查每个记录的范围(索引映射:0x6)”,其中如同MySQL 5.5一样,它使用有没有限制。我发现如果我将force index(Asset_Link_Id)
添加到最后一个连接,它将始终使用“Range checked ...”然后查询将完成。
Whilte修改和优化所有查询是理想的解决方案,在这种情况下它不是最好的。我真的不想修改查询的原因,因为我正在迁移的服务器上有数百个不同的脚本/应用程序,如果我必须在很多不同的应用程序中修改很多查询,那么这将带我一个很长一段时间,我将无法满足迁移截止日期。所以在这一点上,如果这种行为无法通过设置来控制,那么我可能会在新服务器而不是MariaDB 10.1上使用MySQL 5.5。
是否可以解释为什么查询优化器选择5.7中具有大/未定义限制的不同路由,而不是5.5?在阅读了关于动态范围和join_buffer之后,为什么在该范围内使用缓冲区会更慢?根据我的阅读,我会认为这是更高效的?
(本答案并没有直接解决“为什么会这么慢”的问题,而是作为安慰奖来解决其他性能问题。)
我看到两个多对多映射表看起来像什么。这种情况的典型实施效率低于可能的效率。
请按照https://mariadb.com/kb/en/library/building-the-best-index-for-a-given-select/#many-to-many-mapping-table中的提示进行操作 - 然后查看无论是否使用LIMIT
,性能都会提高。
EXPLAIN
可能会改变;让我们来看看它。
Profiling
- 是的,这通常是无用的;它有几条无法提供信息的消息,它花费99%的时间。
不要将缓冲区大小增加到导致交换的程度;这会伤害很多。
在允许这样的版本,请提供EXPLAIN FORMAT=JSON SELECT ...
OR
通常是表演杀手;把它变成UNION
:
( SELECT ...
JOIN US.Asset_Links al ON al.Asset_link_Id = p.Product_id
...
) UNION DISTINCT
( SELECT ...
JOIN US.Asset_Links al al.Asset_link_Id = c.Contributor_ID
...
)
(由于GROUP BY
,我可能没有正确地将OR
映射到UNION
。)