优化此查询1000000+行

问题描述 投票：0回答：1

我需要提取数据并将其写入csv文件，但它需要花费太多时间和太多内存。有什么问题，我该怎么办？此外，我觉得查询本身存在冗余。我是用PHP做的。

这是查询

CREATE TEMPORARY TABLE temp1 SELECT * FROM vicidial_closer_log
USE INDEX(call_date)
WHERE call_date BETWEEN '1980-01-01 00:00:00' AND '2019-03-12 23:59:59';
CREATE TEMPORARY TABLE temp2 SELECT * FROM vicidial_closer_log
USE INDEX(call_date)
WHERE call_date BETWEEN '1980-01-01 00:00:00' AND '2019-03-12 23:59:59';
SELECT a.call_date,
   a.lead_id,
   a.phone_number
   AS customer_number,
   IF(a.status != 'DROP', 'ANSWERED', 'UNANSWERED')
   AS status,
   IF(a.lead_id IS NOT NULL, 'inbound', 'outbound')
   AS call_type,
   a.USER
   AS agent,
   a.campaign_id
   AS skill,
   NULL
   AS campaign,
   a.status
   AS disposition,
   a.term_reason
   AS Hangup,
   a.uniqueid,
   Sec_to_time(a.queue_seconds)
   AS time_to_answer,
   Sec_to_time(a.length_in_sec - a.queue_seconds)
   AS talk_time,
   Sec_to_time(a.park_sec)
   AS hold_sec,
   Sec_to_time(a.dispo_sec)
   AS wrapup_sec,
   From_unixtime(a.start_epoch)
   AS start_time,
   From_unixtime(a.end_epoch)
   AS end_time,
   c.USER
   AS
   transfered,
   a.comments,
IF(a.length_in_sec IS NULL, Sec_to_time(a.queue_seconds),
Sec_to_time(a.length_in_sec + a.dispo_sec))
   AS duration,
Sec_to_time(a.length_in_sec - a.queue_seconds + a.dispo_sec)
   AS handling_time
FROM   temp1 a
   left outer join temp2 c
                   ON a.uniqueid = c.uniqueid
                   AND a.closecallid < c.closecallid
GROUP  BY a.closecallid

我上传了表结构和索引的截图。 Table Structure Indices of Table

谢谢。

更新：显示创建表vicidial_closer_log

vicidial_closer_log     CREATE TABLE `vicidial_closer_log` (
`closecallid` int(9) unsigned NOT NULL AUTO_INCREMENT,
`lead_id` int(9) unsigned NOT NULL,
`list_id` bigint(14) unsigned DEFAULT NULL,
`campaign_id` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`call_date` datetime DEFAULT NULL,
`start_epoch` int(10) unsigned DEFAULT NULL,
`end_epoch` int(10) unsigned DEFAULT NULL,
`length_in_sec` int(10) DEFAULT NULL,
`status` varchar(6) COLLATE utf8_unicode_ci DEFAULT NULL,
`phone_code` varchar(10) COLLATE utf8_unicode_ci DEFAULT NULL,
`phone_number` varchar(18) COLLATE utf8_unicode_ci DEFAULT NULL,
`user` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`comments` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`processed` enum('Y','N') COLLATE utf8_unicode_ci DEFAULT NULL,
`queue_seconds` decimal(7,2) DEFAULT 0.00,
`user_group` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`xfercallid` int(9) unsigned DEFAULT NULL,
`term_reason`       enum('CALLER','AGENT','QUEUETIMEOUT','ABANDON','AFTERHOURS','HOLDRECALLXFER',    'HOLDTIME','NOAGENT','NONE','MAXCALLS') COLLATE utf8_unicode_ci DEFAULT   'NONE',
 `uniqueid` varchar(20) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
 `agent_only` varchar(20) COLLATE utf8_unicode_ci DEFAULT '',
 `queue_position` smallint(4) unsigned DEFAULT 1,
 `called_count` smallint(5) unsigned DEFAULT 0,
 `nopaperform` varchar(5) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'NO',
 `park_sec` int(3) DEFAULT 0,
 `dispo_sec` int(3) DEFAULT 0,
 `record_file` text COLLATE utf8_unicode_ci DEFAULT NULL,
 PRIMARY KEY (`closecallid`),
 KEY `lead_id` (`lead_id`),
 KEY `call_date` (`call_date`),
 KEY `campaign_id` (`campaign_id`),
 KEY `uniqueid` (`uniqueid`),
 KEY `phone_number` (`phone_number`),
 KEY `date_user` (`call_date`,`user`),
 KEY `closecallid` (`closecallid`)
) ENGINE=MyISAM AUTO_INCREMENT=1850672 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci

EXPLAIN QUERY（仅限第三个查询）：

id  select_type     table   type    possible_keys   key     key_len     ref     rows    Extra   
1   SIMPLE  a   ALL     NULL    NULL    NULL    NULL    664640  Using temporary; Using filesort
1   SIMPLE  c   ALL     NULL    NULL    NULL    NULL    662480  Using where; Using join buffer (flat, BNL join)

更新（更新的查询）：

SELECT a.call_date,
       a.lead_id,
       a.phone_number
       AS customer_number,
       IF(a.status != 'DROP', 'ANSWERED', 'UNANSWERED')
       AS status,
       IF(a.lead_id IS NOT NULL, 'inbound', 'outbound')
       AS call_type,
       a.user
       AS agent,
       a.campaign_id
       AS skill,
       NULL
       AS campaign,
       a.status
       AS disposition,
       a.term_reason
       AS Hangup,
       a.uniqueid,
       Sec_to_time(a.queue_seconds)
       AS time_to_answer,
       Sec_to_time(a.length_in_sec - a.queue_seconds)
       AS talk_time,
       Sec_to_time(a.park_sec)
       AS hold_sec,
       Sec_to_time(a.dispo_sec)
       AS wrapup_sec,
       From_unixtime(a.start_epoch)
       AS start_time,
       From_unixtime(a.end_epoch)
       AS end_time,
       c.user
       AS
transfered,
a.comments,
IF(a.length_in_sec IS NULL, Sec_to_time(a.queue_seconds),
Sec_to_time(a.length_in_sec + a.dispo_sec))
       AS duration,
Sec_to_time(a.length_in_sec - a.queue_seconds + a.dispo_sec)
       AS handling_time
FROM   vicidial_closer_log a
       LEFT OUTER JOIN vicidial_closer_log c
                    ON a.closecallid <> c.closecallid
                       AND a.uniqueid = c.uniqueid
                       AND a.closecallid < c.closecallid
WHERE a.call_date BETWEEN '2018-01-01 00:00:00' AND '2019-03-13 23:59:59'

解释更新的查询：

id  select_type     table   type    possible_keys   key     key_len     ref     rows    Extra   
1   SIMPLE  a   ALL     call_date,date_user     NULL    NULL    NULL    662829  Using where
1   SIMPLE  c   ref     PRIMARY,uniqueid,closecallid    uniqueid    62  aastell_bliss.a.uniqueid    1   Using where

更新的查询执行结果：

Number of rows present between given time range: 155016 rows
Time taken: 0.0149 secs

有用！

php mysql pdo web-applications crud
1个回答
0
投票

导致答案的评论摘要：

  • CREATE TEMPORARY TABLE ... SELECT不会在临时表上创建索引
  • 显式使用临时表，特别是大型临时表，很少会带来性能提升。
  • 在连接中使用表别名允许自联接
  • 在连接左侧按主键分组不会增加太多，因为它已经是唯一的，并且JOIN没有聚合表达式。 GROUP BY添加了一个隐式的ORDER BY，因此如果使用二级索引来加入表，表达式可能会变慢。
  • 虽然查询的日期范围很大，但是当它很小时，准备将它作为一个重要的过滤器会使call_date更有利于作为索引。为了使这更有利，连接键被添加到索引的末尾，因此只需查看索引就可以实现连接的大部分工作。
  • 当PK在列上时，不需要同一列上的二级索引。
最新问题
© www.soinside.com 2019 - 2024. All rights reserved.