改善导入数据时的内存使用

Question

我实际上是在寻找建议，而不是解决方案。

我制作了一项服务，可以比较远程数据库中的数据，并在必要时插入、更新或删除本地数据。 源数据库中有超过 50k 的条目，本地内存使用减慢了脚本的速度，使其变得非常长。

这是服务的一部分：

function compare( $uosLocal, $uosDistant ) {

    $idsLocal = array_keys($uosLocal);
    $idsDistant = array_keys($uosDistant);

    $toInsert   = array_diff( $idsDistant, $idsLocal );
    $toDisable  = array_diff( $idsLocal, $idsDistant );


    $this->insert( array_map( function ($uoid) use ($uosDistant) {
        return ($uosDistant[$uoid]);
    }, $toInsert )  );

    $this->delete( array_map( function ($uoid) use ($uosLocal) {
        return ($uosLocal[$uoid]);
    }, $toDisable )  );

    $uosLocal = $this->getLocal();
    $uosDistant = $this->getDistant();
    $idsLocal = array_keys($uosLocal);
    $idsDistant = array_keys($uosDistant);

    $toUpdate   = array_intersect( $idsDistant, $idsLocal );

    $this->update( $toUpdate, $uosDistant, $uosLocal  );



}

function insert( $array_uos ) {

    $em = $this->doctrine->getManager();

    $count = 0;
    $total = count($array_uos);
    $precision = 2;
    $suffixes = array('', 'k', 'M', 'G', 'T');

    if ($total > 0)
        $this->output->writeln(" $total to insert : ");

    foreach ($array_uos as $simple_uo) {

        $uo = new UO();
        $uo
            ->setName($simple_uo['NAME'])
            ->setId($simple_uo['ID'])
        ;

        $em->persist($uo);
        unset($uo);

        $count++;

        if ($count % 100 === 0) {

            $base = log(memory_get_usage(), 1024);

            $em->flush();
            $this->output->writeln('['. (new \DateTime())->format('H:i:s') .']     ' . floor( $count / $total * 100 *100) /100 . "% ($count / $total)  (memory : " . round(pow(1024, $base - floor($base)), $precision) . $suffixes[floor($base)] . ") ");

        }

    }

    $em->flush();

    $this->output->writeln( " $count was inserted " );

}

正是

insert

方法导致了巨大的内存占用。

查看日志：

Starting data extraction from referentiel :  56215 to insert :
[09:49:55]     0.17% (100 / 56215)  (memory : 55.37M)
[09:49:55]     0.35% (200 / 56215)  (memory : 56.43M)
[09:49:55]     0.53% (300 / 56215)  (memory : 56.98M)
[09:49:56]     0.71% (400 / 56215)  (memory : 57.47M)
[09:49:56]     0.88% (500 / 56215)  (memory : 57.95M)
[09:49:57]     1.06% (600 / 56215)  (memory : 58.45M)
[09:49:57]     1.24% (700 / 56215)  (memory : 59.11M)
[09:49:57]     1.42% (800 / 56215)  (memory : 59.59M)
[09:49:58]     1.6% (900 / 56215)  (memory : 60.07M)
[09:49:58]     1.77% (1000 / 56215)  (memory : 60.56M)
[09:49:59]     1.95% (1100 / 56215)  (memory : 61.09M)
[09:50:00]     2.13% (1200 / 56215)  (memory : 61.59M)
[09:50:00]     2.31% (1300 / 56215)  (memory : 62.09M)
[09:50:01]     2.49% (1400 / 56215)  (memory : 62.83M)
[09:50:01]     2.66% (1500 / 56215)  (memory : 63.32M)
[09:50:02]     2.84% (1600 / 56215)  (memory : 63.82M)
[09:50:03]     3.02% (1700 / 56215)  (memory : 64.32M)
[09:50:03]     3.2% (1800 / 56215)  (memory : 64.81M)
[09:50:04]     3.37% (1900 / 56215)  (memory : 65.3M)
[09:50:05]     3.55% (2000 / 56215)  (memory : 65.79M)
[09:50:05]     3.73% (2100 / 56215)  (memory : 66.31M)
[09:50:06]     3.91% (2200 / 56215)  (memory : 66.84M)
[09:50:07]     4.09% (2300 / 56215)  (memory : 67.33M)
[09:50:08]     4.26% (2400 / 56215)  (memory : 67.82M)
[09:50:09]     4.44% (2500 / 56215)  (memory : 68.31M)
[09:50:10]     4.62% (2600 / 56215)  (memory : 68.81M)
[09:50:10]     4.8% (2700 / 56215)  (memory : 69.8M)
[09:50:11]     4.98% (2800 / 56215)  (memory : 70.28M)
[09:50:12]     5.15% (2900 / 56215)  (memory : 70.78M)
[09:50:13]     5.33% (3000 / 56215)  (memory : 71.27M)
[09:50:14]     5.51% (3100 / 56215)  (memory : 71.77M)
[09:50:15]     5.69% (3200 / 56215)  (memory : 72.26M)
[09:50:16]     5.87% (3300 / 56215)  (memory : 72.75M)
[09:50:17]     6.04% (3400 / 56215)  (memory : 73.24M)
[09:50:18]     6.22% (3500 / 56215)  (memory : 73.74M)
[09:50:20]     6.4% (3600 / 56215)  (memory : 74.23M)
[09:50:21]     6.58% (3700 / 56215)  (memory : 74.72M)
[09:50:22]     6.75% (3800 / 56215)  (memory : 75.21M)
[09:50:23]     6.93% (3900 / 56215)  (memory : 75.7M)
[09:50:24]     7.11% (4000 / 56215)  (memory : 76.2M)
[09:50:26]     7.29% (4100 / 56215)  (memory : 76.73M)
[09:50:27]     7.47% (4200 / 56215)  (memory : 77.3M)
[09:50:28]     7.64% (4300 / 56215)  (memory : 77.79M)

是否正常，可以改善吗？

非常感谢！

Answer 1

正如评论中所说，我在

$em->clear()

之后添加了

$em->flush()

并且它有效