让php爬虫跳过特定的URLs。

Question

我有一个GenerateSitemap.php文件，在这个文件中我可以配置爬虫，但我不明白我应该如何让爬虫跳过一些特定的URL，例如(https:/example.comnoindex-url。). 我看过这个，但我无法理解。https:/github.comspatielaravel-sitemap）。

namespace Example\Console\Commands;

use Illuminate\Console\Command;
use Spatie\Sitemap\SitemapGenerator;
use Spatie\Sitemap\Tags\Url;
use Spatie\Crawler\Crawler;

class GenerateSitemap extends Command
{
    /**
     * The console command name.
     *
     * @var string
     */
    protected $signature = 'sitemap:generate';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Generate the sitemap.';

    /**
     * Execute the console command.
     *
     * @return mixed
     */
    public function handle()
    {   
        $siteURL = 'https://example.com';
        SitemapGenerator::create($siteURL)
            ->configureCrawler(function (Crawler $crawler) {
                $crawler->ignoreRobots();
            })
            ->hasCrawled(function (Url $url) {
                if ((string)$url->path() === '/') {
                    return;
                }

                $this->output->writeln('Crawled: ' . (string)$url->path());

                return $url;
            })
            ->writeToFile(public_path('sitemap.xml'));

Answer 1

if ((string)$url->path() === '/noindex-url') {
    return;
}

这就成功了!

这就是现在的样子。

namespace Example\Console\Commands;

use Illuminate\Console\Command;
use Spatie\Sitemap\SitemapGenerator;
use Spatie\Sitemap\Tags\Url;
use Spatie\Crawler\Crawler;

class GenerateSitemap extends Command
{
    /**
     * The console command name.
     *
     * @var string
     */
    protected $signature = 'sitemap:generate';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Generate the sitemap.';

    /**
     * Execute the console command.
     *
     * @return mixed
     */
    public function handle()
    {   
        $siteURL = 'https://example.com';
        SitemapGenerator::create($siteURL)
            ->configureCrawler(function (Crawler $crawler) {
                $crawler->ignoreRobots();
            })
            ->hasCrawled(function (Url $url) {
                if ((string)$url->path() === '/') {
                    return;
                }

                if ((string)$url->path() === '/noindex-url') {
                    return;
                }
                $this->output->writeln('Crawled: ' . (string)$url->path());

                return $url;
            })
            ->writeToFile(public_path('sitemap.xml'));