When using puppeteer-core, make sure to also provide puppeteerOptions.executablePath. If not set, will default to using puppeteer. puppeteer In case you want to use a different puppeteer library (like puppeteer-core or puppeteer-extra), pass the object here.You can use this to prevent a network peak right at the start. Set this to a value like 100 (0.1 seconds) in case you want some time to pass before another worker is created. workerCreationDelay Time between creation of two workers.monitor If set to true, will provide a small command line output to provide information about the crawling process.timeout Specify a timeout for all tasks.If you use this field, the queued data must be your URL or data must be an object containing a field called url. skipDuplicateUrls If set to true, will skip URLs which were already crawled by the cluster.
sameDomainDelay How much time should pass at minimum between two requests to the same domain.Ignored by tasks queued via Cluster.execute. retryDelay How much time should pass at minimum between the job execution and its retry.retryLimit How often do you want to retry a job before marking it as failed.Defaults to undefined (meaning that puppeteerOptions will be used). If set, puppeteerOptions will be ignored. perBrowserOptions > Object passed to puppeteer.launch for each individual browser.We then queue two jobs and wait for the cluster to finish. Then a task is defined which includes going to the URL and taking a screenshot. A cluster is created with 2 concurrent workers. The following is a typical example of using puppeteer-cluster. Npm install -save puppeteer-cluster Usage Install puppeteer (if you don't already have it installed): Progress view and monitoring statistics (see below).Different concurrency models to choose from (pages, contexts, browsers).Auto restarts the browser in case of a crash.Typings for input/output (via TypeScript Generics).Puppeteer Cluster takes care of reusing Chromium and restarting the browser in case of errors. This is helpful if you want to crawl multiple pages or run tests in parallel. This library spawns a pool of Chromium instances via Puppeteer and helps to keep track of jobs and errors.
0 kommentar(er)
