Scrapy and Custom Pipelines

scrapy is a popular web-crawling framework for Python that allows you to create web crawlers out of the box. It's a powerful general-purpose tool that, since it allows a lot of customization, has naturally found its way into professional security workflows. Projects such as XSScrapy, an XSS and SQLi scanning tool built on Scrapy, show the underlying base code's adaptability. Unlike the Burp Suite Spider, whose virtue is that it integrates easily with other Burp tools, and Striker, whose value comes in collecting DNS and networking info from its default configuration, Scrapy's appeal is that it can be set up easily and then customized to create any kind of data pipeline.