datalad-crawler extension

This is a DataLad extension that allows you to crawl external web resources into an automated data distribution. It provides functionality for tracking data on a website and make its files available on a local machine, as well as for querying for potential updates to the website and obtaining any changes.