HultigCrawler is a text crawler that crawls all the text from given website recursively. The crawled data is then saved as items. These items are URL, Title, Tags and Text. This data is then saved into database using scrapy pipelines.
This Software was supported by project C4 - Cloud Computing Competences Centre, financed by the P2020.
