How to crawl links on all pages of a web site with Scrapy
I'm learning about scrapy and I'm trying to extract all links that contains: " http://lattes.cnpq.br/andasequenceofnumbers " , example: http://lattes.cnpq.br/0281123427918302 But I don't know what is the page on the web site that contains these information. For example this web site: http://www.ppgcc.ufv.br/ The links that I want are on this page: http://www.ppgcc.ufv.br/?page_id=697 What could I do? I'm trying to use rules but I don't know how to use regular expressions correctly. Thank you 1 EDIT---- I need search on all pages of the main (ppgcc.ufv.br) site the kind of links (http://lattes