So, I have come up with the need to collect all the website’s urls crawled by Google. And a great tool I used for that was Netpeak Spider.
You can easily find the download url on the web, I will tell you about some of it’s features:
First of all, – you can scan website, directory, or Google.
That means you are able to collect urls from the whole website (the crowler will go from page to page and find all pages available by links inside your website).
Directory scan will collect all urls within a given directory, for example mysite/directory/
Google scan will find all the pages collected by Google (in it’s main cache, not supplementary cache).
This feature becomes extremely useful, when you have some old pages cached by Google, but not available within your website:
This was really useful while I was scanning a business website Form.com.
The company specializes on producing it’s own tool for data collection and management. Using Form.com tool you are able to collect data, manage users, fields, workflows, forms – all in one place. The process of collecting data can be split within your team, thus making the tool ideal for team works and works in the field. You can visit the website here. You can also see Key Survey for more simplified solutions, focused on survey solutions.