It could be a basic need for many data science projects to extract all the links present in the search engine results. The task can be achieved through many way including Google API’s, Scrappers etc.
One simple and easy way is to use the BeautifulSoup. Following is the python code to extract all the links present in the google search for the user supplied keyword.
– Python 3.5
– Beautiful Soup 4
– Internet connection
[Note: the code might need some syntactic changes if other versions are used]
Code can also be downloaded from the github HERE.
from bs4 import BeautifulSoup import urllib.request # Collect the relevant urls from the search engine for the # user supplied keyword def collect_urls(): # Define and set an agent opener = urllib.request.build_opener() opener.addheaders = [('User-agent', 'Mozilla/5.0')] # Accept a keyword from the user print("Enter the Keyword. [use - instead of space for more than one word]") key = input() # Prepare the search engine query # We are going to visit the google search engine # URL is prepared accordingly # Our result will be from the first page of the search engine only url = "http://www.google.com/search?q="+ key +"&start=" # Open the page and parse through the BeautifulSoup # Parser is set to html. Any other suitable could be used as well page = opener.open(url) soup = BeautifulSoup(page, "html.parser") # Open a file and write all the links found file = open("links.txt", "w") for cite in soup.find_all('cite'): file.write(cite.text) file.write("\n") # Close the file file.close() collect_urls()