One of our readers asked me how to scrap stock price from the MarketWatch website using Ruby. Many would say – use tools like wget or curl, which makes total sense, and there are many tutorials on those tools. For example this is a good tutorial.
However, I will do something different today, just to bring another tool to your arsenal.
The tool we will use is called watir.
Watir is a web-driver that uses a live browser to go to the page and interact with it. It can do many cool things like fill in the form, get content of the page elements etc.
For those of you who are familiar with Selenium – watir is just another layer on top of Selenium.
Here is the code to get stock price from MarketWatch website in your terminal .
Scrapping stock price with watir
First thing we need to do is to install watir gem:
gem install watir
Once watir is installed we will do the following:
- Allow user to pass an argument through command line which in our case will be a stock symbol
- Ask watir to go to the MarketWatch website and grab stock price for the symbol
- Scrap stock price and output to the user
- Handle the use case when element cannot be found, or argument is not passed.
Lets jump straight to the code, which in our case is about 15 lines long:
require 'watir' begin if ARGV[0] browser = Watir::Browser.new browser.goto("http://www.marketwatch.com/investing/stock/#{ARGV[0]}") puts browser.element(class: 'lastprice').text.gsub( "n", "" ) browser.close else puts "Please pass a stock symbol" end rescue Watir::Exception::UnknownObjectException => e puts "This stock symbol is not valid!" browser.close end
As you see we are getting an argument (ARGV[0]) and if it exists we open watir browser. Watir then opens website, gets CSS class called ‘lastprice’ (I researched class name by inspecting the website) that holds the price, formats the output by stripping out any new lines, and outputs formatted stock price to the user.
We are also handling exceptions when stock symbol is not valid.
Lets save our ruby code into something like – scrapper.rb, and run it:
ruby scrapper.rb aapl
My output is:
$> ruby scrapper.rb aapl $102.79
Try it and let me know what you think!
Anatoly
Thanks for installing the Bottom of every post plugin by Corey Salzano. Contact me if you need custom WordPress plugins or website design.