Sunday, September 8, 2013

Running Scrapy from a Shell Script

Running Scrapy from a Shell Script

I'm trying to call a number of spiders from a shell script like so:
#!/bin/bash
source /home/pi/.bashrc
source /usr/local/bin/virtualenvwrapper.sh
workon email-scraper
cdvirtualenv
#Change to dir containing scrapy.cfg
cd ./sitecode/jobscraper
pwd
scrapy crawl site1
scrapy crawl site2
scrapy crawl site3
# Email me the jobs.csv file
python '/home/pi/Documents/jobscraper/scripts/jobs-mail.py'
# Delete the file so a new one is created on next scrape
# sudo rm /home/pi/Documents/jobscraper/csv/jobs.csv
The only part it executes properly is the python script at the end, which
emails me an empty csv file as none of the scrapers have run. I get the
following output when running the above script from BASH
(email-scraper)pi@raspberrypi ~/.virtualenvs/email-scraper/sitecode $ sudo
sh runspiders.sh
ERROR: Environment 'email-scraper' does not exist. Create it with
'mkvirtualenv email-scraper'.
ERROR: no virtualenv active, or active virtualenv is missing
runspiders.sh: line 9: cd: ./sitecode/jobscraper: No such file or directory
/home/pi/.virtualenvs/email-scraper/sitecode
runspiders.sh: line 13: scrapy: command not found
runspiders.sh: line 14: scrapy: command not found
runspiders.sh: line 15: scrapy: command not found
runspiders.sh: line 16: scrapy: command not found
runspiders.sh: line 17: scrapy: command not found
runspiders.sh: line 18: scrapy: command not found
runspiders.sh: line 19: scrapy: command not found
runspiders.sh: line 20: scrapy: command not found
runspiders.sh: line 21: scrapy: command not found
runspiders.sh: line 22: scrapy: command not found
I'm very new to shell scripting. Can anyone illustrate how to make sure I
activate the virtualenv and change into the correct dir before calling the
first spider?

No comments:

Post a Comment