Possible to automate download from web?

S

stdavid

New Member
Hello all.

After doing this for the umpteenth time, I was wondering whether it would be possible to automate the process, say once a day?

Currently I'm doing this manually via telnet

Code: 
cd /media/"My Video"/downloads
wget -r -A.avi -nd http://domain.com/dir

Is it possible to run this as a cronjob on the humax (although I've no idea how to set it!)

Thoughts?
 
stdavid said:
I was wondering whether it would be possible to automate the process, say once a day?
Click to expand...
Yes.
Code: 
cd /media/"My Video"/downloads
wget -r -A.avi -nd http://domain.com/dir
Is it possible to run this as a cronjob on the humax (although I've no idea how to set it!)
Click to expand...
Add this to /mod/var/spool/cron/crontabs/root
Code: 
mm hh * * * wget -r -A.avi -nd -P "/media/My Video/downloads" http://domain.com/dir
replacing mm and hh with the minutes and hours respectively of the time you want it to fire.
 
Thanks prpr

I don't know much, just enough to mess things up!

Do I add the line with

Code: 
 nano /mod/var/spool/cron/crontabs/root
?

And I guess I'll get lots of duplicates if I set the cron job too frequent before deleting from the web server?

Thanks.
 
Saying you don't know is not the same as guessing that something will happen.
(Actually I probably do know, but without knowing exactly what it was the OP was trying to do, then I thought it safest to be non-committal.)
 
It does download duplicates (xxx.avi.1 files) but they seem to disappear after a successful download. I've managed to get around it by using the no clobber flag. So, for any future google searchers looking to do something similar, put the following in your cron

Code: 
30 3 * * * wget -r -A.avi -nd -nc -o "/media/My Video/downloads/wget.log" -P "/media/My Video/downloads" http://domain.co.uk/folder
 
Logs are probably better put in /mod/tmp or /var/log depending on whether you want them to persist across a reboot or not.
 
If you are often changing the download URL's I'd suggest sticking it all in one script & running that from cron. Any typo's are less likely to break other cronjobs when adding new URL's and you don't have to set each job to a new time, they will just run sequentially in a script. I'm assuming you'd prefer sequential not parallel downloads because of less file fragmentation (<-maybe?) and to be kinder to other network users.

Assuming /usr/bin is the best place for user scripts…
Code: 
nano /usr/bin/wgetter.sh
#!/bin/sh
wget -r -A.avi -nd -nc -o "/var/wgetter.log" -P "/media/My Video/downloads" http://domain.co.uk/folder
wget -r -B.avi -nd -nc -o "/var/wgetter.log" -P "/media/My Video/downloads" http://domain.co.uk/folder2
Save that & make it executable …
Code: 
chmod a+x /usr/bin/wgetter.sh
Set the cronjob to run the script.
Code: 
30 3 * * * /usr/bin/wgetter.sh
Then you can just change the script & leave the crontab alone when you have new files to get :)
 
prpr said:
/usr/bin is read-only.
/mod/usr/bin is a much better bet.
Click to expand...

Thanks, I saw /mod/… were in the $PATH but /mod/usr/ doesn't exist for me. I guess a
Code: 
mkdir -p /mod/usr/bin
will resolve that.

I have read-write for /usr/bin…
drwxr-xr-x 2 root root 127 Jan 4 2012 /usr/bin
but then I am ssh'd as root. Do you have another user setup prpr?
 
There is only one security context - everything effectively operates as root (FTP aside). /usr/bin is in flash and is thus not writeable for normal operations. The permissions you quoted are irrelevant!
I think I created /mod/usr/bin for my own use several months ago.
 
You must log in or register to reply here.
Back
Top