[Andante]

We regularly batch download files, but recently came across an unusual snag that we can't resolve.

The files online are named like this:
www.sitename.com/report.txt&id=29483
www.sitename.com/report.txt&id=19536
www.sitename.com/report.txt&id=09872

There are many, and each file has a unique five-digit id number.

Batch download programs tend to look at the filename only (in this case "report.txt"), and ignore any extended phrase (one after an ampersand). As such, when we try to download several of the files in question, we end up with report001.txt, report002.txt, report003.txt, ad infinitem.

BUT WE NEED THAT ID NUMBER TO IDENTIFY WHICH FILE IS WHICH.

 

(In the future, we plan to have the programmers name the files differently. But this is what we have to deal with right now.)

 

There are two solutions to this problem. We need a batch download program that will:
1) save the entire URL in the resulting filename, OR
2) give you a history afterwards of which filename came from which URL.

I hope this is clear. Appreciate any help! Our project is on hold until we can resolve this.
 

Edited by Andante, 28 March 2017 - 10:21 PM.

[Advertisements]

Hi,

 

What program do you use to download?

 

I think that can be resolved if you use batch scripting for example and command line download tools like wget or curl

[SleepyDude]

Hi,

 

What program do you use to download?

 

I think that can be resolved if you use batch scripting for example and command line download tools like wget or curl

[Andante]

Yep, Sleepy, you are the second person to recommend wget and curl.

 

We regularly use GetRight and DownThemAll (a Firefox add-on with lots of naming options, but sadly none that works).

 

Trouble is, I don't have the time or energy right now to learn a new scripting language. I looked at each of those, and the number of complex options appears over my head.

 

Sounds like there is not a batch downloader that already does this. And if I go to something like wget or curl, I might have to find someone else to create the script for us.

 

But thanks for the input! Much appreciated.  :-)

[SleepyDude]

Hi,

 

Just to understand the problem a little better presently you have the list of URL's exactly on the format that you posted?

 

Wget and curl have many options because they are powerful tools but you don't have to use them all.

[Andante]

Thanks. Yes, I do have the list of URLs we hoped to download.

 

And of course I know you don't have to use all the options.....it's just I was a bit overwhelmed by the complexity of it all. I would have to wade through a ton to find the right ones we need!  :-)

Edited by Andante, 29 March 2017 - 04:46 PM.

[Andante]

Bad news, SleepyDude. I eventually found some helpful tips on a website to test wget on my machine. I tried downloading just one single file from the web, and it didn't work.  :-(

 

Check the format in my initial post above, and you'll see that each URL ends in a phrase set off by an ampersand. wget considers this a separate command, unfortunately, and keeps saying it doesn't understand "id."

 

Darn!

 

Guess I should check out curl next. Do you know much about it?

[SleepyDude]

Can you provide a live link for testing?

 

On wget you probably have to use -O reportxxx.txt to save the download to a specific file.

[Andante]

Some of the other links may be confidential, but here's an image that isn't:
https://afremov.com/...type=P&id=19194

 

I don't think -O will solve the problem, since wget doesn't even understand the URL. It sees the ampersand, and thinks it's a separate command (not part of the URL).

 

But I hope I'm wrong and you find a way!

 

Thanks for your help.  :-)

[SleepyDude]

Hi,

 

This works

wget -O test.jpeg "https://afremov.com/image.php?type=P&id=19194"

[Andante]

Sleepy, did you open the downloaded file on your machine? On my machine your link worked, but the downloaded file was 0 bytes.

[SleepyDude]

Sleepy, did you open the downloaded file on your machine? On my machine your link worked, but the downloaded file was 0 bytes.

 

Yes 300kb

[Andante]

Weird! I just tried again a few times (both with and without quote marks), and I keep getting a file of 0 bytes. What could be different on our machines I wonder.....??

Could it be our version of wget? My executable is dated 12-31-08.

 

Any other options (other than -O) I might try?

[SleepyDude]

Tested with latest wget from here https://eternallybored.org/misc/wget/

 

used that command from the Command Prompt

[Andante]

The latest version worked, Sleepy! Yay! In fact, I was even able to download without the -O option.

So my problem was apparently the version of wget I had downloaded. It didn't operate according to the instructions that came with it (!).

 

I'm very grateful for your help. Please let me know if you have any questions I can answer about Excel (my expertise).

Cheers!

Edited by Andante, 30 March 2017 - 06:46 PM.

[SleepyDude]

The latest version worked, Sleepy! Yay! In fact, I was even able to download without the -O option.

 

Good.

 

So my problem was apparently the version of wget I had downloaded. It didn't operate according to the instructions that came with it (!).

 

Sometimes it happens because those are tools ported from Linux...

 

I'm very grateful for your help. Please let me know if you have any questions I can answer about Excel (my expertise).

 

If you can keep an eye on the forums, from time to time questions about Excel appear