Jump to content

Welcome to Geeks to Go - Register now for FREE

Geeks To Go is a helpful hub, where thousands of volunteer geeks quickly serve friendly answers and support. Check out the forums and get free advice from the experts. Register now to gain access to all of our features, it's FREE and only takes one minute. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more.

Create Account How it Works
Photo

Issue with batch downloading


  • Please log in to reply

#1
Andante

Andante

    Member

  • Member
  • PipPip
  • 11 posts

We regularly batch download files, but recently came across an unusual snag that we can't resolve.

The files online are named like this:
www.sitename.com/report.txt&id=29483
www.sitename.com/report.txt&id=19536
www.sitename.com/report.txt&id=09872

There are many, and each file has a unique five-digit id number.

Batch download programs tend to look at the filename only (in this case "report.txt"), and ignore any extended phrase (one after an ampersand). As such, when we try to download several of the files in question, we end up with report001.txt, report002.txt, report003.txt, ad infinitem.

BUT WE NEED THAT ID NUMBER TO IDENTIFY WHICH FILE IS WHICH.

 

(In the future, we plan to have the programmers name the files differently. But this is what we have to deal with right now.)

 

There are two solutions to this problem. We need a batch download program that will:
1) save the entire URL in the resulting filename, OR
2) give you a history afterwards of which filename came from which URL.

I hope this is clear. Appreciate any help! Our project is on hold until we can resolve this.
 


Edited by Andante, 28 March 2017 - 10:21 PM.

  • 0

Advertisements


#2
SleepyDude

SleepyDude

    Trusted Helper

  • Malware Removal
  • 4,374 posts

Hi,

 

What program do you use to download?

 

I think that can be resolved if you use batch scripting for example and command line download tools like wget or curl


  • 0

#3
Andante

Andante

    Member

  • Topic Starter
  • Member
  • PipPip
  • 11 posts

Yep, Sleepy, you are the second person to recommend wget and curl.

 

We regularly use GetRight and DownThemAll (a Firefox add-on with lots of naming options, but sadly none that works).

 

Trouble is, I don't have the time or energy right now to learn a new scripting language. I looked at each of those, and the number of complex options appears over my head.

 

Sounds like there is not a batch downloader that already does this. And if I go to something like wget or curl, I might have to find someone else to create the script for us.

 

But thanks for the input! Much appreciated.  :-)


  • 0

#4
SleepyDude

SleepyDude

    Trusted Helper

  • Malware Removal
  • 4,374 posts

Hi,

 

Just to understand the problem a little better presently you have the list of URL's exactly on the format that you posted?

 

Wget and curl have many options because they are powerful tools but you don't have to use them all.


  • 0

#5
Andante

Andante

    Member

  • Topic Starter
  • Member
  • PipPip
  • 11 posts

Thanks. Yes, I do have the list of URLs we hoped to download.

 

And of course I know you don't have to use all the options.....it's just I was a bit overwhelmed by the complexity of it all. I would have to wade through a ton to find the right ones we need!  :-)


Edited by Andante, 29 March 2017 - 04:46 PM.

  • 0

#6
Andante

Andante

    Member

  • Topic Starter
  • Member
  • PipPip
  • 11 posts

Bad news, SleepyDude. I eventually found some helpful tips on a website to test wget on my machine. I tried downloading just one single file from the web, and it didn't work.  :-(

 

Check the format in my initial post above, and you'll see that each URL ends in a phrase set off by an ampersand. wget considers this a separate command, unfortunately, and keeps saying it doesn't understand "id."

 

Darn!

 

Guess I should check out curl next. Do you know much about it?


  • 0

#7
SleepyDude

SleepyDude

    Trusted Helper

  • Malware Removal
  • 4,374 posts

Can you provide a live link for testing?

 

On wget you probably have to use -O reportxxx.txt to save the download to a specific file.


  • 0

#8
Andante

Andante

    Member

  • Topic Starter
  • Member
  • PipPip
  • 11 posts

Some of the other links may be confidential, but here's an image that isn't:
https://afremov.com/...type=P&id=19194

 

I don't think -O will solve the problem, since wget doesn't even understand the URL. It sees the ampersand, and thinks it's a separate command (not part of the URL).

 

But I hope I'm wrong and you find a way!

 

Thanks for your help.  :-)


  • 0

#9
SleepyDude

SleepyDude

    Trusted Helper

  • Malware Removal
  • 4,374 posts

Hi,

 

This works

wget -O test.jpeg "https://afremov.com/image.php?type=P&id=19194"

  • 0

#10
Andante

Andante

    Member

  • Topic Starter
  • Member
  • PipPip
  • 11 posts

Sleepy, did you open the downloaded file on your machine? On my machine your link worked, but the downloaded file was 0 bytes.


  • 0

Advertisements


#11
SleepyDude

SleepyDude

    Trusted Helper

  • Malware Removal
  • 4,374 posts

Sleepy, did you open the downloaded file on your machine? On my machine your link worked, but the downloaded file was 0 bytes.

 

Yes 300kb


  • 0

#12
Andante

Andante

    Member

  • Topic Starter
  • Member
  • PipPip
  • 11 posts

Weird! I just tried again a few times (both with and without quote marks), and I keep getting a file of 0 bytes. What could be different on our machines I wonder.....??

Could it be our version of wget? My executable is dated 12-31-08.

 

Any other options (other than -O) I might try?


  • 0

#13
SleepyDude

SleepyDude

    Trusted Helper

  • Malware Removal
  • 4,374 posts

Tested with latest wget from here https://eternallybored.org/misc/wget/

 

used that command from the Command Prompt


  • 0

#14
Andante

Andante

    Member

  • Topic Starter
  • Member
  • PipPip
  • 11 posts

The latest version worked, Sleepy! Yay! In fact, I was even able to download without the -O option.

So my problem was apparently the version of wget I had downloaded. It didn't operate according to the instructions that came with it (!).

 

I'm very grateful for your help. Please let me know if you have any questions I can answer about Excel (my expertise).

Cheers!


Edited by Andante, 30 March 2017 - 06:46 PM.

  • 0

#15
SleepyDude

SleepyDude

    Trusted Helper

  • Malware Removal
  • 4,374 posts

The latest version worked, Sleepy! Yay! In fact, I was even able to download without the -O option.

 

Good.

 

So my problem was apparently the version of wget I had downloaded. It didn't operate according to the instructions that came with it (!).

 

Sometimes it happens because those are tools ported from Linux...

 

I'm very grateful for your help. Please let me know if you have any questions I can answer about Excel (my expertise).

 

If you can keep an eye on the forums, from time to time questions about Excel appear :)


  • 0






Similar Topics

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

As Featured On:

Microsoft Yahoo BBC MSN PC Magazine Washington Post HP