[Valentino14]

Hello, 

 

Some time ago I received very good help on this forum on a problem i ran into. The solution with cURL worked very well for some time, but now it doesn't anymore. What I'm trying to do is to call a URL with the aplication cURL which leads to download of an excel file. The original thread was 

http://www.geekstogo...le-task-online/

 

The cURL code that worked until last week is 

@echo off
setlocal

:: cut time to only HH:MM:SS
set _TIME=%time:~0,8%
:: replace ':' by '_' because ':' isn't valid in file names
set _TIME=%_TIME::=_%
set FN=%date%-%_TIME%
:: File named: c:/Users/Name/Dropbox/Data/RBS NL_yyyy-mm-dd-hh_mm_ss.xls
curl.exe -L -v -J -o "c:/Users/Name/Dropbox/Data/RBS NL_%FN%.xls" "http://www.bnpparibasmarkets.nl/NL/ExcelExport.ashx?ListName=ProductList&config=ActiveProducts&st_search=smVUClJgyncg54s%%2fXHulQrrA%%2blYEamThFGaF9vbZiuJIg20zaZF6lNpn7UgiO%%2b1w5X5yJSeb%%2brJMuLVMD4lJuKs4HzFyqw2QkyMqQOuSjy8JZLSnGk721HD%%2bIy0Nhwb0YIyOzcwVXKZFgAA9NssHoD0ujOsmAwJIX%%2bIYEjGlSKe%%2batJCRfo3YYJ8cFt81ytLMQQNr5HV6CyfX6UkR04utQ%%2bTcc7TqcLKJHDpJcgJexKWarN5Ke5sJm4YFAqzV1BxaVT6JZ8fO0AvQkfkteXrPXsqHR3%%2f0kXZ7%%2bTMXP4ERlw%%3d"

I've also tried with a line to a cookies.txt file, but no succes

:: cut time to only HH:MM:SS
set _TIME=%time:~0,8%
:: replace ':' by '_' because ':' isn't valid in file names
set _TIME=%_TIME::=_%
set FN=%date%-%_TIME%
:: File named: c:/Users/Name/Dropbox/Data/iShares DE_yyyy-mm-dd-hh_mm_ss.xls
curl.exe -L -v -J -b cookiesRBSNL.txt -o "c:/Users/Name/Dropbox/Data/iShares DE_%FN%.xls" "http://www.bnpparibasmarkets.nl/NL/ExcelExport.ashx?ListName=ProductList&config=ActiveProducts&st_search=smVUClJgyncg54s%%2fXHulQrrA%%2blYEamThFGaF9vbZiuJIg20zaZF6lNpn7UgiO%%2b1w5X5yJSeb%%2brJMuLVMD4lJuKs4HzFyqw2QkyMqQOuSjy8JZLSnGk721HD%%2bIy0Nhwb0YIyOzcwVXKZFgAA9NssHoD0ujOsmAwJIX%%2bIYEjGlSKe%%2batJCRfo3YYJ8cFt81ytLMQQNr5HV6CyfX6UkR04utQ%%2bTcc7TqcLKJHDpJcgJexKWarN5Ke5sJm4YFAqzV1BxaVT6JZ8fO0AvQkfkteXrPXsqHR3%%2f0kXZ7%%2bTMXP4ERlw%%3d"

Is there something which i'm overlooking here...? Can you help me to get cURL code working again?

 

Many thanks, help is very much appreciated!

 

Alexandra

 

[Advertisements]

Hello, Alexandra !

 

I have tried different options of Curl / and also Wget tools, but the result is a same: your link redirected to another xls (1,7 MB) file than browser (4,1 MB).

 

I can offer you another solution that is works for me fine:

 

1. Create somewhere script, for instance: c:\test\my.bat

with a following code:

@echo off
SetLocal EnableExtensions
 
cd /d "%~dp0"
 
set xOS=x64& If "%PROCESSOR_ARCHITECTURE%"=="x86" If Not Defined PROCESSOR_ARCHITEW6432 set xOS=x32
 
if "%xOS%"=="x32" (set "PF=%ProgramFiles%") else (set "PF=%ProgramFiles(x86)%")
 
for /F "UseBackQ delims=" %%a in ("URL.txt") do (
 
  start "" /min "%PF%\Download Master\dmaster.exe" "%%a"
 
)

2. Next, create text file near it with name "URL.txt" 

Note: check URL.txt file properties - ensure it has name "URL.txt", not "URL.txt.txt" !

Note 2: You should NOT use Unicode codepage ! Ensure it on Notepad's File -> Save as... menu. By default, it doesn't use.

 

Place http links inside URL.txt you want to download.

Each link should be placed on new line.

 

3. Download tool "Download Master". It is available here: http://westbyte.com/dm/

It's Russian tool. So, don't be scary. Click link "Закачать Download Master".

Warning! Install: Uncheck all checkboxes on the last pages of installer (you don't need adware).

Run the tool. It available as icon on tray notify area.

 

4. Menu "Tools" -> Options -> Plugins -> Select "Advanced Scheduler" -> Button "Options" -> click "Add"

a. Task type -> Start Program.

b. Program. Write: cmd.exe

c. Parameters. Write: /c "c:\test\my.bat"

where c:\test\my.bat is a full path to script you saved earlier. 

d. When to run: dialy (I think)

e. Start time: set what you need.

OK -> OK -> OK.

 

5. Then you should configure this tool to download files on certain location: your Dropbox folder

Tools -> Options -> Downloads -> Current folder

 

6. And optionally, you can mark checkboxes on its options to download files without warnings about file replacing). I'm sure, you'll find it ('User interface' category).

 

If you do it right, scheduler should work well.

 

[Dragokas]

Hello, Alexandra !

 

I have tried different options of Curl / and also Wget tools, but the result is a same: your link redirected to another xls (1,7 MB) file than browser (4,1 MB).

 

I can offer you another solution that is works for me fine:

 

1. Create somewhere script, for instance: c:\test\my.bat

with a following code:

@echo off
SetLocal EnableExtensions
 
cd /d "%~dp0"
 
set xOS=x64& If "%PROCESSOR_ARCHITECTURE%"=="x86" If Not Defined PROCESSOR_ARCHITEW6432 set xOS=x32
 
if "%xOS%"=="x32" (set "PF=%ProgramFiles%") else (set "PF=%ProgramFiles(x86)%")
 
for /F "UseBackQ delims=" %%a in ("URL.txt") do (
 
  start "" /min "%PF%\Download Master\dmaster.exe" "%%a"
 
)

2. Next, create text file near it with name "URL.txt" 

Note: check URL.txt file properties - ensure it has name "URL.txt", not "URL.txt.txt" !

Note 2: You should NOT use Unicode codepage ! Ensure it on Notepad's File -> Save as... menu. By default, it doesn't use.

 

Place http links inside URL.txt you want to download.

Each link should be placed on new line.

 

3. Download tool "Download Master". It is available here: http://westbyte.com/dm/

It's Russian tool. So, don't be scary. Click link "Закачать Download Master".

Warning! Install: Uncheck all checkboxes on the last pages of installer (you don't need adware).

Run the tool. It available as icon on tray notify area.

 

4. Menu "Tools" -> Options -> Plugins -> Select "Advanced Scheduler" -> Button "Options" -> click "Add"

a. Task type -> Start Program.

b. Program. Write: cmd.exe

c. Parameters. Write: /c "c:\test\my.bat"

where c:\test\my.bat is a full path to script you saved earlier. 

d. When to run: dialy (I think)

e. Start time: set what you need.

OK -> OK -> OK.

 

5. Then you should configure this tool to download files on certain location: your Dropbox folder

Tools -> Options -> Downloads -> Current folder

 

6. And optionally, you can mark checkboxes on its options to download files without warnings about file replacing). I'm sure, you'll find it ('User interface' category).

 

If you do it right, scheduler should work well.

 

[terry1966]

the command
 

curl.exe -L -v -J -o "c:/Users/Name/Dropbox/Data/RBS NL_%FN%.xls" "http://www.bnpparibasmarkets.nl/NL/ExcelExport.ashx?ListName=ProductList&config=ActiveProducts&st_search=smVUClJgyncg54s%%2fXHulQrrA%%2blYEamThFGaF9vbZiuJIg20zaZF6lNpn7UgiO%%2b1w5X5yJSeb%%2brJMuLVMD4lJuKs4HzFyqw2QkyMqQOuSjy8JZLSnGk721HD%%2bIy0Nhwb0YIyOzcwVXKZFgAA9NssHoD0ujOsmAwJIX%%2bIYEjGlSKe%%2batJCRfo3YYJ8cFt81ytLMQQNr5HV6CyfX6UkR04utQ%%2bTcc7TqcLKJHDpJcgJexKWarN5Ke5sJm4YFAqzV1BxaVT6JZ8fO0AvQkfkteXrPX

works fine for me except i changed
 

"c:/Users/Name/Dropbox/Data/RBS NL_%FN%.xls"

to a local location on my linux pc,

 

so without knowing what error message your getting i'm assuming your problem has something to do with where your trying to save the file.

 

sorry don't have a windows pc to test things on, but have sent sleepydude a link to this topic to make sure he sees it and maybe he can work out where your problem may be.

 

[SleepyDude]

Hi Alexandra,

 

It seems that some changes happened on the site and now the original URL is redirect to https, curl complains about this but it could be solved by using the switch to ignore the certificates.

 

The following works for me.

curl.exe -k -L -v -J -o Report.xls "https://www.bnpparibasmarkets.nl/NL/ExcelExport.ashx?ListName=ProductList&config=ActiveProducts&st_search=smVUClJgyncg54s^%^%2fXHulQrrA^%^%2blYEamThFGaF9vbZiuJIg20zaZF6lNpn7UgiO^%^%2b1w5X5yJSeb^%^%2brJMuLVMD4lJuKs4HzFyqw2QkyMqQOuSjy8JZLSnGk721HD^%^%2bIy0Nhwb0YIyOzcwVXKZFgAA9NssHoD0ujOsmAwJIX^%^%2bIYEjGlSKe^%^%2batJCRfo3YYJ8cFt81ytLMQQNr5HV6CyfX6UkR04utQ^%^%2bTcc7TqcLKJHDpJcgJexKWarN5Ke5sJm4YFAqzV1BxaVT6JZ8fO0AvQkfkteXrPXsqHR3^%^%2f0kXZ7^%^%2bTMXP4ERlw^%^%3d"

[terry1966]

just curious but did you need to use the -k option, just adding the s to http using the original code also works for me on linux.

 

Edited by terry1966, 05 May 2016 - 03:57 PM.

[SleepyDude]

Not for me!

* Hostname was NOT found in DNS cache
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*
  Trying 217.110.44.164...
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*
Connected to www.bnpparibasmarkets.nl (217.110.44.164) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS alert, Server hello (2):
} [data not shown]
* SSL certificate problem: unable to get local issuer certificate
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

[Valentino14]

first of all, thanks guys for all your hard thoughts again! SleepyDude, hope you are doing well??

 

Adding the -k flag as suggested by SleepyDude really did the trick. I've been playing around with other options such as cookies but this wasn't working. 

Not sure what this -k does or which changes were made to the website...? 

 

Thanks again, so happy that it works again!

 

Alexandra

[terry1966]

the -k option turns off certificate checking/verification.

 

http is unsecured connection to web sites and https is secure connection to web sites. more info on the difference here :- https://www.instants...ucts/https.html