Jump to content

Welcome to Geeks to Go - Register now for FREE

Geeks To Go is a helpful hub, where thousands of volunteer geeks quickly serve friendly answers and support. Check out the forums and get free advice from the experts. Register now to gain access to all of our features, it's FREE and only takes one minute. Once registered and logged in, you will be able to create topics, post replies to existing threads, give reputation to your fellow members, get your own private messenger, post status updates, manage your profile and so much more.

Create Account How it Works
Photo

cURL code stopped working!


  • Please log in to reply

#1
Valentino14

Valentino14

    Member

  • Member
  • PipPip
  • 16 posts

Hello, 

 

Some time ago I received very good help on this forum on a problem i ran into. The solution with cURL worked very well for some time, but now it doesn't anymore. What I'm trying to do is to call a URL with the aplication cURL which leads to download of an excel file. The original thread was 

http://www.geekstogo...le-task-online/

 

The cURL code that worked until last week is 

@echo off
setlocal

:: cut time to only HH:MM:SS
set _TIME=%time:~0,8%
:: replace ':' by '_' because ':' isn't valid in file names
set _TIME=%_TIME::=_%
set FN=%date%-%_TIME%
:: File named: c:/Users/Name/Dropbox/Data/RBS NL_yyyy-mm-dd-hh_mm_ss.xls
curl.exe -L -v -J -o "c:/Users/Name/Dropbox/Data/RBS NL_%FN%.xls" "http://www.bnpparibasmarkets.nl/NL/ExcelExport.ashx?ListName=ProductList&config=ActiveProducts&st_search=smVUClJgyncg54s%%2fXHulQrrA%%2blYEamThFGaF9vbZiuJIg20zaZF6lNpn7UgiO%%2b1w5X5yJSeb%%2brJMuLVMD4lJuKs4HzFyqw2QkyMqQOuSjy8JZLSnGk721HD%%2bIy0Nhwb0YIyOzcwVXKZFgAA9NssHoD0ujOsmAwJIX%%2bIYEjGlSKe%%2batJCRfo3YYJ8cFt81ytLMQQNr5HV6CyfX6UkR04utQ%%2bTcc7TqcLKJHDpJcgJexKWarN5Ke5sJm4YFAqzV1BxaVT6JZ8fO0AvQkfkteXrPXsqHR3%%2f0kXZ7%%2bTMXP4ERlw%%3d"

I've also tried with a line to a cookies.txt file, but no succes

:: cut time to only HH:MM:SS
set _TIME=%time:~0,8%
:: replace ':' by '_' because ':' isn't valid in file names
set _TIME=%_TIME::=_%
set FN=%date%-%_TIME%
:: File named: c:/Users/Name/Dropbox/Data/iShares DE_yyyy-mm-dd-hh_mm_ss.xls
curl.exe -L -v -J -b cookiesRBSNL.txt -o "c:/Users/Name/Dropbox/Data/iShares DE_%FN%.xls" "http://www.bnpparibasmarkets.nl/NL/ExcelExport.ashx?ListName=ProductList&config=ActiveProducts&st_search=smVUClJgyncg54s%%2fXHulQrrA%%2blYEamThFGaF9vbZiuJIg20zaZF6lNpn7UgiO%%2b1w5X5yJSeb%%2brJMuLVMD4lJuKs4HzFyqw2QkyMqQOuSjy8JZLSnGk721HD%%2bIy0Nhwb0YIyOzcwVXKZFgAA9NssHoD0ujOsmAwJIX%%2bIYEjGlSKe%%2batJCRfo3YYJ8cFt81ytLMQQNr5HV6CyfX6UkR04utQ%%2bTcc7TqcLKJHDpJcgJexKWarN5Ke5sJm4YFAqzV1BxaVT6JZ8fO0AvQkfkteXrPXsqHR3%%2f0kXZ7%%2bTMXP4ERlw%%3d"

Is there something which i'm overlooking here...? Can you help me to get cURL code working again?

 

Many thanks, help is very much appreciated!

 

Alexandra

 


  • 0

Advertisements


#2
Dragokas

Dragokas

    Malware Expert

  • Expert
  • 29 posts
Hello, Alexandra !
 
I have tried different options of Curl / and also Wget tools, but the result is a same: your link redirected to another xls (1,7 MB) file than browser (4,1 MB).
 
I can offer you another solution that is works for me fine:
 
1. Create somewhere script, for instance: c:\test\my.bat
with a following code:
@echo off
SetLocal EnableExtensions
 
cd /d "%~dp0"
 
set xOS=x64& If "%PROCESSOR_ARCHITECTURE%"=="x86" If Not Defined PROCESSOR_ARCHITEW6432 set xOS=x32
 
if "%xOS%"=="x32" (set "PF=%ProgramFiles%") else (set "PF=%ProgramFiles(x86)%")
 
for /F "UseBackQ delims=" %%a in ("URL.txt") do (
 
  start "" /min "%PF%\Download Master\dmaster.exe" "%%a"
 
)
2. Next, create text file near it with name "URL.txt
Note: check URL.txt file properties - ensure it has name "URL.txt", not "URL.txt.txt" !
Note 2: You should NOT use Unicode codepage ! Ensure it on Notepad's File -> Save as... menu. By default, it doesn't use.
 
Place http links inside URL.txt you want to download.
Each link should be placed on new line.
 
3. Download tool "Download Master". It is available here: http://westbyte.com/dm/
It's Russian tool. So, don't be scary. Click link "Закачать Download Master".
Warning! Install: Uncheck all checkboxes on the last pages of installer (you don't need adware).
Run the tool. It available as icon on tray notify area.
 
4. Menu "Tools" -> Options -> Plugins -> Select "Advanced Scheduler" -> Button "Options" -> click "Add"
a. Task type -> Start Program.
b. Program. Write: cmd.exe
c. Parameters. Write: /c "c:\test\my.bat"
where c:\test\my.bat is a full path to script you saved earlier. 
d. When to run: dialy (I think)
e. Start time: set what you need.
OK -> OK -> OK.
 
5. Then you should configure this tool to download files on certain location: your Dropbox folder
Tools -> Options -> Downloads -> Current folder
 
6. And optionally, you can mark checkboxes on its options to download files without warnings about file replacing). I'm sure, you'll find it ('User interface' category).
 
If you do it right, scheduler should work well.
 

  • 0

#3
terry1966

terry1966

    Member 1K

  • Member
  • PipPipPipPip
  • 1,012 posts

the command
 

curl.exe -L -v -J -o "c:/Users/Name/Dropbox/Data/RBS NL_%FN%.xls" "http://www.bnpparibasmarkets.nl/NL/ExcelExport.ashx?ListName=ProductList&config=ActiveProducts&st_search=smVUClJgyncg54s%%2fXHulQrrA%%2blYEamThFGaF9vbZiuJIg20zaZF6lNpn7UgiO%%2b1w5X5yJSeb%%2brJMuLVMD4lJuKs4HzFyqw2QkyMqQOuSjy8JZLSnGk721HD%%2bIy0Nhwb0YIyOzcwVXKZFgAA9NssHoD0ujOsmAwJIX%%2bIYEjGlSKe%%2batJCRfo3YYJ8cFt81ytLMQQNr5HV6CyfX6UkR04utQ%%2bTcc7TqcLKJHDpJcgJexKWarN5Ke5sJm4YFAqzV1BxaVT6JZ8fO0AvQkfkteXrPX

works fine for me except i changed
 

"c:/Users/Name/Dropbox/Data/RBS NL_%FN%.xls"

to a local location on my linux pc,

 

so without knowing what error message your getting i'm assuming your problem has something to do with where your trying to save the file.

 

sorry don't have a windows pc to test things on, but have sent sleepydude a link to this topic to make sure he sees it and maybe he can work out where your problem may be.

 

:popcorn:


  • 0

#4
SleepyDude

SleepyDude

    Trusted Helper

  • Malware Removal
  • 4,149 posts

Hi Alexandra,

 

It seems that some changes happened on the site and now the original URL is redirect to https, curl complains about this but it could be solved by using the switch to ignore the certificates.

 

The following works for me.

curl.exe -k -L -v -J -o Report.xls "https://www.bnpparibasmarkets.nl/NL/ExcelExport.ashx?ListName=ProductList&config=ActiveProducts&st_search=smVUClJgyncg54s^%^%2fXHulQrrA^%^%2blYEamThFGaF9vbZiuJIg20zaZF6lNpn7UgiO^%^%2b1w5X5yJSeb^%^%2brJMuLVMD4lJuKs4HzFyqw2QkyMqQOuSjy8JZLSnGk721HD^%^%2bIy0Nhwb0YIyOzcwVXKZFgAA9NssHoD0ujOsmAwJIX^%^%2bIYEjGlSKe^%^%2batJCRfo3YYJ8cFt81ytLMQQNr5HV6CyfX6UkR04utQ^%^%2bTcc7TqcLKJHDpJcgJexKWarN5Ke5sJm4YFAqzV1BxaVT6JZ8fO0AvQkfkteXrPXsqHR3^%^%2f0kXZ7^%^%2bTMXP4ERlw^%^%3d"

  • 0

#5
terry1966

terry1966

    Member 1K

  • Member
  • PipPipPipPip
  • 1,012 posts

just curious but did you need to use the -k option, just adding the s to http using the original code also works for me on linux.

 

:popcorn:


Edited by terry1966, 05 May 2016 - 03:57 PM.

  • 0

#6
SleepyDude

SleepyDude

    Trusted Helper

  • Malware Removal
  • 4,149 posts

Not for me!

* Hostname was NOT found in DNS cache
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*
  Trying 217.110.44.164...
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*
Connected to www.bnpparibasmarkets.nl (217.110.44.164) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS alert, Server hello (2):
} [data not shown]
* SSL certificate problem: unable to get local issuer certificate
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

  • 0

#7
Valentino14

Valentino14

    Member

  • Topic Starter
  • Member
  • PipPip
  • 16 posts

first of all, thanks guys for all your hard thoughts again! SleepyDude, hope you are doing well??

 

Adding the -k flag as suggested by SleepyDude really did the trick. I've been playing around with other options such as cookies but this wasn't working. 

Not sure what this -k does or which changes were made to the website...? 

 

Thanks again, so happy that it works again!

 

Alexandra


  • 0

#8
terry1966

terry1966

    Member 1K

  • Member
  • PipPipPipPip
  • 1,012 posts

the -k option turns off certificate checking/verification.

 

http is unsecured connection to web sites and https is secure connection to web sites. more info on the difference here :- https://www.instants...ucts/https.html

 

:popcorn:


  • 0






Similar Topics

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

As Featured On:

Microsoft Yahoo BBC MSN PC Magazine Washington Post HP