[JosVa]

On my powerconverter for solarpanels with WIFI I can read in Firefox or IE all the variable data like device, serialnumber, timestamp, amount of Kw now, amount of KWh today, etc. I can copy/paste and save the page and/or the data only on selection and then mouseclick.

Using WGET however I only get a file that I could not open until I discovered it to be a ZIPped file. Using 7-ZIP now it shows as content a file called home.html containing only the html program text/lines BUT without any variable data.

 

Why is the content of home.html showing up different in Firefox and after downloading with WGET?

Is there some secured setting I am not yet aware off?

Should I tell WGET do something extra to get those variable data too?

[Advertisements]

Sounds like the "home.html" file is simply a blank form that gets populated with the current data when you access the page with a browser.

[SpywareDr]

Sounds like the "home.html" file is simply a blank form that gets populated with the current data when you access the page with a browser.

[JosVa]

That is new to me. Thought that webpages would be updating itself in automated cycles or by trigger like changing values..But why not?

So you think that when Firefox is asking for access to http://160.190.0.1it triggers the html to look for the data too? To be honest. Did not look at it that way. Will give it a try and see if the data changes without me asking by pushing F5 "renew page". Have to say that in the attached to home.html folder after copy/paste by mouseclick two javascripts are present. And when I run WGET that folder is not not copied/saved next to the zipped and data-empty html. I could imagine that the builders do not like me to scrap the website in the converter. They offer me to send the data to China in a cloud to access it from the other side off the planet.

Do I have to start Firefox with its startup page set to http://160.190.0.1 via for instance a cycling batchfile. Get the data some how out of Firefox and close Firefox. Extract the data I need and let the computer do the next cycle after let us say half an hour? Or Is there a trick to mime this asking for data behaviour with WGET or CURL?

Edited by JosVa, 03 February 2018 - 03:52 PM.

[SpywareDr]

Many web pages are assembled with the latest data, (from various servers, databases, devices, ad servers, etc.), when your browser requests the page.

 

FWIW, 160.190.0.1 is the National Cancer Center in Tokyo, which is maybe 3,200 km from China? http://ipaddress.is/160.190.0.1

[AstraNut]

On my powerconverter for solarpanels with WIFI I can read in Firefox or IE all the variable data like device, serialnumber, timestamp, amount of Kw now, amount of KWh today, etc. I can copy/paste and save the page and/or the data only on selection and then mouseclick.

Using WGET however I only get a file that I could not open until I discovered it to be a ZIPped file. Using 7-ZIP now it shows as content a file called home.html containing only the html program text/lines BUT without any variable data.

 

Why is the content of home.html showing up different in Firefox and after downloading with WGET?

Is there some secured setting I am not yet aware off?

Should I tell WGET do something extra to get those variable data too?

Might take a look at these. Some may be a little old, but you can find a newer source maybe:

 

 

This one lists frequently used options that can be set: Beginners Guide: How To Use Wget

This has more info than just how to download. Might look into how it's setup and whether it is correctr.: How to download your website using WGET for Windows 

Videos of how to use wget

wget vs curl: How to Download Files Using wget and curl

Wget Manual - Examples

 

These may be of NO help whatsoever.

[JosVa]

The IP-adress is the standard "hardware" adress on the converter to get your first connection. Its WIFI is a weak signal that only works within a few meters range to the device. As soon as the distance becomes to big the WIFI drops dead and also in Firefox. In the manual is writen that you should change it to your local router IP and than how to get it and me connected to China.

 

Checked the refreshing of the variable data in Firefox and it looks to be adjusted every half a minute.The versions of wget and curl I downloaded are the latest ones for both Windows and Linux. Next step now was to study the manuals. And then trying out the most suitable commands. And sorry, seached deep in the internet already and found lots more information than offered. My goal is to have a script downloading the data in the background all day long instead of copy/printing from Firefox by hand or reading a cloud far away.

 

My problem is that I do not understand why Firefox is able to read it all day long getting the changing data too and wget does not see those at all?

Edited by JosVa, 04 February 2018 - 07:05 AM.

[SpywareDr]

WGET doesn't access the home.html file same way that a browser (like Firefox) does. Firefox opens it, WGET downloads it. The code within home.html is what causes Firefox to populate the form with the latest data.

[JosVa]

Thanks SpywareDr,

This is what I was seeing and starting to expect. Now I have to find me a way to copy/paste the Firefox context from ram, disk or screen. Made a start with Procmon.exe to see where Firefox uses diskspace and found the folder users/.../appdata/roaming/mozilla/firefox. Will take some time and as we say over here in Holland: Keeps you of the road   .

Edited by JosVa, 04 February 2018 - 11:20 AM.

[SpywareDr]

[JosVa]

In the end I managed to get a readable download of the page using HTTRACK.exe. This is the command line version of WINHTTRACK.EXE.

In the mirror-folder I found the file home-2.html with an oneliner containing data only like devicenumber, versionnumber, Watt coming in, Kwatt generated today, etc. Extracting the W and KW is an easy task and adding the right day and time plus saving it sequential into a file seperated with comma's is done by Pascal v5.5 (grew up with it). All done just by running in a looped batch-file using the ping commans for counting seconds to wait. All is running on an old laptop in XP-home-sp3 and only connected to the secured WIFI  with IP 160.190.0.1.

Reading this CSV file in Excell and creating graphs was the goal of all the labour. For me the command "httrack.exe 160.190.0.1" did the trick running the Java-scripts too where WGET and CURL fail to start those up.

Edited by JosVa, 12 February 2018 - 02:47 AM.

[SpywareDr]

Cool! Great to hear you finally got what you were after.