X10 Community Forum
📱🖥️PiX10Hub => 🏥Help and Trouble Shooting => Topic started by: soxfan1966 on May 30, 2020, 07:35:58 AM
-
Good morning,
As I periodically do, I checked for updates for my Raspberry Pi this morning. Seeing these seemed to be an update (see here: https://www.raspberrypi.org/downloads/) I went thru the process of using Putty to connect to my Pi and then go thru the update.
Everything appeared to work fine, and it boots and HG Loads.
However, I am getting an error with the Weather Widget now - its reported that it cannot find "sys" in a string - but after some troubleshooting, appears that this command in the Weather program (both the "standard" and my updated version):
var weatherData = Net.WebService(webserviceurl_conditions).GetData();
is no longer returning any data, so when the rest of the program tries to parse the results it ends up with the missing sys error.
If you add this line:
Program.Notify("weatherData = " + weatherData);
to the weather program right after the line above, and then under Maintenence options turn on Event History you can see that weatherData is nil - it should be a list of data returned from the OpenWeatherMap API call.
I'm not sure if perhaps the Net.WebService on the pi was not started / enabled after the update, or if something in the update changed / broke how that is supposed to work. But I'm hoping someone here may have an answer (or even a suggestion).
Thanks :)
-
Have you tried restarting OW. Have you tried recompiling the OW Program, saving it, disabling and re enabling it. I have seen this problem before usually when the OW server was playing up and the call was returning a null value. As it’s a HG specific issue and not an X10 one you’re probably best reporting it over on HomeGenie.Club.
-
Yes, tried restarting the program and also recompiling. Also did a restart of the pi. None of that fixed the problem.
-
I posted the same question / problem over on the HG Club site too.
-
I posted the same question / problem over on the HG Club site too.
Have you tried to make any of those calls manually via a browser bar. I don’t use the OW program program in HG but I’ll enable it and have a look. What version of Mono are you using. Is it definitely the Stable Release. Are you running upgrade or dist-upgrade when updating your system.
-
Yes, I made the same API calls via the browser and they work fine. The weather program was also getting results before I updated the OS. I'm wondering if a service didn't start.
-
Yes, I made the same API calls via the browser and they work fine. The weather program was also getting results before I updated the OS. I'm wondering if a service didn't start.
Ok I'm getting the same error. I reckon there's a problem at the OW server end. If calls are working via a browser bar the only logical explanation would be an issue validating the key via HG. Changes in security certs can sometimes throw an error like this, but I haven't updated in a while so updating is unlikely to be the source of your problem. I'd give it a while before carrying out any major surgery on your system. Bear in mind you're modifying that program quite a bit too.
-
Ok, thanks for checking - I was getting the same error on the standard OW program too (I thought it was something I did at first). I'm out for the day now so will check it later.
-
I'm at my off grid place and a bit behind with my updates. You may wish to check the jkutils open weather map program just to see if it distributes the same behavior.
-
I'm at my off grid place and a bit behind with my updates. You may wish to check the jkutils open weather map program just to see if it distributes the same behavior.
Is your OW program throwing the same error. The problem isn’t updating as I haven’t done a system update in quite a while. Try restarting your OW program to see if it is fetching current forecasts. OW have made changes recently to their api which clearly won’t get picked up unless the author or someone else edits the OW program in HG to reflect this. From what I can gather the last valid call HG made to OW was at 06.30 your time this morning.
-
Just logged on to my HG and your correct last update time for OW was 6:41 am today. :-[
I should have paid more close attention when I pulled it up last. :-[
-
I can make the same API calls for conditions and forecasts from a browser with no errors and data returned.
If the api changes you refer to at for one all, that also works (I'm using that in addition to the other calls for historical rain data).
The issue seems to be the API calls from within HG using the Net.WebService calls - there is no error in the program when you compile but when it runs you get the missing sys reported because there is no weather data to parse.
I tried again while ago and same error happened. But it appears others, who did not update their OS are getting the same error so it does not appear related to that.
I'm at my off grid place and a bit behind with my updates. You may wish to check the jkutils open weather map program just to see if it distributes the same behavior.
Is your OW program throwing the same error. The problem isn’t updating as I haven’t done a system update in quite a while. Try restarting your OW program to see if it is fetching current forecasts. OW have made changes recently to their api which clearly won’t get picked up unless the author or someone else edits the OW program in HG to reflect this. From what I can gather the last valid call HG made to OW was at 06.30 your time this morning.
-
I can make the same API calls for conditions and forecasts from a browser with no errors and data returned.
If the api changes you refer to at for one all, that also works (I'm using that in addition to the other calls for historical rain data).
The issue seems to be the API calls from within HG using the Net.WebService calls - there is no error in the program when you compile but when it runs you get the missing sys reported because there is no weather data to parse.
I tried again while ago and same error happened. But it appears others, who did not update their OS are getting the same error so it does not appear related to that.
I'm at my off grid place and a bit behind with my updates. You may wish to check the jkutils open weather map program just to see if it distributes the same behavior.
Is your OW program throwing the same error. The problem isn’t updating as I haven’t done a system update in quite a while. Try restarting your OW program to see if it is fetching current forecasts. OW have made changes recently to their api which clearly won’t get picked up unless the author or someone else edits the OW program in HG to reflect this. From what I can gather the last valid call HG made to OW was at 06.30 your time this morning.
There's been no changes to HG (obviously). I've not updated HG recently but from what I've been able to determine from other users is that OW stopped fetching weather forecast updates via HG earlier today. The only logical explanation is that there's been some form change in how OW is handling 3rd party service calls to their server. Yes you're API key is working fine from your browser but if their server is not validating the HG app this may be the issue. Best at this stage if you post this problem over at Genes GitHub and see if he responds to it. That's a problem you'll encounter when an app like HG is not regularly maintained.
-
I posted a new issue over on GitHub
https://github.com/genielabs/HomeGenie/issues/403
-
Since I currently only have access via my phone it is difficult to do much digging. However I was able to load the jkutils open weather version and it works so it would be worth looking into its code for comparisons.
-
The JK utils program does not make an API call - it uses the latitude and longitude user defined values and then a bunch of math calculations to get the data. The issue with the weather program is the API call does not have any data returned.
I'm going to try another API call in a HG program to something like ifttt when I get a min - probably tomorrow. Something I want to play with anyway and I can verify if it may be an HG thing or just a weather API call issue.
-
The JK utils program does not make an API call - it uses the latitude and longitude user defined values and then a bunch of math calculations to get the data. The issue with the weather program is the API call does not have any data returned.
I'm going to try another API call in a HG program to something like ifttt when I get a min - probably tomorrow. Something I want to play with anyway and I can verify if it may be an HG thing or just a weather API call issue.
actualy it does use the api & key, you maybe confusing the open weather program with the other jkutils program which does math to calculate dusk dawn.
-
Ahh, yes you are correct. I don't have the JK utils weather program to look at however.
-
Since I currently only have access via my phone it is difficult to do much digging. However I was able to load the jkutils open weather version and it works so it would be worth looking into its code for comparisons.
You can restart OW program via the phone. Have you done this. What are you using for weather. Are you using the OW that comes as standard with HG. I don't use OW but am trying to help soxfan1966 troubleshoot his issue.
-
Both the standard OpenWeather program and my version have the same issue. I have confirmed that. I do not have the same the JK utils weather program. But I looked on GitHub and the version I saw there uses the same Net.WebService call as the other two programs.
-
Ahh, yes you are correct. I don't have the JK utils weather program to look at however.
I dont use jkutils but as far as I remember it stopped working a while back. It hadn't been updated in quite a while. I'm interested to see that it's still working now. Maybe @bkenobi might have a bit more detail on this. You've put a lot of work into OW recently so I'd like to see this problem resolved for you. If you haven't posted the OW code over on HomeGenie.Club it might be worth doing so. Remember it's not included in the HGBE version of HG.
-
Both the standard OpenWeather program and my version have the same issue. I have confirmed that. I do not have the same the JK utils weather program. But I looked on GitHub and the version I saw there uses the same Net.WebService call as the other two programs.
Yes thats why I was surprised to see jkutils was still working.
-
Both the standard OpenWeather program and my version have the same issue. I have confirmed that. I do not have the same the JK utils weather program. But I looked on GitHub and the version I saw there uses the same Net.WebService call as the other two programs.
Yes thats why I was surprised to see jkutils was still working.
I was surprised to see it reporting correctly myself. I have no problem restarting OW what I was referring to was the inability to play with code sorry I wasn't more clear.
-
Both the standard OpenWeather program and my version have the same issue. I have confirmed that. I do not have the same the JK utils weather program. But I looked on GitHub and the version I saw there uses the same Net.WebService call as the other two programs.
Yes thats why I was surprised to see jkutils was still working.
I was surprised to see it reporting correctly myself. I have no problem restarting OW what I was referring to was the inability to play with code sorry I wasn't more clear.
So you’re saying that the OW program is fetching updated forecasting for you. That’s interesting. You’re the only one reporting back that it’s working for so far. When is the last time you did a system update system.
-
Looked a bit again this morning. The GetData() command appears to be defined in the HG code, which I can't see. There is also a Get Bytes() command that is supposed to return a server code. This from the helper doc:
https://genielabs.github.io/HomeGenie/api/ape/class_home_genie_1_1_automation_1_1_scripting_1_1_net_helper.html
However, that returns nil too.
Not sure what to do at this point.
-
@petera my HG included OW doesn't update, it hasn't since 6:41 am yesterday.
However the jkutils openweather program does update though not as accurate as Genes version was. Temperature is off by +3°C I'll have to sèe if it is pulling from same station.
-
I managed to view the jkutils pull request and it does use net.webservice getdata command as well but works, so I believe there is something else at play here.
-
Over on GitHub, Gene looked into it and found the issue to be the certificate on the OpenWeatherMap side expired. You can see the details here
https://github.com/genielabs/HomeGenie/issues/403
-
I emailed OpenWeatherMap support to see when and if we can expect this to be renewed.
-
As I suspected in my Reply #6 the problem appears to be a validation issue on the OW end. If any program stops working suddenly and so widespread it usually points to security certs issues. According to Genes reply to @soxfan1966 in the GitHub certs did expire on the OW end on May 30 so that would indeed confirm my suspicions. It's now a case of OW updating their security store. I have no doubt that other applications using OW as their weather source will flag this issue too. Maybe Gene as the author might alert them to the problem he is having with his application. For the moment it's a case of sitting back until OW resolve this issue.
@soxfan1966 I noted from your GitHub issue you are using HG 1.1 v527. While its not necessary to be running the latest version there's a number a features you are currently missing present in HG 1.3 v7. A number of them relate directly to X10. You can see what they are from the release change logs on the GitHub.
-
I believe I am running the latest version of HG - I updated recently when Gene had posted his last update.
-
I believe I am running the latest version of HG - I updated recently when Gene had posted his last update.
Ok. You mentioned in the issue you were running 1.1 r527 https://github.com/genielabs/HomeGenie/issues/403
-
While you’re waiting on a fix to your problem you might want to try change the time and date on the Pi to pre 30th May as an experiment and try making calls to the OW server via HG. It should confirm the expiry certs issue for you if you receive back valid data in HG.
-
When you post a issue HG github fills in HG version as 1.1-stable 527 many users over look this and don't change it.
-
It’s a game changer as far a security issues are concerned. There are still users on v527 particularly those who have a long time stable setup and don’t want to tempt fate by upgrading it. Totally understandable.
I don’t see the OW issue being resolved any time soon. I don’t think it’s as straightforward as Gene makes it out to be. Previous security cert issues with HG in the past related to Mono but I think issue relates to Java. I maybe wrong. I can’t see a service like OW letting their certs lapse like that without having previously attended to them.
-
I don’t see the OW issue being resolved any time soon. I don’t think it’s as straightforward as Gene makes it out to be. Previous security cert issues with HG in the past related to Mono but I think issue relates to Java. I maybe wrong. I can’t see a service like OW letting their certs lapse like that without having previously attended to them.
I hope your wrong, however since the jkutils openweathermap works( at least for me) there is an alternative to get OW data. I usually like to have 2 weather pull options available so I may have to look into other sources. Mayby code so if one source fails HG switches to the other for data.
-
Can you share the JK utils weather program with me - I'd like to try that for now
-
Can you share the JK utils weather program with me - I'd like to try that for now
Did you see the workaround I posted on the GitHub. Try it and let me know if it works.
-
Yes, I was actually going to try it right now.
-
Yes, I was actually going to try it right now.
Just make sure you’ve taken an image of your SD card before you try it. The workaround should tie you over until a permanent solution is found. It’s always better to try resolve the issue than look for an alternative. I’m surprised the author Gene didn’t attempt to resolve the issue instead of confirming what we already knew.
-
Can you share the JK utils weather program with me - I'd like to try that for now
you can get it from inside HG using the package manager.
-
OK, I can confirm that the work around (which is what it is right, not a fix - that will hopefully come from OpenWeatherMap when they do something to update their certificate) posted on GitHub here will allow the weather program and widget to work again:
https://github.com/genielabs/HomeGenie/issues/403#issuecomment-636527604
Thank you :)
Tuicemen - I will take a look at the jk utils version, I'm curious why that one still worked and it may be possible to work in a backup using that into the OpenWeatherMap Program.
-
OK, I can confirm that the work around (which is what it is right, not a fix - that will hopefully come from OpenWeatherMap when they do something to update their certificate) posted on GitHub here will allow the weather program and widget to work again:
https://github.com/genielabs/HomeGenie/issues/403#issuecomment-636527604
Thank you :)
Tuicemen - I will take a look at the jk utils version, I'm curious why that one still worked and it may be possible to work in a backup using that into the OpenWeatherMap Program.
Could you update your post to reflect this solution on HomeGenie.Club and GitHub. I don’t use OW but there’s nothing more annoying than looking at a program that doesn’t work.
-
Yup, I did. Thanks
-
Yup, I did. Thanks
Not a bad turnaround with a solution. I imagine you’d have been waiting a while otherwise rofl
-
You were looking for JKUtils which I assume Tuicemen obtained from the old board. Also check the Package Manager in Settings for Earth Tools and Solar Altitude. More useful utilities http://old.homegenie.club:8080/www.homegenie.it/forum/index945d.html?topic=1132.0
-
I downloaded the Jk Utils OpenWeatherData Program (not to be confused with the standard OpenWeatherMap program now in HG).
Looks like the code is doing the same thins, in calling Net.WebService(webserviceurl).GetData();
but in the JK Utils version, that call is within a function called _checkWeather and has a Program.RunAsyncTask(()=> definition to it (I'm not sure what that actually means).
That function is called at the end of the program in this code:
When.WebServiceCallReceived(Program.Module.Domain {...}
and again in this code:
while (Program.IsEnabled) ...
In the OpenWeatherMap program (the one that stopped working because of the certificate issue on Saturday), the call to:
var weatherData = Net.WebService(webserviceurl_conditions).GetData();
is just done within the while (Program.IsEnabled) {...} loop.
Not sure why the JK Utils version did not run into the certificate issue - but I am going to keep it installed (just inactive) in case I need a backup weather program again.
I also have the JK Utils Solar Altitude program already and run / use that one.
Thanks
-
@soxfan1966, once I get back to coding ( in the fall) I plan to dig into this a bit more. Just to difficult (for me) to code from a phone. I had always ran two weather pulls up till HG switched from WU to OW. The HG OW program may be able to be edited with a try/catch and if the SSL pull fails use the jkutils pull or a pull from another source. My preference would be another source as the free sources tend to change things without warning.
Although petera's workarround does work for you disabling security certificates for any length of time is not a good idea in todays world.
-
The difference in the HG included OW pull and the jkutil OW pull is the actual call HG uses https jkutils uses http. So to get arround the security certificate issue with out disabling it in HG change the call to a http address.
I just tested this as it was simple to do from my phone and I now have the HG included OW working. :)%
-
So thats in the variable definition of the url then ?
-
@soxfan1966, once I get back to coding ( in the fall) I plan to dig into this a bit more. Just to difficult (for me) to code from a phone. I had always ran two weather pulls up till HG switched from WU to OW. The HG OW program may be able to be edited with a try/catch and if the SSL pull fails use the jkutils pull or a pull from another source. My preference would be another source as the free sources tend to change things without warning.
Although petera's workarround does work for you disabling security certificates for any length of time is not a good idea in todays world.
What exactly do you code. I’m a little confused here. The OW program has experienced expired security certs as did a number of other major applications this weekend and until the matter is resolved I’ve simply bypassed the need to do a key and cypher handshake between HG and OW.
Clearly the JK Utils Program is not very secure if it didn’t go through a similar experience so I wouldn’t recommend anyone use this if we are talking security matters .In fact as I type here I cannot establish a secure connection to this forum so maybe that’s something you should investigate too.
In the meantime instead of scratching my ass and sending emails to people I’ve got @soxfan1966 back on the road. I don’t use weather in HG but saw a user who had put a lot of effort into developing it for his own needs and it didn’t look like the author was in much of a hurry to provide a solution and I imagine soxfan1966 wanted to get back on track before the Fall.
-
Just to ally any fears about security or contrary to what may have been intimated, my workaround only relates to the security connection between OW and HG. Every other aspect of HG will function as normal. Dropping the s from the https url effectively opens an unsecured connection anyway.
-
@petera, what I code is primarily for my own use though I do share some of my creations. Like you and many others, my HG is very customized. I put forth what I planed to do with my version of OW so others could try it if they wished.
Your post suggests that you were the only one that actualy took time to look into this which is totaly false. It also implies I'm belittling your workarround which I'm not. The fact is jkutils just doesn't use HG certificates designed for https calls. Part of the requiement for Https sites is all http calls get forwarded to https. If you wish Jkutils to use security certificates you just need to change the pull address.
My workarround suggestion is just another option. Users like soxfan1966 are free to use either, use their own or wait (hopefuly not till fall as you suggest)
-
OpenWeaterMap support just got back to me about the expired certificate stating it should be fixed now.
So no need for a workarround now. >!
-
I strongly suggest that anyone using OpenWeather seriously for automation etc takes out a paid subscription. Besides the fact that you have direct access to premium product support, you have additional forecast data that the free subscription doesn’t include and of course you are contributing to a project not owned by one of the big data mining giants. I personally don’t use it but would gladly pay if I did. The “something for nothing” concept that some may find admirable is slowly disappearing, being gobbled up by the “big guys”, Dark Sky weather being the latest acquisition. Pay modestly now or pay big later.
-
Many of the weather sites rely on user PWS info something they get for nothing. For users that rely on accurate weather info for their automation they should look into a PWS then forward the info to a weather site. Most times this will give you access to more content not to mention the most up to date and accurate weather info. I'm not sure what OW supplies to users submiting PWS data as my PWS is no longer operational so I never looked into it.