Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[soxfan1966]

Hello X10 world, hope all is well.  I've noticed some inaccuracies with the forecasts for my Weather Widget from OpenWeatherSource lately and finally had a chance to dig into it.

I've been delving into the world of API calls at work so I thought it was a good chance to play around with some of the API calls that HomeGenie was using for the weather.

Essentially, the calls to OopenWeatherMap are done as follows:

1) Current Conditions:
"https://api.openweathermap.org/data/2.5/weather?q=" + location + "&lang=" + language + "&units=metric&appid=" + apikey

so:
https://api.openweathermap.org/data/2.5/weather?q=Charlton, US&lang=EN&units=metric&appid=<YOUR API KEY HERE>

That returns:
{"coord":{"lon":-71.97,"lat":42.14},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04d"}],"base":"stations","main":{"temp":8.9,"feels_like":0.48,"temp_min":8,"temp_max":11.11,"pressure":1001,"humidity":65},"visibility":16093,"wind":{"speed":9.8,"deg":180,"gust":16.5},"clouds":{"all":90},"dt":1587484856,"sys":{"type":1,"id":5231,"country":"US","sunrise":1587463015,"sunset":1587512138},"timezone":-14400,"id":4932823,"name":"Charlton","cod":200}

Then another call is made for the forecast:
"https://api.openweathermap.org/data/2.5/forecast?q=" + location + "&lang="+language+"&units=metric&appid=" + apikey

so:
https://api.openweathermap.org/data/2.5/forecast?q=Charlton, US&lang=EN&units=metric&appid=<YOUR API KEY HERE>

that returns a bunch of stuff - and if you look at the data and specifically this field:
"dt_txt":"2020-04-22 06:00:00"

you can see that it appears to be a forecast in increments of 3 hours

In the Program code for OpenWeatherMap, there is this:

var nextDay = DateTime.UtcNow.AddDays(1); nextDay = nextDay.Date + new TimeSpan(12, 0, 0);

I believe that is setting the time portion for the date/time stamp to be noon?  Or is it 12 hours from the current time ?

The code then appears to try and find the matching date/time entry for the next 3 days, matching the date and time.  I'm not 100% certain but I am wondering if the way this is trying to match may be off or not.  I would assume the forecast would just be at a set time every day for the next 3 days ?

NOTE: I am also going to post this over on the HomeGenie Board

[soxfan1966]

I played around with the Program code a little yesterday, comparing the results against the hourly data that Wunderground weather uses (here is an https://www.wunderground.com/hourly/us/ma/charlton/KMACHARL11 for my town for hourly for a day as well as this one which shows a graph with hourly forecast for the next few days
https://www.wunderground.com/forecast/us/ma/charlton/KMACHARL11

By changing this line of code in the Weather program:

var nextDay = DateTime.UtcNow.AddDays(1); nextDay = nextDay.Date + new TimeSpan(12, 0, 0);

and changing 12 to 15, I get the data from 3pm.  Changing to 18 gives me the data from 6pm.  Using 6pm, that seems to more closely match the forecast high temp for the days from weather.com.

I think the most accurate solution would actually be to take all of the data returned from the OpenWeatherMap API for the forecast and then comb the hourly entries for the day and pull out the min and max temps from the whole day and then use those values.  However, to get the overall weather forecast (i.e. rain. snow, partly cloudy, etc) you still need a single entry so not sure if that is worth it or not.

Overall, at least for me, I don't think noon makes sense to use as the time of day to get that day's forecast from - its not the coldest nor warmest part of the day so what you end up with is what looks like inaccurate forecasts for the upcoming 3 days.  I think 3pm probably makes the most sense since that is closer to when the warmest part of the day is (though that may also depend on the time of year).

At least knowing that the forecast info is either from noon (using the current Program code) or some other time if you change it allows you to put it in perspective.

Interested in what others may think.  Thanks

[petera]

If weather is really important to you just use this OW option. https://openweathermap.org/api/hourly-forecast

[soxfan1966]

The Hourly API call requires a paid API Key (I had looked at that) - and it would actually make the issue with the widget worse since we would now have 24 hourly forecasts to chose from as opposed to 8.  I am still trying to figure out how to use the 3 hour blocks most effectively.  One idea I had was to have 2 entries for each day - morning and afternoon.

[Tuicemen]

The open weather program is a widget that new users notice first and although requires a setup to pull correct info it should work correctly.
Many don't  utilize the full potential of the OW program. Gene quickly converted the Wunderground wiget to this as free api keys were no longer being issued. Saddly many aspects of the widget were over looked in the conversion.
Several of us have modified this widget to fix missing  code which we telied on and some even posted pull requests.
I've been hoping a HG update would include some of these but even with the latest update ( stable 6) these haven't  been added

[soxfan1966]

I'm going to play around with a few ideas I have when I get some time - either something along the lines of getting the min from one point in the day and the max at another, or possibly displaying two sets of info - one for the morning and one for the afternoon.  Now that I at least understand how the info is being compiled I can work to try and make some improvements with it, minimally for my own interest.  It may be worth it at that point for me to create a new Weather Widget with the change I have, but will see how it goes.

[soxfan1966]

I Finally had some time to play around with this yesterday (what else to do when confined to home and its snowing in May).

Anyway, what I did was update the OpenWeatherMap program and widget so that they compile two sets of data for each forecast day. The first, the morning, is from 9am. The second, the afternoon, is from 3pm. Theoretically, those could be updated as desired and/or additional segments added.

Here is what I have for the widget as of this afternoon, still need to do some debugging.

I also spent some time trying to find a way to fix the temp_min reporting issue (OpenWeatherMap returns the same value for the min temp as the max temp, something tuicemen had pointed out and modified the widget to include the humidity instead).

I tried to post an update yesterday afternoon but the site was down.  The image of the widget below is from my changes yesterday:

[soxfan1966]

I made some more tweaks last night and this morning. I added another row of data - for evening. So now I have morning, afternoon and evening. Originally I had morning at 9am and afternoon at 3pm but this morning I decided to change as follows:
morning = 6am (I am usually up by then so its more useful to me than 9am)
afternoon = noon (I may use 3pm still - will see)
evening = using 6pm.

I was thinking I could add options for these values and allow a choice for the morning as either 6am or 9am, afternoon as either noon or 3pm and evening as either 6pm or 9pm.

I may do that today.

Also - a bug (or maybe an oversight) in the original version of the widget and carried over to mine. The last update info at the bottom is quirky - the date can actually end up being tomorrow in the late afternoon. What I think was happening is that once the next forecast time is past today it will use tomorrow’s date (but still the current time) - its all in the way the string for last update is built. So in mine I removed the date portion. The days being forecast will also get thrown off - so late in the afternoon when the last update date turned to Sunday even though it was still saturday, it was doing the forecast for Monday, Tuesday and Wednesday and not Sunday, Monday and Tuesday. I have fixed this as well - though now it will show the forecast for Sunday even when it is Sunday, at least until the last forecast time if met.

Here is my latest version of the widget:

[Tuicemen]

Nice work.
I'm shocked that Gene has yet to fix issues with the open weather map program and widget as this is what most newbies look at first.
I look foward to trying your  version of this.

[soxfan1966]

As you suggested, I removed the forecast text under the date as it did not apply to all the time frames.  Here is an updated pic:

[petera]

A very useful exercise to familiarise yourself with the inner workings of HG and the results look very impressive indeed.

Contrary to what may be the perceived wisdom, weather forecast data as supplied by the likes of OpenWeather, Dark Sky, Weather Underground etc can be at the best of times arbitrary and can vary quite dramatically depending on what source you choose to get your data from. I certainly wouldn't base any mission critical tasks around their results. In fairness what can people expect from a service provided free.

To get the best out of any weather results in HG I'd recommend building one of those simple Arduino weather projects, connecting it to your Raspberry Pi and extract the results via the Serial I/O program. The results can be read directly into OW or any other weather program you choose.

From reports received back via Discourse during the week, the latest update to HG was made purely to fix the installation package manager and we are not likely to see any further changes to HG. It's not really a problem as HG has gone as far as it can in it's current format so if anyone wants to make any changes to it they are free to do so locally.

Personally I've added two of my own MIG's I've written along with numerous C# automation scripts to such an extent HG is hardly recognisable now. Of course with Python support built in there's so much reusable code out there you could use with minor modifications. I even resolved that old Alexa chestnut that everyone craves so much. 

[Tuicemen]

Thats what I liked with WU and maybe is possible with Open Weather.(I haven't looked into it as my PWS is out of service)
WU allowed you to use data from any PWS so you could select the one closest to your location even your own.
The PWS is the best solution for critical weather triggering as your info pull can be seconds apart.
There are several Commercial PWS available that users don't  have to make their own ( unless they wish too)  Gihub is full of code to implement most of the popular ones into a users setup ( Windows or Linux)

[petera]

No but the challenge is cooking your own. I'm years in Linux and open source software and could never understand all the money people shelled out to the likes of Microsoft and its affiliates to do something they could have done so easily themselves. Who in their right mind uses a desktop computer an the likes of an i5 or i7 to run a home automation system. One particularly that runs an energy monitoring system.

My advice to anyone still running HG is to get motivated and start to learn how to code. Something simple like Python is a good starting point. Plenty of Python libraries about to reuse and it works very well with HG. C# is not as cryptic as you may think and as it is compiled it will run you automations quicker. What code is available on the Old Homegenie Forum will suffice for most if not all automations.

I always thought that HG development was a bit like the weather, unpredictable. All I can add is that if you intend to stick with HG, experiment a little. You'd be amazed how quickly you'll learn the coding side of things and how productive your system will become as a result.

[Tuicemen]

Cooking your own PWS maybe a fun project for some but for many the comercial ones are least expensive in the long run.
I never was a fan of Linux until I started playing with a SBC like a pi.
The open source community is great. However to me it feels like leeching unless I try to improve and or expand upon it and share my works back with the author. Much like soxfan1966 has done.
If it gets implemented in a release or by anyone is just a bonus at least I haven't kept things to myself and I feel I've contributed to the project.

[petera]

The whole point of the X10 hub I imagined was to control X10 devices. That's certainly been achieved. Anything beyond that remit is a completely different project and takes time and energy neither of which I have available at the moment.

As you are retired now maybe you could look into expanding the X10 hub concept yourself. There's enough out there on the web to begin basic coding in C# and any other of the languages that HG supports. It's really the only way to take HG in the direction you feel you wish it to go.

I put many requests in to enhance HG over the years and one or two were implemented as can be seen from the GitHub. I ended up getting my hands dirty and doing most of them in the end myself. You'll just have to do it yourself. Few if any are in a position to contribute any longer and those who are improving HG are doing so for their own purposes.

I really do suggest getting stuck into the coding side of things. It's quite rewarding in the end and if you feel those efforts are worthy enough of course you can share them with others here. You could even set up your own fork of HG and put your own spin on it.