Charts Compiler

[nossis]

I figured it out! Yippee!!!! It has been on my to do list for a while now (over 6 months).
I use utils more in a shall we say "modular" form.
I have created and are in production state
range charts https://nossis.com.au/weather/rangecharts.htm
miscellaneous charts https://nossis.com.au/weather/misccharts.htm

The bulk of Cumulitis I have sitting here https://nossis.com.au/weather/climate.htm

What got me to this point was using the CutilsCharts.def file that Hans provides as an example https://www.cumuluswiki.org/a/Charts_De ... uage_(CDL) (down the bottom) and also putting in it anything else that I could find that came with the install pack for utils. I wanted to see all that was on offer!
There were some problems in CutilsCharts.def that I had compiled that would cause it not to create the output file/s. The way I found them was to comment everything first, and then uncomment bit by bit. I now have a pretty much fully functional CutilsCharts.def, with some additions and modifications.

My CutilsCharts.def attached for all to slice, dice and modify as they see fit.

CutilsCharts.zip

Using this .def file creates 3 output files as per the sections in the .def file
cumuluscharts.txt - to the unpublished web page https://nossis.com.au/weather/charts.htm (this is a play/testing page)
misccharts.txt - to https://nossis.com.au/weather/misccharts.htm
rangecharts.txt - https://nossis.com.au/weather/rangecharts.htm

Just like to say a big thanks for this module to Hans. It is an awesome tool!
What I have as one of the best graphs that I go to is the Monthly Temperature and Monthly Rainfall. These I think are exceptional!

Would love to see more plot variables added, when the time permits, and also to able to make more charts like Monthly Temperature and Monthly Rainfall using other variables.

Postlude:
To get to run when I choose (just the compiler) I created a batch script file (since I run from a windows computer)
Also setup a scheduled task to automate that to run every 10 minutes

Code: Select all

REM This runs the "cumulusutils.exe" and uploads components as per switches, only if the process CumulusMX.exe is running
cd\
cd C:\CumulusMX
tasklist /FI "IMAGENAME eq CumulusMX.exe" 2>NUL | find /I /N "CumulusMX.exe">NUL
if "%ERRORLEVEL%"=="0" GOTO runutils
GOTO end
:runutils
START /D C:\CumulusMX C:\CumulusMX\utils\bin\cumulusutils.exe CompileOnly
:end
exit

Again thanks to the time and effort that has gone into this
Hope that this can be a bit useful for some.

[HansR]

Hi Barry,

I figured it out! Yippee!!!! It has been on my to do list for a while now (over 6 months).

A post like this makes me truly happy. Thanks for the effort working with the compiler.

There were some problems in CutilsCharts.def [...]

If you think it useful let me know what needs to be changed in the example CutilsCharts.def (on the other hand it made you study )

My CutilsCharts.def attached for all to slice, dice and modify as they see fit.

Thanks!

Just like to say a big thanks for this module to Hans. It is an awesome tool!

(not oops but blushing )

What I have as one of the best graphs that I go to is the Monthly Temperature and Monthly Rainfall. These I think are exceptional!

Thanks but these cannot be made by the compiler and are pre-constructed and parametrized. It would require quite some work to have these made through the compiler.

Would love to see more plot variables added, when the time permits, and also to able to make more charts like Monthly Temperature and Monthly Rainfall using other variables.

Let me know which variables you are thinking of because I think currently (with the addition of the CustomLogs) all values that come out of CMX can be plotted in timeseries now.

Postlude: [...]

Again thanks to the time and effort that has gone into this
Hope that this can be a bit useful for some.

Thanks for the effort using the compiler and sharing your results, that really motivates.
Additional functional and improvement suggestions are always welcome!

I have one additional question: I see you use and include cumulusutils.js as a library in your own website. I think you are the only one doing so (bravo!) . However, as I never anticipated the modular use of the library, you must generate that library by running the website command, a site you don't use. Does that bother you (would you prefer a separate generation of that library)?

[Edit:] Note that there exists a type Height for the Axis. Specifically interesting for the CloudBase (for which now apparently uses Free). It was missing in the Wiki, I corrected that. You use feet as unit for Height. Nobody did that before so that is a first

[nossis]

If you think it useful let me know what needs to be changed in the example CutilsCharts.def (on the other hand it made you study)

made changes in the file example to get it to run out of box.
Removed the separate output file entries to have it all output to the one file for ease of testing. But good to have at least one in the example file to see how that works for learning.

CutilsCharts.website.zip

Noted the difficulty's on the monthly's, figured it was a bit complicated

Let me know which variables you are thinking of because I think currently (with the addition of the CustomLogs) all values that come out of CMX can be plotted in timeseries now.

In the ALL/DAILY ApparentTemp, InsideTemp, Humidex and Heatindex? Are they needed, not really, but nice to have since we could.

I wasn't able to get EVT and EvapoTranspiration, hadn't looked at the logs yet RE that one though.
Also cooling and heating Deg days it tells me I am missing CUserdataALL.json, how is that one generated or created?

Would be nice to have the ability to add a space in Plot label and also the chart label. For example chart drop down list label of "DailyTemp" to be "Daily Temp"

Would be nice to able to specify some more zoom level buttons for the chart. On my CMX generated charts I set some more there.

Again thank you sir

[HansR]

made changes in the file [..]
Noted the difficulty's on the monthly's, figured it was a bit complicated

OK, thanks.

In the ALL/DAILY ApparentTemp, InsideTemp, Humidex and Heatindex? Are they needed, not really, but nice to have since we could.

Well, that was a choice I made a long time ago as the JSON's grow very fast if you start using these as well and as you say: not really needed. But if technology changes (replace the JSON with some more efficient storage/data transfer) I may include those. For now only the important variables - yes, my choice - are used. And Feels Like is there

I wasn't able to get EVT and EvapoTranspiration, hadn't looked at the logs yet RE that one though.
Also cooling and heating Deg days it tells me I am missing CUserdataALL.json, how is that one generated or created?

Those variables are all there (DayEVT!... that name changed). The CUserdataALL.json is created automatically by the UserAskedData command if you use ALL/DAILY plotvariables which CMX does not supply (or did not supply... CMX may have started to supply some later). CUserdataALL.json is only transferred once per day.

Would be nice to have the ability to add a space in Plot label and also the chart label. For example chart drop down list label of "DailyTemp" to be "Daily Temp"

As you will have noticed: the space is the separator character in the CDL definition. That makes it tricky - not impossible - to use a space in those labels. I understand the n eed for that space in the labels though. I'll put it on the ToDo list (low). NOTE: it may be impossible because of javascript variable limitations but I have to look into that.

Would be nice to able to specify some more zoom level buttons for the chart. On my CMX generated charts I set some more there.

I see. As the Zoom specifier exists already I could rework that to specify explicit zoomlevels i.s.o. only the active one of the predefined levels. However those levels are different for RECENT, DAILY and ALL so it is quite detailed and therefore complex in the syntax. I'll put it on the ToDo list (low) with the current situation as default.

[packman2008]

Thank you very much for the examples Barry

I've just downloaded them to investigate how to use them on my site.

[nossis]

Ah good to see that it may be useful for others in some way

Thanks Hans and very much understood in terms of the priority level. No stress if to hard (to long to do), was more if there was things to ask for that would be it.

I need to look into what I am missing as to why I am not getting created the CUserdataALL.json, with using the switch UserAskedData. More than likely something I have missed in the configuration. As much as it is a pain to have something not work, it is also great when one finds the issue, as this is a much greater learning journey as to how it all works

[HansR]

I need to look into what I am missing as to why I am not getting created the CUserdataALL.json, with using the switch UserAskedData. More than likely something I have missed in the configuration. As much as it is a pain to have something not work, it is also great when one finds the issue, as this is a much greater learning journey as to how it all works

I'll look into the CutilsCharts.def and see if there is something wrong. CUserdataALL.json should be generated automatically.
Note it will only be generated once per day!

[nossis]

I was running it manually to see if I get the file.
C:\CumulusMX\utils\bin\cumulusutils.exe UserAskedData
Is it set to only run at a certain time? Or is it a background thing that runs and then when I run utils with switch it uploads it? Or something else?
I should have look at the logs, anything in there that I should be looking for?

[HansR]

I was running it manually to see if I get the file.
C:\CumulusMX\utils\bin\cumulusutils.exe UserAskedData
Is it set to only run at a certain time? Or is it a background thing that runs and then when I run utils with switch it uploads it? Or something else?
I should have look at the logs, anything in there that I should be looking for?

USerAskedData is meant to be run at a certain frequency where there is data (assuming you use the PHP uoload).
If you have a log frequency of e.g. 5 minutes and an Interval (upload frequency) of 10 minutes, I would run it every 10 minutes. There may be an issue when the run does not contain data (e.g. if you run it every minute for the above example) but I can not confirm that atm.

Could you say which frequencies you use?

You can reinitialise the data upload system by removing the parameter LastUploadTime (section [General]) and the parameter DoneToday (section [Compiler]). The parameters will be recreated and all data will be fully uploaded, not incrementally.

The Logs ([...] anything in there that I should be looking for?)... haha, yes there is a lot in there and you're welcome to look at it and analyse, but posting it and me looking at it is probably the quickest way if there is an issue.

See also the Wiki (revised)

[nossis]

Ah yes sir, that was it.
Added the switch UserAskedData to the regular 10 minute scheduled task for cumulusutils (this just calls the batch file to run it with switches)
Removed the time stamp from cumulusutils.ini file for
[General]
LastUploadTime=
and also
[Compiler]
DoneToday=
Ran that task as above
Made the time stamp deletes again
Then ran my compileonly scheduled task
file CUserdataALL.json created at web host

[HansR]

Ah yes sir, that was it.

Good

[HansR]

And Barry, I am working with Chris (see this thread) on the messages through intermediate releases. If you encounter issues with charts and non-understandable error messages, please join in there and let me know. I will then try to improve error messaging for that specific case and release a new beta.

This, while I am working through all possible messages for improvement but that takes some time.

[packman2008]

Hi Barry

First of all, thank you for publishing your def files. They're really helping my understanding of how the Chart Compiler works. I'm past the basics now and trying to understand equations, evals and how they're used in plots. To do this I'm looking at each of the charts you've created to understand how it works.

When looking at the DailyCloudbase chart I noticed a weird thing happening. Sometimes CloudbaseMin is higher than CloudbaseMax which didn't make sense. Having looked at the chart definition and associated equations I now understand why that happens and as a result I think that this chart doesn't really work in practice.

Take CloudbaseMax for example...it's calculated using the maximum temperature and maximum dewpoint which initially seems reasonable. However, as these are DAILY plots there's a pretty good chance that the daily max temp occurs at a different time to the max dewpoint. A good example of this was yesterday at my location. The maximum temp was at midnight and the maximum dewpoint was at 11:45. The maximum cloudbase is therefore being calculated using two values that occurred 12 hours apart. The same happens with minimum cloudbase (my min temp and min dewpoint are currently 3 hours apart) which means that the max and min calculated both have problems. The average cloudbase is calculated from the average of the max and min, so that in turn is also incorrect.

Does my reasoning make sense? It's possible I've still misunderstood the way the compiler is working! I don't see a way to fix this, but if there is a way that would be good to know how to do it.

[packman2008]

Apologies for hi-jacking this thread, but as it's Chart Compiler related and both Barry and myself are experimenting with the compiler I thought this was the best place to put this.

I'm thinking of the example charts, and those that Barry has created, and started to think of potential changes/additions to the compiler. As I don't know how the code works I don't know if these suggestions would be easy, difficult or even impossible to achieve but I'll include them to allow discussion.

1) Contrary to what I said a couple of days ago, I believe that having multiple SMA averaging periods is probably a good idea. The current default of 180 days/periods is probably OK for long-term averaging, but there are other averaging periods that would probably be useful too. For example, I wondered whether averaging FWI over a week or two would be better than taking a 'current' value. If FWI has been high for 2 weeks but on the chart is show somewhat lower in the last day or two that doesn't necessarily indicate that it's much safer. Similarly, averaging temperatures over 180 days probably isn't very meaningful because you'll have averaged across two or probably 3 different season, so that would need a shorter averaging period, and indeed averaging temperatures over 365 days is really getting into climate change territory.

Defining a few extra averaging periods in the .ini file would probably help achieve this, but being able to define the averaging period for a specific chart/plot would also be good, although I don't know how feasible that would be.

2) The next thought is also related to averaging. A week or so ago I made a comment about FWI in this area only having exceeded 250 three times in 45 years. That was based on taking the 45 year FWI plot and looking for the peaks. I now know that was wrong. I think I'm right in saying that the longer period ALL charts take the data set, i.e. the 16425 daily FWI values and average them down to fit on a 650 (?) pixel plot or about 25 days per pixel for my data. This has the effect of diluting any highs (or lows on a low plot). If you look at smaller parts of the 45 year plot there are many more peaks that have 'disappeared' through averaging.

So I wondered whether it would be possible to have additional plot parameters that determined how data is 'compressed' from a larger data set into something that can be plotted. The default would still be AVERAGE which would just take an average value across the compressed period's values. MAXIMUM would take the largest value from the compressed period and MINIMUM would take the smallest value.

3) I also wondered whether it would be possible to create custom plots like the default Monthly Temperature graph? But possibly with different offset periods. For example, plot the current week maximum temperatures and also from the same week of the previous year. I thought of an example for this as I was going to sleep last night but after sleeping all that remains is that I wanted to use data from two different periods. I may have just dreamed this though!

4) Final comment for now...in the wiki it says "The text in the CutilsCharts.def is case independent". I think I've found a case where it's not. If you declare an equation for "TestValuemax" and then use that value in a chart but enter it as "TestValueMax" you get an error on compilation...

2024-01-15 14:52:38.151 Error Term : TestValueMax is neither an existing Plotvariable nor a predefined equation.
2024-01-15 14:52:38.151 Error Parsing User Charts: No Equation found for TestValueMax

[HansR]

@packman2008:

1) I have no problem explaining the ChartsCompiler anywhere

2) I think multiple SMA periods would be smart. I'll think about it on how to do it and put it (low) on the ToDo list

3) I see you remarks about averaging related to the pwsFWI and I don't grasp them fully. But be aware the the pwsFWI if more than an average of the daily output of the formula. The warning signal (the value that is output and determines the colour) is determined by the formula, previous values and rain (a course estimate). So it is not just averaging daily values. So I would have to reread your point 2 more than several times to fully understand what you are trying to do.

4) whether it would be possible to create custom plots like the default Monthly Temperature graph?
I would have to think about that one. First we need to establish the value for the monthly temperature (which is available as webtag from CMX), webtags. You wpuld have to log those values, plot them and put them in an equation and you might get what you want. I think a lot is possible. The CustomLogs give endless possibilities. The best trick is to customlog webtags (offsets possible), generate the charts and copy the plotline to a chart where you can combine it with other lines and use equation s to manipulate a result to what you want. First thing is to define properly what you want to see, then define your basic needs and the equations and then build it. If you have a clear definition of what you want to try I can help you there.

On the other hand: The monthly temperature graphs is not a timeseries but has a totally different x-axis which cannot be created by the compiler. That would require an extension which currently is not foreseen though not impossible.

5) The text in the CutilsCharts.def is case independent. Yes and No It is the keywords that are definitely case independent. Variable names in equations and user text like the titles seem to be case dependent. I now realise and checked in the code, that plot variables in the PLOT statement can be case independent but when used in an equation they are suddenly case dependent. Sorry, I would really have to check to see if this is a bug or intended (probably a bug). To be sure atm and type variables identical always for now (so treat them as case dependent).