Welcome to the Cumulus Support forum.

Latest Cumulus MX V3 release 3.28.6 (build 3283) - 21 March 2024

Cumulus MX V4 beta test release 4.0.0 (build 4017) - 17 March 2024

Legacy Cumulus 1 release v1.9.4 (build 1099) - 28 November 2014 (a patch is available for 1.9.4 build 1099 that extends the date range of drop-down menus to 2030)

Download the Software (Cumulus MX / Cumulus 1 and other related items) from the Wiki

Automating the upload of dayfile.txt

Discussion and support for 3rd-party (non-Sandaysoft) tools for Cumulus
User avatar
daj
Posts: 2041
Joined: Tue 29 Jul 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

Automating the upload of dayfile.txt

Post by daj »

I've written a little article in the Wiki on how to automatically upload dayfile.txt to your website once per day.

It's a simple method, using the Windows scheduler task. I have tested it on Windows XP and Windows 7

Enjoy

http://wiki.sandaysoft.com/a/Upload_Dayfile
David
kippfordweather.uk
Cumulus MX & Raspberry Pi
Gina
Posts: 1885
Joined: Sat 21 Feb 2009 12:41 pm
Weather Station: Nothing working ATM - making one
Operating System: OS X, Linux Mint, Win7 & XP
Location: Devon UK

Re: Automating the upload of dayfile.txt

Post by Gina »

That looks very useful, thank you :) I must get round to improving my Cumulus web pages - it's all very basic ATM.
Gina

Sorry, no banner - weather station out of action. Hoping to be up and running with a new home-made one soon.
User avatar
Super-T
Posts: 890
Joined: Tue 09 Sep 2008 3:37 am
Weather Station: wh-1081
Operating System: Weather Laptop - Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: Automating the upload of dayfile.txt

Post by Super-T »

GMTA David
I was doing exactly the same thing yesterday but ran into a problem.
I could get all the way to the PUT command, the file is created on the server but then it just sits there showing 150 Ok to send data. ctrl C will abort and it will exit.
I think something is blocking at the server as the file has been created but with 0 content. I haven't got back to it yet to find the problem but any suggestions would be gratefully received. :-)
User avatar
mcrossley
Posts: 12689
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Automating the upload of dayfile.txt

Post by mcrossley »

Terry, do you need to upload 'put' to a temporary filename, 'delete' dayfile.txt, 'rename' temp to dayfile?

Cumulus uses two different mechanisms for its FTPs because of variations in server support for rename/overwrite etc

ultimate fall back would be 'delete', 'put' I guess.
User avatar
Super-T
Posts: 890
Joined: Tue 09 Sep 2008 3:37 am
Weather Station: wh-1081
Operating System: Weather Laptop - Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: Automating the upload of dayfile.txt

Post by Super-T »

Hi Mark
My Cumulus settings are "Auto update" & "Use FTP Rename" ticked.

Delete and put sounds good....i'll give it a try.

OK....tried adding delete although that probably wasn't a problem because I had tried removing the file from the server before running the script.
Partial result:-
230 Login successful.
ftp> debug
Debugging On .
ftp> lcd c:\!Downloads
Local directory now C:\!Downloads.
ftp> quote PASV
---> PASV
227 Entering Passive Mode (202,174,112,121,240,209)
ftp> binary
---> TYPE I
200 Switching to Binary mode.
ftp> cd www/weather/data
---> CWD www/weather/data
250 Directory successfully changed.
ftp> delete www/weather/data/test.txt
---> DELE www/weather/data/test.txt
550 Delete operation failed.
ftp> put c:\!Downloads\test.txt
---> PORT 10,1,1,200,216,157
200 PORT command successful. Consider using PASV.
---> STOR test.txt
150 Ok to send data.

Hangs on that last line and ctrl C to get out of it.
Creates the file but no content.

Also ran the delete line initially as delete test.txt as I was already in the correct directory, but with no joy.


Later in the day. My Host says there is a problem with PASV mode not supported in Windows FTP programme. http://support.microsoft.com/kb/271078/en-us. I've also tried "MoveMeFreely FTP programme that does support PASV but still no joy.

It looks like there is a problem when the host runs PASV mode which most probably do.
tjaliwalpa
Posts: 293
Joined: Sun 18 Apr 2010 9:47 am
Weather Station: Davis VP2
Operating System: Linux Lite Ubuntu 16.04
Location: Karoonda, SA
Contact:

Re: Automating the upload of dayfile.txt

Post by tjaliwalpa »

daj wrote:I've written a little article in the Wiki on how to automatically upload dayfile.txt to your website once per day.

It's a simple method, using the Windows scheduler task. I have tested it on Windows XP and Windows 7

Enjoy

http://wiki.sandaysoft.com/a/Upload_Dayfile
Thanks for this information. It is just what I was looking for. I have a little problem with it however.

I've set up the script and batch file as per the wiki instructions. I have checked and rechecked the variables and am sure I have them correct.

The dayfile.txt is deleted on the server and replaced with another version which is empty. The ftp process times out without uploading the complete file.

What am I doing wrong?

Attached is a screen shot of the cmd.
You do not have the required permissions to view the files attached to this post.
Bob
User avatar
daj
Posts: 2041
Joined: Tue 29 Jul 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

Re: Automating the upload of dayfile.txt

Post by daj »

Hi Bob

In your screen shot you see it doing a "cd e:\Cumulus"

This is wrong -- at that point your are on the webserver and there is no e:\cumulus folder. It should be the directory name as set in your Cumulus Internet Config within Cumulus

Not sure if that is the cause of the over-all problem but it certainly will not help as you need to be in the correct place. Of course, your Directory option in Cumulus could be empty, in which case the CD line in the script can be removed
David
kippfordweather.uk
Cumulus MX & Raspberry Pi
User avatar
Super-T
Posts: 890
Joined: Tue 09 Sep 2008 3:37 am
Weather Station: wh-1081
Operating System: Weather Laptop - Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: Automating the upload of dayfile.txt

Post by Super-T »

Should be doing LCD e:\cumulus\data perhaps but if you are using put e:\cumulus\data\dayfile.txt then you don't need to do the lcd e:\cumulus\data I think?
tjaliwalpa
Posts: 293
Joined: Sun 18 Apr 2010 9:47 am
Weather Station: Davis VP2
Operating System: Linux Lite Ubuntu 16.04
Location: Karoonda, SA
Contact:

Re: Automating the upload of dayfile.txt

Post by tjaliwalpa »

daj wrote:Hi Bob

In your screen shot you see it doing a "cd e:\Cumulus"

This is wrong -- at that point your are on the webserver and there is no e:\cumulus folder. It should be the directory name as set in your Cumulus Internet Config within Cumulus

Not sure if that is the cause of the over-all problem but it certainly will not help as you need to be in the correct place. Of course, your Directory option in Cumulus could be empty, in which case the CD line in the script can be removed
I see now. There is no directory as my weather site is in the root directory of the webspace. In other words, the Cumulus directory is blank.

I've changed this line to cd / with the following result:

ftp> cd /
250 OK. Current directory is /
ftp> put e:\cumulus\data\dayfile.txt
200 PORT command successful
425 Could not open data connection to port 52978: Operation timed out
ftp> quit
221-Goodbye. You uploaded 0 and downloaded 0 kbytes.
221 Logout.

I have no idea where to go next.

Edit:
I tried leaving out the cd / command from the ftp script and found it was not necessary.
Last edited by tjaliwalpa on Fri 11 Mar 2011 10:34 am, edited 2 times in total.
Bob
tjaliwalpa
Posts: 293
Joined: Sun 18 Apr 2010 9:47 am
Weather Station: Davis VP2
Operating System: Linux Lite Ubuntu 16.04
Location: Karoonda, SA
Contact:

Re: Automating the upload of dayfile.txt

Post by tjaliwalpa »

tjaliwalpa wrote:
daj wrote:Hi Bob

In your screen shot you see it doing a "cd e:\Cumulus"

This is wrong -- at that point your are on the webserver and there is no e:\cumulus folder. It should be the directory name as set in your Cumulus Internet Config within Cumulus

Not sure if that is the cause of the over-all problem but it certainly will not help as you need to be in the correct place. Of course, your Directory option in Cumulus could be empty, in which case the CD line in the script can be removed
I see now. There is no directory as my weather site is in the root directory of the webspace. In other words, the Cumulus directory is blank.

I've changed this line to cd / with the following result:

ftp> cd /
250 OK. Current directory is /
ftp> put e:\cumulus\data\dayfile.txt
200 PORT command successful
425 Could not open data connection to port 52978: Operation timed out
ftp> quit
221-Goodbye. You uploaded 0 and downloaded 0 kbytes.
221 Logout.

I have no idea where to go next.

Bob
Ah :oops: Got it now :oops:
:bash: :bash: :bash: :bash:

It is a firewall problem at my end with PC Tools Internet Security. I shall have to write a rule to allow port 52978 access.

When I disable the firewall, everything works perfectly.
Bob
User avatar
Super-T
Posts: 890
Joined: Tue 09 Sep 2008 3:37 am
Weather Station: wh-1081
Operating System: Weather Laptop - Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: Automating the upload of dayfile.txt

Post by Super-T »

I get stuck at a point where the file has been created and then it just sits there without uploading the data and then times out with 0 bytes sent.
Your error may be specific to your host as I don't get that problem.

My result:- This is using "moveItFreely" ftp prog.
C:\WeatherAnalysis>ftps -s:c:\weatheranalysis\daily.ftp linux11.digiweb.net.nz
220 (vsFTPd 2.0.5)
Connected to linux11.digiweb.net.nz.
User: 331 Please specify the password.
Password: 230 Login successful.
ftp> Debugging On .
ftp> Local directory now C:\WeatherAnalysis.
ftp> ---> PASV
227 Entering Passive Mode (202,174,112,121,226,207)
ftp> ---> TYPE I
200 Switching to Binary mode.
ftp> ---> CWD www/weather/data
250 Directory successfully changed.
ftp> ---> PORT 10,1,1,200,192,164
200 PORT command successful. Consider using PASV.
---> STOR test.txt
150 Ok to send data.
ftp: 0 bytes sent in 5.07Seconds 0.00Kbytes/sec.
ftp> ---> QUIT
221 Goodbye.

Also a MS note here:-
http://support.microsoft.com/kb/271078/en-us
tjaliwalpa
Posts: 293
Joined: Sun 18 Apr 2010 9:47 am
Weather Station: Davis VP2
Operating System: Linux Lite Ubuntu 16.04
Location: Karoonda, SA
Contact:

Re: Automating the upload of dayfile.txt

Post by tjaliwalpa »

Super-T wrote:I get stuck at a point where the file has been created and then it just sits there without uploading the data and then times out with 0 bytes sent.
Your error may be specific to your host as I don't get that problem.

My result:- This is using "moveItFreely" ftp prog.
C:\WeatherAnalysis>ftps -s:c:\weatheranalysis\daily.ftp linux11.digiweb.net.nz
220 (vsFTPd 2.0.5)
Connected to linux11.digiweb.net.nz.
User: 331 Please specify the password.
Password: 230 Login successful.
ftp> Debugging On .
ftp> Local directory now C:\WeatherAnalysis.
ftp> ---> PASV
227 Entering Passive Mode (202,174,112,121,226,207)
ftp> ---> TYPE I
200 Switching to Binary mode.
ftp> ---> CWD www/weather/data
250 Directory successfully changed.
ftp> ---> PORT 10,1,1,200,192,164
200 PORT command successful. Consider using PASV.
---> STOR test.txt
150 Ok to send data.
ftp: 0 bytes sent in 5.07Seconds 0.00Kbytes/sec.
ftp> ---> QUIT
221 Goodbye.

Also a MS note here:-
http://support.microsoft.com/kb/271078/en-us
Actually, my issue very similar. When I have the firewall enabled, to dayfile.txt is deleted and created as a blank file. The process hangs for a time and the replacement file is not transferred. Eventually, it gives up and the following is displayed:

425 Could not open data connection to port 53496: Operation timed out
ftp> quit
221-Goodbye. You uploaded 0 and downloaded 0 kbytes.
221 Logout.

The port number is not always the same, so I can't figure out the rule to create.

Edit:
Don't know if I've done the right thing, but if I allow Outbound and Inbound for the the File Transfer Program in my Firewall, everything works beautifully.
Bob
User avatar
mcrossley
Posts: 12689
Joined: Thu 07 Jan 2010 9:44 pm
Weather Station: Davis VP2/WLL
Operating System: Bullseye Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Automating the upload of dayfile.txt

Post by mcrossley »

Super-T wrote:I get stuck at a point where the file has been created and then it just sits there without uploading the data and then times out with 0 bytes sent.
...
200 PORT command successful. Consider using PASV.
...
Also a MS note here:-
http://support.microsoft.com/kb/271078/en-us
So the MS FTP does not support PASV mode, you may have to use something like WinSCP - it's nice to have it around anyway!

Or as Bob has done, open up your firewall for two way sessions with the FTP client (computer and possibly router).
Last edited by mcrossley on Fri 11 Mar 2011 1:27 pm, edited 1 time in total.
User avatar
daj
Posts: 2041
Joined: Tue 29 Jul 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

Re: Automating the upload of dayfile.txt

Post by daj »

Yeah, the MS FTP tool is a very simple implementation, but it is one that everyone has on their system so is a good starting point.

Some useful reading in this topic, so I will add a link from the Wiki article to this for other who may have problems.
David
kippfordweather.uk
Cumulus MX & Raspberry Pi
User avatar
Super-T
Posts: 890
Joined: Tue 09 Sep 2008 3:37 am
Weather Station: wh-1081
Operating System: Weather Laptop - Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: Automating the upload of dayfile.txt

Post by Super-T »

Bob
I did try turning off firewall at one time but your experience means I will re-visit that today.
My Host support seemed to think it was all to do with them requiring PASV as that is meant to avoid the firewall problem but in my case, there is something else involved.
Post Reply