If you use Cumulus, please donate Image

Please read this post before posting

Latest Cumulus release v1.9.4 (build 1099) - Nov 28 2014
Latest Cumulus MX release - v3.0.0 build 3041 Aug 17 2016

Automating the upload of dayfile.txt

Discussion and support for 3rd-party (non-Sandaysoft) tools for Cumulus
User avatar
daj
Posts: 2040
Joined: Tue Jul 29, 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

Automating the upload of dayfile.txt

Postby daj » Thu Mar 10, 2011 3:41 pm

I've written a little article in the Wiki on how to automatically upload dayfile.txt to your website once per day.

It's a simple method, using the Windows scheduler task. I have tested it on Windows XP and Windows 7

Enjoy

http://wiki.sandaysoft.com/a/Upload_Dayfile
David
kippfordweather.uk
Cumulus MX & Raspberry Pi

Gina
Posts: 1885
Joined: Sat Feb 21, 2009 12:41 pm
Weather Station: Nothing working ATM - making one
Operating System: OS X, Linux Mint, Win7 & XP
Location: Devon UK

Re: Automating the upload of dayfile.txt

Postby Gina » Thu Mar 10, 2011 3:57 pm

That looks very useful, thank you :) I must get round to improving my Cumulus web pages - it's all very basic ATM.
Gina

Sorry, no banner - weather station out of action. Hoping to be up and running with a new home-made one soon.

User avatar
Super-T
Posts: 767
Joined: Tue Sep 09, 2008 3:37 am
Weather Station: wh-1081
Operating System: Weather Laptop - Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: Automating the upload of dayfile.txt

Postby Super-T » Thu Mar 10, 2011 9:14 pm

GMTA David
I was doing exactly the same thing yesterday but ran into a problem.
I could get all the way to the PUT command, the file is created on the server but then it just sits there showing 150 Ok to send data. ctrl C will abort and it will exit.
I think something is blocking at the server as the file has been created but with 0 content. I haven't got back to it yet to find the problem but any suggestions would be gratefully received. :-)

User avatar
mcrossley
Posts: 4615
Joined: Thu Jan 07, 2010 9:44 pm
Weather Station: Davis VP2
Operating System: Jessie Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Automating the upload of dayfile.txt

Postby mcrossley » Thu Mar 10, 2011 10:20 pm

Terry, do you need to upload 'put' to a temporary filename, 'delete' dayfile.txt, 'rename' temp to dayfile?

Cumulus uses two different mechanisms for its FTPs because of variations in server support for rename/overwrite etc

ultimate fall back would be 'delete', 'put' I guess.

User avatar
Super-T
Posts: 767
Joined: Tue Sep 09, 2008 3:37 am
Weather Station: wh-1081
Operating System: Weather Laptop - Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: Automating the upload of dayfile.txt

Postby Super-T » Fri Mar 11, 2011 1:14 am

Hi Mark
My Cumulus settings are "Auto update" & "Use FTP Rename" ticked.

Delete and put sounds good....i'll give it a try.

OK....tried adding delete although that probably wasn't a problem because I had tried removing the file from the server before running the script.
Partial result:-
230 Login successful.
ftp> debug
Debugging On .
ftp> lcd c:\!Downloads
Local directory now C:\!Downloads.
ftp> quote PASV
---> PASV
227 Entering Passive Mode (202,174,112,121,240,209)
ftp> binary
---> TYPE I
200 Switching to Binary mode.
ftp> cd www/weather/data
---> CWD www/weather/data
250 Directory successfully changed.
ftp> delete www/weather/data/test.txt
---> DELE www/weather/data/test.txt
550 Delete operation failed.
ftp> put c:\!Downloads\test.txt
---> PORT 10,1,1,200,216,157
200 PORT command successful. Consider using PASV.
---> STOR test.txt
150 Ok to send data.

Hangs on that last line and ctrl C to get out of it.
Creates the file but no content.

Also ran the delete line initially as delete test.txt as I was already in the correct directory, but with no joy.


Later in the day. My Host says there is a problem with PASV mode not supported in Windows FTP programme. http://support.microsoft.com/kb/271078/en-us. I've also tried "MoveMeFreely FTP programme that does support PASV but still no joy.

It looks like there is a problem when the host runs PASV mode which most probably do.

tjaliwalpa
Posts: 290
Joined: Sun Apr 18, 2010 9:47 am
Weather Station: Davis VP2
Operating System: Linux Lite Ubuntu 16.04
Location: Karoonda, SA
Contact:

Re: Automating the upload of dayfile.txt

Postby tjaliwalpa » Fri Mar 11, 2011 7:49 am

daj wrote:I've written a little article in the Wiki on how to automatically upload dayfile.txt to your website once per day.

It's a simple method, using the Windows scheduler task. I have tested it on Windows XP and Windows 7

Enjoy

http://wiki.sandaysoft.com/a/Upload_Dayfile


Thanks for this information. It is just what I was looking for. I have a little problem with it however.

I've set up the script and batch file as per the wiki instructions. I have checked and rechecked the variables and am sure I have them correct.

The dayfile.txt is deleted on the server and replaced with another version which is empty. The ftp process times out without uploading the complete file.

What am I doing wrong?

Attached is a screen shot of the cmd.
You do not have the required permissions to view the files attached to this post.
Bob

User avatar
daj
Posts: 2040
Joined: Tue Jul 29, 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

Re: Automating the upload of dayfile.txt

Postby daj » Fri Mar 11, 2011 9:13 am

Hi Bob

In your screen shot you see it doing a "cd e:\Cumulus"

This is wrong -- at that point your are on the webserver and there is no e:\cumulus folder. It should be the directory name as set in your Cumulus Internet Config within Cumulus

Not sure if that is the cause of the over-all problem but it certainly will not help as you need to be in the correct place. Of course, your Directory option in Cumulus could be empty, in which case the CD line in the script can be removed
David
kippfordweather.uk
Cumulus MX & Raspberry Pi

User avatar
Super-T
Posts: 767
Joined: Tue Sep 09, 2008 3:37 am
Weather Station: wh-1081
Operating System: Weather Laptop - Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: Automating the upload of dayfile.txt

Postby Super-T » Fri Mar 11, 2011 9:31 am

Should be doing LCD e:\cumulus\data perhaps but if you are using put e:\cumulus\data\dayfile.txt then you don't need to do the lcd e:\cumulus\data I think?

tjaliwalpa
Posts: 290
Joined: Sun Apr 18, 2010 9:47 am
Weather Station: Davis VP2
Operating System: Linux Lite Ubuntu 16.04
Location: Karoonda, SA
Contact:

Re: Automating the upload of dayfile.txt

Postby tjaliwalpa » Fri Mar 11, 2011 9:50 am

daj wrote:Hi Bob

In your screen shot you see it doing a "cd e:\Cumulus"

This is wrong -- at that point your are on the webserver and there is no e:\cumulus folder. It should be the directory name as set in your Cumulus Internet Config within Cumulus

Not sure if that is the cause of the over-all problem but it certainly will not help as you need to be in the correct place. Of course, your Directory option in Cumulus could be empty, in which case the CD line in the script can be removed


I see now. There is no directory as my weather site is in the root directory of the webspace. In other words, the Cumulus directory is blank.

I've changed this line to cd / with the following result:

ftp> cd /
250 OK. Current directory is /
ftp> put e:\cumulus\data\dayfile.txt
200 PORT command successful
425 Could not open data connection to port 52978: Operation timed out
ftp> quit
221-Goodbye. You uploaded 0 and downloaded 0 kbytes.
221 Logout.

I have no idea where to go next.

Edit:
I tried leaving out the cd / command from the ftp script and found it was not necessary.
Last edited by tjaliwalpa on Fri Mar 11, 2011 10:34 am, edited 2 times in total.
Bob

tjaliwalpa
Posts: 290
Joined: Sun Apr 18, 2010 9:47 am
Weather Station: Davis VP2
Operating System: Linux Lite Ubuntu 16.04
Location: Karoonda, SA
Contact:

Re: Automating the upload of dayfile.txt

Postby tjaliwalpa » Fri Mar 11, 2011 9:56 am

tjaliwalpa wrote:
daj wrote:Hi Bob

In your screen shot you see it doing a "cd e:\Cumulus"

This is wrong -- at that point your are on the webserver and there is no e:\cumulus folder. It should be the directory name as set in your Cumulus Internet Config within Cumulus

Not sure if that is the cause of the over-all problem but it certainly will not help as you need to be in the correct place. Of course, your Directory option in Cumulus could be empty, in which case the CD line in the script can be removed


I see now. There is no directory as my weather site is in the root directory of the webspace. In other words, the Cumulus directory is blank.

I've changed this line to cd / with the following result:

ftp> cd /
250 OK. Current directory is /
ftp> put e:\cumulus\data\dayfile.txt
200 PORT command successful
425 Could not open data connection to port 52978: Operation timed out
ftp> quit
221-Goodbye. You uploaded 0 and downloaded 0 kbytes.
221 Logout.

I have no idea where to go next.

Bob


Ah :oops: Got it now :oops:
:bash: :bash: :bash: :bash:

It is a firewall problem at my end with PC Tools Internet Security. I shall have to write a rule to allow port 52978 access.

When I disable the firewall, everything works perfectly.
Bob

User avatar
Super-T
Posts: 767
Joined: Tue Sep 09, 2008 3:37 am
Weather Station: wh-1081
Operating System: Weather Laptop - Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: Automating the upload of dayfile.txt

Postby Super-T » Fri Mar 11, 2011 10:02 am

I get stuck at a point where the file has been created and then it just sits there without uploading the data and then times out with 0 bytes sent.
Your error may be specific to your host as I don't get that problem.

My result:- This is using "moveItFreely" ftp prog.
C:\WeatherAnalysis>ftps -s:c:\weatheranalysis\daily.ftp linux11.digiweb.net.nz
220 (vsFTPd 2.0.5)
Connected to linux11.digiweb.net.nz.
User: 331 Please specify the password.
Password: 230 Login successful.
ftp> Debugging On .
ftp> Local directory now C:\WeatherAnalysis.
ftp> ---> PASV
227 Entering Passive Mode (202,174,112,121,226,207)
ftp> ---> TYPE I
200 Switching to Binary mode.
ftp> ---> CWD www/weather/data
250 Directory successfully changed.
ftp> ---> PORT 10,1,1,200,192,164
200 PORT command successful. Consider using PASV.
---> STOR test.txt
150 Ok to send data.
ftp: 0 bytes sent in 5.07Seconds 0.00Kbytes/sec.
ftp> ---> QUIT
221 Goodbye.

Also a MS note here:-
http://support.microsoft.com/kb/271078/en-us

tjaliwalpa
Posts: 290
Joined: Sun Apr 18, 2010 9:47 am
Weather Station: Davis VP2
Operating System: Linux Lite Ubuntu 16.04
Location: Karoonda, SA
Contact:

Re: Automating the upload of dayfile.txt

Postby tjaliwalpa » Fri Mar 11, 2011 10:10 am

Super-T wrote:I get stuck at a point where the file has been created and then it just sits there without uploading the data and then times out with 0 bytes sent.
Your error may be specific to your host as I don't get that problem.

My result:- This is using "moveItFreely" ftp prog.
C:\WeatherAnalysis>ftps -s:c:\weatheranalysis\daily.ftp linux11.digiweb.net.nz
220 (vsFTPd 2.0.5)
Connected to linux11.digiweb.net.nz.
User: 331 Please specify the password.
Password: 230 Login successful.
ftp> Debugging On .
ftp> Local directory now C:\WeatherAnalysis.
ftp> ---> PASV
227 Entering Passive Mode (202,174,112,121,226,207)
ftp> ---> TYPE I
200 Switching to Binary mode.
ftp> ---> CWD www/weather/data
250 Directory successfully changed.
ftp> ---> PORT 10,1,1,200,192,164
200 PORT command successful. Consider using PASV.
---> STOR test.txt
150 Ok to send data.
ftp: 0 bytes sent in 5.07Seconds 0.00Kbytes/sec.
ftp> ---> QUIT
221 Goodbye.

Also a MS note here:-
http://support.microsoft.com/kb/271078/en-us


Actually, my issue very similar. When I have the firewall enabled, to dayfile.txt is deleted and created as a blank file. The process hangs for a time and the replacement file is not transferred. Eventually, it gives up and the following is displayed:

425 Could not open data connection to port 53496: Operation timed out
ftp> quit
221-Goodbye. You uploaded 0 and downloaded 0 kbytes.
221 Logout.

The port number is not always the same, so I can't figure out the rule to create.

Edit:
Don't know if I've done the right thing, but if I allow Outbound and Inbound for the the File Transfer Program in my Firewall, everything works beautifully.
Bob

User avatar
mcrossley
Posts: 4615
Joined: Thu Jan 07, 2010 9:44 pm
Weather Station: Davis VP2
Operating System: Jessie Lite rPi
Location: Wilmslow, Cheshire, UK
Contact:

Re: Automating the upload of dayfile.txt

Postby mcrossley » Fri Mar 11, 2011 12:13 pm

Super-T wrote:I get stuck at a point where the file has been created and then it just sits there without uploading the data and then times out with 0 bytes sent.
...
200 PORT command successful. Consider using PASV.
...
Also a MS note here:-
http://support.microsoft.com/kb/271078/en-us

So the MS FTP does not support PASV mode, you may have to use something like WinSCP - it's nice to have it around anyway!

Or as Bob has done, open up your firewall for two way sessions with the FTP client (computer and possibly router).
Last edited by mcrossley on Fri Mar 11, 2011 1:27 pm, edited 1 time in total.

User avatar
daj
Posts: 2040
Joined: Tue Jul 29, 2008 8:00 pm
Weather Station: WH1081
Operating System: Pi & MX
Location: SW Scotland
Contact:

Re: Automating the upload of dayfile.txt

Postby daj » Fri Mar 11, 2011 12:56 pm

Yeah, the MS FTP tool is a very simple implementation, but it is one that everyone has on their system so is a good starting point.

Some useful reading in this topic, so I will add a link from the Wiki article to this for other who may have problems.
David
kippfordweather.uk
Cumulus MX & Raspberry Pi

User avatar
Super-T
Posts: 767
Joined: Tue Sep 09, 2008 3:37 am
Weather Station: wh-1081
Operating System: Weather Laptop - Windows 10 Pro
Location: Auckland, New Zealand
Contact:

Re: Automating the upload of dayfile.txt

Postby Super-T » Fri Mar 11, 2011 7:52 pm

Bob
I did try turning off firewall at one time but your experience means I will re-visit that today.
My Host support seemed to think it was all to do with them requiring PASV as that is meant to avoid the firewall problem but in my case, there is something else involved.


Return to “3rd Party Tools”

Who is online

Users browsing this forum: No registered users and 1 guest