Please read the posts in the Announcements section about the current status of Cumulus development now that I have retired

Please read this post before posting

Latest Cumulus release v1.9.4 (build 1099) - Nov 28 2014
Latest Cumulus MX release - v3.0.0 build 3043 Jan 20 2017. See this post for download

The handshake failed due to un unexpected packet format

Discussion of version 3 of Cumulus, which runs on Windows, Linux, and OS X. All Cumulus MX queries in here, please.
Dinant
Posts: 51
Joined: Sat Aug 17, 2013 9:11 am
Weather Station: Davis Vantage Pro2
Operating System: Windows 2012 R2
Location: Markelo
Contact:

The handshake failed due to un unexpected packet format

Postby Dinant » Wed Apr 04, 2018 10:40 am

The provider which host my website has disabled SSLv2 and SSLv3 on their servers. Now CumulusMX cannot upload files with SFTP anymore.
I got an error saying: "Error connecting ftp - The handshake failed due to an unexpected packet packet format".
In the ftp log it says:

2018-04-03 20:25:26.536 Uploading realtime.txt to /weer/cumulus/realtime.txt
There is stale data on the socket, maybe our connection timed out. Re-connecting.
Testing connectivity using Socket.Poll()...
220 ::ffff:x.x.x.x FTP server ready
AUTH TLS
234 AUTH TLS successful
2018-04-03 20:25:26.583 Uploading web\realtimegauges.txt to /weer/cumulus/realtimegauges.txt
There is stale data on the socket, maybe our connection timed out. Re-connecting.
220 ::ffff:x.x.x.x FTP server ready
AUTH TLS
234 AUTH TLS successful

Two questions:
Is Cumulus capable of using TLS 1.0, 1.1 or 1.2 for SFTP?
Is there a work-around?

User avatar
Toxic17
Posts: 740
Joined: Mon May 19, 2014 10:45 pm
Weather Station: Davis VPro2 Plus
Operating System: Deb 9.4 Stretch
Location: Bangor, NI
Contact:

Re: The handshake failed due to un unexpected packet format

Postby Toxic17 » Wed Apr 04, 2018 12:12 pm

it is in the wiki if you care to look.

http://wiki.sandaysoft.com/a/SFTP
Regards Simon

https://www.conligwx.org
https://www.conligwx.org/pws/
https://twitter.com/conligwx
Davis Vantage Pro2+ - CumulusMX v3.0.0 (build 3043) + Saratoga/PWS
Image

User avatar
steve
Cumulus Author
Posts: 26493
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: The handshake failed due to un unexpected packet format

Postby steve » Wed Apr 04, 2018 1:14 pm

Dinant wrote:The provider which host my website has disabled SSLv2 and SSLv3 on their servers. Now CumulusMX cannot upload files with SFTP

To clarify, MX does not support SFTP (sometimes called SSH FTP), it supports FTP over TLS (sometimes called FTPS).

Is Cumulus capable of using TLS 1.0, 1.1 or 1.2 for SFTP?

it already uses TLS for FTPS (as confirmed by the "AUTH TLS" command that it issues). I'm afraid I can't tell you what version of TLS it uses (it's all handled by a third party component).
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

Dinant
Posts: 51
Joined: Sat Aug 17, 2013 9:11 am
Weather Station: Davis Vantage Pro2
Operating System: Windows 2012 R2
Location: Markelo
Contact:

Re: The handshake failed due to un unexpected packet format

Postby Dinant » Wed Apr 04, 2018 1:50 pm

@Toxic17: Maybe this is a workaround for me. I 'll test this. Thanks for your reply.

@Steve: Steve is right. I mean FTP over SSL so FTPS. Because my provider has disabled SSL v2 and SSL v3 support yesterday and the FTPS connection is not working since yesterday I must concluded the third party component Cumulus uses is not using TLS 1.0, 1.1 or 2.0.

User avatar
steve
Cumulus Author
Posts: 26493
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: The handshake failed due to un unexpected packet format

Postby steve » Wed Apr 04, 2018 2:50 pm

It must be using either TLS 1.0, 1.1, or 1.2, because those are the only versions which existed at the time the ftp component was written. My conclusion is that you have some other issue. Are you using active or passive ftp mode, for example?
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

Dinant
Posts: 51
Joined: Sat Aug 17, 2013 9:11 am
Weather Station: Davis Vantage Pro2
Operating System: Windows 2012 R2
Location: Markelo
Contact:

Re: The handshake failed due to un unexpected packet format

Postby Dinant » Wed Apr 04, 2018 6:11 pm

steve wrote:It must be using either TLS 1.0, 1.1, or 1.2, because those are the only versions which existed at the time the ftp component was written. My conclusion is that you have some other issue. Are you using active or passive ftp mode, for example?

I am using passive FTP mode (Active FTP mode not checked).
This afternoon I've contacted my provider. They have rejected the modifications made yesterday (disabling SSL v2 and v3). Now FTPS is working again. I haven't change anything.

User avatar
steve
Cumulus Author
Posts: 26493
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: The handshake failed due to un unexpected packet format

Postby steve » Wed Apr 04, 2018 6:14 pm

Can they explain what the problem is, as Cumulus is clearly using TLS and not SSL?
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

Dinant
Posts: 51
Joined: Sat Aug 17, 2013 9:11 am
Weather Station: Davis Vantage Pro2
Operating System: Windows 2012 R2
Location: Markelo
Contact:

Re: The handshake failed due to un unexpected packet format

Postby Dinant » Wed Apr 04, 2018 7:02 pm

steve wrote:Can they explain what the problem is, as Cumulus is clearly using TLS and not SSL?

This is what they told me:
For security reasons SSLv2 and SSLv3 support is disabled. There is only support for TLSv1.0, 1.1 and 1.2.

I did some tests with WinSCP and SSLv3 only. This is what i see in the logging of WinSCP:

Code: Select all

. 2018-04-04 20:36:09.071 FTPS: Explicit TLS/SSL [Client certificate: No]
. 2018-04-04 20:36:09.071 FTP: Passive: Yes [Force IP: Auto]; MLSD: Auto [List all: Auto]; HOST: Auto
. 2018-04-04 20:36:09.071 Session reuse: Yes
. 2018-04-04 20:36:09.071 TLS/SSL versions: SSLv3-SSLv3
. 2018-04-04 20:36:09.071 Local directory: default, Remote directory: home, Update: Yes, Cache: Yes
. 2018-04-04 20:36:09.071 Cache directory changes: Yes, Permanent: Yes
. 2018-04-04 20:36:09.071 Recycle bin: Delete to: No, Overwritten to: No, Bin path:
. 2018-04-04 20:36:09.071 Timezone offset: 0h 0m
. 2018-04-04 20:36:09.071 --------------------------------------------------------------------------
. 2018-04-04 20:36:09.071 Connecting to xxx.xxxx.xx ...
. 2018-04-04 20:36:09.087 Connected with xxx.xxxx.xx, negotiating TLS connection...
< 2018-04-04 20:36:09.102 220 ::ffff:x.x.x.x FTP server ready
> 2018-04-04 20:36:09.102 AUTH TLS
< 2018-04-04 20:36:09.118 234 AUTH TLS successful
. 2018-04-04 20:36:09.243 Server asks for authentication with a client certificate.
. 2018-04-04 20:36:09.243 SSL3 alert write: warning: no certificate
. 2018-04-04 20:36:09.274 Verifying certificate for "" with fingerprint xx and 20 failures
. 2018-04-04 20:36:09.274 Certificate common name "*.xxxx.xx" matches hostname
. 2018-04-04 20:36:09.290 Certificate verified against Windows certificate store
. 2018-04-04 20:36:09.290 Using SSLv3, cipher TLSv1/SSLv3: DHE-RSA-AES256-SHA, 4096 bit RSA, DHE-RSA-AES256-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA1
. 2018-04-04 20:36:09.290 TLS connection established. Waiting for welcome message...
> 2018-04-04 20:36:09.290 USER ********************
< 2018-04-04 20:36:09.305 331 Password required for ********************
> 2018-04-04 20:36:09.305 PASS ********************
< 2018-04-04 20:36:09.321 230 User ******************** logged in
> 2018-04-04 20:36:09.321 SYST
< 2018-04-04 20:36:09.337 215 UNIX Type: L8
> 2018-04-04 20:36:09.337 FEAT
< 2018-04-04 20:36:09.337 211-Features:
< 2018-04-04 20:36:09.337  MDTM
< 2018-04-04 20:36:09.337  MFMT
< 2018-04-04 20:36:09.337  LANG ko-KR;zh-CN;fr-FR;ru-RU;bg-BG;ja-JP;en-US;it-IT;zh-TW
< 2018-04-04 20:36:09.337  TVFS
< 2018-04-04 20:36:09.337  UTF8
< 2018-04-04 20:36:09.337  AUTH TLS
< 2018-04-04 20:36:09.337  MFF modify;UNIX.group;UNIX.mode;
< 2018-04-04 20:36:09.337  MLST modify*;perm*;size*;type*;unique*;UNIX.group*;UNIX.mode*;UNIX.owner*;
< 2018-04-04 20:36:09.337  PBSZ
< 2018-04-04 20:36:09.337  PROT
< 2018-04-04 20:36:09.368  REST STREAM
< 2018-04-04 20:36:09.368  SIZE
< 2018-04-04 20:36:09.368 211 End
> 2018-04-04 20:36:09.368 OPTS UTF8 ON
< 2018-04-04 20:36:09.368 200 UTF8 set to on
> 2018-04-04 20:36:09.368 PBSZ 0
< 2018-04-04 20:36:09.368 200 PBSZ 0 successful
> 2018-04-04 20:36:09.368 PROT P
< 2018-04-04 20:36:09.383 200 Protection set to Private
. 2018-04-04 20:36:09.383 Connected


In the logfile I see "AUTH TLS" when only SSLv3 can be and also is being used. So if AUTH TLS is being shown this doesn't say anything about which protocol is being used.
The Cumulus logfile also shows AUTH TLS but still the SSLv2 or SSLv3 can be used i think. Are you sure Cumulus is NOT using SSLv2 or SSLv3?

User avatar
steve
Cumulus Author
Posts: 26493
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: The handshake failed due to un unexpected packet format

Postby steve » Wed Apr 04, 2018 7:27 pm

All I can tell you is that MX is telling the ftp client code to use Explicit ftp encryption mode. According to the documentation (and the source code) Explicit=TLS, Implicit=SSL
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

Dinant
Posts: 51
Joined: Sat Aug 17, 2013 9:11 am
Weather Station: Davis Vantage Pro2
Operating System: Windows 2012 R2
Location: Markelo
Contact:

Re: The handshake failed due to un unexpected packet format

Postby Dinant » Thu Apr 05, 2018 10:01 am

steve wrote:All I can tell you is that MX is telling the ftp client code to use Explicit ftp encryption mode. According to the documentation (and the source code) Explicit=TLS, Implicit=SSL


I'm not an expert in FTPS but in the meanwhile I have read some web pages about it.
https://blogs.iis.net/robert_mcmurray/ftp-clients-part-2-explicit-ftps-versus-implicit-ftps
https://nl.wikipedia.org/wiki/FTP_over_SSL
http://www.serv-u.com/kb/1088/implicit-vs-explicit-ssl
https://blog.ftptoday.com/explicit-ftps-vs-implicit-ftps-what-you-need-to-know
http://www.jscape.com/blog/bid/82991/Choosing-Between-SSL-Implicit-Explicit-and-Forced-Explicit-Modes
https://www.rebex.net/kb/tls-ssl-explicit-implicit/

I think that the last part of your message "Explicit=TLS, Implicit=SSL" is not true.
I think MX uses Explicit FTPS with SSLv2 or SSLv3 otherwise the MX upload was working after disabling SSLv2 and SSLv3.

In the meantime I am busy testing a batch script to upload the Cumulus files with WinSCP. I'll post the script in a few days.

User avatar
steve
Cumulus Author
Posts: 26493
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: The handshake failed due to un unexpected packet format

Postby steve » Thu Apr 05, 2018 12:53 pm

I’m certainly no expert at FTPS either, but neither am I in the habit of telling lies. What I was telling you, and what is certainly true, is what the documentation and the comments in the ftp component source code say, that setting “explicit” causes the component to use TLS. Whether that actually happens or not in practice, I can’t say.

When I get chance I’ll do some experiments with my own ftp server to see if I can see what might be happening.
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

Dinant
Posts: 51
Joined: Sat Aug 17, 2013 9:11 am
Weather Station: Davis Vantage Pro2
Operating System: Windows 2012 R2
Location: Markelo
Contact:

Re: The handshake failed due to un unexpected packet format

Postby Dinant » Sun Apr 08, 2018 10:04 am

I did some tests with FTPS on 2 webservers of my provider. One with SSLv2 and SSL v3 disabled and one with the original settings (all enabled).
On the servers with everything enabled there are no login problems detected. On the server with SSLv2 and SSLv3 disabled I see the following results:
SSL 3.0: LOGIN DENIED
TLS 1.0: LOGIN DENIED
TLS 1.1 LOGIN SUCCESSFULL
TLS 1.2 LOGIN SUCCESSFULL

My conclusion is that they have also disabled TLS 1.0! This was in contract what they have told me. I have send them an e-mail and wait for their response.
So if MX is using TLS 1.0 this may explain why FTPS in MX is not working anymore!

User avatar
steve
Cumulus Author
Posts: 26493
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: The handshake failed due to un unexpected packet format

Postby steve » Sun Apr 08, 2018 3:12 pm

I’m hoping to do some testing with my own ftp server later today. What I’m not sure about is what part of the system decides what protocol to use - all I know is that it isn’t my code, I leave it to the ftp component. What I would assume happens is that the ftp component asks the server which protocols it supports, and it uses the latest one that it can support itself.
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job

Dinant
Posts: 51
Joined: Sat Aug 17, 2013 9:11 am
Weather Station: Davis Vantage Pro2
Operating System: Windows 2012 R2
Location: Markelo
Contact:

Re: The handshake failed due to un unexpected packet format

Postby Dinant » Mon Apr 09, 2018 9:10 am

This is what i have found on the Wiki page: https://en.wikipedia.org/wiki/FTPS

Explicit
In explicit mode (also known as FTPES), an FTPS client must "explicitly request" security from an FTPS server and then step up to a mutually agreed encryption method. If a client does not request security, the FTPS server can either allow the client to continue in insecure mode or refuse the connection.

The mechanism for negotiating authentication and security with FTP was added under RFC 2228, which included the new FTP command AUTH. While this RFC does not explicitly define any required security mechanisms, e.g. SSL or TLS, it does require the FTPS client to challenge the FTPS server with a mutually known mechanism. If the FTPS client challenges the FTPS server with an unknown security mechanism, the FTPS server will respond to the AUTH command with error code 504 (not supported). Clients may determine which mechanisms are supported by querying the FTPS server with the FEAT command, although servers are not necessarily required to be honest in disclosing what levels of security they support. Common methods of invoking FTPS security included AUTH TLS and AUTH SSL.

The explicit method is defined in RFC 4217. In the later versions of the document, FTPS compliance required that clients always negotiate using the AUTH TLS method.


In RFC 4217 is says:
The TLS protocol is a development of the Netscape Communication
Corporation's SSL protocol and this document can be used to allow the
FTP protocol to be used with either SSL or TLS. The actual protocol
used will be decided by the negotiation of the protected session by
the TLS/SSL layer. This document will only refer to the TLS
protocol; however, it is understood that the Client and Server MAY
actually be using SSL if they are so configured.


The FEAT command (introduced in [RFC-2389]) allows servers with
additional features to advertise these to a client by responding to
the FEAT command. If a server supports the FEAT command, then it
MUST advertise supported AUTH, PBSZ, and PROT commands in the reply,
as described in section 3.2 of [RFC-2389]. Additionally, the AUTH
command should have a reply that identifies 'TLS' as one of the
possible parameters to AUTH. It is not necessary to identify the
'TLS-C' synonym separately.

Example reply (in the same style as [RFC-2389])

C> FEAT
S> 211-Extensions supported
S> AUTH TLS
S> PBSZ
S> PROT
S> 211 END


The AUTH command takes a single parameter to define the security
mechanism to be negotiated. As the SSL/TLS protocols self-negotiate
their levels, there is no need to distinguish between SSL and TLS in
the application layer. The mechanism name for negotiating TLS is the
character string identified in {TLS-PARM}. This allows the client
and server to negotiate TLS on the control connection without
altering the protection of the data channel. To protect the data
channel as well, the PBSZ command, followed by the PROT command
sequence, MUST be used.


I understand that the client and server negotiate which protocol to use. First the AUTH SSL/TLS command decides between SSL and TLS. After that the negotiating between server and client desides which TLS protocol (1.0, 1.1 and 1.2) to use.

TLS is supported for Microsoft .NET Framework 3.5 and higher.
I also read
TLS v1.1 and v1.2 are not available in Windows Vista or Windows Server 2008


So the OS and Microsoft .NET Framework version also deside which protocol versions are available!

User avatar
steve
Cumulus Author
Posts: 26493
Joined: Mon Jun 02, 2008 6:49 pm
Weather Station: None
Operating System: None
Location: Vienne, France
Contact:

Re: The handshake failed due to un unexpected packet format

Postby steve » Mon Apr 09, 2018 10:17 am

I did think that perhaps the OS played some part in the protocol negotiation. What version of Windows are you using - 2003 as in your profile?

I still haven’t had chance to do any testing on my server. It does seem to be pretty conclusive that this is not an issue with MX.
Steve
-----
Hosting available for Cumulus web sites. See http://sandaysoft.com/forum/viewtopic.php?f=2&t=11876

Please read the posts in the Announcements section about the current status of Cumulus development since I have retired from my day job


Return to “Cumulus MX”

Who is online

Users browsing this forum: No registered users and 5 guests