Schannel: fix endless loop in TlsCryptographSchannel::transmit()

We've encountered that under some circumstances DecryptMessage may
return undocumented SEC_E_DECRYPT_FAILURE status. It's probably a bug
in Schannel on Win 11, as there is no such problem on Win 10.

In such cases, since we didn't handle this specifically, we can end
up in an infinite loop in TlsCryptographSchannel::transmit() when
`intermediateBuffer` is not empty and `bytesRead` is 0.

To fix this, we just add SEC_E_DECRYPT_FAILURE status handling and
will disconnect from the host and emit the error respectively.

Change-Id: I340669a967be420c74cb01296629f94c4720c958
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
(cherry picked from commit ed0eb386dad22b0984151b92168dcbf9b57c381b)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
This commit is contained in:
Vladimir Belyavsky 2024-11-21 15:11:08 +03:00 committed by Qt Cherry-pick Bot
parent 395dd7a173
commit 5df8379b18

View File

@ -670,6 +670,8 @@ QString schannelErrorToString(qint32 status)
return QSslSocket::tr("Unexpected or badly-formatted message received");
case SEC_E_ENCRYPT_FAILURE:
return QSslSocket::tr("The data could not be encrypted");
case SEC_E_DECRYPT_FAILURE:
return QSslSocket::tr("The data could not be decrypted");
case SEC_E_ALGORITHM_MISMATCH:
return QSslSocket::tr("No cipher suites in common");
case SEC_E_UNKNOWN_CREDENTIALS:
@ -1945,6 +1947,13 @@ void TlsCryptographSchannel::transmit()
setErrorAndEmit(d, QAbstractSocket::SslInternalError,
schannelErrorToString(status));
break;
} else if (status == SEC_E_DECRYPT_FAILURE) {
// It's not documented as a possible return value for DecryptMessage,
// but we see that this may happen - supposed to be a bug in Schannel (with TLS 1.3?)
shutdown = true; // skips sending the shutdown alert
disconnectFromHost();
setErrorAndEmit(d, QAbstractSocket::SslInternalError, schannelErrorToString(status));
break;
} else if (status == SEC_I_CONTEXT_EXPIRED) {
// 'remote' has initiated a shutdown
disconnectFromHost();