Version 58 of HTTPS

Updated 2020-08-27 16:00:31 by oehhar

HTTPS stands for HyperText Transfer Protocol Secure . It's running HTTP over (a profile of) a SSL or TLS secured socket, and it means that the client can talk confidentially with the server and the client can know for sure the identity of the server that it is talking to.

http package

The http core package itself does not feature https support but contains a plugin interface to delegate the TLS communication to another package. The following packages may be used:

tls package

The http package also can do secure HTTP (HTTPS) with the help of the tls package, as Michael A. Cleverly's example from the http manual illustrates:

package require http 2
package require tls 1.7
http::register https 443 [list ::tls::socket -autoservername true]
set token [http::geturl https://my.secure.site/]

Also see Matt Newman's example: http://tls.sourceforge.net/tls.htm#HTTPS%20EXAMPLE .

dbohdan 2015-01-02: Many websites are disabling SSLv3 these days because the protocol is vulnerable . This means you will run into the error message sslv3 alert handshake failure when trying to connect unless you have support for the newer TLS protocol enabled. Despite what the package is called you have to enable support for TLS 1.x in ::tls::socket manually like so: ::tls::socket -tls1 1. I've updated the example at the top of the page to reflect this. RoyKeene I removed the mentions to -tls1 1 since newer versions of TclTLS (1.7+) deal with this correctly, I also added "-autoservername" which enables SNI, which is typically desirable.

twapi

HaO 2016-05-24: On windows, twapi (thanks Ashok!) may be used instead of the tls package as described above. For me, the main point are security fixes. Using twapi, fixes are installed with the operating system. With the tls package, the openssl library is statically linked and the application programmer must care about a current version. On Unix, this issue is not present if the openssl library is dynamically linked to the tls package.

Here is an extension of the upper example using twapi if present:

package require http 2
if {[catch {package require twapi_crypto}]} {
    package require tls 1.7
    http::register https 443 [list ::tls::socket -autoservername true]
} else {
    http::register https 443 [list ::twapi::tls_socket]
}
set token [http::geturl https://my.secure.site/]

I tested successfully twapi version 4.0.61 with http 2.8.9 (tcl 8.6.5).

To use self-signed certificates, one may switch-off certificate verification also for TWAPI:

package require http 2
if {[catch {package require twapi_crypto}]} {
    package require tls 1.7
    http::register https 443 [list ::tls::socket -autoservername true]
} else {
    proc ::twapi_verify args {return 1}
    http::register https 443 [list ::twapi::tls_socket -verifier ::twapi_verifier]
}
set token [http::geturl https://my.secure.site/]

Stunnel

stunnel can be used as a secure layer over an existing socket.

proxy

HaO 2018-04-17: The tcllib command '::autoproxy::tunnel_connect' allows to tunnel by proxies using the tls package. This command falls back to '::tls::socket', if no proxy host is set or the currently requested URL is within the excluded hosts.

Here is my code, which uses:

  • twapi if loadable, tls package otherwise
  • only routes via autoproxy, if a proxy host is set. This is only required for performance reasons. The direct calls may be removed without functional issues.
package require tls
package require http
package require autoproxy 1.7+ ; # autoproxy 1.7 supports twapi tls
if { I have program-internal proxy settings } {
    ::autoproxy::configure -proxy_host example.com -proxy_port 880
    ::autoproxy::configure -basic -username sampleuser -password samplepassword
} else {
    ::autoproxy::init
}
if { [catch {package require twapi_crypto}] } {
    # No TWAPI -> use tls
    package require tls
    if {[::autoproxy::cget -host] ne ""} {
        # Proxy set -> use autoproxy
        http::register https 443 [list autoproxy::tls_socket -tls1 1]
    } else {
        # Proxy set -> use direct call to tls package
        http::register https 443 [list ::tls::socket]
    }
} else {
    # TWAPI present
    if {[::autoproxy::cget -host] ne ""} {
        # Switch autoconf tls to twapi mode
        autoproxy::configure -tls_package twapi
        http::register https 443 autoproxy::tls_socket
    } else {
        # Direct twapi call
        http::register https 443 [list ::twapi::tls_socket]
    }
}

Alexandru has remarked on clt that the parameter "-tls 1" is ignored for "autoproxy::tls_socket". I looked to the source code and was not convinced. Any comments welcome.

TclCurl

TclCurl features https queries.

Checking your TLS protocol version

dbohdan 2017-02-02: The following code will tell you what version of the TLS protocol your HTTPS client uses when it talks to an up-to-date server. It should work with any TLS protocol handler, which in practice means either tls or twapi, as long as the free service it relies on is up.

package require http
package require json

# Register a TLS protocol handler and configure it as you wish here.
package require tls
::http::register https 443 [list ::tls::socket]

# Make a request to the API.
set token [::http::geturl https://www.howsmyssl.com/a/check]
puts [dict get [::json::json2dict [::http::data $token]] tls_version]
::http::cleanup $token

 Obsolete discussion on https CONNECT proxying

This has been supported by autoproxy for Quite Some Time. aspect (as of 2015-01-03) suspects all this section should simply be deleted.

[But none of this stuff knows how to tunnel proxies, as of December 2001.] [But maybe TclCurl helps?]


Erik Leunissen, Pat Thoyts, and David Bleicher dive deeply into HTTP 1.0 vs. 1.1, persistent connections, certificates, ... in a revealing thread [L1 ] on comp.lang.tcl.


Dave Griffin's proposal for tunnel proxies:

Add this about 90 lines into http::geturl:

    :
    :
    set state(url) $url
    if {![catch {$http(-proxyfilter) $host} proxy]} {
        set phost [lindex $proxy 0]
        set pport [lindex $proxy 1]
    }

    # If a timeout is specified we set up the after event
    # and arrange for an asynchronous socket connection.

    if {$state(-timeout) > 0} {
        set state(after) [after $state(-timeout) \
                [list http::reset $token timeout]]
        set async -async
    } else {
        set async ""
    }

    # If we are using the proxy, we must pass in the full URL that
    # includes the server name.


    if {[info exists phost] && [string length $phost]} {
        #
        # Use SSL tunneling for https proxy
        #
        if {$proto == "https"} {
            # No async connection yet...
            set conStat [catch {socket $phost $pport} s]
            if {$conStat} {
                # something went wrong while trying to establish the connection
                # Clean up after events and such, but DON'T call the command callback
                # (if available) because we're going to throw an exception from here
                # instead.
                Finish $token "" 1
                cleanup $token
                return -code error $s
            }
            fconfigure $s -translation {auto crlf} -buffersize $state(-blocksize)
            puts $s "CONNECT $host:$port HTTP/1.0"
            puts $s "User-Agent: $http(-useragent)"
            puts $s ""
            flush $s
            set proxyOK 0
            #
            # This is incredibly lame, but we're hoping for success and
            # will at least throw an error if there is a problem -- the details
            # of which will be haphazard at best.
            #
            # Read back the proxy server's response and single-mindedly
            # hunt for the connection ok status line -- ignoring everything else.
            #
            while {[gets $s proxyLine] > 0} {
                if {[regexp {^HTTP/.* 200 } $proxyLine]} {
                    set proxyOK 1
                }
            }

            #
            # If we could not detect a good connection, raise an error.
            #
            if {!$proxyOK} {
                close $s
                Finish $token "" 1
                cleanup $token
                return -code error "Unable to connect via proxy: $proxyLine"
            }

            # We've got a good proxy connection.
            # Switch the socket over to SSL for further communication.
            #
            # We're going to assume much about TLS right now.  For example,
            # the normal protocol registration would consist of the ::tls::socket
            # command and all of its options.  We're going to grab any of those
            # options and apply them to the ::tls::import command and hope for the
            # best. The idea here is to no worse than the non-proxied SSL support.
            #
            set conStat [catch {eval ::tls::import [lrange $defcmd 1 end] $s} s]
            if {$conStat} {
                # something went wrong while trying to establish the SSL protocol
                # Clean up after events and such, but DON'T call the command callback
                # (if available) because we're going to throw an exception from here
                # instead.
                Finish $token "" 1
                cleanup $token
                return -code error "Unable to establish SSL connection: $s"
            }

        } else {
            set srvurl $url
            set conStat [catch {eval $defcmd $async {$phost $pport}} s]
        }
    } else {
        set conStat [catch {eval $defcmd $async {$host $port}} s]
    }



    if {$conStat} {

        # something went wrong while trying to establish the connection
        # Clean up after events and such, but DON'T call the command callback
        # (if available) because we're going to throw an exception from here
        # instead.
        Finish $token "" 1
        cleanup $token
        return -code error $s
    }
    set state(sock) $s
    :
    :

We're testing this out now. If it holds together (and nobody has any other suggestions on how to do the error handling better) I'll see if the Tcl Core group would like it.


Of course, the Apache Tcl stuff runs fine on an SSL-enabled Apache.


Jos Decoster : The following code takes Dave Griffin's proposal for tunnel proxies and adds proxy authentication. The proxy authentiation is based on information from Pat Thoyts found in http authentication.

This modified http geturl command takes an additional '-proxy_auth' argument specifying the base64 encoded authentication method (only Basic currently supported), username and password needed to do the proxy authentication. You can use it like this:

    % http::register https $https_port ::tls::socket
    % http::config -proxyhost $proxy_host -proxyport $proxy_port
    % http::geturl $url -proxy_auth [concat "Basic" [base64::encode $proxy_auth_user:$proxy_auth_password]]

I added the proxy authentication to the http::geturl function because I already had to modify it. It may be better to add it to the http::config command.

The modified http::geturl command:

  proc http::geturl { url args } {

    if { $::TctInstall::debug_mode } {
        puts "\n\n\nUSING PATCHED GETURL!\n\n\n"
    }

    variable http
    variable urlTypes
    variable defaultCharset

    # Initialize the state variable, an array.  We'll return the
    # name of this array as the token for the transaction.

    if {![info exists http(uid)]} {
        set http(uid) 0
    }
    set token [namespace current]::[incr http(uid)]
    variable $token
    upvar 0 $token state
    reset $token

    # Process command options.

    array set state {
        -binary              false
        -blocksize           8192
        -queryblocksize      8192
        -validate            0
        -headers             {}
        -timeout             0
        -type                application/x-www-form-urlencoded
        -queryprogress       {}
        -proxy_auth          ""
        state                header
        meta                 {}
        coding               {}
        currentsize          0
        totalsize            0
        querylength          0
        queryoffset          0
        type                 text/html
        body                 {}
        status               ""
        http                 ""
    }
    # These flags have their types verified [Bug 811170]
    array set type {
        -binary         boolean
        -blocksize      integer
        -queryblocksize integer
        -validate       boolean
        -timeout        integer
        -timeout        integer
    }
    set state(charset)  $defaultCharset
    set options {-binary -blocksize -channel -command -handler -headers \
            -progress -query -queryblocksize -querychannel -queryprogress\
            -validate -timeout -type -proxy_auth}
    set usage [join $options ", "]
    set options [string map {- ""} $options]
    set pat ^-([join $options |])$
    foreach {flag value} $args {
        if {[regexp $pat $flag]} {
            # Validate numbers
            if {[info exists type($flag)] && \
                    ![string is $type($flag) -strict $value]} {
                unset $token
                return -code error "Bad value for $flag ($value), must be $type($flag)"
            }
            set state($flag) $value
        } else {
            unset $token
            return -code error "Unknown option $flag, can be: $usage"
        }
    }

    # Make sure -query and -querychannel aren't both specified

    set isQueryChannel [info exists state(-querychannel)]
    set isQuery [info exists state(-query)]
    if {$isQuery && $isQueryChannel} {
        unset $token
        return -code error "Can't combine -query and -querychannel options!"
    }

    # Validate URL, determine the server host and port, and check proxy case
    # Recognize user:pass@host URLs also, although we do not do anything
    # with that info yet.

    set exp {^(([^:]*)://)?([^@]+@)?([^/:]+)(:([0-9]+))?(/.*)?$}
    if {![regexp -nocase $exp $url x prefix proto user host y port srvurl]} {
        unset $token
        return -code error "Unsupported URL: $url"
    }
    if {[string length $proto] == 0} {
        set proto http
        set url ${proto}://$url
    }
    if {![info exists urlTypes($proto)]} {
        unset $token
        return -code error "Unsupported URL type \"$proto\""
    }
    set defport [lindex $urlTypes($proto) 0]
    set defcmd [lindex $urlTypes($proto) 1]

    if {[string length $port] == 0} {
        set port $defport
    }
    if {[string length $srvurl] == 0} {
        set srvurl /
    }
    if {[string length $proto] == 0} {
        set url http://$url
    }
    set state(url) $url
    if {![catch {$http(-proxyfilter) $host} proxy]} {
        set phost [lindex $proxy 0]
        set pport [lindex $proxy 1]
    }

    # If a timeout is specified we set up the after event
    # and arrange for an asynchronous socket connection.

    if {$state(-timeout) > 0} {
        set state(after) [after $state(-timeout) \
                [list http::reset $token timeout]]
        set async -async
    } else {
        set async ""
    }

    # If we are using the proxy, we must pass in the full URL that
    # includes the server name.

    if {[info exists phost] && [string length $phost]} {
        #
        # Use SSL tunneling for https proxy
        #
        if {$proto == "https"} {
            # No async connection yet...
            set conStat [catch {socket $phost $pport} s]
            if {$conStat} {
                # something went wrong while trying to establish the connection
                # Clean up after events and such, but DON'T call the command callback
                # (if available) because we're going to throw an exception from here
                # instead.
                Finish $token "" 1
                cleanup $token
                return -code error $s
            }
            fconfigure $s -translation {auto crlf} -buffersize $state(-blocksize)
            puts $s "CONNECT $host:$port HTTP/1.0"
            puts $s "User-Agent: $http(-useragent)"
            if { [string length $state(-proxy_auth)] } {
                puts $s "Proxy-Authorization: $state(-proxy_auth)"
            }
            puts $s ""
            flush $s
            set proxyOK 0
            #
            # This is incredibly lame, but we're hoping for success and
            # will at least throw an error if there is a problem -- the details
            # of which will be haphazard at best.
            #
            # Read back the proxy server's response and single-mindedly
            # hunt for the connection ok status line -- ignoring everything else.
            #
            while {[gets $s proxyLine] > 0} {
                if {[regexp {^HTTP/.* 200 } $proxyLine]} {
                    set proxyOK 1
                }
            }

            #
            # If we could not detect a good connection, raise an error.
            #
            if {!$proxyOK} {
                close $s
                Finish $token "" 1
                cleanup $token
                return -code error "Unable to connect via proxy: $proxyLine"
            }

            # We've got a good proxy connection.
            # Switch the socket over to SSL for further communication.
            #
            # We're going to assume much about TLS right now.  For example,
            # the normal protocol registration would consist of the ::tls::socket
            # command and all of its options.  We're going to grab any of those
            # options and apply them to the ::tls::import command and hope for the
            # best. The idea here is to no worse than the non-proxied SSL support.
            #
            set conStat [catch {eval ::tls::import [lrange $defcmd 1 end] $s} s]
            if {$conStat} {
                # something went wrong while trying to establish the SSL protocol
                # Clean up after events and such, but DON'T call the command callback
                # (if available) because we're going to throw an exception from here
                # instead.
                Finish $token "" 1
                cleanup $token
                return -code error "Unable to establish SSL connection: $s"
            }

        } else {
            set srvurl $url
            set conStat [catch {eval $defcmd $async {$phost $pport}} s]
        }
    } else {
        set conStat [catch {eval $defcmd $async {$host $port}} s]
    }
    if {$conStat} {

        # something went wrong while trying to establish the connection
        # Clean up after events and such, but DON'T call the command callback
        # (if available) because we're going to throw an exception from here
        # instead.
        Finish $token "" 1
        cleanup $token
        return -code error $s
    }
    set state(sock) $s

    # Wait for the connection to complete

    if {$state(-timeout) > 0} {
        fileevent $s writable [list http::Connect $token]
        http::wait $token

        if {[string equal $state(status) "error"]} {
            # something went wrong while trying to establish the connection
            # Clean up after events and such, but DON'T call the command
            # callback (if available) because we're going to throw an 
            # exception from here instead.
            set err [lindex $state(error) 0]
            cleanup $token
            return -code error $err
        } elseif {![string equal $state(status) "connect"]} {
            # Likely to be connection timeout
            return $token
        }
        set state(status) ""
    }

    # Send data in cr-lf format, but accept any line terminators

    fconfigure $s -translation {auto crlf} -buffersize $state(-blocksize)

    # The following is disallowed in safe interpreters, but the socket
    # is already in non-blocking mode in that case.

    catch {fconfigure $s -blocking off}
    set how GET
    if {$isQuery} {
        set state(querylength) [string length $state(-query)]
        if {$state(querylength) > 0} {
            set how POST
            set contDone 0
        } else {
            # there's no query data
            unset state(-query)
            set isQuery 0
        }
    } elseif {$state(-validate)} {
        set how HEAD
    } elseif {$isQueryChannel} {
        set how POST
        # The query channel must be blocking for the async Write to
        # work properly.
        fconfigure $state(-querychannel) -blocking 1 -translation binary
        set contDone 0
    }

    if {[catch {
        puts $s "$how $srvurl HTTP/1.0"
        puts $s "Accept: $http(-accept)"
        if {$port == $defport} {
            # Don't add port in this case, to handle broken servers.
            # [Bug #504508]
            puts $s "Host: $host"
        } else {
            puts $s "Host: $host:$port"
        }
        puts $s "User-Agent: $http(-useragent)"
        foreach {key value} $state(-headers) {
            set value [string map [list \n "" \r ""] $value]
            set key [string trim $key]
            if {[string equal $key "Content-Length"]} {
                set contDone 1
                set state(querylength) $value
            }
            if {[string length $key]} {
                puts $s "$key: $value"
            }
        }
        if { [string length $state(-proxy_auth)] } {
            puts $s "Proxy-Authorization: $state(-proxy_auth)"
        }
        if {$isQueryChannel && $state(querylength) == 0} {
            # Try to determine size of data in channel
            # If we cannot seek, the surrounding catch will trap us

            set start [tell $state(-querychannel)]
            seek $state(-querychannel) 0 end
            set state(querylength) \
                    [expr {[tell $state(-querychannel)] - $start}]
            seek $state(-querychannel) $start
        }

        # Flush the request header and set up the fileevent that will
        # either push the POST data or read the response.
        #
        # fileevent note:
        #
        # It is possible to have both the read and write fileevents active
        # at this point.  The only scenario it seems to affect is a server
        # that closes the connection without reading the POST data.
        # (e.g., early versions TclHttpd in various error cases).
        # Depending on the platform, the client may or may not be able to
        # get the response from the server because of the error it will
        # get trying to write the post data.  Having both fileevents active
        # changes the timing and the behavior, but no two platforms
        # (among Solaris, Linux, and NT)  behave the same, and none 
        # behave all that well in any case.  Servers should always read thier
        # POST data if they expect the client to read their response.

        if {$isQuery || $isQueryChannel} {
            puts $s "Content-Type: $state(-type)"
            if {!$contDone} {
                puts $s "Content-Length: $state(querylength)"
            }
            puts $s ""
            fconfigure $s -translation {auto binary}
            fileevent $s writable [list http::Write $token]
        } else {
            puts $s ""
            flush $s
            fileevent $s readable [list http::Event $token]
        }

        if {! [info exists state(-command)]} {

            # geturl does EVERYTHING asynchronously, so if the user
            # calls it synchronously, we just do a wait here.

            wait $token
            if {[string equal $state(status) "error"]} {
                # Something went wrong, so throw the exception, and the
                # enclosing catch will do cleanup.
                return -code error [lindex $state(error) 0]
            }
        }
    } err]} {
        # The socket probably was never connected,
        # or the connection dropped later.

        # Clean up after events and such, but DON'T call the command callback
        # (if available) because we're going to throw an exception from here
        # instead.
        # instead.
        # if state(status) is error, it means someone's already called Finish
        # to do the above-described clean up.
        if {[string equal $state(status) "error"]} {
            Finish $token $err 1
        }
        cleanup $token
        return -code error $err
    }

    return $token
  }

CJN wanted something simpler for https thru proxies, and seeking to be able to do this:

    http::config -proxyhost someproxy.my.net -proxyport 8888
    http::register https 443 secureSocket

    http::geturl "https://www.securesite.com"

And thanks the fine examples earlier on this page, came up with this: (Although, the authentication piece hasn't actually been tested... no access to an authenticating proxy!)

    proc secureSocket {args} {
        set phost [::http::config -proxyhost]
        set pport [::http::config -proxyport]
        upvar host thost
        upvar port tport

        # if a proxy has been configured
        if {[string length $phost] && [string length $pport]} {
                # not a gret way to do authentication, but it works here
                set auth ""
                if [regexp {([^:]+):([^@]+)@(.+)} $phost x user pass phost] {
                        set auth "Proxy-Authorization: Basic [base64::encode $user:$pass]\n"
                }

                # create the socket to the proxy
                set socket [socket $phost $pport]
                fconfigure $socket -blocking 1 -buffering full -translation crlf
                puts $socket "CONNECT $thost:$tport HTTP/1.1"
                puts $socket $auth
                flush $socket

                while {[gets $socket r] > 0} {
                        append reply $r
                }

                # be sure there's a valid response code
                if {! [regexp {^HTTP/.* 200} $reply]} {
                        return -code error $reply
                }

                # now add tls to the socket and return it
                fconfigure $socket -blocking 0
                return [::tls::import $socket]
        }

        # if not proxifying, just create a tls socket directly
        return [::tls::socket $thost $tport]
    }

Authentication is used like this:

    http::config -proxyhost username:[email protected] -proxyport 8888

You'll also need:

    package require base64
    package require http
    package require tls