HTTPS stands for HyperText Transfer Protocol Secure . It's running HTTP over (a profile of) a SSL or TLS secured socket, and it means that the client can talk confidentially with the server and the client can know for sure the identity of the server that it is talking to.
The http core package itself does not feature https support but contains a plugin interface to delegate the TLS communication to another package. The following packages may be used:
The http package also can do secure HTTP (HTTPS) with the help of the tls package, as Michael A. Cleverly's example from the http manual illustrates:
package require http 2 package require tls 1.7 http::register https 443 [list ::tls::socket -autoservername true] set token [http::geturl https://my.secure.site/]
Also see Matt Newman's example: http://tls.sourceforge.net/tls.htm#HTTPS%20EXAMPLE .
dbohdan 2015-01-02: Many websites are disabling SSLv3 these days because the protocol is vulnerable . This means you will run into the error message sslv3 alert handshake failure when trying to connect unless you have support for the newer TLS protocol enabled. Despite what the package is called you have to enable support for TLS 1.x in ::tls::socket manually like so: ::tls::socket -tls1 1. I've updated the example at the top of the page to reflect this. RoyKeene I removed the mentions to -tls1 1 since newer versions of TclTLS (1.7+) deal with this correctly, I also added "-autoservername" which enables SNI, which is typically desirable.
HaO 2016-05-24: On windows, twapi (thanks Ashok!) may be used instead of the tls package as described above. For me, the main point are security fixes. Using twapi, fixes are installed with the operating system. With the tls package, the openssl library is statically linked and the application programmer must care about a current version. On Unix, this issue is not present if the openssl library is dynamically linked to the tls package.
Here is an extension of the upper example using twapi if present:
package require http 2 if {[catch {package require twapi_crypto}]} { package require tls 1.7 http::register https 443 [list ::tls::socket -autoservername true] } else { http::register https 443 [list ::twapi::tls_socket] } set token [http::geturl https://my.secure.site/]
I tested successfully twapi version 4.0.61 with http 2.8.9 (tcl 8.6.5).
To use self-signed certificates, one may switch-off certificate verification also for TWAPI:
package require http 2 if {[catch {package require twapi_crypto}]} { package require tls 1.7 http::register https 443 [list ::tls::socket -autoservername true] } else { proc ::twapi_verify args {return 1} http::register https 443 [list ::twapi::tls_socket -verifier twapi_verifier] } set token [http::geturl https://my.secure.site/]
stunnel can be used as a secure layer over an existing socket.
HaO 2018-04-17: The tcllib command '::autoproxy::tunnel_connect' allows to tunnel by proxies using the tls package. This command falls back to '::tls::socket', if no proxy host is set or the currently requested URL is within the excluded hosts.
Here is my code, which uses:
package require tls package require http package require autoproxy 1.7+ ; # autoproxy 1.7 supports twapi tls if { I have program-internal proxy settings } { ::autoproxy::configure -proxy_host example.com -proxy_port 880 ::autoproxy::configure -basic -username sampleuser -password samplepassword } else { ::autoproxy::init } if { [catch {package require twapi_crypto}] } { # No TWAPI -> use tls package require tls if {[::autoproxy::cget -host] ne ""} { # Proxy set -> use autoproxy http::register https 443 [list autoproxy::tls_socket -autoservername true] } else { # Proxy set -> use direct call to tls package http::register https 443 [list ::tls::socket -autoservername true] } } else { # TWAPI present if {[::autoproxy::cget -host] ne ""} { # Switch autoconf tls to twapi mode autoproxy::configure -tls_package twapi http::register https 443 autoproxy::tls_socket } else { # Direct twapi call http::register https 443 [list ::twapi::tls_socket] } }
TclCurl features https queries.
dbohdan 2017-02-02: The following code will tell you what version of the TLS protocol your HTTPS client uses when it talks to an up-to-date server. It should work with any TLS protocol handler, which in practice means either tls or twapi, as long as the free service it relies on is up.
package require http package require json # Register a TLS protocol handler and configure it as you wish here. package require tls ::http::register https 443 [list ::tls::socket] # Make a request to the API. set token [::http::geturl https://www.howsmyssl.com/a/check] puts [dict get [::json::json2dict [::http::data $token]] tls_version] ::http::cleanup $token
This has been supported by autoproxy for Quite Some Time. aspect (as of 2015-01-03) suspects all this section should simply be deleted.
[But none of this stuff knows how to tunnel proxies, as of December 2001.] [But maybe TclCurl helps?]
Erik Leunissen, Pat Thoyts, and David Bleicher dive deeply into HTTP 1.0 vs. 1.1, persistent connections, certificates, ... in a revealing thread [L1 ] on comp.lang.tcl.
Dave Griffin's proposal for tunnel proxies:
Add this about 90 lines into http::geturl:
: : set state(url) $url if {![catch {$http(-proxyfilter) $host} proxy]} { set phost [lindex $proxy 0] set pport [lindex $proxy 1] } # If a timeout is specified we set up the after event # and arrange for an asynchronous socket connection. if {$state(-timeout) > 0} { set state(after) [after $state(-timeout) \ [list http::reset $token timeout]] set async -async } else { set async "" } # If we are using the proxy, we must pass in the full URL that # includes the server name. if {[info exists phost] && [string length $phost]} { # # Use SSL tunneling for https proxy # if {$proto == "https"} { # No async connection yet... set conStat [catch {socket $phost $pport} s] if {$conStat} { # something went wrong while trying to establish the connection # Clean up after events and such, but DON'T call the command callback # (if available) because we're going to throw an exception from here # instead. Finish $token "" 1 cleanup $token return -code error $s } fconfigure $s -translation {auto crlf} -buffersize $state(-blocksize) puts $s "CONNECT $host:$port HTTP/1.0" puts $s "User-Agent: $http(-useragent)" puts $s "" flush $s set proxyOK 0 # # This is incredibly lame, but we're hoping for success and # will at least throw an error if there is a problem -- the details # of which will be haphazard at best. # # Read back the proxy server's response and single-mindedly # hunt for the connection ok status line -- ignoring everything else. # while {[gets $s proxyLine] > 0} { if {[regexp {^HTTP/.* 200 } $proxyLine]} { set proxyOK 1 } } # # If we could not detect a good connection, raise an error. # if {!$proxyOK} { close $s Finish $token "" 1 cleanup $token return -code error "Unable to connect via proxy: $proxyLine" } # We've got a good proxy connection. # Switch the socket over to SSL for further communication. # # We're going to assume much about TLS right now. For example, # the normal protocol registration would consist of the ::tls::socket # command and all of its options. We're going to grab any of those # options and apply them to the ::tls::import command and hope for the # best. The idea here is to no worse than the non-proxied SSL support. # set conStat [catch {eval ::tls::import [lrange $defcmd 1 end] $s} s] if {$conStat} { # something went wrong while trying to establish the SSL protocol # Clean up after events and such, but DON'T call the command callback # (if available) because we're going to throw an exception from here # instead. Finish $token "" 1 cleanup $token return -code error "Unable to establish SSL connection: $s" } } else { set srvurl $url set conStat [catch {eval $defcmd $async {$phost $pport}} s] } } else { set conStat [catch {eval $defcmd $async {$host $port}} s] } if {$conStat} { # something went wrong while trying to establish the connection # Clean up after events and such, but DON'T call the command callback # (if available) because we're going to throw an exception from here # instead. Finish $token "" 1 cleanup $token return -code error $s } set state(sock) $s : :
We're testing this out now. If it holds together (and nobody has any other suggestions on how to do the error handling better) I'll see if the Tcl Core group would like it.
Of course, the Apache Tcl stuff runs fine on an SSL-enabled Apache.
Jos Decoster : The following code takes Dave Griffin's proposal for tunnel proxies and adds proxy authentication. The proxy authentiation is based on information from Pat Thoyts found in http authentication.
This modified http geturl command takes an additional '-proxy_auth' argument specifying the base64 encoded authentication method (only Basic currently supported), username and password needed to do the proxy authentication. You can use it like this:
% http::register https $https_port ::tls::socket % http::config -proxyhost $proxy_host -proxyport $proxy_port % http::geturl $url -proxy_auth [concat "Basic" [base64::encode $proxy_auth_user:$proxy_auth_password]]
I added the proxy authentication to the http::geturl function because I already had to modify it. It may be better to add it to the http::config command.
The modified http::geturl command:
proc http::geturl { url args } { if { $::TctInstall::debug_mode } { puts "\n\n\nUSING PATCHED GETURL!\n\n\n" } variable http variable urlTypes variable defaultCharset # Initialize the state variable, an array. We'll return the # name of this array as the token for the transaction. if {![info exists http(uid)]} { set http(uid) 0 } set token [namespace current]::[incr http(uid)] variable $token upvar 0 $token state reset $token # Process command options. array set state { -binary false -blocksize 8192 -queryblocksize 8192 -validate 0 -headers {} -timeout 0 -type application/x-www-form-urlencoded -queryprogress {} -proxy_auth "" state header meta {} coding {} currentsize 0 totalsize 0 querylength 0 queryoffset 0 type text/html body {} status "" http "" } # These flags have their types verified [Bug 811170] array set type { -binary boolean -blocksize integer -queryblocksize integer -validate boolean -timeout integer -timeout integer } set state(charset) $defaultCharset set options {-binary -blocksize -channel -command -handler -headers \ -progress -query -queryblocksize -querychannel -queryprogress\ -validate -timeout -type -proxy_auth} set usage [join $options ", "] set options [string map {- ""} $options] set pat ^-([join $options |])$ foreach {flag value} $args { if {[regexp $pat $flag]} { # Validate numbers if {[info exists type($flag)] && \ ![string is $type($flag) -strict $value]} { unset $token return -code error "Bad value for $flag ($value), must be $type($flag)" } set state($flag) $value } else { unset $token return -code error "Unknown option $flag, can be: $usage" } } # Make sure -query and -querychannel aren't both specified set isQueryChannel [info exists state(-querychannel)] set isQuery [info exists state(-query)] if {$isQuery && $isQueryChannel} { unset $token return -code error "Can't combine -query and -querychannel options!" } # Validate URL, determine the server host and port, and check proxy case # Recognize user:pass@host URLs also, although we do not do anything # with that info yet. set exp {^(([^:]*)://)?([^@]+@)?([^/:]+)(:([0-9]+))?(/.*)?$} if {![regexp -nocase $exp $url x prefix proto user host y port srvurl]} { unset $token return -code error "Unsupported URL: $url" } if {[string length $proto] == 0} { set proto http set url ${proto}://$url } if {![info exists urlTypes($proto)]} { unset $token return -code error "Unsupported URL type \"$proto\"" } set defport [lindex $urlTypes($proto) 0] set defcmd [lindex $urlTypes($proto) 1] if {[string length $port] == 0} { set port $defport } if {[string length $srvurl] == 0} { set srvurl / } if {[string length $proto] == 0} { set url http://$url } set state(url) $url if {![catch {$http(-proxyfilter) $host} proxy]} { set phost [lindex $proxy 0] set pport [lindex $proxy 1] } # If a timeout is specified we set up the after event # and arrange for an asynchronous socket connection. if {$state(-timeout) > 0} { set state(after) [after $state(-timeout) \ [list http::reset $token timeout]] set async -async } else { set async "" } # If we are using the proxy, we must pass in the full URL that # includes the server name. if {[info exists phost] && [string length $phost]} { # # Use SSL tunneling for https proxy # if {$proto == "https"} { # No async connection yet... set conStat [catch {socket $phost $pport} s] if {$conStat} { # something went wrong while trying to establish the connection # Clean up after events and such, but DON'T call the command callback # (if available) because we're going to throw an exception from here # instead. Finish $token "" 1 cleanup $token return -code error $s } fconfigure $s -translation {auto crlf} -buffersize $state(-blocksize) puts $s "CONNECT $host:$port HTTP/1.0" puts $s "User-Agent: $http(-useragent)" if { [string length $state(-proxy_auth)] } { puts $s "Proxy-Authorization: $state(-proxy_auth)" } puts $s "" flush $s set proxyOK 0 # # This is incredibly lame, but we're hoping for success and # will at least throw an error if there is a problem -- the details # of which will be haphazard at best. # # Read back the proxy server's response and single-mindedly # hunt for the connection ok status line -- ignoring everything else. # while {[gets $s proxyLine] > 0} { if {[regexp {^HTTP/.* 200 } $proxyLine]} { set proxyOK 1 } } # # If we could not detect a good connection, raise an error. # if {!$proxyOK} { close $s Finish $token "" 1 cleanup $token return -code error "Unable to connect via proxy: $proxyLine" } # We've got a good proxy connection. # Switch the socket over to SSL for further communication. # # We're going to assume much about TLS right now. For example, # the normal protocol registration would consist of the ::tls::socket # command and all of its options. We're going to grab any of those # options and apply them to the ::tls::import command and hope for the # best. The idea here is to no worse than the non-proxied SSL support. # set conStat [catch {eval ::tls::import [lrange $defcmd 1 end] $s} s] if {$conStat} { # something went wrong while trying to establish the SSL protocol # Clean up after events and such, but DON'T call the command callback # (if available) because we're going to throw an exception from here # instead. Finish $token "" 1 cleanup $token return -code error "Unable to establish SSL connection: $s" } } else { set srvurl $url set conStat [catch {eval $defcmd $async {$phost $pport}} s] } } else { set conStat [catch {eval $defcmd $async {$host $port}} s] } if {$conStat} { # something went wrong while trying to establish the connection # Clean up after events and such, but DON'T call the command callback # (if available) because we're going to throw an exception from here # instead. Finish $token "" 1 cleanup $token return -code error $s } set state(sock) $s # Wait for the connection to complete if {$state(-timeout) > 0} { fileevent $s writable [list http::Connect $token] http::wait $token if {[string equal $state(status) "error"]} { # something went wrong while trying to establish the connection # Clean up after events and such, but DON'T call the command # callback (if available) because we're going to throw an # exception from here instead. set err [lindex $state(error) 0] cleanup $token return -code error $err } elseif {![string equal $state(status) "connect"]} { # Likely to be connection timeout return $token } set state(status) "" } # Send data in cr-lf format, but accept any line terminators fconfigure $s -translation {auto crlf} -buffersize $state(-blocksize) # The following is disallowed in safe interpreters, but the socket # is already in non-blocking mode in that case. catch {fconfigure $s -blocking off} set how GET if {$isQuery} { set state(querylength) [string length $state(-query)] if {$state(querylength) > 0} { set how POST set contDone 0 } else { # there's no query data unset state(-query) set isQuery 0 } } elseif {$state(-validate)} { set how HEAD } elseif {$isQueryChannel} { set how POST # The query channel must be blocking for the async Write to # work properly. fconfigure $state(-querychannel) -blocking 1 -translation binary set contDone 0 } if {[catch { puts $s "$how $srvurl HTTP/1.0" puts $s "Accept: $http(-accept)" if {$port == $defport} { # Don't add port in this case, to handle broken servers. # [Bug #504508] puts $s "Host: $host" } else { puts $s "Host: $host:$port" } puts $s "User-Agent: $http(-useragent)" foreach {key value} $state(-headers) { set value [string map [list \n "" \r ""] $value] set key [string trim $key] if {[string equal $key "Content-Length"]} { set contDone 1 set state(querylength) $value } if {[string length $key]} { puts $s "$key: $value" } } if { [string length $state(-proxy_auth)] } { puts $s "Proxy-Authorization: $state(-proxy_auth)" } if {$isQueryChannel && $state(querylength) == 0} { # Try to determine size of data in channel # If we cannot seek, the surrounding catch will trap us set start [tell $state(-querychannel)] seek $state(-querychannel) 0 end set state(querylength) \ [expr {[tell $state(-querychannel)] - $start}] seek $state(-querychannel) $start } # Flush the request header and set up the fileevent that will # either push the POST data or read the response. # # fileevent note: # # It is possible to have both the read and write fileevents active # at this point. The only scenario it seems to affect is a server # that closes the connection without reading the POST data. # (e.g., early versions TclHttpd in various error cases). # Depending on the platform, the client may or may not be able to # get the response from the server because of the error it will # get trying to write the post data. Having both fileevents active # changes the timing and the behavior, but no two platforms # (among Solaris, Linux, and NT) behave the same, and none # behave all that well in any case. Servers should always read thier # POST data if they expect the client to read their response. if {$isQuery || $isQueryChannel} { puts $s "Content-Type: $state(-type)" if {!$contDone} { puts $s "Content-Length: $state(querylength)" } puts $s "" fconfigure $s -translation {auto binary} fileevent $s writable [list http::Write $token] } else { puts $s "" flush $s fileevent $s readable [list http::Event $token] } if {! [info exists state(-command)]} { # geturl does EVERYTHING asynchronously, so if the user # calls it synchronously, we just do a wait here. wait $token if {[string equal $state(status) "error"]} { # Something went wrong, so throw the exception, and the # enclosing catch will do cleanup. return -code error [lindex $state(error) 0] } } } err]} { # The socket probably was never connected, # or the connection dropped later. # Clean up after events and such, but DON'T call the command callback # (if available) because we're going to throw an exception from here # instead. # instead. # if state(status) is error, it means someone's already called Finish # to do the above-described clean up. if {[string equal $state(status) "error"]} { Finish $token $err 1 } cleanup $token return -code error $err } return $token }
CJN wanted something simpler for https thru proxies, and seeking to be able to do this:
http::config -proxyhost someproxy.my.net -proxyport 8888 http::register https 443 secureSocket http::geturl "https://www.securesite.com"
And thanks the fine examples earlier on this page, came up with this: (Although, the authentication piece hasn't actually been tested... no access to an authenticating proxy!)
proc secureSocket {args} { set phost [::http::config -proxyhost] set pport [::http::config -proxyport] upvar host thost upvar port tport # if a proxy has been configured if {[string length $phost] && [string length $pport]} { # not a gret way to do authentication, but it works here set auth "" if [regexp {([^:]+):([^@]+)@(.+)} $phost x user pass phost] { set auth "Proxy-Authorization: Basic [base64::encode $user:$pass]\n" } # create the socket to the proxy set socket [socket $phost $pport] fconfigure $socket -blocking 1 -buffering full -translation crlf puts $socket "CONNECT $thost:$tport HTTP/1.1" puts $socket $auth flush $socket while {[gets $socket r] > 0} { append reply $r } # be sure there's a valid response code if {! [regexp {^HTTP/.* 200} $reply]} { return -code error $reply } # now add tls to the socket and return it fconfigure $socket -blocking 0 return [::tls::import $socket] } # if not proxifying, just create a tls socket directly return [::tls::socket $thost $tport] }
Authentication is used like this:
http::config -proxyhost username:[email protected] -proxyport 8888
You'll also need:
package require base64 package require http package require tls