DDG 2020-03-19: (WIP) I had to play around a little bit with the GO programming language by compiling the webview go library . I found it quite nice to directly download packages from github to your package directory using go get. This used git in the background, so the go approach needed a working installation of git which again needed installs of Perl, Tcl, ... around 6000 individual files and 650MB hard disk space. Can we do some small scale thing for Tcl here which works in a similar way. We need a Tcl interpreter, the package tls for the https protocol and finally the json library of Tcllib to parse the json output of the github api . The goal is to download only the required parts of other packages to a parallel directory of your package you are developing.
DDG 2020-03-20: Added recursive download of folders and command line interface for direct download of individual github folders from repositories.
DDG 2020-03-22: Making a package dgtools::repo out of this code which as wells supports chiselapp repositories .
Here is some preliminary code which allows to download individual folders from a github repo:
#!/usr/bin/env tclsh
# file github/github.tcl
# chicken and egg problem we need non-standard packages tls and json ...
package require tls
package require http
::http::register https 443 ::tls::socket
namespace eval ::github {
variable libdir [file normalize [file join [file dirname [info script]] ..]]
if {[lsearch $::auto_path $libdir] == -1} {
lappend auto_path $libdir
}
}
# I already placed the json folder below of the github folder
package require json
package provide github::github 0.2
package provide github 0.2
# Tcl package download
proc ::github::github {cmd owner repo folder package} {
variable libdir
set url https://api.github.com/repos/$owner/$repo/contents/$folder/$package
#puts $url
#puts [lindex $d 1]
set folder [file join [lindex $::auto_path end] $package]
if {$cmd eq "import" && [file exists $folder]} {
return
} elseif {$cmd eq "update" && [file exists $folder]} {
file delete $folder
}
download $url $folder
}
# Folder download
proc ::github::download {url folder {debug true}} {
if {![file exists $folder]} {
file mkdir $folder
}
set data [http::data [http::geturl $url]]
set d [json::json2dict $data]
set l [llength $d]
set files [list]
for {set i 0} {$i < $l} {incr i 1} {
set dic [dict create {*}[lindex $d $i]]
set file [dict get $dic download_url]
set type [dict get $dic type]
if {$file eq "null" && $type eq "dir"} {
set file [dict get $dic url]
set file [regsub {.ref=master} $file ""]
}
lappend files [list $type $file]
}
# TODO subfolders (done)
foreach item $files {
set file [lindex $item 1]
set type [lindex $item 0]
if {$debug} {
puts "fetching $file"
}
if {$type eq "file"} {
set fname [file tail $file]
set fname [file join $folder $fname]
set f [open $fname w]
fconfigure $f -translation binary
set tok [http::geturl $file -channel $f]
set Stat [::http::status $tok]
flush $f
close $f
http::cleanup $tok
} else {
if {$debug} {
puts "fetch new folder $file ..."
}
set nfolder [file join $folder [file tail $file]]
download $file $nfolder $debug
}
}
}
if {[info exists argv0] && $argv0 eq [info script] && [regexp {github} $argv0]} {
set debug true
set idx [lsearch -regexp $argv -silent]
if {$idx > -1} {
set debug false
}
set argv [lsearch -all -inline -regexp -not $argv -silent]
if {[llength $argv] == 0} {
puts "Usage: $argv0 ?--silent? github-url ?directory?\nif directory is not given, current directory is assumed"
exit 0
}
set url [lindex $argv 0]
# removing trailing slash
set url [regsub {/$} $url ""]
set folder [file tail $url]
if {[llength $argv] > 1} {
set folder [lindex $argv 1]
}
if {[regexp {https://github.com/([^/]+)/([^/]+)/tree/master/(.+)} $url -> owner repo gfolder]} {
set url https://api.github.com/repos/$owner/$repo/contents/$gfolder
::github::download $url $folder $debug
} else {
puts stderr "Unkown url type $url"
exit 0
}
exit 0
}This code can be used as Tcl package. Let's use snit and tablelist directly from github without installing tcllib or tklib:
package require github github::github import tcltk tcllib modules snit github::github import tcltk tklib modules tablelist package require snit package require tablelist
There is as well a command line interfce so that you can download any github folder regardless if it is a Tcl package or anything else:
$ github.tcl --silent https://github.com/tcltk/tklib/tree/master/modules/tablelist out
Should download all files in the tablelist folder of github in a folder out.
Please discuss here, I am open for suggestions:
DDG 2020-03-19: I know you can install snit by installing all of tcllib. But I think this approach has as well its charm, as it only takes what you really need. And sometimes it is nice to fetch packages which are not in TEA repositories but only on github.
DDG 2020-03-20: Adding recursive download for any github folder. So this is the Tcl solution to this lengthy discussion on stackoverflow.com