Version 25 of A versioning virtual filesystem

Updated 2004-09-07 20:19:51 by SEH

SEH A versioning virtual filesystem.

First untested rough draft.

Usage: Mount <existing directory> <virtual versioning directory>

Similar to historical versioning filesystems, each edited version of a file is saved separately; each version file is tagged with a timestamp. A tag is a semi-colon and tag info appended to the file name. A deleted file is represented by a new zero-length file with timestamp and a tag reading "deleted". By default only the latest versions of files are visible. If the latest version is marked deleted, it is invisible. Choice of the visible version can be customized in the VersionSort procedure. All versions can be accessed simply by looking into the existing directory used in the Mount command.

Based on A template virtual filesystem


SEH 19 Jul 2004 - I added three optional mount flags: -keep , -project , and -time

Usage: Mount <existing directory> -keep <integer> -project <string> -time <seconds or string acceptable to clock scan> <virtual versioning directory>

-keep: sets the maximum number of versions per file to retain. If the -project tag is used, other projects' versions are ignored.

-project: specify a named project and all new edits get tagged with this name string. As long as you're mounted with this option, versions with the corresponding tag will be preferred, and edits tagged by other projects will be invisible. Multiple -project tags can be used; new versions will tagged with all specified project tags - thus you can share edits among several projects.

-time: specify a time in the past and the vfs will appear just as it did at that time. You can make new edits, but they will be invisible until you remount without a time specified. Can be used in conjunction with -project.

Beware! Needs testing!


 package require Tcl 8.4
 package require vfs 1

 namespace eval ::vfs::template::version {}

 proc ::vfs::template::version::Mount {args} {
        set pathto [eval MountProcedure $args]
        set path [lindex $pathto 0]
        set to [lindex $pathto 1]
        ::vfs::filesystem mount $to [list ::vfs::template::version::handler $path]
        ::vfs::RegisterMount $to [list ::vfs::template::version::Unmount]
        return $to
 }

 proc ::vfs::template::version::Unmount {to} {
        set path [lindex [::vfs::filesystem info $to] end]
        UnmountProcedure $path $to
        ::vfs::filesystem unmount $to
 }

 proc ::vfs::template::version::handler {path cmd root relative actualpath args} {
        switch -- $cmd {
                access {
                        set mode [lindex $args 0]
                        set error [catch {Access $path $root $relative $actualpath $mode}]
                        if $error {vfs::filesystem posixerror $::vfs::posix(EACCES) ; return -code error $::vfs::posix(EACCES)}
                }
                createdirectory {
                        CreateDirectory $path $root $relative $actualpath
                }
                deletefile {
                        DeleteFile $path $root $relative $actualpath
                }
                fileattributes {
                        set index [lindex $args 0]
                        set value [lindex $args 1]
                        array set attributes [FileAttributes $path $root $relative $actualpath]
                        if {$index == {}} {
                                return [lsort [array names attributes]]
                        }
                        set attribute [lindex [lsort [array names attributes]] $index]
                        if {$value == {}} {
                                return $attributes($attribute)
                        }
                        FileAttributesSet $path $root $relative $actualpath $attribute $value
                }
                matchindirectory {
                        set pattern [lindex $args 0]
                        set types [lindex $args 1]
                        return [MatchInDirectory $path $root $relative $actualpath $pattern $types]
                } open {
                        set mode [lindex $args 0]
                        if {$mode == {}} {set mode r}
                        set permissions [lindex $args 1]
                        set channelID [Open $path $root $relative $actualpath $mode $permissions]
                        switch -glob -- $mode {
                                "" -
                                "r*" -
                                "w*" {
                                        seek $channelID 0
                                }
                                "a*" {
                                            seek $channelID 0 end
                                }
                                default {
                                        ::vfs::filesystem posixerror $::vfs::posix(EINVAL)
                                        return -code error $::vfs::posix(EINVAL)
                                }
                        }
                        set result $channelID
                        lappend result [list ::vfs::template::version::Close $channelID $path $root $relative $actualpath $mode]
                        return $result
                }
                removedirectory {
                        set recursive [lindex $args 0]
                        if !$recursive {
                                if {[MatchInDirectory $path $root $relative $actualpath * 0] != {}} {
                                        ::vfs::filesystem posixerror $::vfs::posix(EEXIST)
                                        return -code error $::vfs::posix(EEXIST)
                                }
                        }
                        RemoveDirectory $path $root $relative $actualpath
                }
                stat {
                        return [Stat $path $root $relative $actualpath]
                }
                utime {
                        set atime [lindex $args 0]
                        set mtime [lindex $args 1]
                        Utime $path $root $relative $actualpath $atime $mtime
                }
        }
 }

 namespace eval ::vfs::template::version {

 proc MountProcedure {args} {
        set to [file normalize [lindex $args end]]
        set args [lrange $args 0 end-1]
        set path [lindex $args 0]
        set args [lrange $args 1 end]
        if {![catch {vfs::filesystem info $to}]} {
                # unmount old mount
                Unmount $to
        }

        set argsLength [llength $args]
        for {set i 0} {$i < $argsLength} {incr i} {
                switch -- [lindex $args $i] {
                        -keep {
                                set keep [lindex $args [incr i]]
                                if ![string is digit -strict $keep] {continue}
                                set ::vfs::template::version::keep($to) $keep
                        }
                        -project {
                                set project [lindex $args [incr i]]
                                set project [string map {; &s & &a} $project]
                                lappend ::vfs::template::version::project($to) $project
                        }
                        -time {
                                set time [lindex $args [incr i]]
                                if ![string is digit -strict $time] {catch {set time [clock scan $time]}}
                                if ![string is digit -strict $time] {continue}
                                set time "[clock scan [clock format $time -format %Y%m%dT%H%M%S -gmt 1]]000"
                                set ::vfs::template::version::time($to) $time
                        }
                }
        }

        file mkdir $path

        lappend pathto $path
        lappend pathto $to
        return $pathto
 }

 proc UnmountProcedure {path to} {
        array unset ::vfs::template::version::keep $to
        array unset ::vfs::template::version::project $to
        array unset ::vfs::template::version::time $to
        return
 }

 proc Access {path root relative actualpath mode} {
        set modeString [::vfs::accessMode $mode]
        if {$modeString == "F"} {set modeString RWX}
        set modeString [split $modeString {}]
        set fileName [VAcquireFile $path $root $relative $actualpath]
        if [file readable $fileName] {lappend fileString R}
        if [file writable $fileName] {lappend fileString W}
        if [file executable $fileName] {lappend fileString X}
        foreach mS $modeString {
                set errorMessage "not [string map {R readable W writable X executable} $mS]"
                if {[lsearch $fileString $mS] == -1} {error $errorMessage}
        }
 }

 proc CreateDirectory {path root relative actualpath} {
        file mkdir [file join $path $relative]
 }

 proc DeleteFile {path root relative actualpath} {
        set fileName [VFileNameEncode [file join $path $relative]]\;[VCreateTag]
        set fileName [split $fileName \;]
        set fileName [linsert $fileName 2 "deleted"]
        set fileName [join $fileName \;]
        close [open $fileName w]
 }

 proc FileAttributes {path root relative actualpath} {
        set fileName [VAcquireFile $path $root $relative $actualpath]
        file attributes $fileName
 }

 proc FileAttributesSet {path root relative actualpath attribute value} {
        set fileName [VAcquireFile $path $root $relative $actualpath]
        set newFileName [VFileNameEncode [file join $path $relative]]\;[VCreateTag]
        file copy $fileName $newFileName
        file attributes $newFileName $attribute $value
 }

 proc MatchInDirectory {path root relative actualpath pattern types} {
        if [::vfs::matchDirectories $types] {lappend typeString d}
        if [::vfs::matchFiles $types] {lappend typeString f}

        set globList [glob -directory [file join $path $relative] -nocomplain -types $typeString *]
        set pathLength [expr [string length $path] - 1]
        set newGlobList {}
        set acquireAttempts {}
        foreach gL $globList {
                if [file isfile $gL] {
                        set gL [VFileNameDecode $gL]
                        if {[lsearch $acquireAttempts $gL] > -1} {continue}
                        lappend acquireAttempts $gL
                        set acquiredFile [VAcquireFile $path $root [list $relative/[file tail $gL]] $actualpath]
                        if [string equal $acquiredFile $gL] {continue}
                }
                set gL [string replace $gL 0 $pathLength $root]
                if [string match $pattern [file tail $gL]] {lappend newGlobList $gL}
        }
        return [lsort -unique $newGlobList]
 }

 proc Open {path root relative actualpath mode permissions} {
        set fileName [VAcquireFile $path $root $relative $actualpath]
        if {$fileName == [file join $path $relative]} {
                set fileName [VFileNameEncode [file join $path $relative]]\;[VCreateTag]
                set channelID [open $fileName $mode]
                catch {file attributes $fileName -permissions $permissions}
                return $channelID
        }
        set f [open $fileName r]
        fconfigure $f -translation binary
        set filed [vfs::memchan]
        fconfigure $filed -translation binary
        fcopy $f $filed
        close $f
        fconfigure $filed -translation auto
        seek $filed 0
        return $filed
 }

 proc Close {channelID path root relative actualpath mode} {
 # Do not close the channel in the close callback!
 # It will crash Tcl!  Honest!
 # The core will close the channel after you've taken what info you need from it.

 #        close $channelID
        if {$mode == "r"} {return}
        set fileName [VFileNameEncode [file join $path $relative]]\;[VCreateTag]
        set f [open $fileName w]
        fconfigure $f -translation binary
        fconfigure $channelID -translation binary
        seek $channelID 0
        fcopy $channelID $f
        close $f
        return
 }

 proc RemoveDirectory {path root relative actualpath} {
        catch {file delete [file join $path $relative]}
 }

 proc Stat {path root relative actualpath} {
        set fileName [VAcquireFile $path $root $relative $actualpath]
        file stat $fileName fs
        return [array get $fs]
 }

 proc Utime {path root relative actualpath atime mtime} {
        set fileName [VAcquireFile $path $root $relative $actualpath]
        set newFileName [VFileNameEncode [file join $path $relative]]\;[VCreateTag]
        file copy $fileName $newFileName
        file atime $newFileName $atime
        file mtime $newFileName $mtime
 }

 proc VAcquireFile {path root relative actualpath} {
        if [file isdirectory [file join $path $relative]] {return [file join $path $relative]}
        set fileName [VFileNameEncode [file join $path $relative]]
        set versions [glob -path $fileName -nocomplain -types f "\;*"]
        if {$versions == {}} {return [file join $path $relative]}

        set checkProject 0
        if [info exists $::vfs::template::version::project($root)] {
                set checkProject 1
        }
        foreach ver $versions {
                set ver $root/[file tail $ver]
                lappend versionFiles $ver
                if $checkProject {
                        foreach project $::vfs::template::version::project($root) {
                                if {[lsearch [lrange [split $ver \;] 1 end] $project] > -1} {lappend projectFiles $ver}
                        }
                }
        }
        unset versions

        if ![catch {if {[llength $projectFiles] <= $::vfs::template::version::keep($root)} {error}}] {
                set keep $::vfs::template::version::keep($root)
                set projectFiles [lsort -decreasing -dictionary $projectFiles]
                set fileNumber [llength $projectFiles]
                for {set i [incr fileNumber -1]} {$i >= 0} {incr i -1} {
                        if {[llength $projectFiles] <= $keep} {break}
                        if ![catch {file delete $path/[lindex $projectFiles $i]}] {set projectFiles [lreplace $projectFiles $i $i]}
                }
        }

        set fileName [file tail [lindex [lsort -command VersionSort $versionFiles] 0]]
        if {[lrange [split $fileName \;] 2] == "deleted"} {return [file join $path $relative]}
        if {$checkProject} {
                set projectMember 0
                set tags [lrange [split $fileName \;] 1 end]
                if {[lindex $tags 1] == "deleted"} {set tags [lreplace $tags 1 1]}
                foreach project $::vfs::template::version::project($root) {
                                if {[lsearch $tags $project] > -1} {set projectMember 1}
                }
                set projectLength [llength $tags]
                if {($projectLength > 1) && !$projectMember} {return [file join $path $relative]}
        }
        set fileTime [lindex [split $fileName \;] 1]
        if [info exists $::vfs::template::version::time($root)] {
                if {$fileTime > $::vfs::template::version::time($root)} {return [file join $path $relative]}
        }
        return [file dirname $path/relative]/$fileName
 }

 proc VCreateTag {} {
        set tag [clock scan [clock format [clock seconds] -format %Y%m%dT%H%M%S -gmt 1]][string range [clock clicks -milliseconds] end-2 end]
        if [info exists ::vfs::template::version::project] {
                set projectTag [join $::vfs::template::version::project \;]
                set tag [join $tag $projectTag \;]
        }
        return $tag
 }

 proc VersionSort {element1 element2} {
        set root [file dirname $element1]
        set element1 [file tail $element1]
        set element2 [file tail $element2]
        if [string equal $element1 $element2] {return 0}
        set sorted [lsort -dictionary -decreasing [list $element1 $element2]]
        if {[lindex $sorted 0] == $element1} {set returnValue -1}
        if {[lindex $sorted 0] == $element2} {set returnValue 1}

I don't understand the rest of this proc ... it will never be executed. Above, we sort a 2 element list, the 0th element is either going to be the former 0th, or the former 1th, so one of the above two conditions will be true, and the proc will return -- CMcC

SEH -- The above lines will only return if element1 and element2 are equal, otherwise only a default return value will be set. The default may be overridden if a preferred time and/or project is specified.

        if [info exists ::vfs::template::version::time($root)] {
                set returnValue -1
                set time $::vfs::template::version::time($root)
                set time1 [lindex [split $element1 \;] 1]
                set time2 [lindex [split $element2 \;] 1]
                if {$time1 > $time} {set time1 [expr $time2 - 1]}
                if {($time2 <= $time) && ($time2 > $time1)} {set returnValue 1}
        }

        if [info exists ::vfs::template::version::project($root)] {
                foreach project $::vfs::template::version::project($root) {
                        set project1 [lsearch [lrange [split $element1 \;] 1 end] $project]
                        set project2 [lsearch [lrange [split $element2 \;] 1 end] $project]
                        incr project1 ; incr project2
                        if {$project2 && !$project1} {set returnValue 1 ; break}
                        if {$project1 && !$project2} {set returnValue -1 ; break}
                        set tagEnd1 [lindex [split $element1 \;] 2]
                        if {$tagEnd1 == "deleted"} {set tagEnd1 [lindex [split $element1 \;] 3]}
                        set tagEnd2 [lindex [split $element2 \;] 2]
                        if {$tagEnd2 == "deleted"} {set tagEnd2 [lindex [split $element2 \;] 3]}
                        if {($tagEnd1 == {}) && !($tagEnd2 == {})} {set returnValue -1}
                        if {!($tagEnd1 == {}) && ($tagEnd2 == {})} {set returnValue 1}
                }
        }
        return $returnValue
 }

 proc VFileNameEncode {filename} {
        set filename [file dirname $filename]/[string map {& &a} [file tail $filename]]
        set filename [file dirname $filename]/[string map {; &s} [file tail $filename]]
 }

 proc VFileNameDecode {filename} {
        set filename [file dirname $filename]/[lindex [split [file tail $filename] \;] 0]
        set filename [file dirname $filename]/[string map {&s ;} [file tail $filename]]
        set filename [file dirname $filename]/[string map {&a &} [file tail $filename]]
 }

 }
 # end namespace eval vfs::template::version

Wow! -jcw

Word - CMcC

escargo 17 Jul 2004 - I remember using the Digital Equipment Corporation's VMS operating system, which had a feature for saving multiple versions of files. You could set a limit on how many past versions you wanted retained, so that there would be some bounds on how much disk space you consumed. It might be worthwhile digging into old VMS manuals to see if there are utility functions related to versioned files that are worth implementing. The first that comes to mind would be purge, where only current versions of files are retained.

GWL 18 Jul 2004 - Actually the purge command took an argument specifing how many versions to keep (/keep=N if my memory is working), the default was just one.

escargo 18 Jul 2004 - It's that kind of detail from 15 years ago that made me suggest going to the manuals. Being able to set the limit on how many versions to keep would also be a good thing.


wcf3 18 Jul 2004 - I found a problem using glob while running on linux that I fixed by removing the file tail from the call to VFileNameDecode in the procedure MatchInDirectory. Everything worked great after that! I didn't want to alter the source until after I've had a chance to test it a bit more.

SEH I fixed the problem in the call to VFileNameDecode in MatchInDirectory. Thanks for the catch.

wcf3 20 Jul 2004 - It just keeps getting better! I would like to see another option that allows you to specify the age depth of the history files. So you could mount a versioning file system and lose history after some length of time, say 3 months old. This way files that are changed frequently over a short time will still have all versions for that time period. It would also be nice to be able to change the time option on the fly by a call to the mount procedures as well as query the versions for a file. I should probably let you know that I invented a versioning file system years back that is being sold as GoBack; I think Norton owns it now...all I can say is that my head is full of ideas for this! (whack me upside the head if I get out of control :-)

SEH GAAAHHH!!! Actually I'd like to limit the number and complexity of pruning options. It slows things down and disrupts the simplicity of the code. The nice thing about a scripted vfs is that the real location is perfectly accessible by standard means, so you could accomplish your age cleanup via a cron script or some such. I anticipate that a lot of administrative tasks are best done by direct operations on the existing file location. That's why I formatted the time tags as simple counts of milliseconds similar to [clock seconds] output.

Also, all configuration settings are accessible via namespace arrays, like ::vfs::template::version::time($root), and they can indeed be hot-swapped on the fly. So you could for example work on three separate projects branched from the main file base, then package each in separate output directories by changing the array value and copying the same directory three successive times.

I'm wondering now if it's worthwhile to make directories version and project aware like files.

AK: Wayback: A User-level Versioning File System for Linux @ http://www.usenix.org/events/usenix04/tech/freenix/cornell.html

wcf3 20 Jul 2004 - I've been playing with Wayback too...very nice indeed! I hope to have some time to build a nice GUI frontend for it and I've talked with the author about some other ideas for it (which he was already working on). As for accessing the internal variables to control the rollback time, I prefer to provide an API for stuff like that. It reminds me of the days when people would access a certain memory location for some value (remember getting the COM I/O port from the PC BIOS?)...easy, but a support/migration nightmare... Using a cron script to clean out old history is a much better idea; I wasn't thinking :-)


Category VFS