escargo 2005-09-21: Can TWAPI be used to inspect the Running Object Table (ROT)?

APN: No, I don't think so. There aren't any COM interfaces in TWAPI since I assumed you would be much better off using the TCOM extension for COM-related stuff.

escargo: From what I read about ROT, it identifies objects, not their COM interfaces. So you need to know what the objects are before you can use tcom on them. Since the ROT is part of system state, it seemed like TWAPI would an appropriate way to access it. (See the sample code at the link on the ROT page.)

escargo 2006-03-10: Can TWAPI be used to read and write to raw disk devices (unformatted disks without partitions)?

APN: No. Looking at the SDK, it looks like a job for DeviceIOControl but seems non-trivial, particularly for testing. Possibly one could also just call WriteFile after opening the raw device \\.\PhysicalDriveN using CreateFile. I'm afraid of trying something like this on my single development machine and trashing it!

Alastair Davies: asks if there is an API to configure wireless networking? For example, I would like to obtain a list of available wireless networks, and to connect to one. I would not be surprised if this were quite straight-forward, in Windows XP at least.

APN: You might want to look at WRAPI [L2 ] but you would have to write a Tcl binding for that.

LV 2006-06-05: Sure would be useful if this extension became a part of ActiveTcl!

APN 2011-02-12: I believe TWAPI is now available via teapot though there might be a version skew.

MHo: It would be nice if there where some way to register an event source to avoid such stupid error messages within eventvwr.exe. APN Yes it would be nice but I'm not sure how. AFAIK, to register an event source requires a resource in a DLL or other file. So the messages would have to be "bound" to the resource. You could do this with a resource compiler but then you would not need TWAPI. Or am I missing something? MHo That's right. Perhaps the necessary minimal resources/definitions could be build right into twapi.dll. I don't know exactly how to do event source registration by myself (with ms api's this is not uncommon.... smile ;-). No, it can't be built into twapi. Let's say your application wants to log a message "Danger, Danger! Resource do-hickey running low!". That message has to be in some resource, but obviously it can't be in twapi.dll which knows nothing about the application errors. The only way would be to create a resource file on the fly on the target system and that's just too much work. MHo Hm.... The actually message itself sure comes from the caller at runtime; what's missing is the registration of twapi as a valid event source... sure twapi cannot conatin application specific things... As an example, me as a tcl programmer can write generic messages to the eventlog through means of Winserv or tclsvc...APN It's not enough to register twapi.dll as a valid event source. It has to actually contain a valid resource string for the message logged as well. Just registering it will not make the message go away.

neb 2010-09-01: Here's a proc that I wrote for that. It won't register your library, but it will shanghai somebody else's. appmgmts.dll should exist on any XP-class machine, and event 401 has a message string of "%1"; which means it will just replace the whole thing with your message.

This proc uses twapi, and checks for a registration of $src. If it doesn't exist, it rigsters it to the app managements string library. Upshot is that you no longer get the disclaimer. HTH

proc logevent {msg {type information} {eventid 401} {src "My Default Source Here"}} {
    package require registry
    package require twapi
    
    if [catch {set h [registry keys HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Services\\Eventlog\\Application $src]}] {
        puts "Big registry troubles! Eventlog Applications appear to be missing!" error
    } elseif {![string length $h]} {
        puts "$src not found. Writing it."
        registry set HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Services\\Eventlog\\Application\\$src EventMessageFile %SYSTEMROOT%\\system32\\appmgmts.dll expand_sz
        registry set HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Services\\Eventlog\\Application\\$src TypesSupported 31 dword
    }

    if !$eventid {set eventid 401}
    #::twapi::eventlog_log $msg -source $src -type $type -category $eventid
    set h [::twapi::eventlog_open -source $src -write]
    ::twapi::eventlog_write $h $eventid -type $type -category 0 -loguser -params [list $msg]
    ::twapi::eventlog_close $h
}

male 2008-10-01: Here one sample of manipulating a Windows message box using TWAPI ... countdownMsgBox

MHo 2009-02-20: I noticed that get_global_group_members does not give back members if that group only contains other groups (that is, no "direct members"). So which way I could perform a recursive resolution of all computers downwards from a given OU? That's the same behaviour as with NET GROUP, indeed. With some complicated combinations of DSQUERY and DSGET I'm not satisfied after hours and thought tcl and twapi could help... APN Do the ADSI COM interfaces not provide any help here?

no-ids 2010-04-21: I see that there is a new function called impersonate. Do you have an example of exactly what it does or how to use it? Background: Under certain conditions, I would like to launch my script under a different user than the one logged in. Would it help in this case? APN No. The impersonate_user command impersonates the user in the *current* thread which is not what you want to do (which is create a new process under a different user). You could use the CreateProcessAsUser call but unfortunately matters are not so simple. You need to additional create an appropriate context by loading appropriate registry hives etc. in the new process. That stuff is not encapsulated yet in TWAPI, mainly because exactly how to do it is somewhat murky and not well documented.

ssing - 2010-12-22 03:24:00:

twapi miss the target when current window is lock or RDP session is in minimize mode.

let me explain it more, I am creating a process via twapi, then a login window appear, I am setting that as a forground window and then sending the credential, problem is that when I run the test & lock the window(local system) or I am running the same test in RDP session & I minimize the RDP session, it will not find out the window with specific text ans unable to set the window as forground and exit with error "Invalid pointer or opaque value: ''."

APN This is not a bug per se. What is unsaid in the documentation, because it is implicitly assumed to be known is that Windows does not allow processes in one desktop session to retrieve information or send messages to windows in another desktop session. That is a Windows security feature and AFAIK there is no way around it. If I misunderstand or if you have reason to believe it is possible, please let me know and I'll investigate further. The specific error you are getting is because you are passing an empty string (returned by find_windows when no window is found) to the set_foreground_window command.

ssing - 2010-12-22 06:09:23:

Correct,

But in the above problem I am not sending information from one desktop to other, both the above (RDP & local) are two diffferent scenario where I encounter this issue. The thing is I am running test on my local system & I locked the system it fails because it will not identified the login window but on the other hand if the system is unlock it will identify the login window and proceed further. RDP is different scenario, it's when I am runing my whole test from remote machine(provided whole env. is there), RDP session session is just to watch how test is going, If I have a watch on the session It will pass, say I minimize the RDP session window it will fail.

The thing why I am saying this most of the record & play tool used for gui regression test have this feature as they keeps on running test ALAR.

APN Sorry, still not sure exactly what the scenario is. For the moment, let us ignore RDP. For the login case, here is what I did - I set up a continuous loop that every second would do a find_windows using -text and then being that window to the foreground using set_foreground_window. I then locked the system (switch user screen) and logged back in. The loop was running while locked out and after logging in. So it all seems to work. I assume what I did is different from what you are doing. Can you provide a script (smaller the better) that demonstrates the problem, or provide more detail as to the exact scenario? When you way it does not identify the login window, which window are you referring to ? What text and other command options are you using to identify this window?

ssing - 2010-12-22 08:00:00:

gui login

sorry about recent changes, it was unintentionally.

ssingh 2010-12-23 03:22:29:

It seems when the desktop is locked, "twapi::set_foreground_window $win" returns 0, and if the desktop is unlocked it returns 1. can we avoid this?

APN: Sorry about the delayed response (holidays and other stuff). Regarding set_foreground_window - that command is a direct call to SetForegroundWindow Win32 API so it is what it is. Also, regarding the original issue, I wrote a small C program to enumerate all windows to verify what twapi is returning as the window list. The results are identical to what twapi returns with and without the login screen. So the only possibility is that either Windows is creating your app window "lazily" (meaning it is only created when it needs to be displayed which would be after the login screen is removed), or something else is going on with you app.

MHo 2011-02-11: A tip regarding services: every path specification should be an absolut path, sind relative paths do not work with services... something I alway forget, running into trouble again and again...

MHo 2011-02-18: What´s this:

(1)

C:\Dokumente und Einstellungen\Hoffmann>tclsh
% package require twapi;namespace import twapi::*
% comobj
invalid command name "comobj"
% twapi::comobj
wrong # args: should be "twapi::comobj comid ..."
%

APN: For reasons lost in the mists of time, you have to explicitly call twapi::import_commands.

package require twapi ; twapi::import_commands
comobj ....

ssing: Is there any functionality present in twapi like "psexec.exe" do. I mean can we start a remote process with the help of twapi.

APN: psexec is an application/program. TWAPI is a library. I believe TWAPI contains all the Win32 functionality you would need to write a program like psexec but it is non-trivial. See the bottom of [L3 ] for an outline of how psexec is implemented. You would need to do the same things using TWAPI.

MHo 2011-12-11: Are there any plans to facilitate more console API functions like ReadConsoleOutput? Otherwise it would be hard (or impossible) to use them as the basis for a console mode window manager (what I really need is an up-to-date CTk, but there's no). APN Not likely any time soon. I am about to release a new version, after which it's unlikely I will have much time to work on TWAPI. And what little time I have will be spent on other more important features.

Googie 2012-07-15: I'm trying to get access for my application to write to Windows registry under Windows 7. I'd like it to use gently an UAC in Windows. I guess that if I call "twapi::enable_privileges" I would get what I want. Problem is that it requires privileges list as an argument and there is no documentation about what are possible privileges. I also tried to locate any enumerated privileges on MSDN, but with no luck.

Anyone has an idea where to find available privilege list?

APN: For a privilege list, see http://msdn.microsoft.com/en-us/library/windows/desktop/aa375728(v=vs.85).aspx#privilege_constants . However, I don't think this would help in what you are trying to do. If you want to run a process with elevated privileges (UAC), it has to be created as such by its *parent* process. You can use the TWAPI shell_execute command for this:

twapi::shell_execute -path notepad.exe -verb runas

This will create notepad with elevated privileges after showing the UAC dialog. The original process still runs in unelevated mode.

Googie 2012-07-16: I've looked up a little more and I found a pretty nice solution here (as an alternative to twapi dependency): [L4 ] Might be useful if somebody doesn't want to add twapi dependency just for shell_execute ;)

MHo 2013-02-16: Using lookup_account_sid from within an "event sink" (monitoring callback of eventlog_monitor_start) leads to skipping of records (seems that something isn't reentrant?). APN lookup_account_sid is a straight Windows call, should not be any reentrancy issues. Can you post the specific code snippet? And are the events being skipped existing events or new ones being logged? MHo: New event What Windows platform winXP SP3 and twapi version The latest stable ? If the lookup_account_sid call is removed, are events read without skipping? Yes.

• Note: The code was moved from here to Tailing Windows Eventlogs Using TWAPI

Also, I don't fully understand this paragraph of the help:

-file EVENTLOGBACKUPFILE         Specifies the name of a backed up event log file. This option may not be used with the -source or -file options.

What does this mean,

-file may not be used with -file

? What can I specify for -source? If I specify Application, I see Application-Events. If I specify System, I see System-events. If I specify Security, I still don't see security events. If I specify blablabla, I see some events logged with source blablabla, which makes no sense for me. Is there a simple method of trapping all events?

Also, the interpretation of the eventID is a mistery for me...

Excuse me, I know this is because of my bad understanding of Windows, not because of TWAPI, which provides a great layer of abstraction and simplification in using the various windows services. The windows API by itself is just too complicated and changes too fast....:-(

Especially the event logs are getting on my nervs. Why not simply using text files, as the whole rest of the world, so a simple tail would help (the motivation for my prog...)?

APN: Microsoft says "under some circumstances" the event signalling the callback may be missed as their notification code uses PulseEvent which is a deprecated Win32 call (funny how they advise applications not to use it and do so themselves). See http://sourceforge.net/p/snare/feature-requests/1/ and the MSDN documentation for NotifyChangeEventLog and PulseEvent. You have two alternatives - wait for the next event at which time you will get signalled, or put a poller in the background in addition to the monitoring. The same is true for file system monitoring as well. Why poll AND monitor ? Well, the monitoring will give you less delay in most cases allowing you to set the polling interval higher (at least 60seconds or more).

Agreed the Windows event logging is more complicated but it does have some advantages. For one, it is language independent so the event log can be read in any language. The eventID is in effect a message id so depending on your language setting, you can see the event in your native language, something that is not possible with text logging. The eventlog_format_message will format the event into the appropriate language. Also since the eventId act as "atoms" for strings, the logs tend to be much smaller. The Windows event log also is a circular buffer so you don't have to roll over or clean up old logs etc.

It is not possible in XP to monitor multiple event log sources without opening up multiple handles. In Vista and later you can do so with quite powerful event filtering capabilities. The twapi development version (4.0a16) has support for that but not fully tested yet (and of course the new api will only work on Vista and later).

Oh, and the documentation is a typo - it should be "-file cannot be used with -source or -system".

MHo: Is there something special in monitoring remote Systems? If I use -system name to open an event log from a remote computer and then do a eventlog_monitor_start, the following error occurs:

Das Handle ist ungültig.                  (invalid handle)
    while executing
"NotifyChangeEventLog $hevl $hevent"
    (procedure "eventlog_monitor_start" line 4)
    invoked from within
"eventlog_monitor_start $hevl($ix) [list eventSink $hevl($ix) $logname]"
    ("foreach" body line 12)
    invoked from within

Unfortunately, the NotifyChangeEventLog Win32 API does not support remote handles. You will have to use WMI but that is non-trivial to set up and call (or at least I don't know the steps required to do it) and probably not reliable. If you are on Vista or later, you may be able to use evt_subscribe instead for notification of events on remote systems. MHo: Thanks. With MS eventvwr.exe it is possible to connect to remote logs, but those viewer also doesn't do realtime updates. Maybe I'll implement some kind of polling instead.

DB: I'd assumed twapi::eventlog_count would get me the current record number. Looks like you need to add that value to twapi::eventlog_oldest to get it.

MHo: Another monitoring problem. Just experimented with begin_filesystem_monitor and couldn't figure out an easy way to securely know when an operation in the monitored directory has finished. For example, if monitoring with -filename 1 -size 1, one "added" and one "modified" event are fired immediately after the file appears in the directory. But if the file is big enough, it lasts some time until the copy is complete and one couldn't operate on the file until then. Maybe a kind of isOpen() would help, but I have no such function.

Meanwhile I've found a combination, -filename 1 -write 1, that gives added-modified-modified for new files, and modified-modified- modified sequences for files which are overwritten, where the last modified signals completion. Together with a little logic this gives what I wanted.

MHo 2013-07-29: Can't start a service to handle PAUSE and CONTINUE signals. In the example code, the callback contains code to handle pause and continue signals. But twapi::run_as_service is called without -controls, so I think only start and stop is handled by default. After adding -controls [list pause continue stop], the service never starts; I always get this error: Service error: can't use non-numeric string as operand of "|". regardless of the TWAPI version I tested. APN Please use -controls [list pause_continue stop] (that is, pause_continue is one token with an underscore, not two words. This is not documented clearly.

MHo 2013-07-30: begin_filesystem_monitor, as the documentation states, calls it's registered callback several times for each fs modification. This makes it nearly impossible to determine when a modification really is complete. I thought I've figured it out, but right now the system sends out one signal less than before (added-modified instead of added-modified-modified) although nothing changed in configuration. So, my program doesn't work anymore! APN As has been indicated elsewhere, Windows does not guarantee anything about notifications, neither the number, nor the type. It is therefore of use only under limited circumstances and with caveats. Also, even in theory it is not clear what is meant by a modification being complete. Is it when a complete log file is written ? Or a DB transaction is done ? Or a file is closed ? Or all handles to a file are closed ? The only way I would use the monitoring mechanism is as a quicker response for updating file system viewers etc. with some kind of a poller being the "fail safe" backup. MHo: Yes, I mostly agree. The question was to eleminate polling, keep cpu usage low and to avoid sleeps nevertheless in an app which monitores "queue"-like folders for incoming files to process further. The files may be large, so my program should only act on files which are fully copied and closed by the sender (could take several seconds). Perhaps I should only monitor the folder itself and use the change signal as a trigger for further testing.

MHo The pattern *.* for begin_filesystem_monitor suppresses files with have no extension. Should we use * instead? With cmd.exe, *.* and * are the same, whereas *. gives files without an extension. APN As the docs state, the pattern rules are identical to those for the string match command. So you need to use * (as in the Unix shells).

MHo: I use a set_console_control_handler to trap ctrl-c etc. This works fine if running from command line. If I start the console prog via task planner and then terminate through task planner, the control handler wouldn't get fired, so my program has no chance to clean up. APN A task manager kill (or any equivalent that uses ProcessTerminate) is no different from a Unix kill -9. There is no way to catch it.

MHo: Another observation regarding set_console_control_handler: even with an event loop alive (vwait var), Control-C sometimes isn't trapped and the program is immediately terminated by the OS (at least with Windows XP). I noticed that when the event loop has not much to do (no callbacks), everything works fine, but when there are multiple event handlers active (even when they return relatively fast), there's a good chance that Control-C isn't noticed by the program. So, I didn't find a way to 100% protect a program from unwanted interruption or for a gracefull shutdown... Even worse, a new variant just notice with the same prog on my home pc (again XP): the program locks after pressing Control-C... Maybe I would better understand this behaviour If I'd know more about whats going on beding the scene. I'd thougt trapping is done by TWAPI "in background", resulting in a schedule of the registered event handler (if the event loop is alive, of course). Addition: Why does this minimal program not work as expected:

puts [package require twapi]
::twapi::import_commands
proc trap args {
     puts <<<$args>>>
     return 1
}
proc repeat {} {
     puts -nonewline stderr "."
     after 1000 [list repeat]
}
set_console_control_handler [list trap]
after 10000 {generate_console_control_event ctrl-c}
trap
repeat
puts "vwait start"
vwait forever

The ctrl-c event isn't trapped, neither if entered via keyboard, nor via generate...

APN: It is interference between twapi::trap and your own trap. Rename your trap to something else and it should work. This is a bug in twapi but for now you can work around it by renaming your trap handler.

MHo: Thanks, I overlooked this... The demo program now 100% works as expected. But my production program (where I used the same logic but named the trap "trapClose"...), when there are many idle events serviced through vwait, still doesn't catch the ctrl-c event, just ends or hangs... APN Are you using after or after idle in your production program ? The ctrl-c handler is a separate thread which queues the event to the Tcl event queue. It then waits for 100ms for a response and calls the default handler if no response is received in that time. I could try increasing that timeout but first it would be good to know if you are continuously calling after idle in which case timer events will not get to run (I think). MHo: Yes, I use after idle to schedule my events. But as far as I know, idle events are only served after all other queued events are processed. E.g., callbacks from begin_filesystem_monitor are all serviced before idle events. Other handlers active in parallel are timer events (after 60000) and fileevents. APN: If this is a show-stopper for you, I can create a version with a configurable timeout to see if that helps. But first it would be good to verify that the timer is the problem. I tried the sample session below without any problem. Despite continuous idle scheduling, the ctrl-c handler still worked. Are you sure your handler is either not erroring out or is returning something other than "1" ?

% package require twapi
4.0b24
% proc x args {puts HANDLER; return 1}
% twapi::set_console_control_handler x
x
% proc idler args {after idle idler}
% idler
after#0
% after 60000 exit
after#1
% vwait forever
HANDLER  <- printed when Ctrl-C pressed
HANDLER
HANDLER
HANDLER

MHo: I can confirm that your example works for me, too. I will try to extend this code until it now longer works... I think a configurable timeout would be great, nevertheless.

MHo: eventlog_read xxx -seek yyy gives error message (4.0b22):

package require twapi_eventlog
set source "System"
set hevl [twapi::eventlog_open -source $source]
set recs [twapi::eventlog_count $hevl]
set events [twapi::eventlog_read $hevl -seek [expr {$recs-10}]]
# print out each record
foreach eventrec $events {
    array set event $eventrec
    set timestamp [clock format $event(-timewritten) -format "%x %X"]
    set source   $event(-source)
    set category [twapi::eventlog_format_category $eventrec -width -1]
    set message  [twapi::eventlog_format_message $eventrec -width -1]
    puts "Time: $timestamp\nSource: $source\nCategory: $category\n$message\n"
}

D:\Home\Arbeit1\pgm\tcl\usr\Tst\logViewer\winlog>tclkitsh860.exe twapitest2.tcl
Falscher Parameter.
    while executing
"ReadEventLog $hevl $flags $offset"
    invoked from within
"trap {
        set recs [ReadEventLog $hevl $flags $offset]
    } onerror {TWAPI_WIN32 38} {
        # EOF - no more
        set recs [list ]
    }"
    (procedure "twapi::eventlog_read" line 32)
    invoked from within
"twapi::eventlog_read $hevl -seek [expr {$recs-10}]"
    invoked from within
"set events [twapi::eventlog_read $hevl -seek [expr {$recs-10}]]"
    (file "twapitest2.tcl" line 9)

APN: The argument to -seek is not the index or position of a record in the event log. It is the record number (MS terminology) which might have been better named record id. It increments for each added record but does not necessarily start at 0/1. For what you are doing, you probably want to use eventlog_oldest to get the record number of the oldest record, then add eventlog_count and subtract 10 from that as the seek position. MHo: Yes, that does the trick. Thanks!

MHo 2013-11-12: Another observation with v4-beta: package require'ing twapi-modular from tcl 8.6: no problem. package require'ing twapi-modular from tcl 8.5 gives this error, e.g.:

couldn't load library "d:/var/temp/TCL63.tmp": this library or a dependent library could not be found in library path
    while executing
"load D:/Home/Hoffmann/pgm/tcl/usr/Src/tclstdlib/tclstdlib.kit/lib/twapi-modular/twapi_service.dll twapi_service"
    ("uplevel" body line 1)
    invoked from within
"uplevel #0 $loadcmd"
    (procedure "twapi::package_setup" line 47)
    invoked from within
"twapi::package_setup D:/Home/Hoffmann/pgm/tcl/usr/Src/tclstdlib/tclstdlib.kit/lib/twapi-modular twapi_service 4.0b22 lo
ad twapi_service {}"
    ("package ifneeded twapi_service 4.0b22" script)
    invoked from within
"package require $pkt"
    ("foreach" body line 2)
    invoked from within
"foreach pkt {
   base64
   bgexec
   crypt1
   elog
   execx2
   Globx
   kill
   lock
   md5
   mime
   rc4
   readprof
   smtp
   socktest
   twapi_..."
    (file "test.tcl" line 3)

My test program is as simple as follows:

source [file join [file dir [info script]] tclstdlib.kit]

foreach pkt {
   base64
   bgexec
   crypt1
   elog
   execx2
   Globx
   kill
   lock
   md5
   mime
   rc4
   readprof
   smtp
   socktest
   twapi_service
   tdbc::odbc
} {
   puts "$pkt -> [package require $pkt]"
}
puts [info loaded]
puts [package names]

And the tclstdlib.kit contains the ususal structure:

\---lib
    +---base64
    +---bgexec
    +---crypt1
    +---elog
    +---execx2
    +---globx
    +---lock
    +---md5
    +---mime
    +---rc4
    +---readprof
    +---socktest
    +---tclkill
    +---tdbc1.0.0
    +---tdbc_odbc1.0.0
    \---twapi-modular

Made the same observation with another starpack.

APN: Do you have twapi_base.dll in your vfs ? All twapi modules need that. Although I do expect 8.6 to also fail if that's missing (unless it gets loaded from elsewhere on the 8.6 package path) MHo Yes, it's there...

MG: Is it possible your TWAPI and Tcl 8.6 are 64-bit, but the 8.5 Tcl is 32-bit? I seem to recall getting that error when trying to load the wrong package that way before once.

MHo: No. In the example shown above, only 32bit modules are involved. With Tclkitsh 8.6, it works, but it did not work with tclkitsh 8.5. In twapi-modular, all modules of the modular-distro are included. The tclkits are as follows:

11.11.2013  11:59           726.519 tclkitsh-cli-8.5.13_32.exe
21.12.2012  17:43           841.274 tclkitsh.exe

The first one (8.5.13):

D:\Home\Arbeit1\pgm\tcl\usr\Src\tclstdlib>tclkitsh-cli-8.5.13_32.exe
% info pa
8.5.13
% parray tcl_platform
tcl_platform(byteOrder)   = littleEndian
tcl_platform(machine)     = intel
tcl_platform(os)          = Windows NT
tcl_platform(osVersion)   = 5.1
tcl_platform(platform)    = windows
tcl_platform(pointerSize) = 4
tcl_platform(threaded)    = 1
tcl_platform(user)        = Arbeit1
tcl_platform(wordSize)    = 4
% info loaded
{{} registry} {{} dde} {{} zlib} {{} vfs} {{} rechan} {{} tclkitpath} {{} vlerq}
%

The second one (8.6):

D:\Home\Arbeit1\pgm\tcl\usr\Src\tclstdlib>tclkitsh.exe
% info pa
8.6.0
% parray tcl_platform
tcl_platform(byteOrder)     = littleEndian
tcl_platform(machine)       = intel
tcl_platform(os)            = Windows NT
tcl_platform(osVersion)     = 5.1
tcl_platform(pathSeparator) = ;
tcl_platform(platform)      = windows
tcl_platform(pointerSize)   = 4
tcl_platform(threaded)      = 1
tcl_platform(user)          = Arbeit1
tcl_platform(wordSize)      = 4
% info loaded
{{} registry} {{} dde} {{} vfs} {{} rechan} {{} tclkitpath} {{} Mk4tcl} {{} Itcl}

Hm, I see now, one is a Metakit-Version, the other vlerq. But is this relevant?

APN: For now you can use the twapi-bin distribution. I verified that works with 8.5. With the twapi-modular distribution, the issue is that each module is dynamically linked with the twapi_base dll. The tclkit code copies the module out to a temp area and tries to load it. Windows cannot find the base dll and hence the error. This is the same issue that was discussed in (I think) clt w.r.t to Img or some other extension. Only workaround if you want to use the the modular distribution to reduce size is to copy the twapi_base.dll and twapi_eventlog.dll to the file system and explicitly load from there. I cannot explain how your 8.6 kit works. Possibly the code that implicitly copies the vfs dll to an external dir before loading has changed so that files are not renamed. That could be a 8.5/8.6 difference or a metakit/vlerq difference.

MHo: I thought in the same direction. But it's irritating, as it works with 8.6... See dllfix and Starkits with Binary Extensions.

MHo 2014-03-18: I noticed that it isn't possible to feed the list of options fetched with get_service_configuration into set_service_configuration:

• List of config otions from get_service_configuration:

-dependencies {Tcpip Afd} -servicetype win32_own_process -starttype demand_start -errorcontrol normal -tagid 0 -command
{D:\home\Hoffmann\bin\apache-tomcat-7.0.50\bin\tomcat7.exe //RS//Tomcat7} -loadordergroup {} -account LocalSystem -displ
ayname {Apache Tomcat 7} -interactive 0 -description {Apache Tomcat 7.0.50 Server - http://tomcat.apache.org/}

• Error Message if this is applied to set_service_configuration:

Invalid option '-tagid'. Must be one of -displayname, -servicetype, -interactive, -starttype, -errorcontrol, -command, -
loadordergroup, -dependencies, -account, -password, -system, -database

APN: TWAPI does not accept setting of -tagid because the underlying ChangeConfig Win32 call prototype marks it as output only. Having said that, I'm sure there will be other situations where the get/set are not symmetrical.win

MHo: Thanks. The same happes with -description. Is it possible set the description programmatically, at all? Oh Microsoft... APN: The description is just stored in the registry

(tcl) 58 % registry get {HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\dxgkrnl} description
Controls the underlying video driver stacks to provide fully-featured display capabilities.

And if I specify -servicetype win32_own_process, I get a expected number but got "win32_own_process"-error... This is not the case if using it with create_service.

APN: Looking into this, looks like it only happens when -interactive is also specified. Without -interaction being specified, -servicetype win32_own_process works. So you could potentially do the set config in two steps. Or wait for the next beta.

So, here's my little tool for migrating a service from machine A to machine B for now:

# svcmig1.tcl 19.03.2014 Matthias Hoffmann
# Migrating a service from Machine A to Machine B
# https://wiki.tcl-lang.org/9886
lappend auto_path [file dirname [info script]]
package require twapi
if {$argc < 3} {
    puts stderr {args: servicename sourcesrv destsrv [password]}
    exit 1
}
if {[catch {::twapi::get_service_configuration [lindex $argv 0] -all -system [lindex $argv 1]} cfg]} {
    puts stderr $cfg
    exit 2
}
# Später als dictionary handhaben.
array set c $cfg
parray c
# puts $cfg; exit
if {[catch {
    ::twapi::create_service [lindex $argv 0] "dummy" -system [lindex $argv 2] -servicetype $c(-servicetype)
    unset c(-tagid);       # not supported with 'set'
    unset c(-description); # not supported with 'set'
    unset c(-servicetype); # expected number but got "win32_own_process" with set_service_configuration
    set cfg [array get c]
    ::twapi::set_service_configuration [lindex $argv 0] -system [
        lindex $argv 2] {*}$cfg -password [lindex $argv 3]
} res]} {
    puts stderr $res
    exit 3
}
if {[catch {::twapi::get_service_configuration [lindex $argv 0] -all -system [lindex $argv 2]} cfg]} {
    puts stderr $cfg
    exit 4
}
puts "======================================================================"
array set c $cfg
parray c

MHo 2014-11-25: feeding write_shortcut with the args of read_shortcut does not work as expected. Even for links to normal files, read_shortcut gives back funny values for -idl, which I think is the reason that after writing them back the link is destroyed. After removing -idl from the dictionary before writing back the link, everything works (which is ok right now as I do not need -idl at all). Additionally, I noticed that the case of the given -path is changed (by TWAPI or OS). So reading it back and comparing with a given (new) -path only works when ignoring the case. See the following wrapper prog as a try to work around all this and to experiment with:

package require twapi
twapi::import_commands

proc writeLink {linkFile args} {
    if {[file exists $linkFile]} {
        set update 0
        set curArgs [dict remove [read_shortcut $linkFile] -idl]
ts $curArgs
        # mit dict merge haben wir keine Kontrolle über Änderungen, also zu Fuß
        foreach {k v} $args {
            puts "$k -> $v"
            if {[string equal $k "-path"]} {
                # der Pfad wird vom OS oder TWAPI irgendwie geändert (LW: uppercase),
                # daher zur Feststellung, ob sich etwas ändert, einen -nocase-Vergleich durchführen!
                set parm "-nocase"
            } else {
                set parm ""
            }
            if {[dict exists $curArgs $k] && [string compare $parm [dict get $curArgs $k] $v]} {
ts "<<<[dict get $curArgs $k]>>>"
ts "<<<$v>>>"
                dict set curArgs $k $v
                set update 1
            }
        }
    } else {
        set update 1
        set curArgs $args
    }
    if {$update} {
ts "doUpdate"
        write_shortcut $linkFile {*}$curArgs
    }
}

# file delete [file join [get_shell_folder csidl_desktopdirectory] test0000.lnk]
# writeLink  [file join [get_shell_folder csidl_desktopdirectory] test0000.lnk]  -path "d:\\temp\\2014-11-21_Buchungen in Projectile.pdf"
writeLink  [file join [get_shell_folder csidl_desktopdirectory] test0000.lnk]  -path "d:\\temp\\HEK_neue_Systemumgebung.pdf"

APN: I've added a ticket https://sourceforge.net/p/twapi/bugs/138/ to track this.

APN: Fixed for 4.1b16. However, the case changing as well as longname to shortname etc. is a Windows thing not in TWAPI's control.

MHo 2015-02-09: Just noticed that in the path contained in -path's value (oh lord...) backslashes have to be doubled, whereas in -args's value this isn't the case... kind of confusing... APN Can you show an example? I don't see this (assuming you mean the write_shortcut command). Thanks MHo Yes, write_shortcut. Half of an an example is alreay above, here's another:

write_shortcut [file join [get_shell_folder csidl_desktopdirectory] {xyz.lnk}] \
 -path c:\\programme\\HEK-Tools\\popsel.exe -args [file nativename $popselfile]

As I doubled the backslashes produced from file nativename $popselfile, they appear in the link and the link didn't work. But maybe there was some other fact I'm too blind for right now (as usual...).

APN: The above double-backslash has nothing to do with the command. When you specify a "literal" Tcl will do backslash substitution and hence you need to double them. When the path is in a variable or returned by a command, this backslash subst does not happen. Try puts in the two cases to see the difference.

MHo: Yes, I know this. -path expects Backslashes, as this value is passed 1:1 to windows, I think. And Backslashes in literals have to be doubled to escape them (would be better if one could use forward slashes here, but...). I thougt the same has to be done with this construct (I did a little simplification in the example above...):

-args "/n [file nativename $popselfile]"

This is a literal (at first glance), but I forgot the fact that Tcl does only one round of parsing: after replacing the [...] with it's result, the string is ready as it is... Oh, shame on me and thanks for your patience.

MHo: Thanks for your wonderfull TWAPI. It provides a tcl'ish way to administer things on Windows! And thanks for your binary builds, especially that with TWAPI included. Unfortunally, the latest 8.6.4 interpreters are blamed again by our SYMANTEC security software as a risk. I downloaded them, and Symantec isolates them away. Maybe this will happen in production environments as well - not to think of......

APN: Mho, could you submit a false positive report at https://submit.symantec.com/false_positive/ I would do so myself but they ask for some details (like Symantec product and version) which I don't know.

MHo: Symantec End Point Protection 12.1.4013.4013; VirusDefs: 150326016. Events-Details (Example, they're all the same):

Sicherheitsrisiko gefunden!WS.Reputation.1 in Datei: D:\home\Hoffmann\Downloads\tclkit-cli-8.6.4-x64.exe von: Auto-Protect-Scan. Aktion: Isolieren erfolgreich : Zugriff verweigert. Beschreibung der Aktion: Die Datei wurde erfolgreich isoliert.

Which means: "I don't know the file yet - it must be dangerous..." - What a f.... logic!

APN: OK, I tried and I give up. Now they want me to fill in the field _Name of detection given by Symantec product_ Is there no way you can just override the quanrantine? Most AV will let you pull the files out of quarantine.

MHo: I will see if we can send the false-positive-reports. At first, I have to explain to several people, what tcl an twap is all about...

MHo 2015-03-30: Meanwhile I've submitted a report to Symantec, but only for one of the nine files isolated away, as an example. Let us see what will happen now...

MHo 2015-03-31: Symantec responded very quick. Unfortunally, they only flagged only one download as okay. Actually, I didn't have the time to submit false positive reports aboput the eight other downloads... So I wait and hope that many people are try to download your tclkits, so that the reputation will automatically rise and grow above our threshold...

"...Upon further analysis and investigation we have verified your submission and as such this detection will be removed from our products...."

APN: Thanks. This whole situation is just absurd. There is no mechanism to get redress against this form of abuse.

MHo: Will there be new combined Helpfiles? Or are there no changes at that level? APN I'm waiting for tcllib 1.17 to be released so I don't have to rebuild again.

Mho: I created a tclstdlib.kit with several included libraries, so that my tcl scripts can source it and can use the libraries they usually need. Now I included the newest twapi 4.1.27 in that starkit. The problem is: if I source the kit using an interpreter with an older TWAPI already build in, I get a version conflict. As I use the starkit from several interpreter/script combinations, I wonder if there is any way to work around this, and if it's possible to force the interpreter (with the build-in, older TWAPI) to use the external (newer) TWAPI out of the sourced starkit. This does not work for me even using package require -exact.... Meanwhile I added the following at the top of the main file of the starkit, and this seems to work for every possible interpreter I use (perhaps an older build-in twapi is used, though, as I found no way to force the use of the newer version, that is, to "overwrite" a statically build-in twapi).

# tclstdlib.vfs/main.tcl 11.10.2013, 17.12.2014, 19.04.2015

# Massnahme gegen conflicting versions provided for package "twapi_base": 4.0.61, then 4.1.27
if {[lsearch -index 1 [info loaded] twapi*] >= 0} {
    #  Interpreter mit eingebauten TWAPI
    catch {package require twapi}
}

package require starkit
starkit::startup

MHo 2015-04-21: Is there any secret in using itaskscheduler_get_tasks? No tasks are listed on my Windows 7-PC (whether local or remote). tested with TWAPI 4.1.27. Snippet:

set hSched [itaskscheduler_new]
puts [itaskscheduler_get_tasks $hSched]; # should show something....

Just tested with a tclkit with TWAPI 4.0.61 build-in on Windows 2008 R2. There it works... But still not on Windows 7. It only seems to work on a server. But the underlying APIs should be available on Windows 7 too? Hm, on Win2008, one task is listed, whereas many exist... Now I think there is some secret...

APN: I think the issue is that Vista and later have two task schedulers - 1.0 (XP compatible) and 2.0. The twapi task interfaces only support the former because the latter already has COM based scripting support that you can access with the twapi COM module. The itaskscheduler_get_tasks will only show tasks that were scheduled with the older API. For using the new 2.0 task scheduler with COM, see https://msdn.microsoft.com/en-us/library/windows/desktop/aa384006(v=vs.85).aspx but note that will not work with XP and Win 2003. You are seeing tasks on Win2k8 because they were probably created using the older API. MHo Thanks for the fast response. Ok, I have to dive into the horrors of COM...;-) But I'm sure the abstraction of your taskscheduler commands is a better one than using com...

MHo 2015-06-08: Are there any potentiell reasons why a call to begin_filesystem_monitor could fail? I have a production programs which ran unobtrusive for months, but since I changed from tcl 8.5 to 8.6 and twapi from 4.0.x to 4.1.x (ok, besides changes in *my* code of course) I see mysterious hangs. I added debug output to narrow down what happens, and today it seems that a call to begin_filesystem_monitor never returned. I don't know how to handle this case. I don't use catch and I don't get a run time error...:

:
:putsLog "main(): Programmstart, arg=$argv, v=$_v"
debugOut "main(): LogKonfig: File=[$::elog getLogFile] MaxSize=[$::elog getLogMaxSz] Buffering=[$::elog getLogBuffering]" <=== appears in the log

set stopFile [file join [file dirname [info nameofexe]] newfileaction.stop!]
set ids [list]
set pCount 0
set aliveCount 0
set aliveOff 0
set endRequested 0
foreach {dir pattern action} $oplist {
    set id [begin_filesystem_monitor $dir [list cb $dir $action] -patterns $pattern {*}$monParms]
    lappend ids $id
    putsLog "main(): [list begin_filesystem_monitor $dir [list cb $dir $action] -patterns $pattern {*}$monParms]" <=== does not appear in the log
}
:
:

Ok, could be that only the log entry is not flushed, etc., but I proved that the program indeed did'nt carried out further action... So, it looks to me that *maybe* the begin_file_system is the last statement the programm processed before "hanging"....

APN: Never say never, but I doubt a change from 4.0.x to 4.1 would cause this as I didn't see anything directly related to file system monitoring in the twapi change log. Can you put 4.0.x back (keeping Tcl 8.6 and your new code) and see if the hang goes away? Is it possible that the patterns you are specifying might have changed causing the internal matching code to may be spin in an infinite loop due to a bug? Also, can you tell if the hang is actually a CPU spin (by looking at task manager). MHo: Thanks. The problem here is that the problem occures only very seldom, from time to time once a month or so ;-) Unfortunally, I don't remember the details, but the "hanging" prog didn't use cpu cycles, and there was no disk i/o. Very strange. The changes I made to my code are minor, the $dir, $action, $monParms didn't change etc.

How can I find examples of using the raw win32 api with TWAPI? APN You can look at the Tcl scripts that implement the high level commands since these make use of the raw api! Note that there is very little of the raw win32 api that TWAPI supports that is not already covered by the high level commands. MHo Thanks. Eventually I need AddFontResource(). APN That you should be able to call as

set nfonts [twapi::AddFontResource c:/path/to/font/resource/file]

Hm, I get "invalid command name twapi::addFontResouce"... But it seems to work with twapi::AddFontResourceEx.

MHo 2015-10-15: Just realized that the tclkits provided on the twapi site do not contain the thread package......are there any reasons for this...? APN No particular reason. It's not there because the line between what should be included and what should not is different for different folks.

MHo 2015-10-19: end_process returns 1 if specifying non-existant PIDs?

get_process_info nnnn -pid returns given pid even if it doesn't exist:

% get_process_info 1144 -noexist -1 -pid; # this is an existent process
-pid 1144                                                      
% get_process_info 1144999 -noexist -1 -pid; # this pid does not exist
-pid 1144999                                                   
%                                                              

APN Both are by design. Semantics of end_process are to return 0 if a process with that PID exists when the call returns and 1 otherwise. This is more useful than generating an error, for example in the case where a process is ensuring all child processes are ended before it itself exits. Regarding -pid, that field is considered the "key" and always returns whatever is passed in. The -noexist or -noaccess options do not apply to it.

MHo hm....

From The Help:

end_process PID ?options? Terminates the process with id PID. This function will first try to gracefully end the process by sending all its toplevel windows a WM_CLOSE command. If the process does not have any toplevels or if it does not end by this means and the -force option is specified, it is forcibly terminated.

The function returns 1 if the process ended and 0 otherwise. Note that if -wait has not been specified, or is too short, the function may return a value of 0 in the case where the process is in the process of terminating but has not fully exited by the time the command returns.

• PID exists and kill succeed --> return 1
• PID exists and kill failed ---> return 0
• PID does not exist -----------> return ?????

Mho 2016-11-04: I noticed, that Tk programs do not handle WM_QUERYENDSESSION. I wonder if TWAPI can help in trapping logoff or shutdown events for GUI programs? Oh, I just looked into the Tk sources and found out that WM_SAVE_YOURSELF is synthesized in such cases. And it seems to work.

MHo 2017-07-03: Scenario:

• Files are incoming with a name of file.tmp
• When complete, files are renamed to file
• So begin_filesystemmonitor should not trigger for *.tmp, but for any other file: -patterns {-*.tmp +*}
• begin_filesystemmonitor indeed ingnores the .tmp-files
• but it also ignores the renaming, so nothing is triggered... Options in effect are -filename 1 -write 1

Oops - I'm sorry - just found the error: only gave the first part of the pattern.... now rename triggers a renamenew event... everythings works as excpected now...

MHo 2017-07-05: It would be usefull, if the pattern matching done by begin_filesystem_monitor would be available per API. I have a scenario where I must pick up accumulated files "manually" when re-starting an interrupted program, which normally picks up the files with begin_filesystem_monitor... For now, I'll try to re-implement the matching logic... First try:

# re-implementation of twapi::begin_filesystem_monitor´s -patterns-Matching:
#
# If the -patterns option is  specified, then the value associated  with the
# option  is  a  pattern  filter  that  is  used  to  match against the name
# component being  reported.   SCRIPT will  be invoked  only if  the pattern
# filter matches.
#
# Each element of the pattern list is a string beginning with either + or  -
# followed by a pattern  that is used for  matching using the same  rules as
# for the Tcl string match -nocase command.  Note that \ path separator need
# to  be  escaped  with  another  \  just  as  in  the  string  match  case.
# Alternatively, you can use / as the path separator in the pattern.
#
# The name component being reported  is matched against each element  of the
# pattern list.  When a match  occurs, further elements of the pattern  list
# are not  matched.   If the  matching element  is prefixed  with a  +, then
# SCRIPT is invoked.  If the matching element is prefixed with a -, then the
# pattern filter is deemed to have not matched and SCRIPT is not invoked for
# the notification.   Even  in this  case, no  further matching is attempted
# against the remaining patterns.   Pattern elements that do not  begin with
# +, or - are treated as having an implicit + prefix.
#
# If a  non-empty pattern  filter is  specified but  no element matches, the
# pattern  filter  is  treated  as  having  not  matched,  and SCRIPT is not
# invoked.  Thus a pattern filter  that only has - type elements  will never
# match anything.  Note a pattern list where no element matches is different
# from the case where -patterns is not specified or PATTERNLIST is empty  in
# which case the pattern filter is treated as matched.
#
# Note that  the pattern  matching is  done only  against the name component
# being reported, not against the full path.
#
# Some  examples  of  pattern  matching  are  below (assume callback is your
# SCRIPT procedure):   begin_filesystem_monitor  c:/temp callback  -patterns
# {*.exe}  will  only  monitor  files  in  c:\temp  with  .exe   extensions.
# begin_filesystem_monitor c:/temp callback  -patterns {-*.foo +*.fo*}  will
# monitor files in  c:\temp with an  extension beginning with  .fo except if
# the extension is .foo. begin_filesystem_monitor c:/temp callback -patterns
# {+*.fo* -*.foo} will monitor files in c:\temp with an extension  beginning
# with .fo including those with  extension .foo.  The second  pattern -*.foo
# has no effect as the matching stops as soon as a match is found.
#
proc tbfm_match {patterns singlefile} {
     if {[llength $patterns] == 0} {
        return 1; # pick the file if patterns empty
     }
     set f [file tail $singlefile]
     foreach pattern $patterns {
             set prefix [string range $pattern 0 0]
             if {$prefix eq "+" || $prefix eq "-"} {
                set pattern [string range $pattern 1 end]
             } else {
                set prefix "+"
             }
             if {[string match -nocase $pattern $f]} {
                return [expr {$prefix eq "+"}]; # no further examinations
             }
     }
     return 0
}

# Some tests
foreach {patterns testfile expRC} {
        {}                shouldmatch  1
        {-*.tmp +*}       one          1
        {-*.tmp +*}       two.dat      1
        {-*.tmp *}        one          1
        {-*.tmp *}        two.dat      1
        {-*.tmp *}        no.tmp       0
        -*                nevermatch   0
        *                 always       1
        {-*.tmp -a*.* *}  ok           1
        {-*.tmp -a*.* *}  a.notok      0
        {-*.tmp -a*.* *}  a            1
} {
        puts [format {%20s %20s %i %i} $patterns $testfile $expRC [tbfm_match $patterns $testfile]]
}

Results:

>tclkitsh tbfm_match.tcl
                              shouldmatch 1 1
           -*.tmp +*                  one 1 1
           -*.tmp +*              two.dat 1 1
            -*.tmp *                  one 1 1
            -*.tmp *              two.dat 1 1
            -*.tmp *               no.tmp 0 0
                  -*           nevermatch 0 0
                   *               always 1 1
      -*.tmp -a*.* *                   ok 1 1
      -*.tmp -a*.* *              a.notok 0 0
      -*.tmp -a*.* *                    a 1 1

APN Although implemented a long while ago, from what I recall, my thinking was at the time that there can be any number of needs in terms of pattern matching and these can be met via implementing them in the callback itself. It's not clear to me that making -patterns customizable would be better, particularly because it needs to be fast. Or perhaps I misunderstood your request.

MHo Thanks. Yes, every individual matching/filtering can be done in the callback. The machting itself is fine as it is. I just need a matching elsewhere which exactly behaves like your's, so the question was if your matching could be exposed to the script-level as something like twapi_base::fsm_match_oneFile which takes a list of patterns like begin_filesystem_monitor and a single filename.

MHo Hm, it's not possible to directly retrieve the remotedesktop-services profilepath account attribute (in german: "Remotedesktopdienste-Benutzerprofil Profilpfad"), I fear... One way I found:

set objectTrans [comobj "NameTranslate"]
$objectTrans -call init 3 ""
$objectTrans -call set 3 "$yourDomain\\$theAccount"
set ou [$objectTrans -call get 1]; # OU from SAMAccountname, or directly start with this if known
set objUser [comobj_object "LDAP://$ou"]
set tsProfPath [$objUser TerminalServicesProfilePath]
$objUser -destroy

Updated the version info at the very top.

Question: is an update of the tclkits at https://sourceforge.net/projects/twapi/files/Tcl%20binaries/ to version 8.6.7 planned?

MHo 2018-01-26: It should be noted that only services always receive the shutdown- or logoff-events (if they registered themself for this). Console programs (via set_console_control_handler) oder Tk-GUIs (via WM_SAVE_YOURSELF) sometimes aren't notified about shutdown (if running via taskplanner in system context, see https://stackoverflow.com/questions/24911125/does-scheduled-task-receive-wm-queryendsession-message ). See https://docs.microsoft.com/en-us/windows/console/handlerroutine . Additional infos are at https://www.apriorit.com/dev-blog/413-win-api-shutdown-events .

MHo 2018-02-01: later windows versions have an (obscure;-) facility to control the order in which services are shutting down.... see https://blogs.technet.microsoft.com/askperf/2008/02/04/ws2008-service-shutdown-and-crash-handling/ . Some additional info from https://answers.microsoft.com/en-us/windows/forum/windows_xp-windows_programs/windows-services-shutdown-order/32547433-9c87-483a-a6d9-8a7131c2e57e?auth=1 :

"Service Control Handler Function looks interesting. Consider handling SERVICE_CONTROL_SHUTDOWN, and coordinating shutdown among your services in that fashion. Use the dwCheckPoint and dwWaitHint members of the SERVICE_STATUS struct to provide feedback to the SCM regarding service shutdown progress."

MHo 2018-02-09: There also is an service_control_preshutdown-Event, but I don't know how to trap it. And a function called SetProcessShutdownParameters() to help determining the order in that processes are shutting down. Both of them, I fear, are not available via twapi...

What I noticed is that simply catching shutdown in a service and trying to call remote functions to other machines eventually does not work (I think, other required services for this are alreay terminated then.). I get an error like this:

-code 1 -level 0 -errorstack {INNER {invokeStk1 OpenSCManager remotesys {} 0x00020000} CALL {get_service_status PatrolAgent -system remotesys} CALL {get_service_state PatrolAgent -system WB101SW0225} CALL onShutdown} -errorcode {TWAPI_WIN32 1722 {Der RPC-Server ist nicht verfügbar.}} -errorinfo {Der RPC-Server ist nicht verfügbar.
    while executing
"OpenSCManager $opts(system) $opts(database) 0x00020000"
    (procedure "get_service_status" line 4)
    invoked from within
"get_service_status $name {*}$args"
    (procedure "get_service_state" line 2)
    invoked from within
"get_service_state $svc -system $otherSrv"} -errorline 1

Windows does not respect service dependencies during shutdown. All Services receive the shutdown signal at the same time, normally. And then, it's simply to late for some actions, as part of the APIs are already gone....:-(

Scheduled tasks are simply killed, so even waiting for an Win32_ComputerShutdownEvent doesn't work. Only programs started by users are notified, it seems.

So, I am badly in need for preshutdown-notifications...

AMG: I am having trouble embedding TWAPI in a Starpack. When the DLL is loaded, it tries to source Tcl files which it expects to be located in the same directory as the DLL. However, a Starpack is not a typical TWAPI installation. The DLL is located in a temporary directory just long enough for Windows to load it, and there are no additional files in that directory. For now I think the solution is to use the twapi-bin distribution which I suspect incorporates the Tcl scripts directly into the DLL, but I would prefer an alternative that allows the Tcl scripts to be sourced from the Starpack VFS.

I'm not sure why it is necessary that the DLL source the Tcl scripts; usually it's the other way around, with the DLL providing the low-level capability and the Tcl scripts providing the end interface. If the DLL actually depends on the Tcl scripts, I feel it is acceptable to have errors should the user manually load the DLL, bypassing the normal script initialization routine (package require, etc.).

MHo 2018-06-19: In future releases, it would be great if TWAPI would support ANSI Escape Sequences (ENABLE_VIRTUAL_TERMINAL_INPUT 0x0200) with modify_console_output_mode, see https://docs.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences and https://docs.microsoft.com/en-us/windows/console/setconsolemode . This is a feature missing for years in modern windows versions (ages ago, in MSDOS and MSDOS-based Windows versions, this function was available by loading the ANIS.SYS device driver...), but it's now back in Windows 10. In a cmd.exe shell it seems that ANSI processing is enabled by default now, but the Console Text Editor in Pure Tcl 2 for example does not work as expected...

APN Could you try

twapi::SetConsoleMode [twapi::get_console_handle stdout] 4

and see if that does the needful? (4 -> ENABLE_VIRTUAL_TERMINAL_PROCESSING)

MHo Hm, no this does not help there...maybe some other reason.... i will try and report further...

APN Just to double check - the input and output mode changes are separate. The 4 refers to output processing. For input you would specify 0x0200 as you stated above.

MHo Until yet I couldn't get it to work...just tried on windows 2016 server...

APN Ok, I see that you also have to enable the PROCESSEDOUTPUT flag. So the correct call is

twapi::SetConsoleMode [twapi::get_console_handle stdout] 5

(4 -> virtual terminal, 1 -> processed output) This worked for me to output color sequences. For example,

puts "\033\[0;32mTEST\033\[0m"

will output a green TEST.

MHo: Aaaah - yes - this does the trick - many thanks!

MHo 2018-10-12:

• Does a downloadable version of the html html help files of TWAPI 4.3.5 exist somewhere?
• Will be ever an updated chm combined help file?
• Will there be updated tclkits available some day with TWAPI 4.3.5 included...?

Unfortunally, although I'm a "hardcore tcl user", I'm not able to help myself regarding the above topics....

APN Updated tclkits will be available concurrent with Tcl 8.6.9 release. Regarding the help files, the Microsoft HTML CHM compiler crashes on my system and I haven't been bothered to figure out why. Given that its last release was sometime around 1998, it's a miracle it's worked for so long.

MHo 2019-01-15:

• I get stucked with SASL and TWAPI. I wand to send SMTP-Mails via Exchange. The only option is NTLM. But I don't want to specify User and Password.... APN I commented on that page.

MHo 2019-07-05: Big Problem. I operate several Tcl programs which run without interruptions several day, weeks or months. In some of them, I recently noticed growing "private bytes" usage (via ProcMon). I tried to narrow down which construct is responsible for this phenomen, which I didn't notice on the OS before Windows 2016 and with a twapi-tclkit before 8.6.9, I think...... If you run simply run this loop, you will notice an ever growing memory consumption.... copying or deleting a large number of files pushes on the growth...:

package require twapi
twapi::import_commands

proc cb {h info} {
     lassign $info op obj
     after idle [list puts [format {%10s %s} $op $obj]]
}

begin_filesystem_monitor c:/ cb -patterns * -access 1 -attr 1 -create 1 -dirname 1 -filename 1 -secd 1 -size 1 -write 1 -subtree 1
vwait forever

My real programs depend on begin_filesystem_monitor, so what option do I have other then periodically (self-)restart the script (and/or interpreter)?

2019-07-29 APN Logged as a bug at https://sourceforge.net/p/twapi/bugs/179/ . This should be fixed in 4.3.7. Please try and re-open the bug if you still see the same memory growth.

MHo 2020-06-02: I've problems enumerating Groups with get_global_groups. The command only return ~2500 groupnames, but thousands more exist. Is there any trick I'm missing?

APN Did you use the -resume option?

MHo: Yes I tested this as well, but the moredata return value says there is no more data (0)....

APN Unfortunately I no longer have a domain environment to test this. Are you sure the account running twapi has access rights to those groups? Can you try running this program - https://nettools.net/netgroupenum/ - and see if you get all the groups.

MHo: Unfortunally, like for many other URLs, the nettools-URL is blocked by our gateway security software.... So I did this:

>dsquery group -s hekintra.de -limit 0 > dsquerygroup.txt # gives 5150 entries
>tclkitsh
% packa re twapi; twapi::import_commands
% set g [get_global_groups -system hekintra.de]; llength $g
2244

APN: dsquery uses ADSI interfaces. I was trying to see if the issue was in TWAPI or a limitation of NetGroupEnum, thus I asked to try that specific tool. Difficult to make progress without a domain to test with. Can you try using TWAPI's COM commands (comobj) to get the groups with ADSI and see if that works better?

MHo I will try this a soon as possible. Meanwhile I asked a coworker to do a NET GROUP on the domain controller, as I believe that this also uses the Win32 functions and not something newer like ADSI. NET GROUP lists 2252 Groups (a few more are added meanwhile).... now what's going on here...... I'll have to compare the results side by side to see what groups are missing. Could it really be true, that the older interfaces have a some coded limits...? I fear, inside windows, *everything* is possible..... I discussed further with my AD guy. Maybe groups of a certain (newer) types are only handled by newer APIs, like Exchange-Groups or so.

Further investigations done: The output of twapi is almost identical with that of NET GROUP: TWAPI lists a groupname beginning with the (I think illegal) character §, while NET GROUP does not, and for another name vice versa (currently I'm seeing this group in the NET GROUP-Output which TWAPI does not include: õ13Abs.2_SGB_V). So, I think TWAPI works perfect! What I guess is that the following groups are not listed by NetGroupEnum&Co:

• Groups with a different Pre-Win-2000-Groupname
• Domain-local-groups

Indeed, to see such groups, one have to use ADSI etc.:

package require twapi
twapi::import_commands

#################
### AD-Suche als Coroutine
# Portiert von: Windows Script Referenz, T.Weltner, S. 306
### Parameter:
#   filter: LDAP-Suchfilter-Ausdruck
#   return: Liste von Attributen (durch , getrennt), die zurückgegeben werden sollen 
#   ADsPath: Optionaler base (root) für Suche; falls fehlend := DefaultNamingKontext
### Achtung: Fehler werden hier aktuell nicht abgefangen, das ist Aufgabe des Aufrufers!
### Siehe auch:
# - http://www.selfadsi.org/search.htm#Prepare
# - http://www.selfadsi.de/ldap-filter.htm
# - https://administrator.de/forum/active-directory-emailadressen-datei-ausgeben-74893.html
# - https://www.faq-o-matic.net/2005/04/27/alle-mailadressen-anzeigen/
# - https://social.technet.microsoft.com/wiki/contents/articles/5392.active-directory-ldap-syntax-filters.aspx
# - https://www.w3schools.com/asp/ado_ref_recordset.asp
#
proc searchAD {filter return {ADsPath ""}} {
     # ADODB-Verbindung einrichten
     set connection [comobj "ADODB.Connection"]
     $connection -set Provider "ADsDSOObject"
     $connection Open 
     
     # Abfrage formulieren
     set command [comobj "ADODB.Command"]
     $command -set ActiveConnection $connection
     $command -set Properties "Page Size" 5000; # 1000
     $command -set Properties "Cache Results" True

     if {$ADsPath eq ""} {
        # ADsPath der Domäne ermitteln
        set rootDSE [comobj_object "LDAP://rootDSE"]
        set ADsPath [$rootDSE -call Get defaultNamingContext]
     }

     # Kommando konstruieren
     set query "<LDAP://${ADsPath}>;$filter;$return;subtree"
     # puts $query

     # Kommando ausführen
     $command -set CommandText $query
     set recordSet [$command Execute]
     # set ret [list]
     yield $recordSet; # oder yield
     if {[$recordSet RecordCount]} {
        while {![$recordSet EOF]} {
            # puts [$recordSet GetString]; # nicht alle Objekte haben offenbar GetString!
            # set d [dict create]
            set l [list]
            set o [$recordSet Fields]
            for {set i 0} {$i < [$o Count]} {incr i} {
                set item [$o item $i]
                # dict set d [$item Name] [$item Value]
                lappend l [$item Value]
            }
            # lappend ret $d
            # lappend ret $l
            # stattdessen:
            yield $l
            # if {![$recordSet EOF] && ![$recordSet BOF]} 
            $recordSet MoveNext
        }
     }
     $recordSet Close
     return -code error "EOF"
}


if {![catch {coroutine getGrp searchAD \
   {(&(objectCategory=group)(objectClass=group))} \
   {sAMAccountName} \
   {dc=hekintra,dc=de}}]} {
   while {![catch {getGrp} rec]} {
      puts $rec
   }
}

Don't know why, but this output again differs from DSQUERY GET. Maybe the search spec I used isn't correct. But all this shows: The simple question "What Groups exist?" has no simple answer...

MHo 2020-10-12: Would it make sense to expose the Windows API-Calls FindFirstFileEx*, FindNextFile*, etc. to the script level, so one can iterate over arbitrary deep nested file hierarchies without using much memory?

MHo 2021-02-27: Just noticed that there is no command like map_file_attributes as referenced in the help of find_file_next.... very great, nevertheless! APN 2021-02-28: Documentation error. The command is decode_file_attributes, not map_file_attributes.

c:\Users\matthiasu\usr\pgm\tcl\usr\Tst\jpgIndexNeu>tclkitsh
% info pa
8.6.11
% packa re twapi
4.5.2
% twapi::import_commands
% map_file_attributes
invalid command name "map_file_attributes"
%

Example: recursively iterating through a directory tree using the new commands:

package require twapi_storage
twapi::import_commands

set more [lassign $argv rootDir]

proc iterateFiles {root} {
     set ff [find_file_open [file join $root *]]
     while {[find_file_next $ff buf]} {
          set n [dict get $buf name]
          set a [dict get $buf attrs]
          set f [file join $root $n]
          if {$n ni {. ..}} {
             if {($a & 16) == 16} {
                iterateFiles $f
             } else {
                puts $f
             }          
          }
     }
     find_file_close $ff
}

iterateFiles $rootDir

MHo 2021-09-17: Recently I've trouble writing to the win eventlog with eventlog-log. The data looks ok in EventVwr in XML-Format, but on the "Allgemein" tab (common) there's only the text "Unzulässige Funktion" (invalid function). Win 10, TWAPI 4.5.2. Don't know what's going on; in the past, I had no problems with that...... Example XML-View:

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="tclkitsh" /> 
  <EventID Qualifiers="0">1</EventID> 
  <Version>0</Version> 
  <Level>4</Level> 
  <Task>0</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2021-09-17T18:37:10.7120837Z" /> 
  <EventRecordID>23524</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="0" ThreadID="0" /> 
  <Channel>Application</Channel> 
  <Computer>DESKTOP-AI70A0O</Computer> 
  <Security /> 
  </System>
- <EventData>
  <Data>test4</Data> 
  </EventData>
  </Event>

Ok, I think it's probably the same issue already discussed before on the page... But then I don't know why it sometimes work as expected...

MHo 2022-02-02: I've quoting trouble reading service configuration, illustrated as follows:

% get_service_configuration appserver -command
-command {D:\Programme\Projectile\bin\appserver2.64.exe //RS//appserver}
% set res [get_service_configuration appserver -command]
-command {D:\Programme\Projectile\bin\appserver2.64.exe //RS//appserver}
% dict get $res -command
D:\Programme\Projectile\bin\appserver2.64.exe //RS//appserver
% set x [dict get $res -command]
D:\Programme\Projectile\bin\appserver2.64.exe //RS//appserver
% lindex $x 0
D:ProgrammeProjectilinppserver2.64.exe

I'm not able yet to retrieve the programs path in that simple way. The only solution found so far is this:

% set y [string map [list \\ /] $x]
D:/Programme/Projectile/bin/appserver2.64.exe //RS//appserver
% lindex $y 0
D:/Programme/Projectile/bin/appserver2.64.exe

In this simple case it's ok since I'm only interested in the program path. There must be some step I'm missing here. I think it's because the -command string is not a list, so lindex fails. But then I've to implement parsing the -command string by myself...

APN 2022-02-03: Yes, what you get back is a string which is may not conform to Tcl string representation. The application has to parse it itself. You can check if the get_command_line_args command helps as that parses using Windows conventions.

HaO 2022-02-02: You may use *file normalize* to convert the native path to TCL's idea what a path is:

% file normalize {D:\Programme\Projectile\bin\appserver2.64.exe}
D:/Programme/Projectile/bin/appserver2.64.exe

But you first have to separate the path and the arguments. I suppose, there might be "" around the path or not.

MHo 2022-02-03: Thank you all. Yes it's not a tcl list so I have to parse it myself. I cannot file normalize beforehand as this would alter possible args, too. First I have to separate the programspec from the rest. And I've overlooked get_command_line_args is perfect for this case... but what is this?:

C:\Users\matthiasu\usr\pgm\tcl\usr\Tst\Projectile>tclkitsh
% packa re twapi
4.6.0
% twapi::import_commands
% get_command_line_args
wrong # args: should be "get_command_line_args cmdline"
% get_command_line_args "D:\Programme\Projectile\bin\appserver2.64.exe //RS//appserver"
D:ProgrammeProjectile inppserver2.64.exe //RS//appserver
% foreach a [get_command_line_args "D:\Programme\Projectile\bin\appserver2.64.exe //RS//appserver"] {
puts $a
}
D:ProgrammeProjectile
inppserver2.64.exe
//RS//appserver
%

Thats the same problem, isn't it?

Done further tests and came to the conclusion that is impossible to ever use the return value of get_service_configuration appserver -command in a tcl script. If the backslash is eg. followed by n, every operation sees a newline. So, the trick with string map above only works accidentally..

APN Your example above is incorrect. You are passing a string literal to get_command_line_args in quotes so Tcl will do backslash substitution even before get_command_line_args is called. This is not the same as passing it a string in a variable. If you enclose the literal in braces instead, you will see results as you expect.

(doc) 4 % foreach a [get_command_line_args {D:\Programme\Projectile\bin\appserver2.64.exe //RS//appserver}] { puts $a }
D:\Programme\Projectile\bin\appserver2.64.exe
//RS//appserver

MHo Thank you. Yes, this construct does what I want:

lindex [get_command_line_args [dict get [get_service_configuration $svc -command] -command]] 0

MHo 2022-03-11: How to recover from errors from find_file_next? I have a situation where, within the loop, the network is (perhaps a short time) not available:

Der Netzwerkpfad wurde nicht gefunden.
    while executing
"FindFirstFileEx $path $detail_level 0 "" $flags"
    (procedure "find_file_open" line 13)
    invoked from within
"find_file_open [file join $root *]"
    (procedure "iterateFiles" line 4)
    invoked from within
"iterateFiles $f"
:
:

It seems that find_file_next must be wrapped with a catch but then what would happen with the next find_file_next when the network is available again? Would it be better if find_file_next returns with a distinct returncode in such case (like -1)? Is the search handle intact in such case, or is it better to cancel the whole operation and to start over with a fresh find_file_open...?

APN Given that the FindFile API does not itself provide for these error conditions, I doubt twapi can do anything about it.

How to set time and date of a Windows desktop

HaO2023-03-03: To set time and date of a windows desktop, elevated priviledges are necessary. The following command will ask for administrator priviledges and then set time and date:

twapi::shell_execute -verb runas -path cmd.exe -params {/c "time 09:46 & date 3.3.2023" }

MHo 2024-07-08: Will there ever be a new tclkit(sh).exe Download (based on Tcl 8.6.14 and later 9.x with TWAPI 5.x) at https://sourceforge.net/projects/twapi/files/Tcl%20binaries/Tclkits%20with%20TWAPI/ or maybe https://github.com/apnadkarni ?