***How to detect text encoding use tcl, anyone know about it?***
There is a python implement called '''chardet''' [https://github.com/chardet/chardet].
It works well.
[gobvip] 2010/07/21
-----
The file command can do it too
======none
bash-5.1$ file --mime-encoding foobar.txt
foobar.txt: iso-8859-1
======
The encguess command from Perl can also do this. There are pathological cases that encguess handles better than the file command, and vice versa.
[pi31415] 2024/01/26
[MiR] 2024/01/30
If its only to decide if a file is a test or binary file, one can use code from file's encoding.c like this:
(proof of concept, extend to your needs if necessary)
======
proc checkFileText {fname} {
# check if this is a text or a binary file
# https://github.com/file/file/blob/f2a6e7cb7db9b5fd86100403df6b2f830c7f22ba/src/encoding.c#L151-L228
#define F 0 /* character never appears in text */
#define T 1 /* character appears in plain ASCII text */
#define I 2 /* character appears in ISO-8859 text */
#define X 3 /* character appears in non-ISO extended ASCII (Mac, IBM PC) */
set text_chars {
F F F F F F F T T T T T T T F F
F F F F F F F F F F F T F F F F
T T T T T T T T T T T T T T T T
T T T T T T T T T T T T T T T T
T T T T T T T T T T T T T T T T
T T T T T T T T T T T T T T T T
T T T T T T T T T T T T T T T T
T T T T T T T T T T T T T T T F
X X X X X T X X X X X X X X X X
X X X X X X X X X X X X X X X X
I I I I I I I I I I I I I I I I
I I I I I I I I I I I I I I I I
I I I I I I I I I I I I I I I I
I I I I I I I I I I I I I I I I
I I I I I I I I I I I I I I I I
I I I I I I I I I I I I I I I I
};
set fn [open $fname rb]
fconfigure $fn -translation binary
set teststring [read $fn 1024]
close $fn
set n [string length $teststring]
binary scan $teststring "c$n" binstr
array get $binstr
for {set i 0} {$i<$n} {incr i} {
set bstr [lindex $binstr $i]
set bstr [expr {$bstr & 0xFF}]; # val == 0x8000
set cstr [lindex $text_chars $bstr]
if {$cstr=="F"} {
return -1
}
}
return 1
}
======
<<categories>>Category Printing | Category HTML | Category Word and Text Processing