An arbitrary precision integer math library (also known as bignums) implemented in pure Tcl under the BSD license can be found at http://wiki.hping.org/108 . Hopefully it will be available in Tcllib in short time. In this page you can also find usage examples about RSA and Diffie-Hellman Key Agreement Protocol.
See also math::bignum
Note that Tcl 8.5 includes extensive bignum support.