MetaKit database. Specification of the format used to serialize the in-memory representation to a file.
The foundation for this description can be found at http://www.equi4.com/metakit/format.html
All data is stored using the endianness of the platform creating the database file. Platforms reading a file written with an endianness different from their own will convert automatically, on the fly.
See also the questions at the end.
Generic format
- Header
- Scattered meta-data and content
Header
- Magic number, 2 byte, also used to determine endianess of the file. Value unknown.
- Reserved area, 2 byte
- Pointer to structural information, 4 bytes, an offset counted from the beginning of the database itself, i.e. the first byte of the header. We cannot say counted from the beginning of the file as the database may be embedded in a larger file, usually attached.
Structural information
- Format code. Size unknown.
- Structure definition string. Defines the structure of all views containing in the file. View == Table in a regular database. Of varying size.
- Per view information about its size, as the number of rows contained in it, followed by the location and size of the data for all its columns. The location of a column is specified as offset relative to the beginning of the database. The size is length of the data area for the column, in bytes. Each integer value is stored using the Adaptive Integer format described later. For columns which are internally actually represented as two physical columns (i.e the binary types) the system will store two location/size pairs.
Structure definition string
- The character set underlying this string is unicode, the character encoding is UTF-8.
- Contains the description of all views.
- The descriptions of the individual views are separated by commas (,).
- A description for a view is of the format "viewName [ column-descriptions ]".
- The column descriptions are separated by commas (,).
- A column description can be either be of the simple format "columnName : columnType", or a full-fledged view description. The name of such a sub-view is then the name of the column.
Column data types
These databases are only for columns which are not subviews. In the column description (see above) they are all represented by a single character. See the description of the column formats below to find the character representing the type.
- String - S
- Unicode, UTF-8 encoded. Column data are the concatenated strings, with the terminating zero bytes separating each item.
- Integer - I
- Use 1..32 bits per value, depending on the largest absolute value.
- Floats - F
- 32-bit floating point - 4 bytes per entry.
- Double - D
- 64-bit floating point - 8 bytes per entry.
- Binary - B
- Untyped binary data - Stored in two columns. The first column contains the actual data, without separators. The second column stores the sizes of the individual entries, as adaptive integers.
- Memo - M
- Like binary, except that the data column actually contains offsets into the database to the locations where the data is stored. This makes for better usage of the available space. When adding data to a memo view existing data doesn't have to be moved, and resizing happens only for the comparatively small table of offsets and not for a large blob of binary data. Even better when entries are deleted or set to NULL. The offsets are relative to the beginning of the database. The offsets are stored as adaptive integers.
- Subview
- Each entry in a subview column contains the location and size for each column in the subview, stored as adaptive integers. This is similar to memo's, except that the refered areas in the file are not unstructured blobs. Another difference is that the location/size data is stored in a single column, and not separated in two. The latter is difficult to handle, given that a subview may contain more than one column.
Adaptive Integer
Stored with the most significant bits first. Each byte stores 6 bits of the original integer value in its bits 0 to 6. Bit 7 is flag and is set for the last byte in the value. For all other bytes bit 7 is not set.
Questions:
- Metakit databases can be attached to an executable (*pack), or a script (*kit). This means the beginning of the database is not known, only the end. It is not clear from the above how to jump from the end of the file to its header.
- It is not quite clear what is stored as the size of a colmn. The description above tries to make a reasonable assumption.
- It is unclear for the two binary column types, i.e. the two types stored as two columns how this affects the offset/size information in the structural information. Does that mean that for a column of type B or M the per view information will contain two offset/size pairs ? Or only one offset/size pair and the referenced location is somehow split ? I guess that it is the former, and described it that way above.
- For integer columns it is not clear if each entry is stored as adaptive integer, or if all entries are of the same size, the size chosen according to the largest absolute value. In case of the latter it is not clear where the information about the size of entries is stored, so that we can separate the values from each other at the bit level.
- Offsets are always described as relative to the beginning of the database, i.e. the first byte of the header, which is at offset 0. This is actually not explicitly stated in the original description. So they could be relative to the location of where the offset is stored. I consider this unlikely, still, the original description does not rule this out.
- The contents of a column for a subview are not described at all in the original. My description above is a reasonable guess.