Liberating data from Encrypted TPS Files

My previous article on Clarion TPS files left one big question unanswered: how do encrypted TPS files work and is it possible to decrypt them. In this post I will dissect the encryption algorithm and explain how it works. It involves quite a bit of binary arithmetic and hexadecimal numbers, so take a deep breath before diving in!

First there is the password. It is passed as a parameter to the TPS driver. Oddly enough it is called the 'owner' parameter. With the password a key is generated which is used to encrypt and decrypt the data. The effect is pretty dramatic.

A close look at TPS files.

What are the differences between an encrypted and decrypted file?

The block below is the header of an unencrypted TPS file, easily identified by its tOpS identifier at position 0x0E. Also notice the repetitive nature of the data after position 0x0030 - this is the jump table for data areas in the file.

00000000h: 00 00 00 00 00 02 00 06 00 00 00 06 00 00 74 4F ; ..............tO
00000010h: 70 53 00 00 00 00 00 12 49 00 00 00 00 00 00 00 ; pS......I.......
00000020h: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ; ................
00000030h: 00 00 00 00 04 00 00 00 04 00 00 00 04 00 00 00 ; ................
00000040h: 04 00 00 00 04 00 00 00 04 00 00 00 04 00 00 00 ; ................
00000050h: 04 00 00 00 04 00 00 00 04 00 00 00 04 00 00 00 ; ................
00000060h: 04 00 00 00 04 00 00 00 04 00 00 00 04 00 00 00 ; ................
00000070h: 04 00 00 00 04 00 00 00 04 00 00 00 04 00 00 00 ; ................
00000080h: 04 00 00 00 04 00 00 00 04 00 00 00 04 00 00 00 ; ................
00000090h: 04 00 00 00 04 00 00 00 04 00 00 00 04 00 00 00 ; ................
000000a0h: 04 00 00 00 04 00 00 00 04 00 00 00 04 00 00 00 ; ................
000000b0h: 04 00 00 00 04 00 00 00 04 00 00 00 04 00 00 00 ; ................

The second block is encrypted data. It looks pretty random. However. If you look more closely you will notice that it is repeating every 0x40 bytes. For example, the data at 0x0040 is the same as at position 0x0080. Identical data leads to identical encrypted data every 64 bytes. Thus the encryption works in blocks of 64 bytes that use the same key (so no key progression or block chaining, fortunately).

00000000h: B4 DC 3C 92 90 BC DF 78 B0 5B AF FF A0 F8 30 41 ; ´Ü<’¼ßx°[¯ÿ ø0A
00000010h: 00 AE FF D8 F0 BF EF A2 E0 DC FA 57 F7 BF FB B3 ; .®ÿØð¿ï¢àÜúW÷¿û³
00000020h: A8 54 DA C0 70 6D 7D EA 30 E9 FD 7A D0 7A FD F4 ; ¨TÚÀpm}ê0éýzÐzýô
00000030h: D8 FF FE F1 50 F9 BE C1 E4 D3 77 E3 F4 7A FF F1 ; ØÿþñPù¾ÁäÓwãôzÿñ
00000040h: B4 DC 5C 92 94 BC DF 78 B4 59 AF F9 A4 F8 DC FA ; ´Ü\’”¼ßx´Y¯ù¤øÜú
00000050h: E4 5D FF D9 E4 BC EF B1 A4 DC FA 73 F4 BF FB 53 ; ä]ÿÙä¼ï±¤Üúsô¿ûS
00000060h: AC 54 DA C0 74 6D FD AA 34 E9 BD FA D4 7A FD D4 ; ¬TÚÀtmýª4é½úÔzýÔ
00000070h: DC FF 7E E1 54 F9 FE C1 E4 D3 77 E3 F4 7A BF F1 ; Üÿ~áTùþÁäÓwãôz¿ñ
00000080h: B4 DC 5C 92 94 BC DF 78 B4 59 AF F9 A4 F8 DC FA ; ´Ü\’”¼ßx´Y¯ù¤øÜú
00000090h: E4 5D FF D9 E4 BC EF B1 A4 DC FA 73 F4 BF FB 53 ; ä]ÿÙä¼ï±¤Üúsô¿ûS
000000a0h: AC 54 DA C0 74 6D FD AA 34 E9 BD FA D4 7A FD D4 ; ¬TÚÀtmýª4é½úÔzýÔ
000000b0h: DC FF 7E E1 54 F9 FE C1 E4 D3 77 E3 F4 7A BF F1 ; Üÿ~áTùþÁäÓwãôz¿ñ

I spent quite some time tinkering with XOR schemes, but there is no fixed relation between the bits and their position in the block. Some pretty clever bit shifting is going on. Time to bring out the big guns: IDA Pro. The Free version worked perfectly on the ancient TPSCOPY tool so I spent some evenings stepping through the assembly. Let me tell you what I found.

Step one: Building the Key from the owner string.

The first step is to convert the owner string (the password) to a 64 byte block that will be the encryption and decryption key. The password is treated as a C string, so a 0x00 is at the end. Funnily enough it starts building this key from the 2nd character in the password, not the first.

Filling the block is done by filling out the block in steps of 0x11 bytes, adding the index to the current value and starting at offset 1:

key[(index * 0x11) & 0x3f] = index + password[(1 + index) % password.length]

For a short and simple password of 'a' this leads to:

0000 : 00 92 22 74 04 96 26 78 08 9a 2a 7c 0c 9e 2e 80  .."t..&x..*|....
0010 : 10 62 32 84 14 66 36 88 18 6a 3a 8c 1c 6e 3e 90  .b2..f6..j:..n>.
0020 : 20 72 02 94 24 76 06 98 28 7a 0a 9c 2c 7e 0e a0   r..$v..(z..,~..
0030 : 30 82 12 64 34 86 16 68 38 8a 1a 6c 3c 8e 1e 70  0..d4..h8..l<..p

Notice how the first value is 0x00 (the end of the C string) and the first character at 0x0011 is 0x62 which is 'a' + 1. The next value is at 0x0022 and is 0x00 + 2 and so on and so on.

Obviously this key is not nearly random enough. The next step is some interesting bit re-shuffling. It works in 32 bit little endian words, so 4 bytes at a time in reverse order. There are 16 of such words in the key. For each position in the key it takes that word (a) and another indexed by the least significant 4 bits of that word (b). Then it performs the following computation:

a' = a + a & b
b' = a + a | b

For the first position in the key this means that:

a = word[0] = 0x74229200
b = word[0x74229200 & 0x0F] = word[0] = 0x74229200
And thus:
a' = 0x74229200 + 0x74229200 & 0x74229200 = 0xE8452400
b' = 0x74229200 + 0x74229200 | 0x74229200 = 0xE8452400

So the first word in the key is replaced by 0xE8452400. Not very spectacular but once the b index is different it becomes quite messy. This shuffle operation is performed twice. Once it is done the key formed from the humble 'a' password is unrecognizable, although there are still some patterns visible:

0000 : 80 a4 52 70 90 18 dd 68 a0 48 ab f1 a0 f8 dc 48  ..Rp...h.H.....H
0010 : a0 5c ee 90 60 b8 ed 21 a0 d4 f8 51 f0 ba fb 10  .\..`..!...Q....
0020 : 40 22 74 e0 30 69 2d aa 20 e9 ac 72 d0 78 fd c0  @"t.0i-. ..r.x..
0030 : 98 2c 6e 60 50 d1 32 c1 e0 d1 72 e1 b0 7a 3c d1  .,n`P.2...r..z<.

This block of data is used to encrypt and decrypt the tps file and thus forms the Key.

Step 2: Encrypting a block.

Now we have a key, we can attempt to encrypt a block. The encryption algorithm is quite similar to the shuffling algorithm and works indexed as well.

k   = key[index]
a = block[index]
b = block[k & 0x0F]

a' = k + (!k & b |  k & a)
b' = k + ( k & b | !k & a)

block[index] = a'
block[k & 0x0F] = b'

It takes two words, one at the current index and one at the index pointed at by the 4 least significant bits of the key value at that position. Then half of the bits of a and half of the bits of b (according to the key mask) are added to the key and assigned to a'. The other half of the bits are added to the key and assigned to b'. Then the values are written back.

Let me show an example on how this works:

key = 0xF0F0F0F0
a = 0xFFFF0000
b = 0x0000FFFF

a' = 0xF0F0F0F0 + ( 0x0F0F0F0F & 0x0000FFFF  | 0xF0F0F0F0 & 0xFFFF0000)
b' = 0xF0F0F0F0 + ( 0xF0F0F0F0 & 0x0000FFFF  | 0x0F0F0F0F & 0xFFFF0000)

a' = 0xF0F0F0F0 + ( 0x00000F0F | 0xF0F00000)
b' = 0xF0F0F0F0 + ( 0x0000F0F0 | 0x0F0F0000)

a' = 0xF0F0F0F0 + 0xF0F00F0F = 0xE1E0FFFF
b' = 0xF0F0F0F0 + 0x0F0FF0F0 = 0x0000E1E0

Ok this explains by my XOR tricks didn't work out. Its much more like the RC5 algorithm as it uses addition for encryption and as we'll see subtraction for decryption.

Step 3 : Decrypting a block

Decrypting is done in reverse order than encrypting, so the index starts at 0x0F and ends at 0x00 as it has to produce the bit distribution in exact the reverse order. The algorithm is as follows:

a' = ( key & (a - key)) | (!key & (b - key))
b' = (!key & (a - key)) | ( key & (b - key))

Yes, that is exactly the opposite of the encryption. As it should be of course.

The example below takes the results from the previous encryption example and reverses the process:

key = 0xF0F0F0F0
a = 0xE1E0FFFF
b = 0x0000E1E0

a - key = 0xF0F00F0F
b - key = 0x0F0FF0F0

a' = 0xF0F0F0F0 & 0xF0F00F0F | 0x0F0F0F0F & 0x0F0FF0F0 
b' = 0x0F0F0F0F & 0xF0F00F0F | 0xF0F0F0F0 & 0x0F0FF0F0

a' = 0xF0F00000 | 0x0F0F0000 = 0xFFFF0000 
b' = 0x00000F0F | 0x0000F0F0 = 0x0000FFFF

And the original values are back!

This is how the encrypting and decrypting works for a single key entry. This process is repeated for all 16 key words. Forwards for encrypting and backwards for decrypting as the same block words need to be selected for the process to be reversible.

Conclusion

The TPS encryption algorithm is remarkably elegant and fast to run while it provides good enough encryption to stop most amateurs. I have implemented the algorithm in my tps-parse library and tps-to-csv tool so that they now support the encrypted files as well. You still have to know the password of course!