Project within the Fido-History-Project 3rd Generation: Reverse Engineering of Nodelist-DIFFs

Back to Fido-History-Project page

 

Reverse Engineering of Nodelist DIFFs ...

[28.4.2007] On doing some recovery of single files or parts of the Fido-History-Project archive, i'd running into a problem were 2 full lists exists with all except 1 diffs in sequence.

In detail:
In 1993 Region 24 has been reorganized in Nodelist #173/1993. The changes hasn't been reflected in the Region Pointlist from beginning, so the distribution with the new structure starts after a few weeks.
In the meanwhile R24 has became a split into R24 Classic (the group that patched the official Nodelist with the Classic R24 structure before #173/1993) and R24 Light.
Also the Pointlist has became a split into a R24 Classic distribution and a R24 Light distribution.
As of system crashes, defective tapes, lost floppys, the archive of old R24 Pointlists was reduced to about 6 Pointlists after #173/1993, and again subsequent R24 Pointlists starting #049/1994. Between the #351 and #365/1993 and #049/1994 List there are subsequent Pr24Diffs starting #007/1994 available.
So it comes to an idea, to reverse engineer the DIFF processing backwards from Points24.049/1994 back to Points24.365/1993 and fill the difference with the information from Points24.351/1993.

Reverse Engineering of FTN Diffs



Theory

Assumption: i have a source FTN List and a FTN Diff file.

List #014Diff #021 List #021
AC1   Copy 1=>A
 A1   Add 1  
 Q=>Q
BC1   Copy 1=>B
CD1   Delete 1X 
DC1   Copy 1=>D

A source List with records A,B,C + D becomes after the Diff process A, Q, B + D


Reverse engineering:


In reverse order, there is a possible chance to re-build from List #021 and Diff #021 a List #014 if i can relate in a 2nd step to a list before (#007):

List #021Diff #021 List #014
AC1   Copy 1=>A
QA1   Add 1 (reverse: Delete 1)X 
 Q     skip ADDs  
BC1   Copy 1=>B
 D1   Delete 1 (reverse: Add 1)=>?
DC1   Copy 1=>D

A source List with records A, Q, B + D becomes after the reverse Diff process A,B,? + D

COPY works for the DIFF in both directions.

An ADD command means, that a record in the old list doesn't exist and a new record will be inserted into the newer List.
From the view of the new list, this record doesn't exist before.
So the ADD command works as a DELETE command in reverse direction.

Now to the DELETE command:
The delete command becomes in reverse mode an Add command.
A record that exists before, has to be deleted. In reverse engineering this record has to be rebuild. As i don't know the content, that has been deleted before, i'm using a placeholder.

The 2nd alignment process:


If i have a list one prior (#007) before the list that have to be recovered (#014), i can compare the older list (#007) record by record with the recovered list (#014) and can replace each placeholder with the content that relates in the older list (#007).

List #007 List #014
A   A
B   B
C =>> ?
D   D

A recovered List with records A,B,? + D becomes after the Lists alignment process A,B,C + D


Known Problems:


If a record is added to the recoverable list and has been deleted later in the diff sequence that is used to get the recoverable list, no infos about the original content of that line exists (not in list prior to the recoverable list, not in the diffs, nor in the starting list for recovery).
Such a problem also occure if a record has been deleted and re-added (applying changes with later delete in the diff sequence).
One goodie:
All such problems can be detected by the CRC check, but this prevents to successfuly recover the list.

LISTPrior
List
#007
List to
recover
#014
1st DIFF
#021
2nd DIFF
#028
3rd DIFF
#035
Starting
List
#035
Real
Content
,1,Point1,..,1,Sysop,..,1,Sysop,..,1,CoSysop,..,1,CoSysop,..,1,CoSysop,..
Diff cmd D1
A1
,1,Sysop,..
C1D1
A1
,1,CoSysop,..
C1C1
Known by
Rev.Eng
process
,1,Point1,..D1 ->
(,1,Point1,..)
A1
?
?D1 -> ?
A1
,1,CoSysop,..
,1,CoSysop,..,1,CoSysop,..


Problems in practice:


By the reverse Recovery try of Points24.365/1993 from Points24.049/1994 with Diffs:
# 007, 014, 021, 028, 035, 042, 049 / 1994
i've successfuly recovered Points24.042/1994 (!), CRC check OK   :-)

But, one list before, Points24.035 results in an error.
There are 40 possible mismatches of Bossnode segments with 1 to 70 mismatches of Pointlistings each. Lines that has been replaced or removed are such possible mismatches.

To resolve this problem, all entries have to be checked, if one of these lines has an ADD or CHANGE reference in one of the DIFFs # 007, 014, 021, 028, 035, 042
so these lines becomes to be no longer possible mismatches.
By visible compare and check i can exclude the first seven Bossnode segments of seven checked segments from the mismatch table, because their lines are referenced in one of the DIFFs #007-042.
Checking step by step each referenced entry reduces the possible mitmatch lines, that can be possibly fixed in a later step to get a correct CRC value for Points24.035.

One note: Sources of Net 244 Bossnode Pointlist Segments sent for Points24.365 helps me, to find some correct lines listings. Other mismatches, i have to figure out.

to be continued ...

'cause its a bit time consuming ... :-/


May 2007: giving up.


 
By Ulrich Schroeter @ ambrosia60 (2:244/1120)
29.4.2007