Topic: PC Game Hacking and Mapping Tutorial: Xargon

[Zerker]

Now for something different. Today is font day!

For ROTT, I made my own font class, which contained an array of images that got pasted into the map. It worked, but this time I'd like to leverage the PIL ImageFont capabilities (especially for the re-colouring). Problem is, I need to get the font data into a format that PIL can use. I see two options here:
1) Look up the format for BDF or PCF font files, and create one
2) Figure out the underlying PIL font code and create a compatible implementation based on the in-game font.

Looking at the definition of BDF, that format doesn't look too bad, so I think that is the most likely avenue here. A side-effect of this is that we can actually use the Xargon font in BDF-supporting applications.

To get the font ready, I'm going to start a new file called xargonfontgen.py. This will create a BDF from both main fonts in the Xargon GRAPHICS file. To do this, we're going to need to convert the font images into 1-bit images. That said, we should get 256-colour exports of the font images to decide which "colours" are used. Let's update the xargongraphics save routines and main function:

Code: [Select]

xargonimages.save('Images')
xargonimages.save('OriginalImages', masked=False)


Code: [Select]

def save(self, outpath, masked=True):
    createpath(outpath)
    for recnum, record in enumerate(self.records):
        record.save(outpath, recnum, masked)


Code: [Select]

def save(self, outpath, recnum, masked=True):
    if self.numimages > 0:
        createpath(outpath)
        if (masked):
            for tilenum, tile in enumerate(self.images):
                tile.save(os.path.join(outpath, '{:02}-{:04}.png'.format(recnum, tilenum)) )
        else:
            for tilenum, tile in enumerate(self.origimages):
                tile.save(os.path.join(outpath, '{:02}-{:04}.png'.format(recnum, tilenum)) )

I actually worked a bit ahead on this yesterday, but knew I wasn't going to finish in one day. Astute followers may have noticed the above updates already in yesterday's zip file.

Looking at one of the font images, they appear to be effectively 3-bit images, using values of 1, 2 and 3 (not 0). Value 3 is transparent. From my observations, the menu text appears to use the two different colours, but all the in-map text (i.e. what we care about) does not.

Next, let's compare the images to an ASCII table to make sure they're in the same order... and they are. Same indices too, although the non-printable characters appear to be re-used. So let's start with a basic class that can convert to 1-bit images, but instead of writing the font at first, let's write some debug images to make sure we're categorizing things correctly.

Code: [Select]

from xargongraphics import createpath, imagefile
import sys, os
from PIL import Image

class xargonfont(object):
    @staticmethod
    def conv1bit(inimage):
        outimage = Image.new('1', inimage.size )
        imgdata = list(inimage.getdata())
        for index, pixel in enumerate(imgdata):
            imgdata[index] = 0 if pixel==3 else 1
        outimage.putdata(imgdata)
        return outimage

    def __init__(self, graphics, recnum):
        self.dimensions = graphics.records[recnum].images[0].size
        self.characters = [self.conv1bit(image) for image in graphics.records[recnum].origimages]

    def debugimages(self, outfolder):
        createpath(outfolder)
        for glyphnum, glyph in enumerate(self.characters):
            if glyphnum >= 32:
                glyph.save(os.path.join(outfolder, '{:02}.png'.format(glyphnum)) )


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print """Usage: python xargongraphics.py [Graphics File]
TODO
"""
    else:
        for filename in sys.argv[1:]:
            graphics = imagefile(filename)
            font1 = xargonfont(graphics, 1)
            font2 = xargonfont(graphics, 2)
            font1.debugimages('font1')
            font2.debugimages('font2')



Looks about right. Note that the PIL 1-bit image has 1 = white, 0 = black. We can ignore this. The BDF format has the reverse definition, and that's what we really want to match.

So here's what I came up with to write out the BDF file:

Code: [Select]

def createbdf(self, outname):
    with open(outname, 'w') as outfile:
        outfile.write("""STARTFONT 2.1
FONT {0}
SIZE {1[0]} 75 75
FONTBOUNDINGBOX {1[0]} {1[1]} 0 0
CHARS {2}\n""".format(self.name, self.size, 95))
        for charnum, char in enumerate(self.characters):
            if charnum >= 32 and charnum < 127:
                outfile.write("""STARTCHAR U+{0:04X}
ENCODING {0}
SWIDTH 0 0
DWIDTH {1[0]} 0
BBX {1[0]} {1[1]} 0 0
BITMAP\n""".format(charnum, self.size))
                for y in range(self.size[1]):
                    value = 0
                    for x in range(self.size[0]):
                        value = value + (char.getpixel((x,y)) << self.size[0]-x-1)
                    # Note: this will break if font width is > 8 bits!
                    outfile.write("{:02X}\n".format(value))
                outfile.write("ENDCHAR\n")
        outfile.write("ENDFONT\n")
Note that the alignment is a little bit funny because I'm using multi-line strings. Python's string format method really helps for filling in the data in the appropriate fields. For the meat of the function, I'm just combining each row of the image into a number by using add and bit shift. I'm sure there's a more efficient/pythonic way of doing it, but I only need this to work ONCE to get me a valid BDF file, then I can just keep using the output.

And here's what I get for the first two characters:

Code: [Select]

STARTFONT 2.1
FONT xargon-font1
SIZE 8 75 75
FONTBOUNDINGBOX 8 8 0 0
CHARS 95
STARTCHAR U+0020
ENCODING 32
SWIDTH 0 0
DWIDTH 8 0
BBX 8 8 0 0
BITMAP
00
00
00
00
00
00
00
00
ENDCHAR
STARTCHAR U+0021
ENCODING 33
SWIDTH 0 0
DWIDTH 8 0
BBX 8 8 0 0
BITMAP
18
3C
3C
18
18
00
18
00
ENDCHAR

Seems promising. Let's try this font out.

Code: [Select]

$ pilfont.py *.bdf
xargonfont1.bdf...
Traceback (most recent call last):
  File "/usr/bin/pilfont.py", line 48, in <module>
    p = BdfFontFile.BdfFontFile(fp)
  File "/usr/lib/python2.7/dist-packages/PIL/BdfFontFile.py", line 114, in __init__
    font[4] = bdf_slant[string.upper(font[4])]
IndexError: list index out of range
Uh oh. Now I need to debug someone ELSE's code. Well, more accurately, why my output is wrong and causing the code to crash. Anyhow, let's go open this up. The which command tells me pilfont.py is in /usr/bin/pilfont.py, so I'm going to open that up and find the line in question.

And that just points to the BdfFontFile class inside of PIL, so let's find that. Ah, it's processing the FONT line. I didn't think it would actually need any useful information from that. Let me try and update that to be more representative. It seems to me that it's just performing some adjustments to the name, so I should just need to make sure I have the right number of fields. I'll just grab the wikipedia example and code to match.

Code: [Select]

outfile.write("""STARTFONT 2.1
FONT -{0}-medium-r-normal--{1[0]}-160-75-75-c-80-us-ascii
SIZE {1[0]} 75 75
FONTBOUNDINGBOX {1[0]} {1[1]} 0 0
CHARS {2}\n""".format(self.name, self.size, 95))

That gets us past that block, but now we get: KeyError: 'FONT_ASCENT'. Well, we omitted FONT_ASCENT because the BDF spec says it's optional, and it doesn't really apply to us. Guess pilfont requires it. Let's just put it in and set it to the height with the DESCENT set to 0.

Code: [Select]

outfile.write("""STARTFONT 2.1
FONT -{0}-medium-r-normal--{1[0]}-160-75-75-c-80-us-ascii
SIZE {1[0]} 75 75
FONTBOUNDINGBOX {1[0]} {1[1]} 0 0
STARTPROPERTIES 2
FONT_ASCENT {1[1]}
FONT_DESCENT 0
ENDPROPERTIES
CHARS {2}\n""".format(self.name, self.size, 95))

Now, pilfont runs with no errors. Let's try out the new .pil files.

Code: [Select]

# Text sprites:
self.addsprite(6, 0, textsprite(ImageFont.load("font2.pil"), graphics))
self.addsprite(7, 0, textsprite(ImageFont.load("font1.pil"), graphics))

We can also take out the alignment corrections we added for the true-type font we picked inside the textsprite class.



Well, one font looks good, but the other is cut off. You know, I bet the BDF spec has left-aligned pixels instead of right-aligned pixels. This means I need to shift my 6-pixel font 2 images over 2 more pixels. For simplicity, I will simply update to align to an 8 bit number in my BDF export:

Code: [Select]

for x in range(self.size[0]):
    value = value + (char.getpixel((x,y)) << 7-x)



There we go. Font victory!

Now, just for fun, I'm going to use the built-in Xargon font for all my debug images and stop including DroidSans.

Oh, BTW, I just noticed additional files that actually contain map data (DEMO*.XR1, INTRO.XR1, STORY.XR1), so I'll be sure to include those too. Unfortunately, my clever hackery for string processing has malfunctioned on the STORY scene:



I'll have to keep that in mind, but I probably won't be changing anything there for a while. Also, the map ID isn't a good way to recognize alternate palettes, as some of the auxiliary files have different palettes but the same map id of 0. I'm going to switch over to filenames.

Code: [Select]

if self.name.upper() in ['BOARD_03', 'BOARD_05', 'BOARD_06',
        'BOARD_07', 'BOARD_09', 'BOARD_10', 'INTRO', 'DEMO1',
        'DEMO2', 'DEMO3']:
    graphics.changepalette(1)
else:
    graphics.changepalette(0)


FYI: day13.zip is available.

[Zerker]

Good evening. The next couple days are likely going to be fairly boring. We have most of the functionality implemented, so I'm going to spend the time playing the game and identifying sprites. Yesterday we noticed a problem with the sprites in the Story map. I'll likely need to get some debug data output to look into that further, so I'm going to defer that to the weekend when I can look into it more.

The first thing I'm going to do is identify any sprite I can find in the menu maps and the demos. Following that, I'll continue where I left off in the game on stage 5.

The story scene sprites all appear continuous, so I can populate all of those with a simple loop:

Code: [Select]

# Story Scenes:
for subtype in range(24):
    self.addsprite(85, subtype, sprite(graphics.records[56].images[subtype]))
    self.addsprite(86, subtype, sprite(graphics.records[57].images[subtype]))



Mapping for a bit, I found out that the Episode 3 demo has a slightly different colour palette, so I'll incorporate that.

Code: [Select]

if self.name.upper() in ['BOARD_03', 'BOARD_05', 'BOARD_06',
        'BOARD_07', 'BOARD_09', 'BOARD_10', 'INTRO', 'DEMO1', 'DEMO2']:
    graphics.changepalette(1)
elif self.name.upper() == 'DEMO3':
    graphics.changepalette(2)
else:
    graphics.changepalette(0)


Code: [Select]

palimage = Image.open('palimage3.png')
self.palette[2] = palimage.getpalette()

For Demo 1, I added an entry for floor/ceiling spears. However, I need to expand the variable sprite to allow offsets, because the floor image spear appears inside the floor. I also need to make the offets a list to match with the lookup.

Code: [Select]

# Ceiling Spear
self.addsprite(43, 0, variablesprite({
    0 : graphics.records[36].images[9],
    1 : graphics.records[36].images[12]},
    offsets={0: (0, 0), 1:(0, -4) },
    field='direction'))


Code: [Select]

class variablesprite(sprite):
    def __init__(self, imagelookup, contents=None, field='apperance', offsets=None):
        # Create a lookup of possible boxes
        self.types = imagelookup
        self.xoffs = 0
        self.yoffs = 0
        self.contents = contents
        self.offsets = offsets
        self.field = field

    def draw(self, mappicture, objrec, mapdata):
        # Pick the correct image then use the parent routine to draw the box
        self.image = self.types[objrec.__dict__[self.field]]
        if self.offsets != None:
            (self.xoffs, self.yoffs) = self.offsets[objrec.__dict__[self.field]]
        super(variablesprite, self).draw(mappicture, objrec, mapdata)

        # Place contents immediately above the current sprite
        if self.contents != None:
            mappicture.paste(self.contents, (objrec.x +self.xoffs,
                objrec.y +self.yoffs - self.contents.size[1]), self.contents)

I also notice that it looks like the flaming jet face has a hidden ice cream cone instead of the sprite that would say "this is a flaming jet face".



Since the face is in the background, I just need to recognize this case and simply not draw anything. Looking at the Object file, it appears that the flame jet instance has a 1 in what I called the "Direction" field, while the other two have 4. I'm going to rename that field to variant, and make a special variablesprite case for sprite 73, type 0 in the spritedb:

Code: [Select]

# Special case for 73, type 0. Variant 4 appears to be the pickup item.
# Other variants (all rendered invisible) appear to be:
# 1 : Flaming Face Jet (Down)
# 2 : Flaming Lava Jet (Up)
# 3 : TBC
self.addsprite(73, 0, variablesprite({
    1 : graphics.records[30].images[19],
    2 : graphics.records[30].images[19],
    3 : graphics.debugimage(73, 'T3', 16, 16),
    4 : sprite(graphics.semitransparent(
        graphics.records[37].images[0], 128))},
    field='variant'))

That works for most stages, but it looks like I somehow broke something in DEMO3:

Code: [Select]

Generating Map 'DEMO3'
Traceback (most recent call last):
  File "xargonmapper.py", line 68, in <module>
    mapper = xargonmapper(xargonimages, tiledata, themap)
  File "xargonmapper.py", line 49, in __init__
    sprites.drawsprite(self.mappicture, objrecord, mapdata)
  File "/data/Projects/Xargon/spritedb.py", line 196, in drawsprite
    self.sprites[objrec.sprtype][objrec.subtype].draw(mappicture, objrec, mapdata)
  File "/data/Projects/Xargon/spritedb.py", line 255, in draw
    super(variablesprite, self).draw(mappicture, objrec, mapdata)
  File "/data/Projects/Xargon/spritedb.py", line 226, in draw
    objrec.y +self.yoffs), self.image)
  File "/usr/lib/python2.7/dist-packages/PIL/Image.py", line 1079, in paste
    "cannot determine region size; use 4-item box"
ValueError: cannot determine region size; use 4-item box

That's a bit cryptic, so I think it's time to make my own exception handler to get more info.

Code: [Select]

def drawsprite(self, mappicture, objrec, mapdata):
    try:
        if objrec.sprtype not in self.sprites or \
                objrec.subtype not in self.sprites[objrec.sprtype]:
            self.addsprite(objrec.sprtype, objrec.subtype, sprite(
                self.graphics.debugimage(objrec.sprtype, objrec.subtype,
                objrec.width, objrec.height)))

        self.sprites[objrec.sprtype][objrec.subtype].draw(mappicture, objrec, mapdata)

        #if objrec.info != 0:
        #    self.drawlabel(mappicture, (objrec.x -8, objrec.y -8), str(objrec.info))
    except:
        print "Problem with Sprite {}, Type {}, Appearance {}, Variant {} at ({}, {})".format(
            objrec.sprtype, objrec.subtype, objrec.apperance, objrec.variant,
            objrec.x, objrec.y)
        traceback.print_exc()

FYI: traceback.print_exc() will print the standard stack trace you get without any exception handler.
When I run it again, I get:

Code: [Select]

Problem with Sprite 73, Type 0, Appearance 0, Variant 4 at (848, 640)
Traceback (most recent call last):
< Snip >

Problem with Sprite 73, Type 0, Appearance 0, Variant 4 at (240, 592)
Traceback (most recent call last):
< Snip >

Okay, so it's something about the NORMAL semi-transparent ice cream cone. Let's look at that line.

Code: [Select]

4 : sprite(graphics.semitransparent(
                graphics.records[37].images[0], 128))},

Gah, I'm declaring a sprite inside a variable sprite declaration! I should just be passing the semi-transparent image in directly. Fixing now.

And that's everything I can access from the main menu. I think I'll stop for today. day14.zip is available.

[Zerker]

Hello again. Today will mostly just be me identifying some more sprites, but I thought I'd fix one general bug I noticed yesterday (and no, not the string order in the STORY stage). It's the problem where sprites are drawn over text, contrary to the in-game appearance.



So what I'm going to do is split the text sprites and non-text sprites into two separate lists, and always draw the text after everything else. The main change is in the map loading:

Code: [Select]

self.objs = [objrecord(struct.unpack(objrecstruct,
    mapfile.read(struct.calcsize(objrecstruct)) ) )
    for i in range(numobjs)]

# Create separate sprite and text lists:
self.text = [obj for obj in self.objs if obj.sprtype in [6, 7] ]
self.sprites = [obj for obj in self.objs if obj.sprtype not in [6, 7] ]

And use these two lists for the map:

Code: [Select]

for objrecord in mapdata.sprites:
    sprites.drawsprite(self.mappicture, objrecord, mapdata)
for objrecord in mapdata.text:
    sprites.drawsprite(self.mappicture, objrecord, mapdata)

Also, I'm going to make a really minor change and make the static flame sprites on the menus screens variable to take into account each version:

Code: [Select]

# Menu Flame Jets:
self.addsprite(47, 0, variablesprite({
    6 : graphics.records[48].images[3],
    8 : graphics.records[48].images[4]},
    field='info'))

And here's the update:



And on a total whim, I decided to check the palette of the title screen and see if that's the correct palette for image record 53. Turns out it is!



Sorry folks, that's about all the excitement today. Now I just get to play the game some more and identify more sprites. Well, that's not true. I found out that invisible platforms can come in multiple variants. I just added one to support stage 6:

Code: [Select]

# Variant of Compound and semi-transparent for hidden platform(s)
self.addsprite(11, 0, variablesprite({
    6: graphics.semitransparent(
       graphics.compositeimage((32, 16), [(0, 0, 25, 14),
       (16, 0, 25, 15)]), 128),
    7: graphics.semitransparent(
       graphics.compositeimage((32, 16), [(0, 0, 51, 10),
       (16, 0, 51, 11)]), 128) }
     ))

Stage 6 complete:


I did the stages out of order, so going back to stage 5 doesn't result in anything extraordinary. It has another palette, so I'll add that to the list. It also has switches and toggle-able walls. I will hide the wall sprites and draw the switches. I'll add the switches and walls to my "maybe label the link" list. A few more trap and item types round out the list.

Stage 5 complete:


Stage 7 complete too. Nothing major here. I'll pencil in the stalactites as another possible sprite to mark with trigger identifiers, so we can tell when they would drop (and what causes them).



day15.zip is available.

[Zerker]

Looks like I forgot to post what I wrote up yesterday. Let me get that out of the way:

Another day, more maps. Today I will be filling in the gaps for the next couple levels in the game. The further we get, the fewer sprites need identification. We'll see when we identify them all .

Another map, another palette. I'm going to rename the palette images to start at 0 and generalize this a bit:

Code: [Select]

self.palette = {}
for i in range(6):
    palimage = Image.open('palimage{}.png'.format(i) )
    self.palette[i] = palimage.getpalette()

Stage 8 is done:



And Stage 9:



Not a big day today. I'll map the final stage of Episode 1 tomorrow, then get to some of the other tasks I've been putting off before submitting them for the site. Episode 2 will follow that. day16.zip is available.

[Zerker]

And here's today's post. Time for the final level(s) and ending! Stage 10 isn't too complicated. There's just another illusionary wall, and the "To Reactor" Sign. There's also some sort of unidentified pickup hidden behind an illusionary wall with the ID of 42:0. Since the yellow gem was 42:3, I'm going to take the educated guess that this is the green gem. I'll comment it as "TBC" in case I actually see one in Episodes 2 or 3.

Stage 10 done:


Now, the Reactor stage. First, it appears to have the wrong palette, so I'll fix that immediately. Second, it has these infinite floating robot spawners. I'm going to create a "composite" sprite containing a couple robots clustered together and see how that looks.

Code: [Select]

# Special case for 73, type 0. Variant 4 appears to be the pickup item.
# Other variants (all rendered invisible) appear to be:
# 1 : Flaming Face Jet (Down)
# 2 : Flaming Lava Jet (Up)
# 3 : Robot Spawner
self.addsprite(73, 0, variablesprite({
    1 : graphics.records[30].images[19],
    2 : graphics.records[30].images[19],
    3 : graphics.compositeimage((32, 32), [(0, 0, 59, 1),
       (16, 0, 59, 4), (8, 12, 59, 1)]),
    4 : graphics.semitransparent(
        graphics.records[37].images[0], 128)},
    field='variant'))

Reactor Stage (32) Done:


Ending sequence time. It looks like the ending has a special font-mode which actually uses the two colours for the fonts. Since my setup doesn't let me do that, I'm going to work-around this by drawing the text twice, once for each colour, to get a somewhat similar effect.  Right now it shows up as dark gray, which is wrong.

Code: [Select]

def draw(self, mappicture, objrec, mapdata):
    pen = ImageDraw.Draw(mappicture)

    if objrec.appearance == 8:
        # Simulate multi-colour appearance by creating a fake shadow effect
        pen.text((objrec.x, objrec.y), mapdata.getstring(objrec.stringref),
                font=self.font, fill=self.graphics.getcolour(14))
        pen.text((objrec.x-1, objrec.y), mapdata.getstring(objrec.stringref),
                font=self.font, fill=self.graphics.getcolour(6))
    else:
        pen.text((objrec.x, objrec.y), mapdata.getstring(objrec.stringref),
                font=self.font, fill=self.graphics.getcolour(objrec.appearance))



Not perfect, but a reasonable facsimile. It will do. With that, the shareware game is fully mapped. It just needs a few tweaks before it's ready for the site.

Oh, but there are a few sprites in the "demo" stages that we haven't identified. And, unfortunately, we CAN'T identify without playing the original versions of those stages. Since the demo stages are essentially identical to their original versions, there's no real reason to submit these.

day17.zip is available.

[Zerker]

Hello folks. It's cleanup day. Here are the outstanding tasks I have written down, in the order I'm going to tackle them:

• The player start and one mountain look misaligned on the map and should be adjusted.
• Re-add switch indicators, with filtering to remove other uses of this field.
• Add correct monster facing.
• Fix the string ordering on the STORY scene.

We'll see if we can get these all done today, or if it will spill over into tomorrow.

Task 1 is easy, we just add an offset of 4 to each sprite:

Code: [Select]

# Map Images that need alignment:
for (sprtype, subtype, recnum, imagenum) in [
        (5, 0, 47, 8), # Map Player
        (88, 4, 47, 22), (88, 5, 47, 23)]:
    self.addsprite(sprtype, subtype, sprite(
        graphics.records[recnum].images[imagenum], xoffs=4))

For task 2, I'm going to start by enabling the existing code, and adding the sprite ID to each string for easy debugging (and filter list population). I'm also going to use the smaller font for this.

Code: [Select]

if objrec.info != 0:
    self.drawlabel(mappicture, (objrec.x -8, objrec.y -8),
        "{} ({}:{})".format(objrec.info, objrec.sprtype, objrec.subtype))

With that done, I need a mechanism for filtering our numbers on some sprites. I'm going to add some optional parameters to the sprite __init__ methods. Then, I move the label code into the sprite code, and adjust accordingly. It also looks like some of the high numbers (>=90) are used for form changes, so I will hide those automatically. I'll also add the ability to specify the label offset, for entries (like the trigger pickups) where the switch is the primary purpose.

Code: [Select]

class sprite(object):
    def __init__(self, image, xoffs=0, yoffs=0, hidelabel=False,
            labelpref='', labeloffs=(-8, -8)):
        self.image = image
        self.xoffs = xoffs
        self.yoffs = yoffs
        self.hidelabel = hidelabel
        self.labelpref = labelpref
        self.labeloffs = labeloffs

    def draw(self, mappicture, objrec, mapdata):
        # When pasting masked images, need to specify the mask for the paste.
        # RGBA images can be used as their own masks.
        mappicture.paste(self.image, (objrec.x +self.xoffs,
            objrec.y +self.yoffs), self.image)

        if objrec.info != 0 and objrec.info < 90 and not self.hidelabel:
            text = "{}{} ({}:{})".format(self.labelpref, objrec.info,
                objrec.sprtype, objrec.subtype)

            # Draw the text 5 times to create an outline
            # (4 x black then 1 x white)
            pen = ImageDraw.Draw(mappicture)
            for offset, colour in [( (-1,-1), (0,0,0) ),
                    ( (-1,1), (0,0,0) ),
                    ( (1,-1), (0,0,0) ),
                    ( (1,1), (0,0,0) ),
                    ( (0,0), (255,255,255) )]:
                pen.text( (objrec.x +self.xoffs +offset[0] +self.labeloffs[0],
                    objrec.y +self.yoffs +offset[1] +self.labeloffs[0]),
                    text, font=markupfont, fill=colour)

Then I just need to actually use hidelabel and labelpref members for some good. Hidelabel is just to remove the clutter of useless information, but labelpref is to ADD info. Specifically, to give the label numbers more context. I'm going to use this for the doorways first:

Code: [Select]

self.addsprite(61, 0, sprite(graphics.records[30].images[19])) # Out Door
self.addsprite(62, 0, sprite(graphics.records[30].images[19], labelpref='To ')) # In Door

And the main ones to hide:

Code: [Select]

for sprtype in [17, 63]:
    for subtype in range(-1, 11):
        self.addsprite(sprtype, subtype, sprite(graphics.records[30].images[19],
            hidelabel=True))

Now I'll just go ahead and hide anything else that looks like it needs it.

And done. However, there are two cases that need to be fixed, and I think they will both require pre-processing. Case 1 is that all locked doors appear to have switch triggers. We don't want to display these, because the use of locked doors is fairly obvious. In order to remove this ONE use of triggers, we will need to first find all locked doors, then erase the info value for any triggers that match.

The second update is very minor, but on stage 5, it appears that two triggers are on the same tile. The pre-processing will simply need to move one down (or up) about 8 pixels.

Code: [Select]

def preprocessmap(self, mapdata):
    switchlocations = []
    doorinfos = []

    # First loop: find all door info values and move doubled up sprites.
    for objrec in mapdata.sprites:
        if objrec.sprtype == 12:
            while (objrec.x + objrec.y*128*16) in switchlocations:
                objrec.y += 8
            switchlocations.append(objrec.x + objrec.y*128*16)

        if objrec.sprtype == 9:
            doorinfos.append(objrec.info)

    # Second loop: Erase switches that align with doors
    for objrec in mapdata.sprites:
        if objrec.sprtype == 12 and objrec.info in doorinfos:
            objrec.info = 0

Not too hard. Now I just take out the debug sprite ID, and the labels are good to go.



My convention is to add "To" for a Doorway transition, "TR" for a pickup trigger, and "SW" for a toggle switch. Let me know if anything is unclear or you have any suggestions. I thought of adding "W" for the toggle walls, but I think they should be okay as-is.


Monster facing is next. Let's start with Stage 3, which obviously has monsters facing both directions. If we filter by monster 55, we get:

Code: [Select]

55  880     400 2   0   32  48  0   0   0   0   0   0   0   0   0
55  560     304 2   0   32  48  0   0   0   0   0   0   0   0   0
55  1440    336 2   0   32  48  0   0   0   0   0   0   0   0   0
55  400     944 -2  0   32  48  0   0   0   0   0   0   0   0   0
55  1904    960 3   0   32  48  0   0   0   0   0   0   0   0   0
55  160     864 3   0   32  48  0   0   0   0   0   0   0   0   0
55  560     864 -1  0   32  48  0   0   0   0   0   0   0   0   0
55  1232    400 -4  0   32  48  0   0   0   0   0   0   0   0   0
55  1712    928 -1  0   32  48  0   0   0   0   0   0   0   0   0
55  304     336 1   0   32  48  0   0   0   0   0   0   0   0   0

So, I don't see a left-right, but I do see a positive-negative correlation. The actual number probably specifies the exact initial sprite, which we will likely have a very hard time guessing. I'm going to assume positive is left, and negative is right. I'm also going to use a different frame of animation for each ID just for the hell of it.

Code: [Select]

# Monsters:
# Brute
self.addsprite(55, 0, variablesprite({
    -4 : graphics.records[61].images[4],
    -3 : graphics.records[61].images[5],
    -2 : graphics.records[61].images[6],
    -1 : graphics.records[61].images[7],
    0 : graphics.records[61].images[8],
    1 : graphics.records[61].images[9],
    2 : graphics.records[61].images[10],
    3 : graphics.records[61].images[11],
    4 : graphics.records[61].images[12],
    } ))

But it's not correct. Compare the below screenshot to the STORY screen:


Well, STORY is easy, because the sprites don't move, so I'll pick exact matches for them. Then I'll go play through Stage 3 again and see if I can figure out the initial facing of each monster.

Unfortunately, that's rather difficult to do. Here's the best I was able to come up with:

Code: [Select]

self.addsprite(55, 0, variablesprite({
    -4 : graphics.records[61].images[12],
    -3 : graphics.records[61].images[11],
    -2 : graphics.records[61].images[10],
    -1 : graphics.records[61].images[9],
    0 : graphics.records[61].images[8],
    1 : graphics.records[61].images[0],
    2 : graphics.records[61].images[1],
    3 : graphics.records[61].images[2],
    4 : graphics.records[61].images[3]
    } ))

Let's go ahead and do some other monsters. I'll leave the centipede as-is, because the only known instances all face in the same direction. Most other monsters appear to just use 0 and 2.

That's done. Still not 100% convinced, but it adds variety, and it's better than we had before.


Now strings. We're going to need more information for this, so let's add another output to the debug mode for xargonmap.py.

Code: [Select]

# Standalone debug string list:
with open(self.name + '_strings.csv', 'wb') as csvfile:
    writer = csv.writer(csvfile)
    for stringnum, lookupval in enumerate(self.stringlookup):
        writer.writerow([stringnum, lookupval, self.strings[stringnum]])

With that listing in place, I will try to identify all the discrepancies by comparing screenshots against my map output. For each discrepancy, to the right of the string that "actually appears", I will put the string that "should be there", as well as the index in the string array for that string. Without a better idea, I think I'm just going to have to correct the alignment discrepancy in the lookup array.

With that in place, I can put together a simple set of corrections to fix the alignment. Most of them are shifted by 4, so we only need to take into account the entries OTHER than the ones that shift by 4. Generally, this just means moving a few of the page number entries to different places in the list.

Code: [Select]

# String adjust for STORY map:
if mapdata.name.upper() == 'STORY':
    page3to5 = mapdata.stringlookup[117:120]
    page6 = mapdata.stringlookup[82]
    page7 = mapdata.stringlookup[81]
    page8 = mapdata.stringlookup[84]
    page9 = mapdata.stringlookup[83]
    page10 = mapdata.stringlookup[116]

    del mapdata.stringlookup[116:120]
    del mapdata.stringlookup[81:85]

    mapdata.stringlookup[81:81] = page3to5 + [page6, page7, page8, page9, page10]

I confused myself a little bit figuring out exactly which value I was moving (original string ID vs string position in array). All is good now, however. This means that Episode 1 maps are complete!

day18.zip is available. Episode 1 maps are also on my website, and submitted to VGMaps. Tomorrow I will start Episode 2.

[Zerker]

Hello again. Today is the day we make the mapper script work with Episodes 2 and 3. The first thing I noticed when firing up the other two episodes is that each one has a different colour palette:



And I suspect the Graphics file will have slightly different contents. Before doing anything else, I'm going to run my Graphics script on the other GRAPHICS files with the episode 1 palette. I should be able to compare the results to see what's different.

And Beyond Compare tells me there's very little that's actually different. A few missing sprites (notably the story scenes) and a few extra. Nothing that should totally break things, but the spritedb will need to be populated based on the episode number. We can't have it populating sprites that don't exist.

Also, there's the nature of the alternate palettes per episode. So let's get the files upgraded to detect episode number from the file extension. To do this, we add this line to xargonmap.py:

Code: [Select]

self.epnum = int(tempext[-1])
and to xargongraphics.py:

Code: [Select]

self.epnum = int(filename[-1])
...
if self.epnum == 2:
    self.activepal = 6
elif self.epnum == 3:
    self.activepal = 7
else:
    self.activepal = 0
And for debug:

Code: [Select]

xargonimages.save('Episode{}Images'.format(xargonimages.epnum))
xargonimages.save('Episode{}OriginalImages'.format(xargonimages.epnum), masked=False)

Let's update xargonmapper.py to tie this together. I'll also clean up the Episode 1 palette choice, since more stages use the "DARK" palette than otherwise:

Code: [Select]

if self.epnum == 2:
    # Episode 2
    graphics.changepalette(6)
elif self.epnum == 3:
    # Episode 3
    graphics.changepalette(7)
else:
    # Episode 1
    if self.name.upper() in ['BOARD_01', 'BOARD_02', 'BOARD_04']:
        graphics.changepalette(0)
    elif self.name.upper() == 'DEMO3':
        graphics.changepalette(2)
    elif self.name.upper() == 'BOARD_05':
        graphics.changepalette(4)
    elif self.name.upper() in ['BOARD_08', 'BOARD_33']:
        graphics.changepalette(5)
    else:
        graphics.changepalette(1)

sprites = spritedb(graphics, mapdata.epnum)

And save to a subfolder:

Code: [Select]

def save(self):
    epfolder = 'Episode{}'.format(self.epnum)
    createpath(epfolder)
    self.mappicture.save(os.path.join(epfolder, self.name + '.png'))

And go by my sprite comparison to see which sprites are not in Episode 2, which looks like 56, 57 and 62. 62 I expect is in Episode 3, though.

Code: [Select]

# Skull Slug!
if epnum != 2:
    self.addsprite(75, 0, variablesprite({
        -1 : graphics.records[62].images[2],
        0 : graphics.records[62].images[0],
        1 : graphics.records[62].images[5],
        2 : graphics.records[62].images[3]
        }, hidelabel=True ))

...

# Story Scenes:
if epnum == 1:
    for subtype in range(24):
        self.addsprite(85, subtype, sprite(graphics.records[56].images[subtype]))
        self.addsprite(86, subtype, sprite(graphics.records[57].images[subtype]))


Then we run it and find out what breaks...


On the whole, it works. There are a few monster sprites with unmatched direction indices. Let me just fill those in now, run it again to confirm, then run it on Episode 3. Episode 3 has a couple more indices to fill in, including new hidden platform types. I will add those as debug images until I can see them in-game.

Code: [Select]

# Variant of Compound and semi-transparent for hidden platform(s)
self.addsprite(11, 0, variablesprite({
    2: graphics.debugimage(11, 'T2', 32, 16),
    4: graphics.debugimage(11, 'T4', 32, 16),
    6: graphics.semitransparent(
       graphics.compositeimage((32, 16), [(0, 0, 25, 14),
       (16, 0, 25, 15)]), 128),
    7: graphics.semitransparent(
       graphics.compositeimage((32, 16), [(0, 0, 51, 10),
       (16, 0, 51, 11)]), 128) }
     ))

Same goes for the new spawner varient (sprite 73).

Well, that's all done. Now we just need to identify new sprites to get the Episode 2 maps up to code. But first, I'm going to make one fix to the Episode 1 maps based on what I've seen of Episodes 2 and 3: Trigger Number of -1. This doesn't ever appear to be linked to anything and doesn't have any specific use for a map. So I will exclude it and only draw positive numbers:

Code: [Select]

if objrec.info > 0 and objrec.info < 90 and not self.hidelabel:

With that, a new palette, and a couple new sprites, Episode 2, Stage 1 is complete:


Oh, btw, the world map was complete before we even started (yay for reuse), but DarkWolf has already submitted the map to the site.

Stage 2 was almost done for me, and only needed a single new treasure box type. That said, one of the doorway labels is obscured by a spider, so I'm going to add doorways and triggers to the "text" list so they are drawn on top.

Code: [Select]

self.text = [obj for obj in self.objs if obj.sprtype in [6, 7, 12, 61, 62] ]
self.sprites = [obj for obj in self.objs if obj.sprtype not in [6, 7, 12, 61, 62] ]

Since switches moved to the "text" list, we also need to adjust our post-processing routines:

Code: [Select]

# First loop: find all door info values
for objrec in mapdata.sprites:
    if objrec.sprtype == 9:
        doorinfos.append(objrec.info)

# Second loop: Erase switches that align with doors and move doubled up sprites.
for objrec in mapdata.text:
    if objrec.sprtype == 12:
        while (objrec.x + objrec.y*128*16) in switchlocations:
            objrec.y += 8
        switchlocations.append(objrec.x + objrec.y*128*16)
        if objrec.info in doorinfos:
            objrec.info = 0

And here's stage 2, illustrating the tricky asymmetric doorways .


Stage 3 introduces two new enemies: evil purple bunnies and mini dinosaur things. Let's add them both to the sprite DB and continue. I'm also going to pick the jumping animation for the bunny because they are SMALL otherwise.

Code: [Select]

# Mini Dino
if epnum != 1:
    self.addsprite(58, 0, variablesprite({
        -1 : graphics.records[56].images[5],
        0 : graphics.records[56].images[4],
        1 : graphics.records[56].images[1],
        2 : graphics.records[56].images[0],
        } ))
...
# Evil Bunny
if epnum != 1:
    self.addsprite(70, 0, variablesprite({
        0 : graphics.records[63].images[4],
        2 : graphics.records[63].images[1],
        } ))

Stage 3 is ALMOST done now. I realize there's a minor glitch, that didn't manifest in Episode 1:


We need to make sure to use the tile mask data, and draw the map with the proper background colour (index 250 in the palette). This should be fairly simple. First, the initial map image should start from this colour, then we need to paste in tiles using their mask.

Code: [Select]

self.mappicture = Image.new("RGB", (128*16, 64*16), graphics.getcolour(250) )

...

tileimg = tiledata.gettile(graphics, tileval)
self.mappicture.paste(tileimg, (x*16, y*16), tileimg )

Better now:


I could keep going, but I think that's enough for today.

day19.zip is available.

[Zerker]

Good evening. From here on our things will likely be pretty straightforward. Stage 4 just has one new sprite to identify, which appears to be a simple bat enemy. Yes, another small hard-to-see enemy. See if you can find them in the image below:



Stage 5 has one new enemy (some sort of goopy creature) and a new palette:



Stage 6 has a new trap and a new treasure box type (containing a red key no less).



The next two stages I did out of order. Stage 8 just has one new treasure box (yellow key):


And Stage 7 I will get to tomorrow. day20.zip is available.

[Zerker]

Hello folks. Time for more maps. Things are getting easier and easier as we go along. Soon everything will be done for us before we even start.

But not stage 7. It has ONE identified sprite, which is just the Eagle character again. And a new palette.



Stage 9 has a few new things. We've got another illusionary wall, more spikey trap guys, a blue key treasure box, and a variant of the bouncy ball trap. The latter I need to switch over to using varients so I can update this accordingly.

Code: [Select]

# Bouncing Balls:
for i in range(2):
    self.addsprite(46, i, variablesprite({
        0 : graphics.records[51].images[4],
        3 : graphics.records[51].images[7]},
        field='info', hidelabel=True))

Oh, and I may as well fill in the Green Key treasure box, since all four keys appear to be in order.



I'm playing stages out-of-order again. Stage 11 has an alternate palette and one new ID for the same spike spear we've already seen. I also realize that I screwed up the shrimp monster, which also appears in Episode 1. I've re-uploaded the fixed copy to my site and get it corrected here after I finish Episode 2 (to make sure we didn't miss anything else).



And Stage 10 just has a different palette, an epic disk in a treasure box and ANOTHER version of the eagle sprite. I think I'm just going to assume all his variants are the same and populate via a loop:

Code: [Select]

# Silvertongue
for i in range (25):
    self.addsprite(23, i, sprite(graphics.records[45].images[1]))



day21.zip is available.

[Zerker]

Another day, more maps. Today, I play Stage 12 only to discover it's already fully mapped:



Same with stage 13:


And 14:


And 15 (once I selected the correct palette):


See what I mean about things becoming easier? Now, there's one new sprite in the reactor level (stage 32), and that's for the larger reactor:


However, there's also misaligned text again for the ending sequence, so I'm going to need to do the same trick as with the Episode 1 Story scene:

Code: [Select]

# String adjust for Episode 2 Ending:
if mapdata.name.upper() == 'BOARD_32' and mapdata.epnum == 2:
    blank = mapdata.stringlookup[-1]

    del mapdata.stringlookup[-1]
    mapdata.stringlookup.insert(8, blank)

And with that, Episode 2 is complete! I'll post it on my site tomorrow.

day22.zip is available.

[Zerker]

Episode 2 is now posted on my web site, and submitted to VGMaps. I also submitted the correction to Episode 1.

Episode 3 time. The first task we need to do is fix THIS error when generating the Episode 3 maps:

Code: [Select]

Generating Map 'BOARD_01'
Traceback (most recent call last):
  File "xargonmapper.py", line 139, in <module>
    mapper = xargonmapper(xargonimages, tiledata, themap)
  File "xargonmapper.py", line 66, in __init__
    sprites = spritedb(graphics, mapdata.epnum)
  File "/data/Projects/Xargon/spritedb.py", line 247, in __init__
    self.addsprite(56, 0, sprite(graphics.records[46].images[2]))
IndexError: list index out of range

Looks to me like that slime creature is an Episode 2 exclusive. So let's change that:

Code: [Select]

if epnum == 2:
    # Goo Monster
    self.addsprite(56, 0, sprite(graphics.records[46].images[2]))

Then it looks like we just need to add keys 1 and 2 for the bouncing ball trap. I will use debug images for now until we ID them:

Code: [Select]

# Bouncing Balls:
for i in range(2):
    self.addsprite(46, i, variablesprite({
        0 : graphics.records[51].images[4],
        1 : graphics.debugimage(46, 'T1', 32, 16),
        2 : graphics.debugimage(46, 'T2', 32, 16),
        3 : graphics.records[51].images[7]},
        field='info', hidelabel=True))

After that, it looks like the first three stages are all identified, excluding their palettes. Let's add a few more entries to our palette list and finish those up:





And that's all for today. day23.zip is available.

[Zerker]

Good evening. There's not a whole lot left, so this is pretty much turning into a daily progress report until I finish. Almost all of the interesting stuff is already done.

Today, I start with Stage 4, which has one new sprite in the ceiling. It appears to be a ceiling switch of some sort:

Also, I died far too many times in that damn lava pit.


And I'm out of order again. Stage 6 appears to have some timers for periodically triggered events, so I will draw them by adding a prefix to their label number and make them otherwise invisible:

Code: [Select]

# Timers:
for i in [30, 40, 50, 60]:
    self.addsprite(73, i, sprite(graphics.records[30].images[19],
        labelpref="Timer ", labeloffs = (-4, 4)) )



Stages 5, 7 and 8 are fully identified. I just needed to find the right palette, and voila:





day24.zip is available.

[Zerker]

Good Morning. Time for a bit more Episode 3. Stage 9 sees the return of those cloak guys from the last level of Episode 1, be it with a different number:


Stage 10 was fully mapped, but it also proved the value of the doorway identifiers when I was playing it:


Stage 11 has a few new sprites: some snake-like creature, a new pickup gem, a new variant of the bouncing balls, high-jump boots in a treasure box, and a new invisible platform. And a different palette. Nothing we haven't already done.



Stage 13 is the demo stage, and just needs a new palette to finish it off:


Stage 12 just needed us to select the correct palette and...


Augh! No! Wrong! Guess the palette isn't *quite* identical to the earlier stage. Let's use a screenshot from this stage directly:


Better.

Next up is Xargon's Castle, i.e. the last set of levels. I will do that tomorrow. However, I can at least fill in the correct sprites for the map image:


Some assembly required? Let's see if we can't get the alignment on some of these sprites fixed up. After a few attempts, and some careful examination of a screenshot, here's what I came up with:

Code: [Select]

# Xargon's castle:
if epnum == 3:
    self.addsprite(88, 7,  sprite(graphics.records[47].images[25], yoffs=6, xoffs=4))
    self.addsprite(88, 8,  sprite(graphics.records[47].images[26], yoffs=6, xoffs=10))
    self.addsprite(88, 9,  sprite(graphics.records[47].images[27]))
    self.addsprite(88, 10, sprite(graphics.records[47].images[28], xoffs=4))
    self.addsprite(88, 11, sprite(graphics.records[47].images[29], xoffs=10))
    self.addsprite(88, 12, sprite(graphics.records[47].images[30]))



day25.zip is available.

[Zerker]

Hello again. Today we finish Episode 3 of Xargon, and thus the entire game. The first thing I need to do is play the last few levels with an eye to what is missing in the maps, and take screenshots of each so I don't forget. Since I can't save between levels, I'm going to just finish the game in one go then go back and update the maps.

And I finished the game. Now, Stage 14 is the castle stage, and it actually only has one missing sprite: another toggle platform. Let me add it now. Since it has a shadow when it appears (even on the right side), I'm going to create it as a 48 x 32 image and also include the shadow:

Code: [Select]

4: graphics.semitransparent(
   graphics.compositeimage((48, 32), [(0, 0, 11, 1),
   (16, 0, 11, 1), (0, 16, 11, 2), (16, 16, 11, 2),
   (32, 0, 11, 19), (32, 16, 11, 19)]), 128),



However, there appears to be a hidden rapid fire over one of the flame spitters, which as far as I can tell, is not there. This looks like the same issue as the ice cream cone thing. I suspect the "mapping" based on the variant field has priority over the subtype when this behaves as an ordinary hidden item. Since this is the only alternate case I have come across, I will simply add it as a special entry in the sprite DB:

Code: [Select]

for i in range(2):
    self.addsprite(73, i, variablesprite({
        0 : graphics.records[30].images[19],
        1 : graphics.records[30].images[19],
        2 : graphics.records[30].images[19],
        3 : graphics.compositeimage((32, 32), [(0, 0, 59, 1),
           (16, 0, 59, 4), (8, 12, 59, 1)]),
        4 : graphics.semitransparent(
            graphics.records[37].images[i], 128),
        ...

Stage 15 only has one new type of bouncing ball, after which it is complete:



The boss & ending stage, Stage 32, has one new sprite for a slug spawner, so we'll clump a couple slugs together for this (also, we need to account for the fact that slugs aren't in Episode 2):

Code: [Select]

if epnum != 2:
    slugspawner = graphics.compositeimage((32, 14),
            [(2, 0, 62, 2), (-3, 0, 62, 0)])
else:
    slugspawner = graphics.records[30].images[19]

...

5 : slugspawner,

Incidentally, Xargon was already identified from the Episode 1 ending. However, the ending picture doesn't show up at all!



Let's look into this further; Going through the entire set of sprites for this map, I can't see anything that looks like it would trigger the ending image. Everything is fairly well identified. The last column in the unidentified header region has the number 143, which is unique. I can only theorize that this must be some special function to draw the ending image. Therefore, we will just have to do it the hard way.

First thing, I'm going to take advantage of my composite sprite code to make up a fake sprite that does not appear in the game to contain the full image. Looking through the graphics output, it appears to be stored in record 57 as 100 16 x 16 pixel chunks (i.e. 10 x 10). Let's put this together:

Code: [Select]

# Fake sprite for the ending scene (which does not appear to have a sprite OR use Tiles):
tilelist = []
for x in range(10):
    for y in range(10):
        tilelist.append( (x*16, y*16, 57, x + 10*y) )
self.addsprite(1000, 0, sprite(graphics.compositeimage((160, 160), tilelist)))

Then we need to go into GIMP and find out the exact upper-left corner. With that determined, we can add this fake sprite to the end of the sprite list for Episode 3, Stage 32:

Code: [Select]

# Fake Sprite for Episode 3 Ending:
if mapdata.name.upper() == 'BOARD_32' and mapdata.epnum == 3:
    mapdata.sprites.append(objrecord( (1000, 48, 240, 0, 0, 160, 160,
        0, 0, 0, 0, 0, 0, 0, 0) ))

That worked:


And that's it. The complete game is mapped. I have posted the Episode 3 maps on my web site and submitted them to VGMaps. The only thing remaining (for me) is to do a bit more code cleanup and documentation before I release the tool "officially".

day26.zip is available if you want to see the complete tool "as-is".

[Zerker]

Hey guys. For the next couple days, I'm just going to be adding better comments and interface documentation to the code, as well as writing usage instructions. I won't be posting a daily log of this process, but I will post when everything is complete and on my web site.

If anyone has any questions, they are of course quite welcome.

EDIT: That actually didn't take long. It's released on my web site now.