Topic: Umihara Kawase [COMPLETE]

[Raccoon Sam]

Well, I was going to test adding sprites with my free scroll patch, but after laying the screenshots over the map to get the positions and getting ready to paste the real sprites on them, I realized that there are no rips of Umihara Kawase sprites, excluding the player sprite! I can't rip them manually myself because the current SNES9x emulates the game incorrectly, and I honestly don't like ZSNES.

So, I though, why not stab two flies in one hit and get rips from each sprite in the game, for the benefit of the collective and to apply fresh clean sprites onto the maps? Here's my plan:

Umihara Kawase Shun: Second Edition Kanzenban for the DS is essentially a 3-in-1 game that also includes the SNES Umihara Kawase game. Since the DS has a filesystem of its own, by decompressing all of the game's files you can access them as if they were real folders and files in Windows Explorer/Finder.
This is particularly exciting because all of the files in the 'sfc' folder are actually the files the SNES game actually uses. You can, for example, search any of the files in the 'sfc/map/' from the original SNES ROM and find the level data. Some data has been re-encoded losslessly, but it's pretty much the same data.
So, instead of searching for the graphics individually in one giant chunk of a file (the SNES ROM), I looked into the 'sfc/obj/' folder and found files of each sprite set in the game. The files also contain palette and tilemap info, so one program that does exactly what I've explained below could easily spew out every single sprite in the game!

.BIN file structure:
All BINs have a $18 byte header that defines eight things:
1. Where the block array begins [BLKADDR, 4 bytes]
2. Where the sprite definition begins [SPRADDR, 4 bytes]
3. Where the palette info begins [PALADDR, 4 bytes]
4. Where the graphics data begins [GFXADDR, 4 bytes]
5. Amount of blocks [BLKAM, 2 bytes]
6. Amount of sprites [SPRAM, 2 bytes]
7. Amount of palettes [PALAM, 2 bytes]
8. Amount of tiles in total [TOTAL, 2 bytes]

Let's take U3_KNG.bin, which is the Goldfish enemy's sprite file. First $18 bytes with appropriate annotations are below:

addr.+------------------------ 
     | BLKADDR____ SPRADDR____
$000 | 18 00 00 00 88 00 00 00
     +
     | PALADDR____ GFXADDR____
$008 | A0 00 00 00 C0 00 00 00
     + 
     | BLKAM SPRAM PALAM TOTAL
$010 | 0E 00 05 00 01 00 40 00
-----+------------------------

• Let's dive into the first thing, BLKADDR. Sprites are built from blocks and one block is 8 bytes. This is the address where the 8-byte block array begins. The program reads the bytes backwards so it gets the correct address, which is 0x00000018.
• The second thing, SPRADDR is the offset where the program starts reading the info of what blocks to use to build a sprite. This time it's at 0x00000088.
• Third thing, PALADDR is self-explanatory. Where the Palette data begins. In this case, from 0x000000A0. Palettes are read the same way as the SNES ones I explained earlier in this thread, 1555 BGR.
• Fourth thing, GFXADDR is also pretty obvious; where the graphics data begins. This time 0x000000C0. Format is 4bpp linear, reverse order.

• Fifth thing, BLKAM is 0x000E. What this means is how many blocks there are in total. This time, 0x000E. Also, since one block is 8 bytes, you now know that the block data starts from 0x00000088 and is 112 bytes big (because BLKAM * 8 = $70).
• Sixth thing, SPRAM is 0x0005. What this means is exactly how many sprites are exported in the end. This time, 0x0005. It takes four bytes to define a sprite, so you now know that the sprite data begins from 0x00000088 and is 20 bytes big (because SPRAM * 4 = $14).
• Seventh thing, PALAM, is again pretty clear. Usually sprites have one or two palettes, but in this case it's 0x0001. One palette is 32, or, $20 bytes. There's one palette this time so obviously it takes only 32 bytes as is.
• The eighth thing serves little purpose. It just tells us how many tiles there are in total. This time 64, or, $40.

So, after the header, we're in address 0x018. This is where the blocks are listed, I've written them down for easy readability and added annotations:

addr.+-xx XX yy YY tt pp BP UU-+--NR--. 
$018 | F3 FF F6 FF 00 00 60 00 | 0000 |
$020 | F3 FF 06 00 08 00 50 00 | 0100 |
$028 | FA FF EE FF 18 00 00 00 | 0200 |
$030 | F2 FF F6 FF 0C 00 60 00 | 0300 |
$038 | F2 FF 06 00 14 00 50 00 | 0400 |
$040 | FA FF EE FF 25 00 00 00 | 0500 |
$048 | F2 FF F6 FF 19 00 60 00 | 0600 |
$050 | F2 FF 06 00 21 00 50 00 | 0700 |
$058 | FA FF EE FF 32 00 00 00 | 0800 |
$060 | F2 FF F6 FF 26 00 60 00 | 0900 |
$068 | F2 FF 06 00 2E 00 50 00 | 0A00 |
$070 | FA FF EE FF 3F 00 00 00 | 0B00 |
$078 | F2 FF F6 FF 33 00 60 00 | 0C00 |
$080 | F2 FF 06 00 3B 00 50 00 | 0D00 |
-----+-------------------------+------'

I'll explain each of the eight bytes here, using the first block an example.
xx XX = X position of block, two bytes, read backwards, so in this case it would be $FFF3. See this grid for reference.
yy YY = Y position of block, two bytes, read backwards, so in this case it would be $FFF6.
tt pp = tile to use and its property^{â€Â} . In this case it would be tile 00 with property 00.
BP = Block type^{‡} and Palette to use for the block. The palette is always 0 so I don't see how this would be so useful.
UU = Unused.
NR = Block Number. Just for reference! This is not in the ROM, I just put it to the right of the table to help visualize what block we're in right now. You'll need to know this later. From the first block definition it's just counting upwards from 00 00, bytes backwards.

†: property byte is kind of funny. Of its 8 bits, only the first two are used. vhUU UUUU, where v means vertical flip of block's tiles and h horizontal flip, respectively (and U=unused).

‡: There are twelve block types. They essentially tell how the block is made, or, arrange a predetermined amount of tiles to a certain arrangement to form the block. See this for reference. $C_, $D_, $E_ and $F_ are unused.

Now, if the program would read the data, it would now have the information of how many blocks there are in total, how to form the correct blocks and where to put them. However, this would lead into all blocks being on top of each other, forming one sprite-cake. The next bytes, SPRADDR tells where to snip them apart, or, how many blocks build a sprite. Again, I've formatted and annotated the data for easy understanding.

addr.+ AMONT FROMB-. 
$088 | 02 00 00 00 |
$08C | 03 00 02 00 |
$090 | 03 00 05 00 |
$094 | 03 00 08 00 |
$098 | 03 00 0B 00 |
-----+-------------'

Because SPRAM is $0005, we know that this file defines only five sprites. It takes four bytes to define a sprite an again, these two-byte combos are read backwards.
AMONT = Amount of blocks this sprite is built from...
FROMB = ...when counting from this Block Number.

In this case, it would be like this:
• Sprite one is two blocks: 0000 and 0100.
• Sprite two is three blocks: 0200, 0300 and 0400.
• Sprite three is three blocks: 0500, 0600 and 0700.
• Sprite four is three blocks: 0800, 0900, 0A00.
• Sprite five is three blocks: 0B00, 0C00, 0D00

And that's it. Load next file and repeat. I did this informational GIF that should help understand what's happening in the sprite creation. If you have a chance, view it frame by frame. Basically it's just "Get X, get Y, get tile, check if flips, use correct block type"

Wow, big post, probably not as clear as I want it to be... I'm tired @_@. Also, apologies again to be that guy who can't do anything without a programmer. I just hope my general plan is acceptable. I can always fall back to ZSNES to get the sprites I need but I thought that this would be great too :G

Here's a link to all the BINs in a ZIP

But yeah, concentrating on getting all 57 ZSTs now. :3
DONE. Check 'em out, the raw files are ready for ogling here. I'm having problems with Dropbox sharing right now so just change the 00.zst to 36.zst or anything.zst to view the map.
Another interesting thing, there are 57 files in the MAP folder of the DS version, though there are only 48 accessible maps in the game. Unused levels, mayhaps?

[TerraEsperZ]

Even more impressive!

*sigh* I wish I had the ability to process this much information as easily as you do. Just thinking about memory offset and reading bytes backward and I already feel lost. It's not that I don't understand what you're doing, but it would take me a lot more time and concentration to achieve even a fraction of all this.

Thank you for sharing this with us in any case. We're not all going to become expert map and sprite rippers overnight but every bit helps.

[Raccoon Sam]

Thank you for your kind words. I don't consider myself an expert hacker though, as pretty much every hacking expedition I dwell into is pretty much trial and error. I screw things up on purpose, see what that changes and write my notes based on the observations. Though my documents prove useful, I think the real wizard here is mechaskrom. After all, without him, all we'd have is a nice recipe but no cake.

EDIT: Well hello there!

[mechaskrom]

The program works exactly as I intended! Using a mono command line tool on Mac for the first time proved no challenge either.

Wonderful! Both that the program and mono is working.

Reverse-engineering the sprite-format isn't that important, but could be interesting. But it seems to be a lot of work and I don't know 658c16 assembly unfortunately. So in the end it's probably not worth the trouble.


Getting the sprites from the DS-version seems to be a novel idea.
I started working on a program to extract the sprites, but the format is complicated so this turned out to be harder to make than the field tilemap extractor. I've got something working, but I'm very uncertain of how correct it is so I'll need more time to check the output of it and so.

Now a few questions to Raccoon Sam, if you don't mind:
-Spriteblocks always use palette 0 you say, so what's the point of PALAM if it can be higher then 1? I think I'm missing something here. Why can a bin-file have multiple palettes if only one is used for all spriteblocks?

-How do you want to specify the input to the program? One file or one folder with files? Running the program for every bin-file seems tedious so it perhaps is better to give it a folder?

-How do you want the output from the program?
-Save every sprite in a bin-file as a png-file?
-Save all sprites in a bin-file in one png-file (spritesheet)?
-Other? I toyed with animated GIFs, but .NET doesn't support that good enough to satisfy me (the output quality is horrible even for small low color images). But if you really want that I could perhaps look into it more.



And TerraEsperZ, don't worry. You have to be a genius to understand all of this instantaneously. My head hurt just looking att Raccoon Sam's wall-of-text post. But if you give it time and process it in small steps it will usually become clear.


FYI, a more correct term for "reading bytes backwards" is little-endian:
http://en.wikipedia.org/wiki/Endianness
and it's very common.


Don't be modest Raccoon Sam. I think you've done some excellent hacking with Umihara Kawase so far.
And don't underestimate trial and error. That's my modus operandi when I'm programming. Powerful stuff.

Though my documents prove useful, I think the real wizard here is mechaskrom. After all, without him, all we'd have is a nice recipe but no cake.

Thank you. Very kind. ....I like cake.


The image of every map is impressive. :O
But why is one field up side down? And some fields seems to be repeats of earlier levels. I haven't played Umihara Kawase so perhaps it should be so?


Finally, I've attached the spritesheet-output from the program of U3_KNG.bin
The dark cyan strips are just spacing between the sprites so you can easily tell them apart.
So the first sprite is 32x24 pixels big, the following 4 are 32x32.


I need some sleep now... 

[Raccoon Sam]

Wow! Looking great! Also, glad to see you're interested! Regarding your questions:

Spriteblocks always use palette 0 you say, so what's the point of PALAM if it can be higher then 1? I think I'm missing something here. Why can a bin-file have multiple palettes if only one is used for all spriteblocks?

Sorry, should've been more thorough there. For example, the snail's PALAM is 2. There are two kinds of snails in the game, blue and brown. The palettes in the BIN are for a blue snail and a brown snail. A sprite sheet should be exported PALAM times, one sheet for each palette.

How do you want to specify the input to the program? One file or one folder with files? Running the program for every bin-file seems tedious so it perhaps is better to give it a folder?

Well, the level exporter works one file at a time, too, so I guess that would be better. I made a batch script to handle all the ZSTs, so I guess doing so for the BINs would work too.

How do you want the output from the program?

Your attachment was perfect. The strip should be transparent in the final version though. I'd go for the 'all-sprites-in-one-PNG'-approach.

Also, yes, some of the later levels are upside-down copies of the first ones. Also one that's exactly the same, but with conveyors going the opposite direction.

Thank you for your kindness.

[mechaskrom]

Made som big progress with the sprite ripper and I'm pretty sure it's working correctly now. But because I dont know how the sprites should look like there could always be some errors left (although they look fine to me).

A sprite sheet should be exported PALAM times, one sheet for each palette.

Aha, that makes sense. Should be easy to fix.

Just to be clear. Do you want a transparent strip/spacing between sprites or no strip/spacing at all?
See EDIT.

Some more questions/remarks:
There are three things that are strange that I discovered when running the ripper on all 55 files.

The amount of sprites specified in the header (sprAm) is often smaller than the amount of sprites that fits between sprAddr and the following address (always(?) palAddr). Take U3_KNG.bin for example.
sprAddr=0x00000088=136
palAddr=0x000000A0=160
sprAm  =0x0005=5
bytes per sprite = 4
(160-136)/4=6
So there is space for 6 sprites but sprAm  = 5. I don't think the extra sprites depict anything, so I guess this doesn't matter anyway. I just thought it was a bit strange. All the other amounts specified in the header conforms to the calculated amounts for all files.


I don't think the sprite's block amount is 2 bytes in size. The sprite format seems to be more like this:

Code: [Select]

addr.+ AM UU FROMB-.
$088 | 02 00 00 00 |
$08C | 03 00 02 00 |
$090 | 03 00 05 00 |
$094 | 03 00 08 00 |
$098 | 03 00 0B 00 |
-----+-------------'
AM = amount, 1 byte
UU = unknown?, 1 byte
FROMB = from block, 2 bytes

If you treat amount as 2 bytes you get values (both signed and unsigned) that can't be right.

I don't know what the unknown byte does, but of all 55 files only the two MSB seems to be used. They could be some kind of flags? So the format of the unknown byte seems to be:
ff------
where
ff=flags?
-= never used?


And last (I treat amount as one byte here).
Block amount is = 0 for some sprites in two files (U3_SMOKE.bin and U3K0_PR0.bin).
What can this mean? Something to do with the flags in the unknown byte? The sprite is disabled?
For now, I treat this as a disabled sprite (8x8 blank/transparent) and that looks fine in the spritesheets at least.
But perhaps I'm wrong with amount = one byte?


I've uploaded the logs from the files that the sprite ripper doesn't like 100%.
And I also included the spritesheets from the two files with "disabled" blank sprites.
The logs are divided into 3 folders and hopefully the foldernames indicate what the ripper didn't like with them. Link:
http://www.mediafire.com/?gvz8nndxwf1gc86

Perhaps they could help you figure things out?


EDIT:
About the spritesheet output again. Isn't it problematic to set all spacings (non sprite pixels) to transparent? I used dark cyan because it makes it easy to see how big the sprites are, but perhaps you don't need to know that to use the sprites? I could always make it an command line option so you could chose between a transparent (default?) or a dark cyan background.

I checked all unused sprites (after sprAm) and they all have block amount = 0 and a size of 1x1, so they are "disabled". I'm just wondering why they are even there?

And before you made the last post I changed the program so it accepts both a single file or a folder.
I needed that to test all bin-files anyway.

[mechaskrom]

The sprite ripper is ready for testing now. Check your messages.
Let me know if you find any errors/bugs.

One thing I noticed. There are no sprites for the player (Kawase?) at least not in the files you sent me. Where is she?

And finally....
You gotta love the sprites in this game. Fishes with legs? The attached sprite always cracks me up.
Ah, good old games. You rarely see this in modern games.

[Raccoon Sam]

The player sprites are, coincidentally, stored in a different format far more easily readable/rippable than any other sprite! The player's sprites are also alone in a different folder.

But, let the testing begin! I honestly can't thank you enough. Good job! I'll keep you informed


EDIT: Well, I tried the ripper...


...and the output is 100% ACCURATE! Fantastic job, mechaskrom!! We're a dream team!  

Now, to dwell on the maps. Expect results... very soon :>


EDIT 2: Hrm. Well, I've got a good framework on the maps, the sprite/starting positions are done, water level has been marked, but I can't help but to ponder—how should the final output be presented?

• Static, unanimated PNG maps with markings like which way the conveyors go, how the platforms behave, how the water level acts (static or tide). Problem: no animation.
• Animated GIF maps with animated conveyors and platforms. Problem: no proper rendering of water and possible enemy/sprite markings.
• HTML pages of maps with several layers: an unmarked PNG with just the level on the back and an animated GIF consisting of just the sprites/starting position/conveyors on the front. I could animate the moving platforms too, and clicking a door could link to the corresponding field. Problem: Would require a decent browser and would be possibly heavy.

[mechaskrom]

I would choose static unanimated png maps.  They are easy to view and are probably what most people want as a map.

But if you have the time and energy it would of course be great to also have animated versions of the maps. I don't know what is the best way to do animated maps though. Besides GIF and HTML, perhaps flash or lossless video? But maybe that could cause a problem with big file sizes?

[Raccoon Sam]

You're exactly right. Also, GIFs would be good for only small animated elements like the conveyors. I wouldn't go for flash nor video either.

But yeah, I've done much progress now, and have about 90% done maps of 11/48 fields. For all sprites with a moving property (moving platforms, rising/falling towers) I've set an alpha of 50%. I also include small textual tips where necessary (which way a conveyor/platform is going, where a door leads to).

Nothing to show though, but here's a demo of field 22

[Raccoon Sam]

Well guys, I'm happy to announce that all maps are done. Sprite placement 100% accurate and guide notes have been added. This is also my first complete mapping project.
In the end, I chose not to go with animated maps, but instead toned down the opacity on some sprites and added some arrows here and there. All in all:
• If a platform has a small arrow in it, it goes continuously to that direction.
• If a platform has a big arrow in it, it stays still until grabbed with the hook, pulling it down.
• If a sprite has a 50% opacity, it means that it is a protruding/receding "pushy" sprite. An arrow next to it denotes its direction.
• Arrows, in general, explain which way conveyors go. Also, see above.
• If the level has a tide, I've set it to its highest point in the map.










If these get added to the site, I feel that mechaskrom must also be added to the 'Author' section as his help was overwhelmingly significant.


here's the title card

[mechaskrom]

I think the maps look great and that you choose a simple and good way to handle "animated" graphics in static maps. Great work and congratulations to your first complete project.

And thank you for considering me as a co-author, it's a real honor. Though I think my part in this project was pretty insignificant, but if you insist.

[JonLeung]

Looks great!  Is this an official submission?

You guys know I prefer being emailed about this if so, rather than being expected to always catch them on the forums...

[Raccoon Sam]

Looks great!  Is this an official submission?

You guys know I prefer being emailed about this if so, rather than being expected to always catch them on the forums...

Thanks! And yes, this is an official submission. I also e-mailed you these earlier today. :3

[TerraEsperZ]

It's too bad the backgrounds are ugly digitized pictures that don't repeat well because the finished maps look kind of empty without them.

Now all you're missing is a "road map" showing which door in each stage leads to which stage under what circumstances, a puzzle in itself judging from the walkthrough I read a few years back.