Page 1 of 3

.hack MAD - Aimed at developers.

Posted: Sun Dec 06, 2009 2:45 pm
by Coldbird
After asking Kuukai whether it would be okay to post about it - I'm posting information about my latest invention in here.
Some people might still remember my .hack//fragment localization project.

Since then I've been spending my time wondering just what would be the best legal way for developers to build a authentic .hack MMO.

And I've figured out a proper way to do it.


To those that don't know the word MAD - it means Media Access Driver, its a layer providing a useable API for accessing third party devices, usually some kind of disc media, or similiar...

This time around, the .hack discs, be it INFECTION up to FRAGMENT or GU 1 to 3, will be the media in use.
And to access the archives properly, my MAD will do the dirty work.

I've spent quite some time reversing the CCSF format in use on all .hack games so far, I figured out its using a database like storage system for its data, splitting a file into multiple objects, and those objects into multiple variable sized sectors.

The library, as of the time of writing, isn't complete yet, but is capable of resolving all those referential links in the data storage of the games, which equals about 60% of the work necessary to access the files on disc in a useable way for game developers.

The reason I'm posting this topic early, even though I'm still working on this library, is because I'm searching for someone to step in my footprints, to take this baby all the way in my stead once this library is complete, to make a full The World MMO - based on the simple system of inserting the ps2 game disc into your computer and being able to play.

And before the flaming starts because I'm not posting code yet, let me say some last words, Cyberconnects users, aka. Joseph Stacko Slaves - I won't accept them as my heirs.

EDIT - Because the general oppinion on dothackers seems to be "not real until someone shows proof" - I'm providing some of the information I've gathered so far.

Code: Select all

All .hack games released so far (.hack INFECTION, MUTATION, OUTBREAK, QUARANTINE, FRAGMENT and GU 1-3) are using a storagesystem made by Crysoft - a japanese company that designs all kind of media tools.

The idea behind this storagesystem is minimizing loading times by effectively using physical properties of DVD discs...
(Outer Cylinders reading faster than Inner Cylinders, effective file splitting for loading only necessary components, etc.)

I can't fully make out what the short-name of the filesystem means - but it seems to be CCSF.

Each CCSF File Storage (About 7800 of those in .hack FRAGMENT, about 15000 in .hack GU 3) contains a variable number of files, mostly .max 3D scenes + animations aswell as materials, and is compressed using the opensource deflate algorithm provided by the zlib library.

After proper decompression the storage is made up of different blocks, each CCSF storage has atleast 3 blocks, 2 header and 1 data block.

The Header Block has the opcode 0xCCCC0001 and contains nothing of real interest, besides the CCSF magic code that identifies the file as CCSF storage.

The second block contains the file and objectlist and decides which objects are linked to which file, it's opcode is 0xCCCC0002 and contains information like "number of files" aswell as "number of objects" and the referential integrity, similiar to a database system.

Each block carries a size field, that contains the blocksize divided by 4 - ored with some binary flags that give further information about the blocks contents.

Each block, except the two information blocks at the beginning carry a reference field, which in turn indicates to which object the block belongs, again - several flags are ored into this field aswell.

So the overall way the system works is like this...
testcharacter.max (file)
  |--arm_left (object)
      |-- block 1 (data block)
      |-- block 2 (data block)
      |-- block 3 (data block)
  |--leg_left (object)
      |-- block 1 (data block)
      |-- block 2 (data block)
  ... 

Of course, the system is a bit more complicated than this display, but it makes the system and how it works understandable to those that have no idea of database systems.

There are many different types of data blocks, each containing different data, etc.
Each block is identified by its opcode, just like the two information blocks at the beginning of the file, each opcode is unique and used to identify the block contents, they all follow the same opcode schemata of 0xCCCCXXXX.

I hope this bit of information I've provided so far is proof I'm actually working on something real, and that the doubters finally get the idea of how much effort it is to fully reverse this structures and make proper use of the data inside.

For those interested in the case... 0xCCCC0300 = Color Table for Materials (Yes, they are indiced, every material in the .hack games has no more than 256 colors).

I hope to get some more work done on the MAD this weekend and hope to provide a opcode table for those interested soon.

Re: .hack MAD - Aimed at developers.

Posted: Mon Dec 07, 2009 3:12 am
by N3IWVC
It's most likely over my head because I'm still trying to get through school for this kind of stuff.
Though, either way I'd love to play with this library of yours and see what I can come up with.
I doubt I could lead this project but I'd like to be a part of it.

Re: .hack MAD - Aimed at developers.

Posted: Mon Dec 07, 2009 6:59 am
by Keyaki
So what your doing is trying to be able to make a .Hack MMO and play it by just inserting a .Hack disc into your computer?

Re: .hack MAD - Aimed at developers.

Posted: Mon Dec 07, 2009 8:26 pm
by Coldbird
Yes, just that I'm not going to code the actual game, just make the disc accessable to the game developers (which I'm still looking for).

This pretty much means... giving them access to 3D data on the disc aswell as textures, animations and similiar resources.

Also, as the game resources the developers would be using, are stored on the disc, only people having the proper disc could play them, making it a pretty legal approach to a "official" The World MMORPG.

Re: .hack MAD - Aimed at developers.

Posted: Wed Dec 09, 2009 7:46 pm
by K3nt0
It is good to read your name in the boards again Coldbird. Your name burns like an azure flame in the scene.

I am helping you to find a successor. Right now I am scanning the net for a list of potential communties that could contain such developers. I also will think about your project and how I could support it. I had a similar idea like i told you. I think it is the best project someone could start.

To interested developers: I also think it will be the best to make a general MMORPG engine that would just use the .hack media as backend database. That way we can also add any other game media and create limitless MMORPGs. For example using Naruto game media for a Naruto MMORPG but still we only need one engine. The other major advantage of such a general engine would be that not only .hack fans would be interested but also any other developer that wishes to create a MMORPG. While the whole .hack scene might contain 5 skilled people an open source MMORPG engine scene would contain hundreds of skilled people.

At first we do not even need a complete MMORPG engine. A simple 3D chat engine would be already great. Later we still can upgrade it to a full MMORPG engine. But the great advantage we get if we start with a simple chat is that it draws a lot attention in the game scene. This attention will be it what leads to new skilled developers. While it takes years to make a good MMORPG engine a simple 3D chat would take only months. And that means much more attention over time.

Re: .hack MAD - Aimed at developers.

Posted: Wed Dec 09, 2009 10:07 pm
by N3IWVC
Coldbird I was just wondering if you have a projected date for finalizing your library.
Also are you planning to release one that can be used with all the discs or will they be separate.
Heh one more thing, If they are separate which discs could we expect to be able to use first?

Re: .hack MAD - Aimed at developers.

Posted: Thu Dec 10, 2009 1:18 pm
by Coldbird
K3nt0 wrote:It is good to read your name in the boards again Coldbird. Your name burns like an azure flame in the scene.
Uhhh... thanks?
K3nt0 wrote:That way we can also add any other game media and create limitless MMORPGs. For example using Naruto game media for a Naruto MMORPG but still we only need one engine.
That's not really the point behind my .hack MAD is it...?
But as coincidence wants it... the Naruto Games made by CC2 Corp really use the same CCSF Storage .hack uses, so for NARUTO GAMES - this really would be possible, even if not intended...
N3IWVC wrote:Coldbird I was just wondering if you have a projected date for finalizing your library.
Not really. As I got student-life to handle aswell and of course my girlfriend.
But if you really want to know the progress and when I usually work on it, weekends - pretty much every weekend...
This weekend I will be cooperating for a little reversal session with my friend Saiks to get the CCSF Blockheader Fields reversed properly.

For those - really interested in seeing the progress I make though, I wouldnt mind providing test-binaries I built using the MAD library, those samples serve no real purpose, as it's just some testing code I write to debug the library, but afterall its just for proof of concept.
N3IWVC wrote:Also are you planning to release one that can be used with all the discs or will they be separate.
Heh one more thing, If they are separate which discs could we expect to be able to use first?
The code is universal - as all .hack games, so far, use the same CCSF storage format, so it will work just fine with all .hack discs.
Right now though, internally I use a .hack Fragment Disc to work on the MAD.

Re: .hack MAD - Aimed at developers.

Posted: Thu Dec 10, 2009 10:07 pm
by N3IWVC
This is all wonderful news, keep up the good work.

Re: .hack MAD - Aimed at developers.

Posted: Sat Dec 12, 2009 10:55 am
by azureeagle
Sweet. Can't wait to see how this'll turn out. My best wishes to you Coldbird.

Re: .hack MAD - Aimed at developers.

Posted: Sun Dec 20, 2009 8:13 pm
by Takai Tenshi
good luck. ill stay updated on this topic.

despite what flamers may say, i wanna see it succeed

Re: .hack MAD - Aimed at developers.

Posted: Mon Dec 21, 2009 1:21 pm
by Coldbird
Thanks to the believers so far, lately I've spent some time taking a closer look at the .hack link demo for PSP.
First off - I'm happy to announce that CC2 Corp is still using the Cri-Storage File System my MAD is ment to support...

Means the work I'm doing here will also work with .hack LINK. :)

Re: .hack MAD - Aimed at developers.

Posted: Mon Dec 21, 2009 7:07 pm
by Takai Tenshi
you said just put a ps2 disc into the PC disk drive, right?

that doesn't work though, right?

and it will work with link?

how would one go about running a umd in the PC?

Re: .hack MAD - Aimed at developers.

Posted: Mon Dec 21, 2009 8:08 pm
by TheSorrow
you said just put a ps2 disc into the PC disk drive, right?

that doesn't work though, right?
Well, putting a PS2 disc on a computer's CD/DVD drive works perfectly; it just reads it as general data and you get to see the files it has and all, like a computer would read any other disc, instead of actually playing it as a game as the respective console would (unless you play the CD on an emulator or something, which would indeed "emulate" the way the console reads the disc and plays it on the screen).

So, yeah, it does "work" and i've done it several times before, but what Coldbird is working on, at least from what i understand, is some sort of program (or API, an interface more specifically) that will read the files from the disc and automaticly does all the file accessing and needed emulation and all, and all the user will do is simply put in the disc and do a few clicks and some waiting. Just the way an emulator will do all the needed functions to play the game, this API will emulate and execute all the needed functions so you can play the game, with your only tool needed being the game's disc (and the program/API, of course)

But yeah, in a general view, putting a PS2 disc in a computer's disc drive works perfectly, just not the desired way as with using the MAD thingy...for now :P

Re: .hack MAD - Aimed at developers.

Posted: Mon Dec 21, 2009 8:25 pm
by Keyaki
Well, putting a PS2 disc on a computer's CD/DVD drive works perfectly; it just reads it as general data and you get to see the files it has and all, like a computer would read any other disc
Huh? I tried that with my Redemption disc and nothing liek that happened o.o

Re: .hack MAD - Aimed at developers.

Posted: Mon Dec 21, 2009 8:49 pm
by TheSorrow
Keyaki wrote:
Well, putting a PS2 disc on a computer's CD/DVD drive works perfectly; it just reads it as general data and you get to see the files it has and all, like a computer would read any other disc
Huh? I tried that with my Redemption disc and nothing liek that happened o.o
Tried putting in the disc and entered to the "D:" drive manually, instead of waiting for your computer to tell you something? Because obviously the computer won't find any files it can normally access to, like music or DVD video files and so it won't "invite" you to do any action like any other disc would do (after all, they're all "unknown files" for any normal OS like Windows or Ubuntu), but after all, it's just a storage device, just something that stores files and it isn't any different to any other CD/DVD, and whether the files are common or unknown for your computer, they're still files your computer can view and access to...of course, all you'll view is mostly garbage data, or raw HEX data depending of how you try to view the files, but you'll still view them :P

I've done that with all of GU's discs (which is how i found all the text information, since Rebirth...and got my high hopes by seeing all the skills and arena features which were NEVER used ;3;), with IMOQ, Final Fantasy IX, and Metal Gear Solid 3, and i think some others too, and they have all worked perfectly, in both my XP and Ubuntu (and also with COUNTLESS of PS1 games, like...over 100 XD), and have had no problems doing it. Sure, most of the time it tells me no message that a DVD is inside, and doesn't try to read much on it's own, but by simply going to the "D:" drive, i can see every file it has. There are other programs which help too, like ISO viewers, but that actually works for the cases when you have an ISO file (which is like a "virtual CD") and you want to view the files it has like you would by inserting the disc.

Re: .hack MAD - Aimed at developers.

Posted: Mon Dec 21, 2009 10:38 pm
by Keyaki
TheSorrow wrote:
Keyaki wrote:
Well, putting a PS2 disc on a computer's CD/DVD drive works perfectly; it just reads it as general data and you get to see the files it has and all, like a computer would read any other disc
Huh? I tried that with my Redemption disc and nothing liek that happened o.o
Tried putting in the disc and entered to the "D:" drive manually, instead of waiting for your computer to tell you something? Because obviously the computer won't find any files it can normally access to, like music or DVD video files and so it won't "invite" you to do any action like any other disc would do (after all, they're all "unknown files" for any normal OS like Windows or Ubuntu), but after all, it's just a storage device, just something that stores files and it isn't any different to any other CD/DVD, and whether the files are common or unknown for your computer, they're still files your computer can view and access to...of course, all you'll view is mostly garbage data, or raw HEX data depending of how you try to view the files, but you'll still view them :P

I've done that with all of GU's discs (which is how i found all the text information, since Rebirth...and got my high hopes by seeing all the skills and arena features which were NEVER used ;3;), with IMOQ, Final Fantasy IX, and Metal Gear Solid 3, and i think some others too, and they have all worked perfectly, in both my XP and Ubuntu (and also with COUNTLESS of PS1 games, like...over 100 XD), and have had no problems doing it. Sure, most of the time it tells me no message that a DVD is inside, and doesn't try to read much on it's own, but by simply going to the "D:" drive, i can see every file it has. There are other programs which help too, like ISO viewers, but that actually works for the cases when you have an ISO file (which is like a "virtual CD") and you want to view the files it has like you would by inserting the disc.

hmmmmm, nope still nothing:/

Re: .hack MAD - Aimed at developers.

Posted: Mon Dec 21, 2009 10:51 pm
by TheSorrow
...You sure you have a DVD drive in your computer? D:

Re: .hack MAD - Aimed at developers.

Posted: Mon Dec 21, 2009 11:55 pm
by Keyaki
TheSorrow wrote:...You sure you have a DVD drive in your computer? D:
That probably it

Re: .hack MAD - Aimed at developers.

Posted: Tue Dec 22, 2009 11:56 am
by Takai Tenshi
so basically, he is working on an emulator that, instead of running all of the graphics, sounds, etc. off of a program, it'll load them all from disks.


well, either way. if your serious about it, can you please make character design a bit... idk BETTER than fragment was??

Re: .hack MAD - Aimed at developers.

Posted: Wed Dec 23, 2009 3:17 am
by Coma
I'm working on a .hack MMO... but not really. For copyright reasons, .hack (or The World) won't be mentioned anywhere, and visual designs will be different. Personally, I think this effort is kind of futile, because when designing a modern game, you really don't want to use graphics from the PS2 era. Even worse is that it will be reliant on multiple DVDs, which will severely restrict the amount of people who can play it -- and between those, those who WILL play it, simply for the hassle of doing so.

As I said, it's a .hack MMO - but not really. That is, it will be designed the same, and make possible all the things that make .hack special (and not just some random MMO with different classes), without making everyone special. The one thing I really haven't figured out is hacking -- because while I want to make it possible for hackers in the .hack sense to exist too, it's quite obvious a tutorial for everything will soon be written and being a hacker will no longer be special. I was going to purposefully leave holes for people to exploit, but even something that required genuine knowledge (in contrast with pseudo-knowledge, ala a simple language made for the game) would have tutorials made for it.

Anyway, the game is still in its baby steps. I'm working on it solo for the time being - I hope to get it to a stage, although still simple, where it has something significant to show, and attract capable individuals. I don't think a game in such an infant stage has any hope of attracting motivated, capable individuals, so I'm going to lay the groundwork down first -- when simple functions are achieved (walking in a root town, chatting, etc), I'll look to build a team. Doing it before that stage seems like a recipe for project death.

It's using OGRE 3D for graphics, FMOD EX for sound, and Project Darkstar for networking, server/client framework, and data storage. They're all freely available so that I may eventually release the entire game and server as open source. I highly doubt I'll be able to support it financially for long, so I plan to release it and let it continue living in the private server domain, which pretty much guarantees immortality.

I know this is somewhat unrelated, but I posted this because I really don't think you should continue working on this, for the reasons stated above. The art is out of date, and it is a huge hassle to use it. I'm not trying to advertise my project or something -- I just really don't think any promising project will spring out of your efforts. It'll likely all be to vain, if as you say, you don't plan to do it yourself.