[Developers] Everything Steam thread

EriX920

Respected
Purpose:
This thread is intended for all the collected information about Steam, how it works and it's protocol. There are a few "wikis" around on the internet with information on Steam but arn't really maintained and don't contain the interesting parts such are remotely mounting GCFs and updating local outdated GCFs at the same time.

If there is information posted in here by me and I haven't included credits please let me know and I will add them.

Introduction:
What is Steam? Simply put, Steam is a platform that can deliver content to anyone, anywhere. Valve has content server's for Steam in over 50 countries so download speeds are always adequate. All in all, Steam is an amazing platform and allows game makers to easily and quickly release their content to a large variety of users. In addition to this, whenever the content needs an update, the developer can push out an update to the users so that they do not have to go out to some third party site to search for a patch.

I'm sure if you're reading this it is for the more advanced topics but it is important to know that Steam can push updates to the users. This means that we can refresh the CDR (see further down in thread) often to see the updates for a file. For example, if we have version 100 of winui.gcf and Steam has a new version of 101, the CDR can be updated minutes after the update is released and then we can see that there is an update available. If developing an updater for your games, you can have your app re-download the CDR every once in a while to see if there are updates. If there is an update, we can mount the remotely updated file and local outdated file and update the file without re-downloading the entire content file. I will go into greater details later on.

Before continuing down this post, make sure you are aware of these terms:

Why do we care?:
There are many ways that Steam can benefit our community. We all know that obtaining GCFs is a complicated process, at least it is for those who do not have rapidshare or megaupload accounts. Not to mention that it is an act of piracy for hosting copyrighted content and distributing it. Here at Freesteam, we do not host ANY GCFs/NCFs or other content connected to Steam, just had to clear that up ;).

Back to the benefits. With Steam we can do the following:
  1. Have a database of accounts with games
  2. Download all games from accounts
  3. Securely host content and distribute
Content file updating:

In the past, we would have have to update the file and then completely re-upload the file to get the update. When steamCooker came along, he created a tool that allowed users to create an archive file of an outdated content file. This includes the header of the GCF file with the file names and checksums for the chunks of the files. With this archive file it would then be possible to compare it to an updated file and create an .update file. The user could then apply this .update file to the outdated GCF and in no time have a fully updated content file without having to re-upload/re-download the entire file. This process saves countless download time/upload time and just about everything else. In addition to this, if the user does own the game, with steamCooker's tool the user could login and update a content file in the same manner Steam does. However, if the user does not own a certain content file and has obtained it through nefarious ways they must wait until someone who owns the content file creates an .update file and uploads it.

Content file downloading:

Just like content file updating, content file downloading requires the community to upload the entire file to a peer before anyone else can obtain it. With steamCooker's CFToolbox, we are able to download some content files but essentials like the ones the contain "client.dll" need an authorized account to access. This is not a fault with CFToolbox, it just simply cannot be done without a valid account.

A solution:
There are two ways to look at our "issue" here. One way is to see that Steam has the files locked away for only authorized accounts to access, which indeed they are. The other way is to see that we CAN access the files IF we have an authorized account. This brings up an interesting question. If we can access these files with authorized accounts, how can get the content out to our clients? This as you can imagine is not an easy question to answer. Here are some ways myself and others have come up with to solve this question.
  1. Post Steam accounts to trusted people and allow them to use it for updating and download of content.
    Pros:
    • Don't have to deal with the client server model of transferring files
    • No worring about transfer speed or bandwidth
    • Cheap
    Cons:
    • High risk of getting accounts disabled or stolen
  2. Create a client server application to securely send Steam login information to clients to allow them to access the account(s).
    Pros:
    • Easy to manage
    Cons:
    • High risk of getting accounts disabled or stolen
    • Application creation
    • Risk of security problems (unauthorized people accessing the accounts)
-- More later! --

What's a CDR?:
First off, CDR stands for Content Description Record and Steam uses it for storing all of the content that it supplies to users in the network. You cannot see all of the content within the CDR unless you are running cracked Steam or if you have a master account. Steam checks to see whether or not your account has permissions to access the subscription. The way it can tell is by simply calling the "SteamIsSubscribed" function within steam.dll after the user has logged in. For every subscription in the CDR it calls this, and if the subscriptionID is subscribed to the account it adds it to the list.

A snippet of code that can represent this functionality is as follows (Credits to Mitsukarina for original code):
Code:
 For Each RawSubscriptionID As UInteger In RawSubscriptionIDs
                    iSteamReturn = SteamIsSubscribed(RawSubscriptionID, isSubSubscribed, isSubReady, SErr)

                    If isSubSubscribed = 1 Then

                        Dim SubNFO As New TSteamSubscription()
                        SubNFO.Name = New String(" "c, 255)
                        SubNFO.MaxNameChars = 255
                        SubNFO.AppIDs = intPtr
                        SubNFO.MaxAppIDs = AppStats.uNumApps

                        iSteamReturn = SteamEnumerateSubscription(RawSubscriptionID, SubNFO, SErr)

                        If Not RawSubscriptionID = 0 Then
                            If Not SubNFO.Name = "" Then
                               ' Add the subscription name to anywhere
                            End If
                        End If

                        Dim ofs As Integer = 0
                        Dim uLoop As Integer
                        For uLoop = 0 To SubNFO.NumApps - 1 'populate our app array from marshaled memory
                            tAppIDs(uLoop) = Marshal.ReadInt32(intPtr, ofs)
                            ofs = ofs + arraysize
                        Next

                        For uLoop = 0 To SubNFO.NumApps - 1 'Clear the relevant array indexes for the next subscription
                            tAppIDs(uLoop) = 0
                        Next

                    End If
                Next
We can see that it will go through each subscriptionID in the CDR and check to see whether or not the it is subscribed. As you can probably guess, removing the check to see if the appid is subscribed or not will allow you to get the info of all the content in the CDR.

Once again we're back to "why do we care?" For instance, if we have two accounts one with a subscription to Counter Strike and one with Team Fortress. If we run the account through our isSubscribed loop, we can determine which files the account has access to. From here we can pass the appids to a downloading function where we can begin to download and queue all subscribed appids for a given account. We have now, in theory, created an automated content downloader / updater.

GCF Basics:
As we know, a GCF stands for "Game Cache File". It can be compared to a zip or rar archive (though winrar or winzip cannot natively open a GCF file). In the GCFs header it contains the file names with their location, their checksums and size. When a GCF is downloaded, a skeleton structure of the GCF is created by remotely mounting the GCF on Steam's content servers. From here, the general information about the file is downloaded to the clients computer. We know that the header contains the most important information so that is what is downloaded first. Before any of the internal files are downloaded, we are left with a "mini-GCF". All this means is that there is space allocated in the header along with all the file names and checksums. This is why when we compress a mini-GCF, it can be dropped 98-99% of the allocated size. For example, winui.gcf is about 40mb. When in the mini-GCF form, it can be compressed to less than a megabyte because there is hardly any real data in there, simply just allocated blank space.

To write to a GCF you must open a file for reading which can be done by mounting the GCF (RAIN_mountGcf("C:\path\to\file.gcf")). It is important to know that you MUST write less than or equal to the amount of allocated space for the given file. If you run RAIN_size(gcf, "/path/to/file.txt") and it returns 10 bytes you can only write 10 bytes of data to that file.

Checksums are stored in blocks. This means that the entire file is one checksum if the file is less than or equal to 32768 bytes. If it is larger, it will consist of checksums for each of the 32768 byte blocks in the file. Because of this, we must return an index of all the checksums for the blocks of the file. See the bottom of this post for the usage of "RAIN_getChecksums".

By using RAIN_size and RAIN_getChecksums we can compare two files very accurately. This is especially important for archive updating. The way that I do the updating is the following:
  1. Mount the remotely updated archive on Steam's content server
  2. Create a new blank GCF with the remote GCF as the reference file
  3. Mount the locally outdated GCF and compare the files
  4. If the checksums don't match between the locally outdated file and remotely updated file then obviously the file was changed in the update so that file must be redownloaded.
  5. If the file in the new updated file does not exist in the old outdated file, then that file has to be downloaded.
By following the above process, the GCF can be updated without having to redownload the entire GCF everytime you want to update it.

Mounting remote GCFs?:
Before recently, this task was unimaginable for most including myself. Thankfully, steamCooker decided to release his new API called "rain" to me. This API contains nothing but useful functions for developers. See the Unofficial RAIN documentation at the bottom.

Available APIs:
Currently there are two APIs available to the public.

The rain API which does such things as downloading the CDR, download content, mounting content and other task.

The CFT api allows you to create .archive files from existing content files. With the .archive file, you can compare it against an updated content from (from which the .archive file was created from) to create a .update file. With this file your old outdated file can be updated without redownload the entire file over again.

Unofficial RAIN API Documentation:
SteamCooker is always busy cooking stuff up for Steam so I figured i'd write an unofficial documentation for him and he could add stuff/fix stuff as he pleases.

The header file for the API is as follows.
Code:
/*
Rain API - by SteamCooker
*/

//#define RAIN_DEBUG

#define RAIN_FS
#define RAIN_CHAT

#ifdef RAIN_DEBUG
#define RAIN_API
#else
#ifdef RAIN_EXPORTS
#define RAIN_API __declspec(dllexport)
#else
#define RAIN_API __declspec(dllimport)
#endif
#endif

RAIN_API void RAIN_enableLogs(char * logFile);

#ifdef RAIN_FS

typedef int GCF;
typedef int GCF_ENTRY;
typedef int CDR;

typedef struct
{
	unsigned int appId;
	bool isNcf;
	char name[200];
	char installDirName[200];
	unsigned int currentVersion;
} APPInfo;

typedef struct
{
	unsigned int appId;
	bool isOptional;
} APPFileSystem;

#define APPFileSystems_MAX	20

typedef struct
{
	unsigned int count;
	APPFileSystem fileSystems[APPFileSystems_MAX];
} APPFileSystems;

typedef struct
{
	unsigned int count;
	unsigned int * checksums;
} Checksums;

RAIN_API GCF RAIN_mountGcf(char * gcfPath);
RAIN_API GCF RAIN_mountGcfOnContentServer(char * gdsServer,unsigned char cellId, char * accountName, char * accountPassword,unsigned int cfId, unsigned int cfVersion);

RAIN_API GCF RAIN_newEmptyGcf(char * gcfPath,GCF toCopy);

RAIN_API bool RAIN_unmountGcf(GCF gcf);

RAIN_API unsigned int RAIN_getAppId(GCF gcf);
RAIN_API unsigned int RAIN_getAppVersion(GCF gcf);

RAIN_API char ** RAIN_listFiles(GCF gcf, char * path);
RAIN_API void RAIN_free(char ** list);

RAIN_API bool RAIN_isFile(GCF gcf, char * path);
RAIN_API bool RAIN_isDirectory(GCF gcf, char * path);

RAIN_API unsigned __int64 RAIN_size(GCF gcf, char * path);
RAIN_API unsigned __int64 RAIN_effectiveSize(GCF gcf, char * path);

RAIN_API bool RAIN_delete(GCF gcf, char * path);

RAIN_API Checksums RAIN_getChecksums(GCF gcf, char * path);
RAIN_API void RAIN_free(Checksums checksums);

RAIN_API GCF_ENTRY RAIN_openToRead(GCF gcf, char * path);
RAIN_API GCF_ENTRY RAIN_openToWrite(GCF gcf, char * path, bool append=true);

RAIN_API bool RAIN_close(GCF_ENTRY file);

RAIN_API unsigned int RAIN_read(GCF_ENTRY file, unsigned char * data, unsigned int size);
RAIN_API unsigned int RAIN_write(GCF_ENTRY file, unsigned char * data, unsigned int size);

RAIN_API CDR RAIN_downloadCDR(const char * gds);
RAIN_API void RAIN_free(CDR cdr);
RAIN_API void RAIN_save(CDR cdr, const char * filename);
RAIN_API CDR RAIN_load(const char * filename);
RAIN_API APPInfo RAIN_getAppInfo(CDR cdr, unsigned int appId);
RAIN_API APPFileSystems RAIN_getAppFileSystems(CDR cdr, unsigned int appId);

#endif

#ifdef RAIN_CHAT

extern "C" typedef const char * (__cdecl *RAIN_OnChatMessageHandler)(unsigned __int64 steamGlobalId, const char * message);
// the handler returns a string to be sent back to the message sender, or 0 if no message is to be sent
// the string will not be freed

enum ChatStatus
{
	ChatStatus_offline=0,
		ChatStatus_online=1,
		ChatStatus_busy=2,
		ChatStatus_away=3
};

RAIN_API bool RAIN_startChatListener(const char * login, const char * password, RAIN_OnChatMessageHandler onMessageHandler);
RAIN_API bool RAIN_isChatListening();
RAIN_API bool RAIN_setChatStatus(ChatStatus status);
RAIN_API void RAIN_stopChatListener();

#endif

  • RAIN_enableLogs(Byval pathToLog As String): This function will start the logger for all the rain funtions.
  • RAIN_mountGcf(Byval gcfPath As String): Mounts the gcf for manipulation. For example, writing files into and reading files out of.
  • RAIN_mountGcfOnContentServer(Byval gdsServer As String, Byval cellId As Integer, Byval accountName As String, Byval accountPass As String, Byval cfId As Integer, Byval cdVersion As Integer): This function is very handy. It will allow you to mount a file remotely on Steam's content server. By doing this you can copy files out of the remote file without downloading it. Say the remote archive is winui.gcf and you want a file out of it. You use this function to mount the archive then you can use RAIN's internal reading and writing functions on it. This way you can mount the remote gcf and only copy the files you need out of it (for remote updating).
  • RAIN_newEmptyGcf(Byval gcfPath As String, Byval gcfToCopy As Integer): This function is meant to be used in conjunction with the mountGcfOnContentServer function. This function must be used if you are planning on downloading a remote file. It creates a mini gcf based on the gcf that you pass to it. Say you run this code:
    Code:
    Dim gcf As Integer
    ' Mount winui.gcf remotely (appid 7)
    gcf = RAIN_mountGcfOnContentServer("gds1.steampowered.com:27030", 5, steamUsername, steamPassword, 7, 270)
    After doing that, you can create a mini a gcf based on winui. The completed code is as follows:
    Code:
    Dim gcf, miniGcf As Integer
    gcf = RAIN_mountGcfOnContentServer("gds1.steampowered.com:27030", 5, steamUsername, steamPassword, 7, 297)
    miniGcf = RAIN_newEmptyGcf("winui_mini.gcf",gcf)
    RAIN_unmountGcf(miniGcf)
    RAIN_unmountGcf(gcf)
  • RAIN_getAppId(Byval gcf As Integer): This function gets the appId of the gcf you pass into it.
    Code:
    Dim gcf, appId As Integer
    gcf = RAIN_mountGcf("winui.gcf")
    appId = RAIN_getAppId(gcf)
    RAIN_unmountGcf(gcf)
  • RAIN_getAppVersion(Byval gcf As Integer): This function gets the version of the mounted gcf file.
    Code:
    Dim gcf, appVersion As Integer
    gcf = RAIN_mountGcf("winui.gcf")
    appVersion = RAIN_getAppVersion(gcf)
    RAIN_unmountGcf(gcf)
  • RAIN_isFile(Byval gcf As Integer, Byval path As String): This function determines whether or not a path within an archive is a file.
    Code:
    Dim gcf As Integer
    Dim isFile As Boolean = False
    gcf = RAIN_mountGcf("winui.gcf")
    isFile = RAIN_isFile(gcf, "/Support/support.dll")
    RAIN_unmountGcf(gcf)
  • RAIN_isDirectory(Byval gcf As Integer, Byval path As String): Similar to isFile except it determines whether or not a path within an archive is a directory.
    Code:
    Dim gcf As Integer
    Dim isDirectory As Boolean = False
    gcf = RAIN_mountGcf("winui.gcf")
    isDirectory = RAIN_isDirectory(gcf, "/Support")
    RAIN_unmountGcf(gcf)
  • RAIN_getChecksums(Byval gcf As Integer, Byval path As String): This function will
    Code:
    ' Vb.Net
        <StructLayout(LayoutKind.Sequential)> Structure Checksums
            Dim count As UInt32
            Dim checksums() As UInt32
        End Structure
    
    Private Sub getChecksums(Byval gcf As Integer, Byval path As String)
            Dim checksumsStruct As Checksums
            checksumsStruct = RAIN_getChecksums(gcf, path)
            For ind As UInteger = 0 To checksumsStruct.count
                MsgBox(checksumsStruct.checksums(ind))
            Next
            RAIN_Free(checksumsStruct)
    End Sub
    
    // C++
    typedef struct
    {
    	unsigned int count;
    	unsigned int * checksums;
    } Checksums;
    
    void printChecksums(GCF gcf, char * filePath)
    {
    	Checksums checksums=RAIN_getChecksums(gcf,filePath);
    	
    	printf("%s has %u checksums :\n",filePath,checksums.count);
    	
    	for (unsigned int ind=0;ind<checksums.count;ind++)
    	{
    		printf("[%08X]",checksums.checksums[ind]);
    	}
    	printf("\n");
    	RAIN_free(checksums);
    }
  • RAIN_size(Byval gcf As Integer, Byval path As String): This function determines the size of a given file within a gcf. Only use this function if the gcf is 100% complete or else it may return an incorrect file size if the file is incomplete.
  • RAIN_effectiveSize(Byval gcf As Integer, Byval path As String): If the gcf is incomplete or complete, this function will work. It will return the actual size of the file in the archive. If the file is complete, RAIN_effectiveSize and RAIN_size will return the same. If the file is incomplete (determined by calculating the completion of the archive), then RAIN_effectiveSize will return the actual written bytes of the file.
I will explain what each function does in depth later on.


-- More later! --
 
Everything Steam thread

Good read!

It has filled in a lot of gaps - thanks Erix920.

cheers
OldFart
 
Everything Steam thread

Thanks for the post buddy, keep it up! Glad to see you being very active around here again.
 
No problem :), I hope to get a few more devs here to help me expand this post.

Latest additions (1/13/10): GCF Basics, RAIN_size, RAIN_effectiveSize
 
Thanks again Erix920!

I'm about to read the updates and post a link to here - on my clan's forums.

cheers
OldFart
 
Back
Top