a lot of people have been asking over the years if we could figure out a way to get Spending Quotient and other interesting stats about our matches. there is an interesting file that i believe contains all the stats that are shown in the Summary Screen after a game completes.
it's the s2gs file. You can trigger arbitrary *.s2gs files to be downloaded to your machine by viewing game summaries in the SC2 client. In SC2, go to your Match History, then click on one of the magnifying glasses. That will trigger a new s2gs file to be downloaded to your machine.
The easiest way to find a s2gs file is to look at some match summary screen in the game client, then go to the directory and sort by date modified.
If we all put our heads together, I bet we can figure out the format.
A fellow SC2 nerd, Nick, was able to get this far:
#!/usr/bin/env python
import zlib
data = open("test.s2gs","rb").read()
print zlib.decompress(data[16:])
Some of you may wonder about what Blizzard's reaction to this project will be. I hope that Blizzard will have the same reaction that they've had to the community's study of the structure of the MPQ and the replay format -- essentially none. Of course, if Blizzard prefers that this project not move forward, I for one will cease immediately.
I'm not sure what's the best way to structure our findings -- I guess people can post what they discover in this thread, and once that becomes unwieldy, someone can set up a wiki or public Google docs page to hold the findings. Or a github project.
You might be asking, what's the point of being able to parse s2gs files? Basically every tool that does anything useful with replays could parse s2gs as well, and all the info that is visible in the end-of-game summary screen could be merged with the replay info. For example, computing Spending Quotient would be trivial. There are about 30 other cool things would be immediately quite doable.
On April 21 2012 06:04 Kralic wrote: I am not sure what to say... so I will say good luck decrypting the encrypted file.
I dont think its encrypted, just compressed. This seams like a cool project, but I cant think of anything you can get from the summery that you couldnt get from the replay file. Just easier parsing perhaps?
spancho, the summary has a lot of things you can't get from the replay, for example: * average unspent resources * resource collection rate * number of workers created * first 64 things built by each player
each of these can be estimated from the replay, but in many cases the estimation will be off, sometimes by a lot.
the replay file contains build commands that fail due to lack of resources, and there's no way to tell which these are without recreating the whole game engine. unspent resources also cannot be calculated without essentially rebuilding the game engine.
also the replay file doesnt tell you when workers are killed. when my workers die it can change my resource collection rate a lot
On April 21 2012 06:04 Kralic wrote: I am not sure what to say... so I will say good luck decrypting the encrypted file.
I dont think its encrypted, just compressed. This seams like a cool project, but I cant think of anything you can get from the summery that you couldnt get from the replay file. Just easier parsing perhaps?
It would be a way to get spending quotient without relying on a sketchy text recognition script.
I'm not sure this file has complete data, I'm only seeing reference points (game speed, difficulty, players) and no game data such as resources/workers.
It would be nice to chat with some others who are working with this data. I'm in #teamliquid on IRC.
After some further discussion, I believe this might be a red herring. A file containing reference points needed to parse the relevant packets coming from Battle.net, which are consumed by the replay scoreboard to populate variables.
On April 22 2012 12:00 zylog wrote: This is awesome. I wonder if this can somehow be integrated into sc2gears to add additional information for replay analyses.
0x05 is a Key-Value object, so average unspent resources is a Key-Value object with 2 entries. First key (key 0x0) is a variable length integer (0x09). Second key (key 0x02) is also a variable length integer (0x09). sc2reader has a parser for this format written in python, and I also have a parser of my own written in javascript: https://github.com/tec27/comsat/blob/master/lib/parsers/blizzerial.js
Hopefully that helps you see what the prefixes are for and stuff
Hi, so I added a couple fixes to the code mentioned above. The wiki hasn't been updated but the code is very simple and can be found on the master branch of the repo (here).
The uncompressed data appears to be a list of structures. I had 9 in the example I was working from, the number may vary based on the number of players, not sure yet. An example python script I am using can be found below:
Ok I installed your python thingy, here is what i got : on 4th structure : { 0: 999, 1: 0} resources [{ 0: ressources, 1: 0, 2: game time}] { 0: 999, 1: 1} units { 0: 999, 1: 2} structures { 0: 999, 1: ...} ... on 5th structure (weird that it is in another structure, especially the razed count) { 0: 999, 1: 10} is structure razed count ? { 0: structure razed count, 1: 0, 2: game time} { 0: 999, 1: 11} is resource graph [{0:army value, 1:0,2: game time}, ...] { 0: 999, 1: 12} is army graph [{0:army value, 1:0,2: game time}, ...] { 0: 999, 1: 16777328} is unknown ??
So we also have the graph data now, only need to find the build order information now...
Edit : I put values of 1: and 2: in Hexadecimal, its easier to get values like this
I think 0x84 is for drones, 0x86 for overlords the 0: is the supply (7,8,9,10) the 1: not sure... the ending A might be the total supply (10) the 2: I think is the time where 0x1C = 00:02 0x32 = 00:03 0x49 = 00:05 0x11C = 00:18 0x12F = 00:19 0x167 = 00:22 0x1C4 = 00:28 0x1DE = 00:30 0x276 = 00:39 0x2C4 = 00:44 0x316 = 00:49 0x378 = 00:56 0x38C = 00:57
anyone to get the general formula ?
Edit : Found ! i's in 16th of seconds ! 0x1DE = 478 and 478/16 = 29.875 so 00:30
On April 22 2012 12:21 Marti wrote: Now you got me curious, but i'm still not sure about the actual use of the information.
It's reliable stats. Think if a tournament like MLG or DH gets all s2gs files. They can then, for each player, calculate the most used build order or stuff like that. It would be possible to see trends in metagame with build orders. It's just more stats which you can't get out of the replay files. altough they are "harder" to retrieve for a big tournaments because s2gs isn't made so you can share the information like replay files are.
I'm only able to parse stats3.s2gs with Graylin's script. + Show Spoiler +
Traceback (most recent call last): File "extract.py", line 89, in <module> main() File "extract.py", line 80, in main buffer.read_data_struct() File "build\bdist.win-amd64\egg\sc2reader\utils.py", line 270, in read_data_struct datatype = self.read_byte() File "build\bdist.win-amd64\egg\sc2reader\utils.py", line 150, in read_byte return ord(self.read_basic(1)) TypeError: ord() expected a character, but string of length 0 found
for i in range(len(pdata)): stats[data_names[i]] = pdata[i][1][index][0][0] stats[data_names[len(pdata)]] = data[4][0][0][1][index][0][0] player['stats'] = stats
return player
def main(): p = sys.argv[2] == "players" with open(sys.argv[1],"rb") as s2gs: raw_data = zlib.decompress(s2gs.read()[16:]) buffer = ReplayBuffer(raw_data)
I don't have time to migrate documentation to the teamliquid wiki right now. Anyone who wants to compile a team liquid wiki can feel free to steal from the sc2reader wiki. Just leave a back reference to the project if you can.
Also, how can I get quoted text to preserve link markup?
On April 22 2012 21:29 RobiTL wrote: Edit : Found ! i's in 16th of seconds ! 0x1DE = 478 and 478/16 = 29.875 so 00:30
Yes, sc2runs 16 frames per second. You can use the games speed conversion to get real time if you wanted.
I should also mention that unit types on the build order are almost certainly integers. They should match up to the hex values in this data file. The types are also accessible programmatically, see the TargetAbilityEvent for an example.
There are a bunch of other oddities that might apply here. Game date is in a windows timestamp which has a wierd conversion function.
At this point I wish I had maintained my documentation better. Given that we're sitting here parsing a compressed byte stream I am hoping that reading the code isn't too much of an issue.
@Prillian You only need to read 5 data structures (the interesting stuff is on the 4th and 5th) On some files if you try to read more it'll make the TypeError ord().
On April 22 2012 22:46 RobiTL wrote: @Prillian You only need to read 5 data structures (the interesting stuff is on the 4th and 5th) On some files if you try to read more it'll make the TypeError ord().
On April 22 2012 22:46 RobiTL wrote: @Prillian You only need to read 5 data structures (the interesting stuff is on the 4th and 5th) On some files if you try to read more it'll make the TypeError ord().
The ord error is an unchecked way of letting you know you hit the end of the buffer. The class could benefit from a more defensive programming style...
Amazing stuff guys keep it up. I have a lay (idiot) question. If I wanted to see my changing sending quotient over my last hundred games, do I have to go back trough my games played and open the stats page 1 by 1 to generate the file and then run a program of some sort at the end of the process or would I need to get the program to hit the files as they were spat out or... I guess I'm basically asking when the cache folder gets cleaned....
On April 22 2012 23:30 Dapper_Cad wrote: Amazing stuff guys keep it up. I have a lay (idiot) question. If I wanted to see my changing sending quotient over my last hundred games, do I have to go back trough my games played and open the stats page 1 by 1 to generate the file and then run a program of some sort at the end of the process or would I need to get the program to hit the files as they were spat out or... I guess I'm basically asking when the cache folder gets cleaned....
Test it out. Grab all the .s2gs files and see how old they are
Note that we're not done yet so all desired information isn't available yet
On April 22 2012 23:30 Dapper_Cad wrote: Amazing stuff guys keep it up. I have a lay (idiot) question. If I wanted to see my changing sending quotient over my last hundred games, do I have to go back trough my games played and open the stats page 1 by 1 to generate the file and then run a program of some sort at the end of the process or would I need to get the program to hit the files as they were spat out or... I guess I'm basically asking when the cache folder gets cleaned....
I think it can be implemented in sc2gears or some program to create that file for every auto saved replay after the update.
I'll have a look when I get home. Regardless, great, GREAT things will come of this, as long as it's going. As an aside it's both odd and depressing that one of the colossal advantages which esports should have over regular sports is automated gathering of statistics with unprecedented breadth and depth and that's exactly what IP acti-bliz's constipated attitude toward IP gets in the way of. Whatever, can't emphasis enough, fantastic work.
On April 22 2012 23:39 Aelfric wrote: I think it can be implemented in sc2gears or some program to create that file for every auto saved replay after the update.
This file is downloaded from the bnet depots. Someone could packet sniff their computer and get the request URL. From there it will probably be possible to automatically download the S2GS files at will based on information from the replay.
I am fairly confident saying this because that is how map downloading currently works. You can grab the exact map file game was played on from the replay file via bnet.
On April 22 2012 23:39 Aelfric wrote: I think it can be implemented in sc2gears or some program to create that file for every auto saved replay after the update.
This file is downloaded from the bnet depots. Someone could packet sniff their computer and get the request URL. From there it will probably be possible to automatically download the S2GS files at will based on information from the replay.
I am fairly confident saying this because that is how map downloading currently works. You can grab the exact map file game was played on from the replay file via bnet.
I'm on it!
EDIT: Found it.
GET /{hash}.s2gs HTTP/1.1 User-Agent: Battle.net Web Client Host: EU.depot.battle.net:1119 Accept: */*
Now to figuring out the hash. Maybe it's contained in the replay somewhere?
GET /{hash}.s2gs HTTP/1.1 User-Agent: Battle.net Web Client Host: EU.depot.battle.net:1119 Accept: */*
Now to figuring out the hash. Maybe it's contained in the replay somewhere?
Awesome. Can you supply a set of replay files and their corresponding .s2gs hashes?
It'd help to have a common set of data for everyone to be working with, possibly with screenshots of the score screens as well. Ideally a mix of 1v1,2v2,4v4, FFA games to see how/if the player count affects the structure.
I haven't booted SC in weeks so this would be particuarly helpful to me...
GET /{hash}.s2gs HTTP/1.1 User-Agent: Battle.net Web Client Host: EU.depot.battle.net:1119 Accept: */*
Now to figuring out the hash. Maybe it's contained in the replay somewhere?
Awesome. Can you supply a set of replay files and their corresponding .s2gs hashes?
It'd help to have a common set of data for everyone to be working with, possibly with screenshots of the score screens as well. Ideally a mix of 1v1,2v2,4v4, FFA games to see how/if the player count affects the structure.
I haven't booted SC in weeks so this would be particuarly helpful to me...
Sure, I need to structure my work a bit first.. It's a mess ^^
On April 23 2012 00:38 Prillan wrote: Do anyone know how the map hash is calculated?
The map hash is stored in the replay.initData file in the .SC2Replay MPQ archive. We don't calculate it, we just use it. The initData file actually stores a bunch of resource identifiers but none of them (that I know of) are this .s2gs file though so it is probably stored elsewhere.
On April 23 2012 00:38 Prillan wrote: Do anyone know how the map hash is calculated?
The map hash is stored in the replay.initData file in the .SC2Replay MPQ archive. We don't calculate it, we just use it. The initData file actually stores a bunch of resource identifiers but none of them (that I know of) are this .s2gs file though so it is probably stored elsewhere.
Sad news: The hash is actually the SHA-256 hash of the file itself.
I'm trying to find some kind of index file to grab from the same source.
I would guess that map files are named the same way. This means that if we can't find an explicit reference from the replay file we might be out of luck with the automatic downloading.
I would guess that map files are named the same way. This means that if we can't find an explicit reference from the replay file we might be out of luck with the automatic downloading.
There is no date or timestamp that could be compared/checked? I'm novice programmer, but I figure there has to be something between the replay file and the s2gs file that would match and that would be unique that could be cross referenced. Date/Time seems like a good place to start..but again i'm novice, and as my sig says "I'm Nub."
I would guess that map files are named the same way. This means that if we can't find an explicit reference from the replay file we might be out of luck with the automatic downloading.
There is no date or timestamp that could be compared/checked? I'm novice programmer, but I figure there has to be something between the replay file and the s2gs file that would match and that would be unique that could be cross referenced. Date/Time seems like a good place to start..but again i'm novice, and as my sig says "I'm Nub."
Yeah, that's not the problem ^^
We wanted to find a way to calculate the hash so we could download the file given a replay. Now we can only use new .s2gs files and those already in the Cache folder. The other ones are lost forever
I would guess that map files are named the same way. This means that if we can't find an explicit reference from the replay file we might be out of luck with the automatic downloading.
There is no date or timestamp that could be compared/checked? I'm novice programmer, but I figure there has to be something between the replay file and the s2gs file that would match and that would be unique that could be cross referenced. Date/Time seems like a good place to start..but again i'm novice, and as my sig says "I'm Nub."
The issue of correlating two files is different from the issue of using one file to get the other.
The two files might both have identical player/time/initdata information that allows us to definitively say that the two files we have are for the same replay. That information is certainly useful but requires that we have physically obtained a copy of each file.
It is a different issue to find one file given that we have the other. Since we know how to grab replay files easily, the issue becomes grabbing the s2gs file from the replay file. If files are named according to a hash of their contents (not reproducable without the file), the only way we could find one from the other is if there were embedded links. Map files for instance, have their id embedded in the replay file for retrieval at a later point.
Great work. I don't know enough to meaningfully contribute to the work but I can say that you are doing a great job.
Implementing the findings into programs like SC2Gears would be amazing for even more replay and multiple replay analysis. Thanks for the good work and I hope you can figure it all out.
I don't think the s2gs hash is in the replay, or computable from the replay. Think about how the SC2 client works. There must be some kind of "player profile" data that the SC2 client retrieves, which must contain a list-like data structure of the Match History. Inside the Match History there must be information for the games that are shown on the Match History. Inside the data for a match must be the s2gs filename.
I expect that only wiresharking will tell us for sure.
On April 23 2012 02:41 dsjoerg wrote: I don't think the s2gs hash is in the replay, or computable from the replay. Think about how the SC2 client works. There must be some kind of "player profile" data that the SC2 client retrieves, which must contain a list-like data structure of the Match History. Inside the Match History there must be information for the games that are shown on the Match History. Inside the data for a match must be the s2gs filename.
I expect that only wiresharking will tell us for sure.
I've done that and didn't find anything interesting. I haven't found any other interesting file in the Cache folder which contains anything close to a list of s2gs files
Searching through hexdumps of the replay datafiles confirms that there isn't an explicit reference in the replay file.
Also, I agree with your analysis @dsjoerg. Wireshark can provide us with the URLs we need to simulate moving through the client application. I'd suspect that most of the files will be in the same serialized data format. Not sure how much Blizzard would appreciate our pinging of their data servers but I suppose we can find out the hard way.
@Prillan want to track down some of the other data feeds triggered by the profile GUI?
Prillan, although the "player profile" may not be in the Cache file, it must be transmitted over the wire to the SC2 client... otherwise it wouldn't know s2gs filename to request.
On April 23 2012 03:02 dsjoerg wrote: Prillan, although the "player profile" may not be in the Cache file, it must be transmitted over the wire to the SC2 client... otherwise it wouldn't know s2gs filename to request.
I know. The problem is that almost all communication is encrypted. Only file downloads are not and from that I can't find anything useful, unfortunately.
As for GUI -> File download triggers. Nothing interesting. + Show Spoiler +
.s2ml - Map localization .s2mi - Map info .s2mv - Map image .s2mh - Map ??? .unlk - JSON - ??? .s2ma - Map file .ldpl - JSON - ??? .pfty - A lot of utf-8 encoded swear words (language filter?) .s2ql - XML - Some kind of localization file .s2qh - JSON - ??? .wafl - (waffles??) - Image file .ldpt - JSON - Something about textures .s2gs - Score screen stats .auth - ??? .achv - JSON - Achivements .xml - Depends on the file .txt - [Breaking News/PTR Available] .dll - Scan.dll (No idea what it does)
and with that I mean "No interesting file downloads when I look around"
When I open a players match history a bunch of files are created in my cache folder. Most of them have the extension "s2ml", and there is also some with extensions "s2mh" and "sc2mi". This type of files are also created when I click "show more entries" at the bottom of the Match history. They could maybe contain information about which s2gs files that are needed.
I do not understand the methods for decoding the files, so I collected the files in the linked zip-file and hope it can be useful to someone else.
The files created by clicking at "show more entries" and some related s2gs files(created when I clicked on matches that appeared after showing more entries): http://folk.ntnu.no/toringb/moreentries.zip
EDIT: Great work guys! Im excited to see if this will be figured out!
I think I have something better! When I sort the match history by type(1v1 etc) only one file changes! It has the file extension "s2mi".
I have gathered the files changed in all sorting versions and some related s2gs files. The files in the folder 1v1 is collected when sorting 1v1. http://folk.ntnu.no/toringb/Sorting.zip
My method is this: Go to profile. Wait a minute so the clock changes. Click match history Search /Users/Shared/Blizzard for files with . in the filename and are changed within the last minute. (Playing on mac)
On April 23 2012 04:29 Mixopteryx wrote: I think I have something better! When I sort the match history by type(1v1 etc) only one file changes! It has the file extension "s2mi".
I have gathered the files changed in all sorting versions and some related s2gs files. The files in the folder 1v1 is collected when sorting 1v1. http://folk.ntnu.no/toringb/Sorting.zip
My method is this: Go to profile. Wait a minute so the clock changes. Click match history Search /Users/Shared/Blizzard for files with . in the filename and are changed within the last minute. (Playing on mac)
I still believe that .s2mi is StarCraft 2 Map Information. This is what I could extract from your
I'm sure it's not 100% correct though (and some bits may be missing). I just added this in utils.py to get the output (not sure it's even reading the right amount of chars or doing anything like the correct thing).
elif datatype == 0x01: #0x01 is an unknown data type... return self.read_chars(3)
Still. You can see that it points to the s2ml, s2mv and s2ma files.
using System; using System.IO; using System.Security.Permissions;
namespace watcher { public class Watcher {
public static void Main() { Run(); }
[PermissionSet(SecurityAction.Demand, Name = "FullTrust")] public static void Run() { // Create a new FileSystemWatcher and set its properties. FileSystemWatcher watcher = new FileSystemWatcher { Path = ".", NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite | NotifyFilters.FileName | NotifyFilters.DirectoryName, Filter = "*.*", IncludeSubdirectories = true, };
watcher.Changed += new FileSystemEventHandler(OnChanged); watcher.Created += new FileSystemEventHandler(OnChanged); watcher.Deleted += new FileSystemEventHandler(OnChanged); watcher.Renamed += new RenamedEventHandler(OnRenamed);
watcher.EnableRaisingEvents = true;
Console.WriteLine("Press \'q\' to quit the sample."); while (Console.Read() != 'q') ; }
private static void OnChanged(object source, FileSystemEventArgs e) { // Specify what is done when a file is changed, created, or deleted. Console.WriteLine("File: " + e.ChangeType + " " + e.FullPath); }
private static void OnRenamed(object source, RenamedEventArgs e) { // Specify what is done when a file is renamed. Console.WriteLine("File: {0} renamed to {1}", e.OldFullPath, e.FullPath); } } }
You guys are amazing. I am only a laymen but im having a great time tracking your progress here. I have a good feeling that what you achieve here will become very useful on future data mining too.
Prillan mentioned that "The problem is that almost all communication is encrypted." One thing that is probably not encrypted are the communications between the SC2 client and its Scaleform/Flash GFX UI. If anyone knows how to get in there, that seems like natural point from which to get a commanding view of the terrain.
I've seen the flash code inside one of the MPQs. gfx files IIRC.
On April 23 2012 19:32 dsjoerg wrote: Prillan mentioned that "The problem is that almost all communication is encrypted." One thing that is probably not encrypted are the communications between the SC2 client and its Scaleform/Flash GFX UI. If anyone knows how to get in there, that seems like natural point from which to get a commanding view of the terrain.
I've seen the flash code inside one of the MPQs. gfx files IIRC.
I looked into it. The .gfx files is only templates that the client fills. We probably need the client code :/
Can anyone upload some of the .s2gs files with the corrisponding basic info (like workers built and average unspent resources)? I'm at work with no access to sc2.
EDIT: Or just the file name / info, since it looks like we can request it from b.net servers
On April 23 2012 23:01 SirPsychoMantis wrote: Can anyone upload some of the .s2gs files with the corrisponding basic info (like workers built and average unspent resources)? I'm at work with no access to sc2.
EDIT: Or just the file name / info, since it looks like we can request it from b.net servers
I'm kinda reluctant to dive into this at all - the hash retrieval, not the s2gs reversing, that is.
I've checked through the traffic preceeding the s2gs GET and it appears to be typical battle.net 2.0 protocol chatter, most of which has been reversed elsewhere. It's probably possible to get into this somehow, of course, but Blizzard isn't friendly to people attempting to reverse their protocols or implementing anything that makes use of it, as we should all know.
And since every other file in the cache, including the index, look clean it's pretty safe to assume that it's somewhere in the payload after the auth - and again, I'd rather not dive into that, especially in public.
I've also taken a look through all the web frontend stuff on battle.net, since they're at least displaying the match history there, but to no avail.
imho, the best bet would be to grab Morhaime on an upcoming Blizzard event and shake him until his team prioritizes a SC2 API. They just recently confirmed a D3 API (http://us.battle.net/wow/en/forum/topic/4217171883#10) and shortly before that confessed that while they'd like to do a SC2 API, it's not happening currently.
They seem to be slacking on these fronts ..I remember when I was racing Thott, reading from WoW memory after the savedvariables got cut in one of the beta pushes. Took them more than 5+ years, scraping terrabytes off the armory, until they had an API going. And all that while they have the massively popular WoW UI community going, necessarily realizing that third party stuff is 100% win-win. Gah. The lengths through which we have to go to obtain these bits of data is excruciating sometimes.
On April 23 2012 23:01 SirPsychoMantis wrote: Can anyone upload some of the .s2gs files with the corrisponding basic info (like workers built and average unspent resources)? I'm at work with no access to sc2.
EDIT: Or just the file name / info, since it looks like we can request it from b.net servers
I did. Look at the page 2.
Not sure how I completely missed that, thanks. I'll do some poking.
Edit : I put values of 1: and 2: in Hexadecimal, its easier to get values like this
I think 0x84 is for drones, 0x86 for overlords the 0: is the supply (7,8,9,10) the 1: not sure... the ending A might be the total supply (10) the 2: I think is the time where 0x1C = 00:02 0x32 = 00:03 0x49 = 00:05 0x11C = 00:18 0x12F = 00:19 0x167 = 00:22 0x1C4 = 00:28 0x1DE = 00:30 0x276 = 00:39 0x2C4 = 00:44 0x316 = 00:49 0x378 = 00:56 0x38C = 00:57
anyone to get the general formula ?
Edit : Found ! i's in 16th of seconds ! 0x1DE = 478 and 478/16 = 29.875 so 00:30
How's it going? I've been trying to parse it but it seems to be incomplete.. Some units are missing.
Also, any idea what the 2nd structure contains? Lots of entries like this:
After carefully recording the times of various actions I took in the SC2 client, and comparing to the timestamps of files in my Cache, I don't think any of the files in the Cache contain a usable reference to an s2gs file.
Try it yourself! If any of you are trying this on OS X I'd be happy to share the command I used for retrieving the cache files in timestamp order.
In particular what you should try is pulling up a friend's profile, then their Match History, then hover your mouse over the magnifying glass, and then click on the match. What I found, and let me know if you find differently, if you do it carefully and make a note of the times of your actions, is that the s2gs is retrieved when you hover your mouse over the magnifying glass. Prior to that, nothing else you've done with the friend causes any files to be retrieved into the cache.
To me, this strongly implies that the s2gs filename must be being sent over the wire, but not stored in any cache file.
On April 24 2012 11:18 dsjoerg wrote: To me, this strongly implies that the s2gs filename must be being sent over the wire, but not stored in any cache file.
Not only that, but the HTTP requests to *.depot.battle.net are also part of the general Battle.net 2.0 protocol. Authentication and Warden utilize that CDN in all current Blizzard games. The only thing obviously referenced are the map information files that are in the index. As I said above, dangerous territory to get into. Let's find a blizzard rep, a blunt object and get an official API from him
I just tought about this : imagine if you could at the start of a ladder game get all your opponent previous build orders of his match history against your race with s2gs files, you could metagame your opponent lol. (oh your last 5 games against my race you did fast expand, then i will fast expand too).
I made a hacky prototype demonstrating how to get the hash files and automating a little of the SC2 client. It's a windows C#.NET app that takes a character name and code as input, and outputs the .s2gs file locations for their last 50 games.
Obviously I don't expect people to actually use the tool by itself, but you can take the code from it and perhaps expose a web service interface to it or integrate it with other tools etc.
The client automation is a very brittle proof of concept - it requires that SC2 be run in windowed mode and positioned in the top left of your screen. If the client lags more than it expects, it'll get confused since it doesn't bother trying to figure out what state the client is in before moving to the next step. Adding that functionality wouldn't be hard, you could just take samples of critical regions of the screen and compare average color values, for example.
On April 27 2012 19:46 Prillan wrote: Nice. But how does it trigger the download? By just looking at the match history?
Yes. Just to clarify though, the download in this case is a ~5 kilobyte payload that just has references to the .s2gs files on their CDN. To download the actual s2gs files, you'd still have to make the http requests (which is fast and trivial so I didn't include it).
I read all of this and still dont understand what is so great about it. What could you possibly get from this that you caouldnt get from a replay? You can get you spending from a replay.
On April 27 2012 19:46 Prillan wrote: Nice. But how does it trigger the download? By just looking at the match history?
Yes. Just to clarify though, the download in this case is a ~5 kilobyte payload that just has references to the .s2gs files on their CDN. To download the actual s2gs files, you'd still have to make the http requests (which is fast and trivial so I didn't include it).
Ok, looked at the source. How did you know where in the memory to look? ^^
On April 27 2012 20:20 treekiller wrote: I read all of this and still dont understand what is so great about it. What could you possibly get from this that you caouldnt get from a replay? You can get you spending from a replay.
No, actually you can't. Stats not possible to get from replays (might be a incomplete list): * Resource Collection Rate * Average Unspent Resources * Workers Created (you can get pretty close with the replay) * Army Value over time (graph) * Income over time (graph) * Units Killed * Structures Razed
Ok, looked at the source. How did you know where in the memory to look? ^^
Well I had a basic understanding of how the client caches the interface data in memory from when I was pulling ladder data from it for starcraftrankings.com - the rest was just taking hex dumps/diffing them/guessing and checking a little. This data was particularly easy since they always have an 8 byte prefix and all you have to do is take the 32 bytes immediately following it.
On April 27 2012 20:20 treekiller wrote: I read all of this and still dont understand what is so great about it. What could you possibly get from this that you caouldnt get from a replay? You can get you spending from a replay.
Nope. A replay file contains only the match information (players, maps, etc.) and user input actions. It does not contain any processed information about the game itself.
Spending, resource collection rate and the other info in score screen all require SC2 game engine to be calculated accurately. The replay data contains a lot of noise and simulating even the simplest things, such as build order from replay is pretty much impossible to do without running SC2.
Ok, looked at the source. How did you know where in the memory to look? ^^
Well I had a basic understanding of how the client caches the interface data in memory from when I was pulling ladder data from it for starcraftrankings.com - the rest was just taking hex dumps/diffing them/guessing and checking a little. This data was particularly easy since they always have an 8 byte prefix and all you have to do is take the 32 bytes immediately following it.
I made some additions to your code ^^. I think you should have a option of skipping the macro process to allow the user to navigate by himself to the match history screen and also automate the downloading process as well. (I could do it but Avengers will take up my time today ^^)
On April 27 2012 22:22 Prillan wrote: I made some additions to your code ^^. I think you should have a option of skipping the macro process to allow the user to navigate by himself to the match history screen and also automate the downloading process as well. (I could do it but Avengers will take up my time today ^^)
Thanks for the pull request, merged! Meh, the code is already separate enough to just do the memory reading if you want (just comment out the call to SC2Macros.openMatchHistory()). You should add the support for sending the http requests if you want it
On April 27 2012 22:25 dsjoerg wrote: Gibybo, that prototype is red-hot. Nice work!
On April 27 2012 22:22 Prillan wrote: I made some additions to your code ^^. I think you should have a option of skipping the macro process to allow the user to navigate by himself to the match history screen and also automate the downloading process as well. (I could do it but Avengers will take up my time today ^^)
Thanks for the pull request, merged! Meh, the code is already separate enough to just do the memory reading if you want (just comment out the call to SC2Macros.openMatchHistory()). You should add the support for sending the http requests if you want it
Just wanted to put a note in here for TLers. Prillan has been making pretty rapid progress on pulling information out of the file, interested parties can follow his/our progress on github. We'll try to keep any significant breakthroughs posted here as well.
On April 30 2012 01:48 ShadesofGraylin wrote: Just wanted to put a note in here for TLers. Prillan has been making pretty rapid progress on pulling information out of the file, interested parties can follow his/our progress on github. We'll try to keep any significant breakthroughs posted here as well.
Indeed I have. But now I need help with parsing the build orders. RobiTL was working on this and I haven't been able to recreate his progress.
EDIT: Btw, come by #sc2reader on FreeNode and help us, ask questions or just say hi ^^
Not sure how useful it is in the grand scheme of things, or if this has been worked out already but:
Looking at the first block of data, it seems lobby data is in [5] and [6]. [5] contains all the possible settings (and the defaults in section 8 of that?), and [6] indexes the actual values.
I noticed that "build order" was one of the things you guys were trying to find with this. This site seems to be able to extract build orders out of the replay file itself: http://starcrafttools.net/
On April 30 2012 20:10 InfCereal wrote: I noticed that "build order" was one of the things you guys were trying to find with this. This site seems to be able to extract build orders out of the replay file itself: http://starcrafttools.net/
It does seem to be able to clean up the commands nicely, has anyone checked the accuracy when dealing with queueing and click spam? If it is accurate, I'll have to speculate that they either do some crazy hope for the best resource simulation to validate the queue and eliminate spam or some speculative rate limiting on the production structures.
If the authors are around or someone knows I'd be very interested in their approach.
On May 01 2012 01:54 dsjoerg wrote: Holy moly nice work Prillan! What is not complete?
Big congratulations to you, very impressive. :D :D
Thanks!
Well, it's all in my experiment script. I'll have to write it into the sc2reader lib (shouldn't take too long).
Also, there seems to be more information in the file than the score screen shows but I have no idea what it means ^^ (and I'm not sure if it's relevant)
I'm kinda proud so I hope you guys like this as much as I do ^^
Everything is on github here: https://github.com/GraylinKim/sc2reader/tree/s2gs ShadesofGraylin will review it before merging it to master but it's possible to work with it right now. So go and write something awesome with this!
Dammit, I wish I had enough programming skills to take advantage of this. I'm begging someone to make up a SQ calculator based off information recorded in the s2gs file after each game ala sc2gears style.
On May 12 2012 15:43 Peleus wrote: Dammit, I wish I had enough programming skills to take advantage of this. I'm begging someone to make up a SQ calculator based off information recorded in the s2gs file after each game ala sc2gears style.
Thank you so much for all your work Prillan.
I'm working on integrating this into Sc2gears, but unfortunately based on a replay file you can't get the name of the s2gs file...
On May 12 2012 15:43 Peleus wrote: Dammit, I wish I had enough programming skills to take advantage of this. I'm begging someone to make up a SQ calculator based off information recorded in the s2gs file after each game ala sc2gears style.
Thank you so much for all your work Prillan.
I'm working on integrating this into Sc2gears, but unfortunately based on a replay file you can't get the name of the s2gs file...
First, awesome that you're working on it - thank you so much for your efforts.
Is it possible to monitor the master folder or s2gs files and order by creation date to correlate with replays? Perhaps matching some unique information within the replay that can be retrieved and cross matched to the score screen in the s2gs files allowing you to match them together correctly? (I.e. player names + map) Not sure if that would be unique enough.
Just throwing ideas out there, they may be to clunky to actually implement. Thanks regardless.
On May 12 2012 15:43 Peleus wrote: Dammit, I wish I had enough programming skills to take advantage of this. I'm begging someone to make up a SQ calculator based off information recorded in the s2gs file after each game ala sc2gears style.
Thank you so much for all your work Prillan.
I'm working on integrating this into Sc2gears, but unfortunately based on a replay file you can't get the name of the s2gs file...
First, awesome that you're working on it - thank you so much for your efforts.
Is it possible to monitor the master folder or s2gs files and order by creation date to correlate with replays? Perhaps matching some unique information within the replay that can be retrieved and cross matched to the score screen in the s2gs files allowing you to match them together correctly? (I.e. player names + map) Not sure if that would be unique enough.
Just throwing ideas out there, they may be to clunky to actually implement. Thanks regardless.
I could match those, that's not the problem. The problem is that s2gs files do not magically appear on your hard drive unless you individually check match result for all games in your match history inside Sc2 (or you use an automated program for it).
On May 12 2012 15:43 Peleus wrote: Dammit, I wish I had enough programming skills to take advantage of this. I'm begging someone to make up a SQ calculator based off information recorded in the s2gs file after each game ala sc2gears style.
Thank you so much for all your work Prillan.
I'm working on integrating this into Sc2gears, but unfortunately based on a replay file you can't get the name of the s2gs file...
First, awesome that you're working on it - thank you so much for your efforts.
Is it possible to monitor the master folder or s2gs files and order by creation date to correlate with replays? Perhaps matching some unique information within the replay that can be retrieved and cross matched to the score screen in the s2gs files allowing you to match them together correctly? (I.e. player names + map) Not sure if that would be unique enough.
Just throwing ideas out there, they may be to clunky to actually implement. Thanks regardless.
I could match those, that's not the problem. The problem is that s2gs files do not magically appear on your hard drive unless you individually check match result for all games in your match history inside Sc2 (or you use an automated program for it).
Ahh ok, I understand a lot better now - thank you for taking the time to explain it to me.
The only thing I can think of now is perhaps wiresharking the connection and trying to identify the traffic sent to blizzard to request match history and simulating that after each game. I seem to remember someone perhaps trying that earlier but I don't know how it ended up.
On May 12 2012 15:43 Peleus wrote: Dammit, I wish I had enough programming skills to take advantage of this. I'm begging someone to make up a SQ calculator based off information recorded in the s2gs file after each game ala sc2gears style.
Thank you so much for all your work Prillan.
I'm working on integrating this into Sc2gears, but unfortunately based on a replay file you can't get the name of the s2gs file...
First, awesome that you're working on it - thank you so much for your efforts.
Is it possible to monitor the master folder or s2gs files and order by creation date to correlate with replays? Perhaps matching some unique information within the replay that can be retrieved and cross matched to the score screen in the s2gs files allowing you to match them together correctly? (I.e. player names + map) Not sure if that would be unique enough.
Just throwing ideas out there, they may be to clunky to actually implement. Thanks regardless.
I could match those, that's not the problem. The problem is that s2gs files do not magically appear on your hard drive unless you individually check match result for all games in your match history inside Sc2 (or you use an automated program for it).
Ahh ok, I understand a lot better now - thank you for taking the time to explain it to me.
The only thing I can think of now is perhaps wiresharking the connection and trying to identify the traffic sent to blizzard to request match history and simulating that after each game. I seem to remember someone perhaps trying that earlier but I don't know how it ended up.
Yes, there is a program for that, but that violates the terms of use, so I will not implement something like that in Sc2gears.
On May 12 2012 15:43 Peleus wrote: Dammit, I wish I had enough programming skills to take advantage of this. I'm begging someone to make up a SQ calculator based off information recorded in the s2gs file after each game ala sc2gears style.
Thank you so much for all your work Prillan.
I'm working on integrating this into Sc2gears, but unfortunately based on a replay file you can't get the name of the s2gs file...
First, awesome that you're working on it - thank you so much for your efforts.
Is it possible to monitor the master folder or s2gs files and order by creation date to correlate with replays? Perhaps matching some unique information within the replay that can be retrieved and cross matched to the score screen in the s2gs files allowing you to match them together correctly? (I.e. player names + map) Not sure if that would be unique enough.
Just throwing ideas out there, they may be to clunky to actually implement. Thanks regardless.
I could match those, that's not the problem. The problem is that s2gs files do not magically appear on your hard drive unless you individually check match result for all games in your match history inside Sc2 (or you use an automated program for it).
Ahh ok, I understand a lot better now - thank you for taking the time to explain it to me.
The only thing I can think of now is perhaps wiresharking the connection and trying to identify the traffic sent to blizzard to request match history and simulating that after each game. I seem to remember someone perhaps trying that earlier but I don't know how it ended up.
I did that but couldn't find anything. It looks like it's encrypted
On May 12 2012 15:43 Peleus wrote: Dammit, I wish I had enough programming skills to take advantage of this. I'm begging someone to make up a SQ calculator based off information recorded in the s2gs file after each game ala sc2gears style.
Thank you so much for all your work Prillan.
I'm working on integrating this into Sc2gears, but unfortunately based on a replay file you can't get the name of the s2gs file...
First, awesome that you're working on it - thank you so much for your efforts.
Is it possible to monitor the master folder or s2gs files and order by creation date to correlate with replays? Perhaps matching some unique information within the replay that can be retrieved and cross matched to the score screen in the s2gs files allowing you to match them together correctly? (I.e. player names + map) Not sure if that would be unique enough.
Just throwing ideas out there, they may be to clunky to actually implement. Thanks regardless.
I could match those, that's not the problem. The problem is that s2gs files do not magically appear on your hard drive unless you individually check match result for all games in your match history inside Sc2 (or you use an automated program for it).
I think people would still appreciate it even if it only worked on new replays
What could be built without too much trouble is a program that scans your computer's directories for s2gs files and loads them to ggtracker or into sc2gears. However this would require you to manually click around within SC2, clicking on each game in your match history in order for the relevant s2gs file to be loaded onto your computer.
On May 16 2012 03:01 dsjoerg wrote: What could be built without too much trouble is a program that scans your computer's directories for s2gs files and loads them to ggtracker or into sc2gears. However this would require you to manually click around within SC2, clicking on each game in your match history in order for the relevant s2gs file to be loaded onto your computer.
Actually, hovering over the magnifying glass is enough.
On May 12 2012 16:02 Dakota_Fanning wrote: I'm working on integrating this into Sc2gears, but unfortunately based on a replay file you can't get the name of the s2gs file...
Usually the score screen is loaded after a game really quickly, so it'd be kinda unavoidable to get the s2gs downloaded after most games though, wouldn't it? I guess it'd be problematic for team games though, since I don't think you can get the sg2s file until the team game is over (which takes longer than just the first person leaving like in 1v1s)
On May 12 2012 16:02 Dakota_Fanning wrote: I'm working on integrating this into Sc2gears, but unfortunately based on a replay file you can't get the name of the s2gs file...
Usually the score screen is loaded after a game really quickly, so it'd be kinda unavoidable to get the s2gs downloaded after most games though, wouldn't it? I guess it'd be problematic for team games though, since I don't think you can get the sg2s file until the team game is over (which takes longer than just the first person leaving like in 1v1s)
After a game no s2gs file is saved on your computer. At the end of a game the Score screen is displayed from in-memory data. S2gs files are only saved to your computer if you browse the match history.
On May 16 2012 05:46 skeldark wrote: i checked allready but better ask you guys. Is there any information about the changepoints or player points in the file?
No.
It has nothing to do with the game. The points you see right after the game is not showed if you enter the score screen from the match history.
On May 12 2012 16:02 Dakota_Fanning wrote: I'm working on integrating this into Sc2gears, but unfortunately based on a replay file you can't get the name of the s2gs file...
Usually the score screen is loaded after a game really quickly, so it'd be kinda unavoidable to get the s2gs downloaded after most games though, wouldn't it? I guess it'd be problematic for team games though, since I don't think you can get the sg2s file until the team game is over (which takes longer than just the first person leaving like in 1v1s)
After a game no s2gs file is saved on your computer. At the end of a game the Score screen is displayed from in-memory data. S2gs files are only saved to your computer if you browse the match history.
Oh OK. It would be sending that information to the battle.net server though. I just presumed it would save the file when doing that. It only makes sense to cache the hsitory of games one's played. I suppose that's why battle.net is so damn laggy — inefficient programming. I'm Not sure if it would be in the same format when the info is being sent, but still I assume that's probably not something you'd want to do (packet reading).
On May 16 2012 05:46 skeldark wrote: i checked allready but better ask you guys. Is there any information about the changepoints or player points in the file?
No. It has nothing to do with the game. The points you see right after the game is not showed if you enter the score screen from the match history.
True, but technically the point change is still shown in the match history screen still (even though not on the score screen), so it's not a huge difference, even though the answer's still 'no'.
On May 12 2012 15:43 Peleus wrote: Dammit, I wish I had enough programming skills to take advantage of this. I'm begging someone to make up a SQ calculator based off information recorded in the s2gs file after each game ala sc2gears style.
Thank you so much for all your work Prillan.
I'm working on integrating this into Sc2gears, but unfortunately based on a replay file you can't get the name of the s2gs file...
First, awesome that you're working on it - thank you so much for your efforts.
Is it possible to monitor the master folder or s2gs files and order by creation date to correlate with replays? Perhaps matching some unique information within the replay that can be retrieved and cross matched to the score screen in the s2gs files allowing you to match them together correctly? (I.e. player names + map) Not sure if that would be unique enough.
Just throwing ideas out there, they may be to clunky to actually implement. Thanks regardless.
I could match those, that's not the problem. The problem is that s2gs files do not magically appear on your hard drive unless you individually check match result for all games in your match history inside Sc2 (or you use an automated program for it).
Ahh ok, I understand a lot better now - thank you for taking the time to explain it to me.
The only thing I can think of now is perhaps wiresharking the connection and trying to identify the traffic sent to blizzard to request match history and simulating that after each game. I seem to remember someone perhaps trying that earlier but I don't know how it ended up.
Yes, there is a program for that, but that violates the terms of use, so I will not implement something like that in Sc2gears.
The Terms of use is a joke.
sc2gears violates the terms of use! Read them careful, you are not allowed to collect information about the client. With this text you break it in the second you install sc2.
EPT
User:Prillan/s2gs
)
rillan/s2gs">
