| 01:25.15 | seebs | Okay, that's an amusing special case. |
| 01:25.55 | seebs | serialized inventory: 25KB. Compressed: 6.2KB. Compressed, after using that EXACT string to warm up the compressor's cache: 205 bytes. |
| 01:26.18 | seebs | This doesn't exactly scale usefully, but it is amusing to me anyway. |
| 01:26.57 | Imhothar | 205 bytes? |
| 01:27.33 | seebs | Yup. |
| 01:28.14 | Imhothar | are you sure? |
| 01:28.23 | seebs | Pretty sure. I have a working test framework for this. |
| 01:28.42 | seebs | The idea is: You have data (the actual payload) and header (a hunk of text intended to "hint" to the compressor). |
| 01:29.09 | seebs | You create a deflater, then do header_bits = deflate(header, 'sync'); data_bits = deflate(data, 'finish') |
| 01:29.23 | seebs | Then you create an inflater, and inflate header_bits first, then data_bits. |
| 01:29.41 | seebs | The result of the second inflation will be exactly the same bytes as data. |
| 01:30.17 | seebs | So if you had two machines which had the *same* header data available, you only need to transmit data_bits to recreate data, but data_bits can be smaller than it would have been if header/header_bits hadn't been involved. |
| 01:30.34 | seebs | For reasonable cases, I can get a 1-2% decrease on large data sets, 5-10% on small ones. |
| 01:30.49 | seebs | But it occurred to me that it would be amusing to try this using the same exact strings for data and header. |
| 01:31.37 | Imhothar | uhm well if you used the same payload for header and data then the second part is simply a reference to the previous, as it's all part of the same compression stream |
| 01:31.40 | seebs | Interestingly, it seems to be partially specific to that data set -- if I do it with all bank slots, I get 48k compressing to either 10k or 9k. |
| 01:31.42 | seebs | Yes. |
| 01:31.53 | seebs | I do not think it is a *useful* test, it's just amusing to me. |
| 01:32.32 | seebs | What I find particularly interesting is that using an empty string as a header seems to consistently save about 2 bytes. |
| 01:33.06 | Imhothar | now the 205 thing fits in my head |
| 01:33.29 | seebs | But it's also interesting to me that, with much larger tables, the savings of the "exactly the same string" get tiny on larger strings. I think the 25k case must fit inside some particular cache size or something. |
| 01:34.08 | Imhothar | actually the point is simply that larger datasets tend to have much more repetitions in them |
| 01:34.49 | seebs | Could be, but... 25k -> 205 bytes is a HECK of a lot more impressive than 48k going from 10k to 9k of compressed output. |
| 01:35.32 | Imhothar | that works only because the 205b are nothing but references into the code tree that you created in the header |
| 01:35.46 | seebs | Right. |
| 01:36.09 | Imhothar | so in truth it's 50k -> 9,2k |
| 01:36.13 | seebs | But when I go to 48k, it should in principle be still nothing but references into that code tree, but it's suddenly barely smaller. |
| 01:36.16 | Imhothar | and 25k -> 9k |
| 01:36.27 | seebs | No, I'm not counting the bytes in the header. |
| 01:36.33 | Imhothar | you have to |
| 01:36.45 | seebs | Not for what I'm testing. |
| 01:37.01 | seebs | I am playing with the *purely theoretical* question of how the header affects the number of additional bytes needed to compress a string. |
| 01:37.21 | seebs | I don't care, at all, how large the header or compressed header is, for my purposes; I just want to experiment and see what happens. |
| 01:37.34 | seebs | If there is no header pass at all, and I just do a raw compression, I get 25k->6k, 48k->10k. |
| 01:37.41 | Imhothar | that really just depends on how "accurately" the header represents the repetitions embedded in the data part |
| 01:38.01 | seebs | If I use the *exact same* string as the header, then I get 25k->205 bytes, 48k->9k. |
| 01:38.27 | seebs | So the header is coming out to 6k in the 25k case, and 10k in the 48k case. Which I am ignoring because it's not really relevant to any real-world use case; this is entirely artificial. |
| 01:38.46 | Imhothar | the distance of back references is rather limited |
| 01:39.12 | Imhothar | well, hmm, that's not entirely true |
| 01:39.37 | seebs | Well, I got the idea to ignore the separate compression phase, and just compress x..x and y..y |
| 01:39.51 | seebs | Which got me 50,154->6,591 and 97,090->19,296. |
| 01:40.49 | seebs | So I think there's some size at which some useful optimization for small but highly duplicative data sets goes away. |
| 01:41.34 | Imhothar | the compression window is usually variable, though all we got is just 1-9 |
| 01:41.40 | Imhothar | not much control |
| 01:41.41 | seebs | Yeah. Oh, hey, here's a data point. |
| 01:41.53 | seebs | I split a bunch of stacks up so my inventory became 33k of text instead of 25k of text. |
| 01:42.01 | seebs | 33k -> 6.5k, 66k -> 12k. |
| 01:42.03 | Imhothar | oh no wait that's the search intensity, not the data window |
| 01:42.12 | seebs | So it is just a little too big for whatever the really cool effect was. |
| 01:42.28 | Imhothar | I suspect the window size |
| 01:42.41 | seebs | Yeah. Because at 29k, it's 6393 for one, 6753 for two. |
| 01:42.49 | seebs | So right around 32k, *boom*. |
| 01:44.32 | seebs | Yeah, 32878 doesn't get the ludicrous savings, 32490 does. |
| 01:48.42 | *** join/#riftuidev delphian (~null@pool-96-251-119-68.lsanca.fios.verizon.net) |
| 01:54.18 | Imhothar | in theory, it might be possible to construct a header and insert it at precomputed points in the stream to reduce the total size, but that's a real challenge |
| 01:56.29 | seebs | Yeah. |
| 01:56.51 | seebs | I was mostly thinking that for the RP addon, I might be able to improve things by caching things that are going to be in nearly every hunk of data. |
| 01:58.18 | seebs | Like, say, the names of all the fields, or something. |
| 02:20.08 | Imhothar | hmm with a struct of arrays appraoch you don't need field names |
| 02:22.01 | seebs | For that addon, there's only one item at a time -- not much point in working really hard to improve that. |
| 02:22.11 | seebs | It was inventory and quests where I was starting to think about making things smaller. |
| 02:24.18 | seebs | I was trying to think of a good way to send information about quest progress back and forth between party members. |
| 02:32.36 | Noshei | why not just ask zorba to make that available to us |
| 02:32.48 | Noshei | the data is already being passed within the game |
| 02:39.39 | seebs | Huh, point. |
| 02:50.22 | seebs | That's odd. There's a thing that looks a little like a red shiny artifact just sort of hovering in mid air in Granite Falls. Not selectable or clickable or anything. |
| 07:22.04 | NuSuey | morning/evening guys! :) |
| 08:15.03 | Ivnedar | hiya :) |
| 12:59.17 | Ivnedar | 14 manning Crucia! |
| 13:47.24 | *** join/#riftuidev Imhothar (miranda@ppp-188-174-74-147.dynamic.mnet-online.de) |
| 13:56.48 | *** join/#riftuidev Wildtide (0fe3b94b@gateway/web/freenode/ip.15.227.185.75) |
| 15:59.49 | ZorbaBeta | you'd be surprised how little information is passed to the rift client :) if it's not explicitly visible in the UI, it's probably not being sent to the client |
| 16:43.14 | *** join/#riftuidev g0urra (~g0urra@unaffiliated/gourra) |
| 18:54.36 | seebs | ZorbaBeta: The client can show how many nearby group members are on a quest, is that information available to the client? I'm trying to figure out how to track shared quest status remotely efficiently, and it occurs to me that it may already be being done. |
| 21:17.34 | *** join/#riftuidev odine (odine@cpe-74-77-168-188.buffalo.res.rr.com) |
| 21:17.51 | odine | afternoon/evening/etc |
| 21:19.27 | odine | i thought it was something of thel ikes where it might be ok if we force the limiter option we have to set amt |
| 21:19.34 | odine | and no, what kind of weird tings? |
| 21:20.13 | odine | so were u? |
| 21:22.05 | seebs | ... mistells? |
| 21:22.13 | seebs | I have no context for "weird tings" |
| 21:22.23 | odine | ya Baanano had a derp moment |
| 21:23.07 | Baanano | dunno, my irc client showed a private message window :S |
| 21:23.12 | odine | weird |
| 21:23.26 | odine | oh well, shit happens! |
| 21:23.38 | odine | hey zorba when ur around give me or baanano a shout |
| 22:18.04 | odine | i should prolly take a break rom addons and farm |
| 22:18.12 | odine | 1plat is not gonnna cut it if imma be raiding this week |
| 23:04.00 | *** join/#riftuidev g0urra_ (~g0urra@unaffiliated/gourra) |