| 00:01.19 | *** join/#maemo LjL (~ljl@unaffiliated/ljl) |
| 00:20.01 | *** join/#maemo infobot (ibot@rikers.org) |
| 00:20.01 | *** topic/#maemo is Welcome to #maemo http://maemo.org/intro/ | N9/harmattan related questions please in #harmattan ! | "firmware" see ~lf ~tabletsdev ~combined ~emmc ~flasher sdk: ~sb | http://wiki.maemo.org/Community_Council | Source: http://wiki.maemo.org/Sources | Chanlog: http://mg.pov.lt/maemo-irclog | #maemo-ssu is where the (few) devels and maintainers of the maemo-future meet |
| 01:42.53 | *** part/#maemo Wikiwide (~Wikiwide@Maemo/community/ex-council/Wikiwide) |
| 02:08.13 | *** join/#maemo Wikiwide (~Wikiwide@Maemo/community/ex-council/Wikiwide) |
| 02:11.28 | Wikiwide | el-v1.db is borked again. Wrong nick for some quotes. Will have to SQL Update Where again, to fix it. Somewhen in offline. |
| 02:19.23 | *** join/#maemo Vajb (~Vajb@85-76-75-251-nat.elisa-mobile.fi) |
| 02:39.01 | Maxdamantus | doesn't think he's had that corrupted since switching to ext4 around two years ago |
| 02:57.41 | *** part/#maemo Wikiwide (~Wikiwide@Maemo/community/ex-council/Wikiwide) |
| 02:58.44 | *** join/#maemo pagurus` (~user@p50837356.dip0.t-ipconnect.de) |
| 03:11.31 | *** join/#maemo Wikiwide (~Wikiwide@Maemo/community/ex-council/Wikiwide) |
| 03:12.21 | Wikiwide | How large is your el-v1.db? Mine is 50MB+ |
| 03:25.51 | Maxdamantus | Mm .. mine's only 833 K |
| 03:31.45 | Wikiwide | Do you use telepathy-idle for IRC? Or something else? |
| 03:32.14 | Wikiwide | Because that's the primary source for "events" in my el-v1.db |
| 03:34.01 | *** join/#maemo povbot (~supybot@office.pov.lt) |
| 03:44.49 | Maxdamantus | No, I just ssh into my computer at home, which is always running irssi in tmux. |
| 03:51.47 | Wikiwide | Fun :-) Anyway, telepathy-idle should speak with NickServ more, because the way it handles nicks makes it difficult to work across nick changes, as well as integrate with Contacts. |
| 04:05.34 | Maxdamantus | O_o |
| 04:06.33 | Maxdamantus | so it messages nickserv `info ..` for lots of nicks and tries to parse out the account name? |
| 04:07.03 | Wikiwide | Nay, it doesn't speak with NickServ at all atm. |
| 04:07.14 | Maxdamantus | imagines the WHOIS information on freenode would be better to deal with .. in either case, the logic is kind of network-specific. |
| 04:07.58 | Maxdamantus | Oh, you're saying it doesn't spak with nickserv, but should. |
| 04:08.18 | Wikiwide | So I have to keep "svetlana", "svetlana-gry", "gry", "gry_" and "evilgry" in Address Book, and it isn't perfect at all. |
| 04:09.11 | Wikiwide | Because 1) Gry may use yet-another-nick; 2) somebody else can use one of her nicks, especially not protected ones (like gry_ ) |
| 04:10.40 | Wikiwide | Also, it doesn't notice servers. It thinks that gry on freenode, moznet and whatever server has to be the same person, regarding integration with Contacts at least. |
| 04:43.21 | *** join/#maemo Wikiwide (~Wikiwide@Maemo/community/ex-council/Wikiwide) |
| 04:55.29 | *** join/#maemo DocScrutinizer05 (~saturn@openmoko/engineers/joerg) |
| 05:23.59 | Maxdamantus | Presumably another good way of avoiding corruption would be to just append messages to log files. |
| 05:24.21 | *** join/#maemo Smily (~SmilzOrg@ip5f5bf6f4.dynamic.kabel-deutschland.de) |
| 05:25.45 | Maxdamantus | Hmm .. just wondering, is your filesystem mounted with data=ordered? Mine is .. presumably there isn't a practical way for sqlite to maintain consistency without it. |
| 05:26.10 | Wikiwide | How to check? |
| 05:26.18 | Maxdamantus | cat /proc/mounts |
| 05:26.34 | Maxdamantus | Look for the line mentioning your main ext3 mount |
| 05:26.54 | Maxdamantus | Probably /home |
| 05:27.20 | Maxdamantus | /dev/mmcblk0p2 /home ext4 rw,noatime,nodiratime,commit=20,barrier=1,data=ordered 0 0 |
| 05:27.37 | KotCzarny | or just finding a way to minimize that db |
| 05:27.57 | Wikiwide | /dev/mmcblk0p2 /home ext3 rw,noatime,nodiratime,errors=continue,commit=1,data=writeback 0 0 |
| 05:28.27 | Maxdamantus | Mm .. I suspect that should really be ordered. |
| 05:28.28 | Wikiwide | How do I change from writeback to ordered? And what's the difference? |
| 05:28.49 | Maxdamantus | wonders if that's something he changed at about the time he switched to ext4 |
| 05:29.24 | *** join/#maemo ceene (~ceene@collins.gmr.ssr.upm.es) |
| 05:29.43 | Maxdamantus | writeback refers to a writeback cache, the important part is that there isn't much guarantee about what order stuff within files is written in. |
| 05:29.50 | Wikiwide | 21.2011.38-1Tmaemo11+thumb0 |
| 05:30.42 | Maxdamantus | with data=ordered, if a process performs two writes to a file and the system crashes, it won'at be left in a state where nthe second write seemed to happen but the first didn't. |
| 05:31.19 | Wikiwide | /me just wants to check whether it is typical for CSSU-thumb to have data=writeback... Or is it just-me problem? |
| 05:31.54 | Maxdamantus | I think it's probably typical. I probably would have changed it myself if it were the default. |
| 05:32.47 | Maxdamantus | anyway, to change it you need to modify some scriupt in /etc so it doesn't generate fstab on boot, then you can just modify the fstab |
| 05:33.27 | Maxdamantus | or else modify the script so it adds data=ordered |
| 05:33.38 | KotCzarny | mine has writeback too |
| 05:33.41 | KotCzarny | (cssu-thumb) |
| 05:33.57 | Wikiwide | Wouldn't it be better to modify script to generate data=ordered fstab ? |
| 05:34.50 | Maxdamantus | /etc/default/mount-opts |
| 05:35.39 | Maxdamantus | Maybe. It seems kind of silly to me generating the same fstab every boot though anyway. |
| 05:36.31 | Wikiwide | In case it gets damaged somehow? |
| 05:36.39 | KotCzarny | o.O |
| 05:37.23 | KotCzarny | for dynamic mounts it should be written as an udev rule |
| 05:37.30 | KotCzarny | not rewriting fstab O.o |
| 05:38.26 | Wikiwide | Is /home a dynamic mount ? |
| 05:39.40 | KotCzarny | unless that script only runs once |
| 05:41.02 | Maxdamantus | Actually, I'm not even mounting based on that fstab .. I have a script that mounts it before preinit |
| 05:42.09 | Wikiwide | No idea. /etc/default/mount-opts ext3_opts="data=writeback" |
| 05:47.22 | Maxdamantus | Eventually we should all just be using CoW filesystems, which work perfectly well with writeback caching (as long as there's an underlying barrier mechanism) |
| 05:48.08 | Maxdamantus | otherwise you need to do journalling, but noone uses data journalling on ext (though it is supported) |
| 05:48.35 | Maxdamantus | (journalling = write everything twice) |
| 05:53.39 | Wikiwide | What is CoW? |
| 05:53.56 | KotCzarny | copy-on-write |
| 05:59.14 | kerio | sadly everyone seems to be fucking it up, as of late :< |
| 05:59.29 | kerio | apple's brand new apfs has no data checksums |
| 06:00.24 | kerio | how do you go "yep, this zfs thing is good, i'ma copy it but i'll remove the best feature" |
| 06:01.09 | KotCzarny | maybe its patented :P |
| 06:12.06 | *** join/#maemo dafox (~dafox@92.69.221.187) |
| 06:28.13 | Maxdamantus | Does their apfs still do unicode normalisation in the filesystem? |
| 06:28.37 | Maxdamantus | Oh good, it doesn't. |
| 06:28.51 | Maxdamantus | So I guess it's no longer worse than Microsoft. |
| 06:31.11 | *** join/#maemo chainsawbike (~chainsawb@unaffiliated/chainsawbike) |
| 06:37.50 | kerio | Maxdamantus: no but they moved unicode normalization in half the APIs |
| 06:38.03 | kerio | so you can have two files with names that normalize to the same thing |
| 06:38.21 | kerio | but you can only ever see and access one in guis |
| 06:40.52 | KotCzarny | whoever thought that using anything else than ascii for file names should sort files in hell |
| 06:40.55 | *** join/#maemo dmth|intevation (~dmth|inte@2001:470:747a:2a:6af7:28ff:fe64:1ded) |
| 06:59.33 | *** join/#maemo spiiroin (~spiiroin@2001:998:2a:dead:4499:3559:7a5d:8a22) |
| 07:02.03 | Wikiwide | Unicode is Unicode. At long as everybody uses the same encoding, it shouldn't a problem. Anyway, file name doesn't matter for tracker-ed files. |
| 07:02.23 | KotCzarny | sure, and then are no-printable characters |
| 07:02.32 | KotCzarny | and whole lots of problems |
| 07:13.18 | Wikiwide | Hmm, I understand that most programs don't treat filenames as "text document" aka "let's change text direction, breakline, and so on", but it should be doable. |
| 07:13.28 | Wikiwide | Granted, non-standard. |
| 07:16.59 | *** join/#maemo florian_kc (~fuchs@Maemo/community/contributor/florian) |
| 07:17.35 | Maxdamantus | Well, text documents aren't stored as some magical normalised unicode object in the filesystem. |
| 07:17.42 | Maxdamantus | Whatever happens, neither should filenames. |
| 07:18.05 | Maxdamantus | They should both be treated the same way by the filesystem: a sequence of bytes. |
| 07:18.13 | KotCzarny | wikiwide: i'm not saying unicode is bad, i'm saying it shouldnt be used for file names |
| 07:18.33 | Maxdamantus | Something outside of the kernel is responsible for interpreting the bytes in some way so as to have various text directions. |
| 07:18.41 | KotCzarny | unless you want confusion, collisions and errors |
| 07:18.50 | Maxdamantus | all the kernel has is byte 0, byte 1, byte 2, .. byte n |
| 07:20.23 | Maxdamantus | I think unicode is perfectly fine for filenames, just make sure the assumptions about unicode stay in userspace, preferably at a high level that only graphical applications interact with. |
| 07:21.45 | Maxdamantus | So if you call `open(2)` on the string ".GIT", you shouldn't be able to read that file back by calling `open(2)` on the string ".git" |
| 07:21.54 | *** join/#maemo geaaru (~geaaru@host170-152-static.43-88-b.business.telecomitalia.it) |
| 07:22.55 | Maxdamantus | Otherwise this happens: https://github.com/blog/1938-git-client-vulnerability-announced |
| 07:23.45 | Maxdamantus | git shouldn't be responsible for handling arbitrary normalisations, since it's not trying to render text. |
| 07:24.03 | Maxdamantus | All it should care about is arrays of bytes, possibly terminated with 0 |
| 07:26.36 | *** join/#maemo vahe (~vahe@unaffiliated/vahe) |
| 07:26.58 | Maxdamantus | I've been wondering how OS X behaves regarding unicode normalisation across OS versions. |
| 07:27.12 | Maxdamantus | because the effects of unicode normalisation change across revisions of unicode. |
| 07:28.04 | Maxdamantus | So what happens if you transfer files back and forth between systems that do and don't use Unicode 9.0? |
| 07:29.33 | Maxdamantus | or worse, if you plug a harddrive into various versions of OS X? |
| 07:30.24 | Maxdamantus | since then you don't really get a chance to do the translations at the protocol level (which people probably don't bother doing anyway, like the CRLF crap in Windows) |
| 07:40.43 | Maxdamantus | (note on the CRLF point: protocols like FTP have a "text" mode that you're meant to use when transferring text files, which will correctly translate to/from things like CRLF form; the idea is that CRLF is just a way of encoding text, so as long as you transcode it every time you move it to a different system, there shouldn't be a problem) |
| 07:42.21 | Maxdamantus | But that's just silly. It should just be stored in the simplest way possible. |
| 07:42.35 | Maxdamantus | s/possible/plausible/ |
| 08:16.42 | *** join/#maemo L29Ah_ (~L29Ah_@31.173.83.147) |
| 08:17.17 | *** join/#maemo jskarvad (jskarvad@redhat/jskarvad) |
| 09:06.05 | *** join/#maemo N-Mi (~nicolas@office.ibrowse.com) |
| 09:06.05 | *** join/#maemo N-Mi (~nicolas@calvix/staff/N-Mi) |
| 09:29.19 | *** join/#maemo mickname (~mickname@myj6.kyla.fi) |
| 09:45.44 | auenfx4 | you mean emoji in file names isnt a good thing? :P |
| 09:46.22 | *** join/#maemo spiiroin (~spiiroin@176-93-3-139.bb.dnainternet.fi) |
| 09:46.28 | KotCzarny | we need emojis for basic unix tools! |
| 09:47.23 | auenfx4 | is there a programming language that is entirely in emojis? |
| 09:47.56 | auenfx4 | http://www.emojicode.org/ |
| 09:47.58 | auenfx4 | aparently ues |
| 09:48.09 | auenfx4 | but, i'd probably use lolcode over it still |
| 09:49.03 | KotCzarny | as if there would be a difference |
| 09:53.26 | L29Ah_ | what about lolicode |
| 09:54.31 | bencoh | -_- |
| 09:59.17 | *** join/#maemo dingen (~henk@unaffiliated/dreamer) |
| 10:13.43 | Maxdamantus | What I'm saying has no effect on whether or not you can use emojis in filenames. |
| 10:14.00 | Maxdamantus | Just that the kernel shouldn't have to know about emojis. |
| 10:14.14 | Maxdamantus | or the definition of the filesystem format. |
| 10:14.52 | Maxdamantus | Unicode already gives a really good way to represent emojis as bytes: UTF-8 |
| 10:16.22 | Maxdamantus | A lot of the time when designing software, the best way to support things is not to cram lots of features in that specifically tend to that thing. |
| 10:16.29 | Maxdamantus | but rather to be generic enough to allow it. |
| 10:17.52 | Maxdamantus | Linux filesystems are perfectly generic enough, since they all handle filenames as bytes, and UTF-8 is a very good way to represent Unicode as bytes, even versions of Unicode that came out after the filesystem was last updated. |
| 10:18.33 | Maxdamantus | That seems to be the direction that Apple's new filesystem is going, since they're presumably well aware by now that what they did was completely wrong. |
| 10:18.58 | L29Ah_ | another filesystem? |
| 10:19.22 | Maxdamantus | Yeah, APFS |
| 10:20.03 | Maxdamantus | HFS+ does two really stupid things WRT unicode: use 16-bit code units (so things are meant to be encoded in UTF-16), and do unicode normalisation at the filesystem level. |
| 10:20.10 | Maxdamantus | afaict, APFS does neither. |
| 10:21.04 | L29Ah_ | https://en.wikipedia.org/wiki/Apple_File_System#Design i'd say there are a lot of features you don't usually see at the filesystem level |
| 10:22.09 | Maxdamantus | Like what? Encryption? |
| 10:22.41 | Maxdamantus | Other than encryption, all the other things listed there should be standard features of new filesystems. |
| 10:23.00 | Maxdamantus | eg, btrfs and ZFS are relatively new filesystems that do all of those things other than encryption. |
| 10:23.09 | L29Ah_ | multikey encryption, snapshots, data deduplication |
| 10:23.16 | Maxdamantus | (ZFS might do encryption, dunno) |
| 10:23.36 | Maxdamantus | actually, ZFS might not have cloning of files .. btrfs does. |
| 10:24.07 | L29Ah_ | i don't see the reason why snapshots should be done at the filesystem level instead of the lower one (as in lvm) |
| 10:24.43 | L29Ah_ | data deduplication/CoW may have been as well (as seen in fossil + venti), yet it didn't go mainstream for some reason |
| 10:24.57 | Maxdamantus | Well, why not just do data integrity at the LVM level? |
| 10:25.27 | Maxdamantus | It's perfectly feasible for something like LVM to be CoW and provide total data ordering to arbitrary filesystems that sit on top of it. |
| 10:25.30 | L29Ah_ | data integrity is done at the hardware level :] |
| 10:25.45 | Maxdamantus | I don't trust data integrity done at the hardware level. |
| 10:26.13 | L29Ah_ | then it's time for you to use raid |
| 10:26.35 | Maxdamantus | er, when I said integrity before, I more meant "crash protection" |
| 10:26.37 | L29Ah_ | i don't see what task the extra checksums at filesystem level would solve |
| 10:26.59 | L29Ah_ | maybe you wanted journaling? |
| 10:27.10 | Maxdamantus | No, journalling is a really bad way of solving that. |
| 10:27.18 | Maxdamantus | Journalling means you have to write everything twice. |
| 10:27.27 | Maxdamantus | Which means your speed is at least halved. |
| 10:27.32 | Maxdamantus | CoW is a much better way to do it. |
| 10:27.34 | L29Ah_ | it doesn't mean that |
| 10:27.43 | Maxdamantus | Yes, it does. That's exactly what journalling is. |
| 10:27.46 | L29Ah_ | CoW doesn't impede journaling |
| 10:27.58 | Maxdamantus | Write the new stuff into a temporary space, then write it to the actual space. |
| 10:28.08 | L29Ah_ | and you don't need to store checksums for CoW |
| 10:28.14 | Maxdamantus | That way if you crash in the middle, you can ignore the stuff in the temporary space. |
| 10:28.29 | Maxdamantus | or ignore the stuff in the actual space (and copy everything from the temporary space) |
| 10:28.46 | Maxdamantus | note: practically noone actually uses full journalling on ext, only metadata. |
| 10:29.12 | Maxdamantus | The best crash protection you can get from ext without full journalling is data=ordered, which has its own performance issues. |
| 10:29.17 | L29Ah_ | well, i meant CoW for data, journaling for metadata |
| 10:30.07 | Maxdamantus | So you're writing the metadata twice? Why? |
| 10:31.10 | L29Ah_ | to avoid data loss on crashes (: |
| 10:31.18 | Maxdamantus | You get that with CoW. |
| 10:31.26 | Maxdamantus | and you don't have to write everything twice. |
| 10:31.53 | L29Ah_ | CoW over metadata? |
| 10:32.00 | Maxdamantus | Also, you mentioned RAID .. RAID is a really bad solution to this as well. |
| 10:32.03 | L29Ah_ | where do i store the metadata about metadata then? |
| 10:32.07 | Maxdamantus | It practically requires .. again .. journalling. |
| 10:32.18 | Maxdamantus | So with RAID you have to write everything twice. |
| 10:32.43 | Maxdamantus | If you have a really expensive hardware RAID controller, it might have hundreds of megabytes of really fast NVRAM to work as its journal. |
| 10:33.11 | Maxdamantus | if you're using a less expensive RAID controller or software RAID, you're just fucked in terms of crash protection. |
| 10:33.28 | Maxdamantus | Every RAID setup that doesn't have a journal is vulnerable to the write hole issue. |
| 10:33.44 | Maxdamantus | Again, you need CoW to do this without journalling. |
| 10:34.24 | Maxdamantus | Things like btrfs and ZFS offer RAID-like features (striping, parity, duplicating, etc), but using CoW instead of journalling. |
| 10:34.42 | Maxdamantus | Well, btrfs' parity mode supposedly still has the write hole, last I checked. |
| 10:35.36 | Maxdamantus | 22:32:03 < L29Ah_> where do i store the metadata about metadata then? |
| 10:35.50 | Maxdamantus | You don't store it until you know all of the metadata is stored. |
| 10:36.02 | Maxdamantus | "it" being the information in the superblocks. |
| 10:36.46 | Maxdamantus | The superblock is a small block of information that is replicated multiple times on a single device. |
| 10:36.57 | L29Ah_ | :* |
| 10:37.19 | Maxdamantus | When mounting the filesystem, you read all the instances of the superblock and look for the latest one that matches its own checksum. |
| 10:38.09 | Maxdamantus | (this is in btrfs, other CoW filesystems might have other solutions, but they'll probably be essentially the same) |
| 10:43.23 | sicelo | OT: what is currently best RSS reader for N900? |
| 10:44.04 | L29Ah_ | uses http://rss2email.blasux.ru/ |
| 10:45.10 | sicelo | i'm not looking for something web based |
| 10:45.45 | bencoh | I'd tend to say cutenews, but... |
| 10:46.00 | L29Ah_ | it is not web based |
| 10:47.09 | sicelo | bencobut ... |
| 10:48.21 | sicelo | L29Ah_: i also don't want them in my mailbox |
| 10:49.52 | bencoh | sicelo: I thought about using it with miniflux (since miniflux would prepare content remotely then provide me with a clean rss feed / clean articles) |
| 10:50.16 | bencoh | I eventually ditched the whole idea and don't use any rss reader |
| 10:50.36 | bencoh | just that cutenews seemed the most clean/efficient at that time |
| 10:50.44 | bencoh | (compared to, say, feedingit) |
| 10:51.45 | sicelo | thanks bencoh. will use cutenews then |
| 10:52.47 | sicelo | i think i do need an rss reader. stock one is fine for me, but it has problems with others |
| 11:01.37 | sicelo | what did you end up using? |
| 11:02.09 | bencoh | no rss, no articles on the go |
| 11:02.13 | bencoh | not a great loss anyway |
| 11:17.36 | Wikiwide | What's the problem with built-in rss reader? |
| 11:24.56 | sicelo | i do use it just fine most of the time - however it can't open for example https://news.ycombinator.com/rss |
| 12:09.02 | DocScrutinizer05 | Wikiwide: everything >1MB for el-v1.db is considered to bring down whole system to a grinding halt, due to sqlite accessing all that stuff and thus bringing swap hell to a system with only 250MB of RAM with only ~50MB of that available to userland |
| 12:13.43 | Wikiwide | Hmm... Interesting... My point of view is that el-v1.db should be separated into many small files (though it would be hell to maintain), so that when user wants the last 10 messages, he doesn't have to load 1million-heavy history into RAM. |
| 12:13.47 | L29Ah_ | does sqlite always read the whole database to ram? |
| 12:14.05 | L29Ah_ | also i see 100MB free on my pretty much stock maemo |
| 12:14.10 | Wikiwide | How do other programs do that? Log and load logs into GUI? |
| 12:14.25 | *** join/#maemo cyphase (~cyphase@unaffiliated/cyphase) |
| 12:14.47 | DocScrutinizer05 | Wikiwide: it's one of the known problems in maemo that Nokia devels never tested maemo for long term sustainability, neither with 1000s of contacts in database, nor for cleaning up any 'logging' files like particularly el-v1.db. You *could* write your own SQL cleanup jobs if you know SQL and understand how the db is structured |
| 12:15.49 | DocScrutinizer05 | for all those questions please refer to RDBMS-101 |
| 12:15.55 | Wikiwide | What would cleanup entail? I know basic SQL, but structure of el-v1.db is confusing. |
| 12:16.07 | DocScrutinizer05 | yes, that's a problem |
| 12:17.05 | DocScrutinizer05 | luckily I think the data isn't incremental at all, in a sense of "you can delete old records without any impact to new data" |
| 12:17.45 | Wikiwide | I don't want to delete records. |
| 12:18.00 | DocScrutinizer05 | it's probably safe to delete just records with old timestamps, out of a set of very many similar types of records |
| 12:18.13 | Wikiwide | Imagine life-logger. |
| 12:18.23 | DocScrutinizer05 | how would you shrink your el-v1.db without doing that? |
| 12:19.06 | Wikiwide | Also, some time in distant future it could be possible to implement Find, and search All Messages for a word, sender, or something. |
| 12:19.19 | DocScrutinizer05 | *shrug* |
| 12:19.55 | Wikiwide | How to shrink? Split it, either chronically or channel-y or both, and load only pieces needed. |
| 12:20.32 | DocScrutinizer05 | you can't keep 1TB of data on a 512MB uSD, and you can'tkeep the data of a 50MB el-v1.db in a el-v1.db <1MB |
| 12:20.40 | Maxdamantus | imo that data shouldn't even all be stored in sqlite. |
| 12:21.28 | Maxdamantus | Simple log files for actual conversation data seems more sensible to me .. easier to maintain/trim. |
| 12:21.33 | DocScrutinizer05 | you can mv el-v1.db el-v1_1.db any time. Instantly reboot after that! |
| 12:21.48 | Wikiwide | Mmm. Sqlite has good tools to go through data. Where and the like. Text file wouldn't be as easy to edit. |
| 12:21.50 | Maxdamantus | and don't need special tools to search .. grep should just work. |
| 12:23.03 | Wikiwide | Grep wouldn't do field search, like who-sender or which-channel or SMS-vs-IM |
| 12:23.50 | L29Ah_ | googling suggests sqlite won't load the whole database into ram unless explicitly requested to |
| 12:24.08 | Wikiwide | I think that Quassel has similar Sql database for logs, or something. |
| 12:24.09 | DocScrutinizer05 | of course not |
| 12:24.15 | Maxdamantus | grep who-sender ~/irclogs/Freenode/#maemo.log |
| 12:24.16 | DocScrutinizer05 | read RDBMS-101 |
| 12:24.33 | L29Ah_ | so i doubt it makes sense to rotate it |
| 12:24.40 | Maxdamantus | grep '^..:.. <.Wikiwide>' ~/irclogs/Freenode/#maemo.log |
| 12:24.47 | bencoh | Wikiwide: tbh when I'm looking for some information in my (say) text messages I just launch sqlite and fire a SELECT query |
| 12:25.07 | Wikiwide | Okay, RDBMS-101. Will look it up. |
| 12:25.11 | DocScrutinizer05 | I doubt this chat makes sense to me, I think I'm missing parts thanks to /ignore |
| 12:26.01 | DocScrutinizer05 | https://www.ixquick.com/do/metasearch.pl?query=RDBMS-101 :-) |
| 12:26.13 | Wikiwide | bencoh: nice! SQL SELECT aka search is a good example of how human-friendly sqlite database is. |
| 12:27.57 | DocScrutinizer05 | a RDBMS is usually b-tree or somesuch, with indizes. depending on the operation you perform, it's necessary to sometimes walk the complete dataset which means you load every part of dataset to RAM, sequentially. Since linux fs and RDBMS both keep buuffers, you will inevitably hog some RAM with that |
| 12:29.23 | *** join/#maemo vahe (~vahe@unaffiliated/vahe) |
| 12:31.09 | DocScrutinizer05 | and mere experience holds evidence that el-v1.db >1or2 MB brings device to a state that's close to "grinding halt", with delays in the range of 10 to 30 seconds to events that are supposed to be realtime, like dialer popping up on inbound call. You don't want that to happen 30s *after* inbound call started |
| 12:32.13 | Maxdamantus | has IRC logs that are gigabytes long |
| 12:32.32 | Maxdamantus | irssi doesn't care .. all it does is open them in append mode and write to them. |
| 12:33.06 | DocScrutinizer05 | bencoh: make that a script, and augemt it so it could work across multiple database files, so users could keep archive in el-v1_*.db ? |
| 12:33.10 | Maxdamantus | (on PC, not on N900, though N900 probably wouldn't suffer from having those large logs either) |
| 12:33.18 | DocScrutinizer05 | bencoh: that would be an extremely useful little app |
| 12:33.30 | bencoh | DocScrutinizer05: it probably would, yeah |
| 12:34.47 | DocScrutinizer05 | my SQL is pretty dusty, thugh I have a comprehensive understanding of how things work and should work |
| 12:35.12 | DocScrutinizer05 | IOW I could do SQL but it takes ages |
| 12:35.13 | Maxdamantus | imo if there were to be more advanced search features, it should just use the logs as a source of truth. |
| 12:35.29 | bencoh | "the logs"? |
| 12:36.06 | Maxdamantus | ie, once you've got logs that are >100 MiB long, you can generate extra search information that's stored in a cache. |
| 12:36.35 | bencoh | some kind of index? |
| 12:36.40 | Maxdamantus | If you think that search information might be corrupted, just delete it and have it regenerated. |
| 12:36.40 | KotCzarny | maxd: that's assuming world is static, ie. doesn't change |
| 12:36.52 | bencoh | KotCzarny: not necessarily |
| 12:37.02 | bencoh | see mairix, notmuch, xapian, the likes |
| 12:37.05 | Maxdamantus | KotCzarny: well, the index data would still be generated incrementally. |
| 12:37.24 | bencoh | (xapian is actually the engine. mairix/notmuch are frontends) |
| 12:37.37 | KotCzarny | there is a saying 'only cow doesnt change it's opinions' |
| 12:37.53 | bencoh | only CoW? |
| 12:37.57 | bencoh | :p |
| 12:38.03 | KotCzarny | moo :P |
| 12:38.21 | DocScrutinizer05 | ~moo |
| 12:38.21 | infobot | mooooooo! I am cow, hear me moo, I weigh twice as much as you. I'm a cow, eating grass, methane gas comes out my ass. I'm a cow, you are too; join us all! type apt-get moo. |
| 12:41.00 | DocScrutinizer05 | ((<Wikiwide> bencoh: nice! SQL SELECT aka search is a good example of how human-friendly sqlite database is.)) also a good example for the RDBMS needing to walk the *complete* dataset if it can't optimize the SELECT condition to use indizes |
| 12:42.43 | DocScrutinizer05 | and I think what wikiwide meant is "human friendly" of 4th generation languges like SQL at large |
| 12:43.23 | DocScrutinizer05 | the tricky part however starts with defining and understanding the data model |
| 12:43.45 | DocScrutinizer05 | *using* it is simple, after that |
| 12:45.12 | DocScrutinizer05 | SELECT responsetext FROM ideal-database WHERE (is-relevant-to-wikiwide = TRUE); ;-P |
| 12:46.16 | DocScrutinizer05 | now is-relevant-to-wikiwide could be a stored procedure AIUI ;-) |
| 12:46.40 | *** join/#maemo frals (~frals@Maemo/community/contributor/frals) |
| 12:47.10 | DocScrutinizer05 | still ineffective until you get an index for it |
| 12:47.36 | DocScrutinizer05 | I prefer to use grep for such stuff, generally |
| 12:48.19 | DocScrutinizer05 | XML isn't all bad for those purposes |
| 12:49.02 | DocScrutinizer05 | not compatible with simple grep usually, but still simpler and more lightweight than the usual RDBMS |
| 12:51.20 | DocScrutinizer05 | for IRC I have plaintext logfiles (not on N900) and grepping in them is a matter of milliseconds, even in the 20+MB sized ones |
| 12:52.33 | DocScrutinizer05 | anyway maemo fremantle lacks any useful means (except for SMS iirc) to clean out old records form el-v1.db |
| 12:52.53 | DocScrutinizer05 | s/form/from/ |
| 12:53.35 | DocScrutinizer05 | heck, I even don't lnow of any simple archiver or export tool for the data in there |
| 12:56.30 | Maxdamantus | I'm pretty sure grep is even going to be faster at most searches you'd want to do. |
| 12:57.05 | Maxdamantus | because of DocScrutinizer05's point; unless you're doing some operation that the schema was specifically designed to make optimal, it's probably just searching through all the rows. |
| 12:58.21 | Maxdamantus | I imagine in the best case you might be able to get faster searches for a particular time period, but if every message is logged with a timestamp and there are no clock skews, you can already do that using `look` from util-linux |
| 12:59.03 | Maxdamantus | (`look` does a line-wise binary search on a file) |
| 12:59.37 | Maxdamantus | hacked together some program himself that did that, before realising `look` already existed |
| 13:01.05 | Maxdamantus | I suspect still in that case, running an entire look process will be far quicker than simply opening an SQLite database |
| 13:03.44 | Maxdamantus | Hm, though `look` doesn't seem to have an option to just produce an offset into the file. |
| 13:04.08 | Maxdamantus | My program did that, so you could just pass the result as an argument to something like `less` to seek to the right position. |
| 13:06.35 | Maxdamantus | Oh hey, my program can actually exec less for you. |
| 13:08.42 | Maxdamantus | Funnily enough, I also have that mechanism built into my terminal. |
| 13:09.20 | KotCzarny | yeah, probably set PAGER variable in bash |
| 13:09.41 | Maxdamantus | in rxvt-unicode you can hold ctrl-shift and drag to show information about a character .. I modified my rxvt-unicode so whenever it calculates that information it does a binary search on UnicodeData.txt to show more interesting stuff. |
| 13:10.24 | Maxdamantus | No noticable latency, because each lookup takes probably less than 1 ms |
| 13:10.40 | Maxdamantus | only a 1.4 MiB file, but obviously binary search scales extremely well. |
| 13:13.58 | Maxdamantus | $ time ./findline /usr/share/unicode-data/UnicodeData.txt 000{0..9}\; >/dev/null |
| 13:13.58 | Maxdamantus | real 0m0.001s |
| 13:14.36 | Maxdamantus | https://gist.github.com/Maxdamantus/2e344f688d9715490cb0ea6dd6ffb8bd in case anyone's interested |
| 13:14.58 | DocScrutinizer05 | SELECT responsetext FROM ideal-database WHERE (is-relevant-to-wikiwide = TRUE) gets you a dozen or a few 10000s of lines/records out of a 500MB database in only a few milliseconds, while SELECT responsetext FROM the-usual-database WHERE (responsetext LIKE "this is what I search for"); takes ages and gets beat by a simple grep hands down |
| 13:15.33 | Maxdamantus | just adapted it from his urxvt code one day at work when analysing some stuff from a database dump |
| 13:23.49 | DocScrutinizer05 | database dudes have a hard time eachs single time I show them a grep is faster than their RDBMS in particular cases ;-P |
| 13:24.46 | DocScrutinizer05 | "but but but...! it's not... errr... wel you can't.... ERRRR but it's NOT SQL!" |
| 13:36.33 | *** join/#maemo Kabouik (~kabouik@158.38.1.70) |
| 13:48.14 | Maxdamantus | fun test: sha1sum <(./findline -e -i /tmp/unicodedata-sorted.txt </tmp/unicodedata-sorted.txt) /tmp/unicodedata-sorted.txt |
| 13:49.13 | Maxdamantus | (for every line in the file, looks for that line in the file using binary search) |
| 13:50.22 | Maxdamantus | about 170k independent searches per second |
| 13:51.39 | Maxdamantus | could easily be optimised, since there are particular hotspots that the binary searches all touch that could remain in memory (ie, the lines at 0.5, 0.25, 0.75, ...) |
| 14:07.27 | *** join/#maemo vahe (~vahe@unaffiliated/vahe) |
| 14:18.50 | *** join/#maemo vahe (~vahe@unaffiliated/vahe) |
| 14:37.03 | parazyd | freemangordon: amprolla is finished i believe :) |
| 14:37.23 | parazyd | should undergo some testing now by the 'big bosses' |
| 14:37.59 | parazyd | will find out more when they wake up (also got a devuan meeting tonight i think) |
| 14:38.23 | *** join/#maemo vahe (~vahe@unaffiliated/vahe) |
| 14:40.12 | *** join/#maemo Kabouik_ (~kabouik@158.38.1.70) |
| 14:45.27 | *** join/#maemo Kabouik_ (~kabouik@158.38.1.70) |
| 15:22.44 | *** join/#maemo Kabouik (~kabouik@158.38.1.70) |
| 15:35.36 | *** join/#maemo Pali (~pali@Maemo/community/contributor/Pali) |
| 15:52.51 | *** join/#maemo dafox (~dafox@92.69.217.28) |
| 15:57.41 | freemangordon | parazyd: great :) |
| 16:03.39 | *** join/#maemo jskarvad (jskarvad@redhat/jskarvad) |
| 16:12.58 | *** join/#maemo jskarvad (jskarvad@redhat/jskarvad) |
| 16:13.17 | Wizzup | parazyd: thanks for going the extra mile and getting it into devuan. onwards |
| 16:13.25 | Wizzup | s/extra/final/ |
| 16:15.04 | *** join/#maemo LauRoman (~LauRoman@5-14-84-204.residential.rdsnet.ro) |
| 16:16.50 | *** join/#maemo spiiroin (~spiiroin@178-55-107-30.bb.dnainternet.fi) |
| 16:22.08 | *** join/#maemo ff_ (~ff_@dzk135.neoplus.adsl.tpnet.pl) |
| 16:23.38 | ff_ | https://www.sqlite.org/lang_vacuum.html el-v1.db might help |
| 16:24.08 | *** join/#maemo florian (~fuchs@Maemo/community/contributor/florian) |
| 16:25.49 | ff_ | I done it, and el-v1.db is 14% smaller now, no data loss, I don't know about speed |
| 16:26.55 | bencoh | :) |
| 16:40.42 | *** join/#maemo hurrian_ (~textual@122.53.252.242) |
| 17:16.54 | *** join/#maemo Milhouse (~Milhouse@kodi/staff/milhouse) |
| 17:19.13 | *** join/#maemo dafox (~dafox@2a02:a448:c25a:1:7061:3b2a:37df:ded3) |
| 17:21.41 | sicelo | back to my rss issue - cutenews doesn't load on my N900. |
| 17:23.20 | sicelo | anyway, guess should try all of them one by one |
| 17:29.53 | sicelo | ah, silly problem. http://talk.maemo.org/showthread.php?t=95797&page=3 |
| 17:41.50 | parazyd | Wizzup: thank you for the help :) |
| 18:10.37 | sixwheeledbeast | sicelo: seems a bit odd for it to be file permissions issue. |
| 18:11.53 | sicelo | indeed, but at least on my installation that worked. i actually first adjusted the .desktop file, which didn't work |
| 18:39.41 | *** join/#maemo Venemo (~venemo@fedora/Venemo) |
| 18:49.16 | sixwheeledbeast | Thanks I'll try it later if I get a minute |
| 18:53.48 | *** join/#maemo heroux (sandroco@gateway/shell/insomnia247/x-krwainrwicivuskj) |
| 19:45.00 | *** join/#maemo how900 (~Neo900@2a03:b0c0:2:d0::f7:f001) |
| 19:45.01 | *** join/#maemo how900 (~Neo900@neo900/coreteam/communications/how) |
| 19:45.40 | *** join/#maemo parazyd (~parazyd@chat.dyne.org) |
| 19:45.40 | *** join/#maemo parazyd (~parazyd@unaffiliated/parazyd) |
| 19:46.52 | *** join/#maemo how900 (~Neo900@2a03:b0c0:2:d0::f7:f001) |
| 19:46.52 | *** join/#maemo how900 (~Neo900@neo900/coreteam/communications/how) |
| 19:47.31 | *** join/#maemo parazyd (~parazyd@chat.dyne.org) |
| 19:47.31 | *** join/#maemo parazyd (~parazyd@unaffiliated/parazyd) |
| 19:51.01 | *** join/#maemo how900 (~Neo900@2a03:b0c0:2:d0::f7:f001) |
| 19:51.02 | *** join/#maemo how900 (~Neo900@neo900/coreteam/communications/how) |
| 19:51.42 | *** join/#maemo parazyd (~parazyd@chat.dyne.org) |
| 19:51.42 | *** join/#maemo parazyd (~parazyd@unaffiliated/parazyd) |
| 20:07.22 | *** join/#maemo spiiroin (~spiiroin@188-67-77-135.bb.dnainternet.fi) |
| 20:38.46 | *** join/#maemo cyteen (~cyteen@cyteen42.plus.com) |
| 21:25.02 | *** join/#maemo brolin_empey_ (~brolin@brolin.be) |
| 21:25.38 | *** join/#maemo TriztAway (~user@ukko.aho.hk) |
| 21:26.33 | *** join/#maemo Wizzup_ (~Wizzup@a82-161-36-93.adsl.xs4all.nl) |
| 21:26.50 | *** join/#maemo sledges_ (~sleleiva@194.157.27.4) |
| 21:27.07 | *** join/#maemo enyc_ (~enyc@muddle.enyc.org.uk) |
| 21:32.54 | *** join/#maemo janus (Janus@bnc.animux.de) |
| 21:33.44 | *** join/#maemo pagurus (~user@p50837356.dip0.t-ipconnect.de) |
| 21:35.56 | *** join/#maemo Xjs|moonshine (moon@chrysostomos.aoide.de) |
| 21:36.31 | *** join/#maemo sledges (~sleleiva@unaffiliated/sledgeas) |
| 21:41.36 | *** join/#maemo ruskie (~ruskie@sourcemage/mage/ruskie) |
| 21:55.02 | *** join/#maemo sunshavi (~user@190.234.106.138) |
| 22:29.37 | *** join/#maemo Cor-Ai (~cor-ai@c-423572d5.06-922-7570702.cust.bredbandsbolaget.se) |
| 22:32.18 | *** join/#maemo NotKit (~nekit@178-45-2-157.saransk.ru) |
| 22:34.46 | *** join/#maemo TTilus (terotil@idan.ihme.org) |
| 22:35.10 | *** join/#maemo MikaT (~mtapoja@projects.sse.fi) |
| 23:54.39 | Maxdamantus | Heh .. I mentined yesterday how HFS+ does unicode normalisation and that I wondered how it handled things between systems with different versions. |
| 23:55.35 | Maxdamantus | Their solution is actually quite simple: the normalisation is just based on a particular version of unicode: 3.2, released in 2002 |
| 23:56.15 | Maxdamantus | so it simply doesn't do normalisation of newer code points. |
| 23:56.47 | Maxdamantus | because if they did .. you'd have that major issue. |
| 23:57.13 | Maxdamantus | (not that it still has major issues) |