| 00:02.44 | DocScrutinizer05 | the best way is to actually never use a maybe 10% of the SD card, by downsizing the partition(s) |
| 00:04.46 | DocScrutinizer05 | once the system wrote to all available blocks (and didn't issue a DISCARD or whatever the equivalent name on SD bus, for all the no longer needed blocks, and made sure the non-used blocks concatenate to a whole erase page), the card will significantly drop on write performance |
| 00:04.54 | *** join/#maemo sunshavi (~user@181.64.192.217) |
| 00:05.14 | Oksanaa | So, use up to 90% or so of the card. |
| 00:05.52 | DocScrutinizer05 | since the card controller can't fortune tell which pages to erase "preemptively" |
| 00:06.20 | DocScrutinizer05 | card always have a few hidden spare pages, but not that many |
| 00:06.59 | DocScrutinizer05 | Oksanaa: (90%) I mere guess by me, but sounds good |
| 00:07.26 | Oksanaa | So, if somebody made a card with high enough quantity of hidden spare pages, that would be easier on user. |
| 00:08.16 | DocScrutinizer05 | yes |
| 00:08.53 | Maxdamantus | Well, presumably it'll usually have around 73 MB per GB of spare blocks. |
| 00:09.16 | Maxdamantus | if its capacity is given in GB. |
| 00:09.44 | Maxdamantus | since 1 GiB is a bit more than 73 MB more than a GB. |
| 00:10.57 | Maxdamantus | flash usually comes in a power-of-2 number of bits, then the SD card will offer a power-of-2 number of GBs or MBs. |
| 00:11.16 | DocScrutinizer05 | maybe interesting (didn't read it completely): https://blogofterje.wordpress.com/2012/01/14/optimizing-fs-on-sd-card/ |
| 00:16.08 | DocScrutinizer05 | https://haydenjames.io/increase-performance-lifespan-ssds-sd-cards/ here it's called TRIM |
| 00:16.09 | Juesto | Do compactation? |
| 00:16.54 | DocScrutinizer05 | though >>Therefore, when you delete a file, your SSD is now able to write data to blocks as if it they were brand new without having to perform the cumbersome deletion process.<< obviously is utter bullshit |
| 00:27.15 | DocScrutinizer05 | MEH! I need to avoid looking into chanlog, always an annoyance to read that crap my /ignore luckily usually saves me to read |
| 00:30.42 | *** join/#maemo sunshavi (~user@181.64.192.217) |
| 00:31.24 | DocScrutinizer05 | Juesto: one of the major problems about uSD embedded controller is: it doesn't know about ext-fs or anything else than VFAT basically. On ext[234] you have inodes scattered all over the complete storage |
| 00:32.39 | DocScrutinizer05 | it's assumed that the SD controller even knows about File allocation Table in VFAT and optimizes using VFAT properties it knows about. Can't do same for any other filesystem |
| 00:33.02 | *** join/#maemo dafox (~dafox@2a02:a448:c25a:1:7061:3b2a:37df:ded3) |
| 00:36.07 | DocScrutinizer05 | Juesto: anyway SSD/SD analysis and optimization is absolutely non-trivial, due to the complex way a flash storage works and the obscure SEKRIT sauce the uSD controller adds on top |
| 00:36.44 | luke-jr | Maxdamantus: SD cards have a virtual block layer on top of the real flash, which adds overhead for remapping bad blocks |
| 00:36.53 | DocScrutinizer05 | that's why for really tight control and fast r/w access you want NAND and a filesystem like ubifs |
| 00:37.11 | DocScrutinizer05 | oh hi luke-jr :-) |
| 00:37.17 | luke-jr | hi DocScrutinizer05 |
| 00:38.39 | DocScrutinizer05 | seems to recall there were plans or even initiatives to reflash the uSd controller firmware to allow using the uSD like it was normal un-managed NAND storage |
| 00:39.49 | DocScrutinizer05 | obviously such a project would be highly brand&make specific on the particular uSD |
| 00:50.33 | DocScrutinizer05 | Oksanaa: re "90%" don't get fooled: see https://www.sandisk.com/home/memory-cards/sd-cards/extremeplus-sd-uhs-i "DISCLOSURES: ... 1MB=1,000,000 bytes. ** 1GB=1,000,000,000 bytes" |
| 00:51.35 | DocScrutinizer05 | *plus*: "Actual user storage less." |
| 00:55.39 | DocScrutinizer05 | so a 32GB uSD has a 32E9 bytes, minus a non-marginal fraction for overhead like spare blocks and prolly also flash controller metadata etc |
| 00:58.57 | *** join/#maemo Kabouik_ (~kabouik@65.76.93.92.rev.sfr.net) |
| 01:02.00 | *** join/#maemo DarioAlejandro (~root@cpe-67-10-7-12.rgv.res.rr.com) |
| 01:19.01 | DocScrutinizer05 | AAAH and there's the term I missed: FTL aka Flash Translation Layer (aka "the uSD embedded controller) http://www.linux-mtd.infradead.org/doc/ubifs.html |
| 01:19.56 | DocScrutinizer05 | s/"the uSD embedded controller("the uSD embedded flash controller"/ |
| 01:21.01 | DocScrutinizer05 | http://www.linux-mtd.infradead.org/doc/ubifs.html#L_raw_vs_ftl |
| 01:21.27 | Maxdamantus | 13:55:39 < DocScrutinizer05> so a 32GB uSD has a 32E9 bytes, minus a non-marginal fraction for overhead like spare blocks and prolly also flash controller metadata etc |
| 01:21.30 | Maxdamantus | No. |
| 01:21.32 | *** join/#maemo cyphase (~cyphase@unaffiliated/cyphase) |
| 01:21.53 | Maxdamantus | It has 32 GiB inside, but provides at least 30 GB that you can write to. |
| 01:23.05 | Maxdamantus | You practically can't get 32 GB in flash chips .. a 32 GB SD card will have 32 GiB of flash inside, but the block device you end up with will be around 32 GB. |
| 01:25.28 | Maxdamantus | Hm, actually, my 128 GB card only has 127865454592 bytes. |
| 01:26.35 | DocScrutinizer05 | the trick with optimizing uSD is:http://www.linux-mtd.infradead.org/faq/general.html#L_mtd_vs_hdd you have an interface/protocol for a block device (left side of table, with extensions like TRIM for example), but you try to implement a "usage pattern" aka filesystem and driver that takes care about properties described on right side half of the table |
| 01:27.56 | DocScrutinizer05 | you need a priori knowledge (or assumptions) about how the FTL firmware works and what it does in response to what you do on the block device interface |
| 01:33.44 | Maxdamantus | The eMMC on my N900 has 32015122432 bytes though. |
| 01:33.51 | DocScrutinizer05 | http://www.linux-mtd.infradead.org/faq/general.html#L_ext2_mtd >>But in many cases using mtdblock is a very bad idea because what it basically does if you change any sector of your mtdblockX device, it reads the whole corresponding eraseblock into the memory, erases the eraseblock, changes the sector in RAM, and writes the whole eraseblock back. This is very straightforward. If you have a power failure when the eraseblock is being erased, |
| 01:33.52 | DocScrutinizer05 | you lose all the block device sectors in it. The flash will likely decay soon because you will wear few eraseblocks out - most probably those ones which contain FAT/bitmap/inode table/etc.<< |
| 01:34.46 | DocScrutinizer05 | eraseblock (I called it erase page, sorry) is as large as several dozen or even hundreds of kB |
| 01:35.18 | DocScrutinizer05 | and to change a singke bit in it, you need to read modify erase write the whole eraseblock |
| 01:35.41 | Maxdamantus | DocScrutinizer05: yeah, that's why you shouldn't use something like mtdblock, as mentioned above. |
| 01:35.51 | Maxdamantus | DocScrutinizer05: something sensible would keep writing to new blocks. |
| 01:36.05 | Maxdamantus | that's basically the principle behind log-based filesystems. |
| 01:36.32 | Maxdamantus | The FTLs in SD cards probably do something similar. |
| 01:36.44 | DocScrutinizer05 | *unless* the bit change is 0->1, assuming all-0s is the result of block-erase (which takes several milliseconds as I already explained) |
| 01:37.47 | Maxdamantus | https://en.wikipedia.org/wiki/List_of_log-structured_file_systems |
| 01:38.15 | Maxdamantus | The filesystems at the bottom are all the standard ones for flash. |
| 01:38.24 | Maxdamantus | including UBIFS, which is used by maemo. |
| 01:38.54 | DocScrutinizer05 | so stuff like FTL and ubifs implements very nifty ways to not always read-modify-write for a few bytes that need to be changed, rather it uses journals etc that simply get extended at end, writing to the already/still all-0 region of a erase block |
| 01:38.59 | Maxdamantus | an SD card or MMC has to do the same sort of stuff, but it just provides a block device instead of a filesystem. |
| 01:39.55 | DocScrutinizer05 | and TRIM deals with telling FTL that a complete eraseblock is 'clean' again and ready to get erased when there's some spare time in background for doing so |
| 01:43.41 | DocScrutinizer05 | (simplified image, since TRIM doesn't really care about erase blocks, it has no notion about them afaik. It's up to the FTL to deduce erase blocks needing a background erase, from what fs blocks (512) TRIM told the FTL are unused |
| 01:43.46 | DocScrutinizer05 | ) |
| 02:11.47 | *** part/#maemo Oksanaa (~Wikiwide@Maemo/community/ex-council/Wikiwide) |
| 03:43.59 | *** join/#maemo RedW (~redw@89-74-189-236.dynamic.chello.pl) |
| 03:51.48 | *** join/#maemo utanapischti (~username@p4FC12E42.dip0.t-ipconnect.de) |
| 05:29.39 | *** join/#maemo DocScrutinizer05 (~saturn@openmoko/engineers/joerg) |
| 05:49.15 | *** join/#maemo chainsawbike (~chainsawb@unaffiliated/chainsawbike) |
| 06:30.24 | KotCzarny | l29ah: yes, see above eraseblock explanation |
| 06:30.50 | KotCzarny | sd cards were created for mobile capture devices |
| 06:31.29 | KotCzarny | nowadays people started using them as rootfs etc, so sandisk/samsung started optimizing 4k random r/w |
| 06:32.03 | KotCzarny | if you want to measure correctly run iozone test, you will see for yourlsef |
| 07:28.55 | *** join/#maemo RedW (~redw@89-74-189-236.dynamic.chello.pl) |
| 08:15.42 | *** join/#maemo florian (~fuchs@Maemo/community/contributor/florian) |
| 11:08.26 | *** join/#maemo Kabouik_ (~kabouik@65.76.93.92.rev.sfr.net) |
| 11:15.31 | *** join/#maemo Pali (~pali@Maemo/community/contributor/Pali) |
| 11:16.48 | *** join/#maemo xorly (~xorly@ip-89-176-102-19.net.upcbroadband.cz) |
| 11:28.26 | *** join/#maemo rosseaux (~BeRo@228.66.238.89.in-addr.arpa.manitu.net) |
| 13:34.24 | *** join/#maemo Kabouik_ (~kabouik@65.76.93.92.rev.sfr.net) |
| 13:54.07 | *** join/#maemo ced117 (~ced117@opensuse/member/ced117) |
| 14:09.02 | *** join/#maemo florian (~fuchs@Maemo/community/contributor/florian) |
| 14:40.35 | *** join/#maemo eMHa (~mh@p200300738E2AF700267703FFFE6CC360.dip0.t-ipconnect.de) |
| 14:45.16 | *** join/#maemo dafox (~dafox@2a02:a448:c25a:1:7061:3b2a:37df:ded3) |
| 16:28.17 | *** join/#maemo shentey (~shentey@x4e34c8c0.dyn.telefonica.de) |
| 16:55.31 | L29Ah | One more step |
| 16:56.20 | *** join/#maemo shentey (~shentey@x55b2f825.dyn.telefonica.de) |
| 17:35.15 | *** join/#maemo fuz_ (nobody@vau75-10-88-164-23-88.fbx.proxad.net) |
| 19:41.32 | *** join/#maemo cyphase (~cyphase@unaffiliated/cyphase) |
| 20:05.31 | *** join/#maemo Roth (~quassel@73.61.17.235) |
| 20:07.28 | *** join/#maemo louisdk (~louisdk@5.103.130.65) |
| 20:08.54 | *** join/#maemo Roth_ (~quassel@73.61.17.235) |
| 20:14.01 | *** join/#maemo Roth (~quassel@73.61.17.235) |
| 20:19.05 | *** join/#maemo Roth_ (~quassel@73.61.17.235) |
| 20:20.16 | *** join/#maemo xes_ (~xes@unaffiliated/xes) |
| 20:24.11 | *** join/#maemo Roth (~quassel@73.61.17.235) |
| 21:00.22 | sunshavi | ~info sunshavi |
| 21:25.18 | *** join/#maemo RzR (~RzR@82.236.136.171) |
| 22:40.53 | *** join/#maemo cyphase (~cyphase@unaffiliated/cyphase) |
| 22:59.59 | DocScrutinizer05 | 'awesome'¡ http://www.openmobilealliance.org/release/SUPL/V3_0-20140916-C/OMA-AD-SUPL-V3_0-20110920-C.pdf >> 5.3.1.8 SUPL Triggering Function (STF)<< |