[meta] high IO usage

[RubenKelevra]

the current approach to pin on each update the full folder structure recursive requires IPFS to do quite a lot of IO, as @Luflosi mentioned here: #39

There's still the other option that I've implemented on version one of this project, which is to pin each file on it's own, as well as each folder non-recursive.

The advantage is a low amount of IO and a much faster processing time of updates on the cluster nodes, as it's easier to be processed by IPFS. But it requires a lot of disk space on the cluster-database since each changed file and each changed folder is an individual transaction. After just some months the database had a file size of 20 GB while there were 400'000 transactions.

Once ipfs-cluster/ipfs-cluster#1008 and ipfs-cluster/ipfs-cluster#1018 are implemented we could explore this possibility to reduce the IO load again, with one transaction per update, but individually pinned files and folders.

[guysv]

not solving the underlying issue but what about updating the pins only every few half a day or so? that + bumping on security updates seems reasonable to me. I'm also pretty sure that's how often official mirrors update anyway.

[RubenKelevra]

@guysv there's no way to tell what a security update is and what isn't.

Apart from this that is highly discouraged, as you can tell from the wiki page:

Check the status of the Arch mirrors by visiting the Mirror Status page. It is recommended to only use mirrors that are up to date, i.e. not out of sync.

I only run an update when there's an update available - so there's no unnecessary IO.

I think I found the issue why it's pretty high in IO load. It does up to 4 tasks concurrently right now.

This doesn't reduce the IO usage, but the load, since ipfs does less concurrently.

[Luflosi]

Wouldn't it be possible to optimise IPFS itself so that for every node in the merle dag, it caches wether or not all nodes below that node are available locally? I would imagine, that this would make a recursive pin, that only has a small difference to another recursive pin, about as fast as a non-recursive pin. This could massively reduce the amount of IO required to recursively pin something if most of the stuff is already available locally.

[FireMasterK]

Not sure if this helps, but we can see if Arch has anything like this: https://wiki.ubuntu.com/Mirrors/PushMirroring

[RubenKelevra]

@Luflosi well... yeah, there might be potential to optimize IPFS itself - which would be great.

But - there are already several ways to improve the situation which are implemented:

• You can use BadgerDB as the block storage, which gives IPFS much more performance than FlatFS - but there's a catch: The cleanup currently doesn't workup very well. See GC: Data may remain after garbage collecting everything ipfs/go-ds-badger#54

• The other option to increase speed of an update is to use GraphSync which should increase the speed of a delta transmission between two states. There are network security concerns - that's why I don't use it yet.

• We could use the pin-update command rather than pin-add for the cluster. I'm not sure why I decided against it some month ago when I wrote version 2, but I think there was a limitation in ipfs-cluster v0.12 which had me use regular pin-add. Not sure how this works out IO wise, thought.

[RubenKelevra]

@FireMasterK wrote:

Not sure if this helps, but we can see if Arch has anything like this: https://wiki.ubuntu.com/Mirrors/PushMirroring

Well, no technical Arch does not. But we use very frequent polling to archive the same thing: Single-digit minute numbers between a package maintainer pushing an update until it has reached the IPFS-pacman-mirror.

But the delay between updates which are available and having them added to the cluster isn't the issue here.

It's more a internal cluster issue, where the changes have to be compared to the old version by each cluster member resulting in very large amounts of IO reading the old data from the disk.

[teknomunk]

Another option besides pinning the root directory and pinning each individual file recursively and the directories non-recursively is to add all the packages in an update to a directory that is not in the normal folder structure but contains the same files and add that to the cluster instead of the individual files. You end up with the same result as the individual file pins, but with fewer cluster entries in every instance except for single updated packages.

[RubenKelevra]

@Luflosi I've pushed the concurrency fix half an hour ago to ipfs. If you restart your follower you should see a drop in IO load.

Please report back. :)

[RubenKelevra]

Another option besides pinning the root directory and pinning each individual file recursively and the directories non-recursively is to add all the packages in an update to a directory that is not in the normal folder structure but contains the same files and add that to the cluster instead of the individual files. You end up with the same result as the individual file pins, but with fewer cluster entries in every instance except for single updated packages.

That's not an option.

You'll end up with a lot of folders with unrelated updates which got pushed together. Say if package a to g got updated, I'll add a, b, c, d, e, f, g to a folder.

Now you got an update of d, f, x and y, so I would have to traverse all folders stored in the cluster to find the folder with d and f, delete those files from the folder, push the new version of the folder and create a new folder with d, f, x and y in it.

Over time you end up with a lot of folders with just a single package in them when they haven't got updated in a while.

Edit:

Just to give you an impression, that would be roughly 5259 folders for /community right now. For just ~8000 packages.