No new pins

[Luflosi]

I'm running a collaborative cluster follower on my server. I wrote a systemd unit that runs ipfs-cluster-follow pkg.pacman.store run so I don't need to use tmux. It worked ok except for high disk IO but that's not the problem here.
There seem to be no new pins added to the cluster since about a week ago. This is the output of journalctl -b -u ipfs-cluster-follow@pkg.pacman.store.service --no-pager since I restarted it yesterday:

ipfs-cluster-follow output through journalctl

Dec 29 17:03:32 ipfs ipfs-cluster-follow[108]: Starting the IPFS Cluster follower peer for "pkg.pacman.store".
Dec 29 17:03:32 ipfs ipfs-cluster-follow[108]: CTRL-C to stop it.
Dec 29 17:03:32 ipfs ipfs-cluster-follow[108]: Checking if IPFS is online (will wait for 2 minutes)...
Dec 29 17:03:32 ipfs ipfs-cluster-follow[108]: waiting for IPFS to become available on /ip4/127.0.0.1/tcp/5001...
Dec 29 17:03:34 ipfs ipfs-cluster-follow[108]: 2020-12-29T17:03:34.430+0100        INFO        config        config/config.go:361        loading configuration from http://127.0.0.1:8080/ipns/cluster.pkg.pacman.store
Dec 29 17:03:35 ipfs ipfs-cluster-follow[108]: 2020-12-29T17:03:35.935+0100        INFO        cluster        ipfs-cluster@v0.13.0/cluster.go:132        IPFS Cluster v0.13.0 listening on:
Dec 29 17:03:35 ipfs ipfs-cluster-follow[108]:         /ip6/::1/tcp/16587/p2p/12D3KooWM2WefGdNzYLkduTBsTMrgtJevP3Wa8HV4tQ9pGfRTVXA
Dec 29 17:03:35 ipfs ipfs-cluster-follow[108]:         /ip6/fd42:dd60:65a4:c374:216:3eff:febf:5066/tcp/16587/p2p/12D3KooWM2WefGdNzYLkduTBsTMrgtJevP3Wa8HV4tQ9pGfRTVXA
Dec 29 17:03:35 ipfs ipfs-cluster-follow[108]:         /ip4/127.0.0.1/tcp/16587/p2p/12D3KooWM2WefGdNzYLkduTBsTMrgtJevP3Wa8HV4tQ9pGfRTVXA
Dec 29 17:03:35 ipfs ipfs-cluster-follow[108]:         /ip4/10.208.171.176/tcp/16587/p2p/12D3KooWM2WefGdNzYLkduTBsTMrgtJevP3Wa8HV4tQ9pGfRTVXA
Dec 29 17:03:53 ipfs ipfs-cluster-follow[108]: 2020-12-29T17:03:53.113+0100        INFO        restapi        rest/restapi.go:515        REST API (HTTP): /unix//home/ipfs/.ipfs-cluster-follow/pkg.pacman.store/api-socket
Dec 29 17:03:53 ipfs ipfs-cluster-follow[108]: 2020-12-29T17:03:53.133+0100        INFO        crdt        go-ds-crdt@v0.1.12/crdt.go:275        crdt Datastore created. Number of heads: 1. Current max-height: 37761
Dec 29 17:03:57 ipfs ipfs-cluster-follow[108]: 2020-12-29T17:03:57.232+0100        INFO        cluster        ipfs-cluster@v0.13.0/cluster.go:619        Cluster Peers (without including ourselves):
Dec 29 17:03:57 ipfs ipfs-cluster-follow[108]: 2020-12-29T17:03:57.233+0100        INFO        cluster        ipfs-cluster@v0.13.0/cluster.go:626            - 12D3KooWDM4BGmkaxhLtEFbQJekdBHtWHo3ELUL4HE9f4DdNbGZx
Dec 29 17:03:57 ipfs ipfs-cluster-follow[108]: 2020-12-29T17:03:57.233+0100        INFO        cluster        ipfs-cluster@v0.13.0/cluster.go:634        ** IPFS Cluster is READY **
Dec 29 17:04:28 ipfs ipfs-cluster-follow[108]: 2020-12-29T17:04:28.115+0100        ERROR        p2p-gorpc        go-libp2p-gorpc@v0.1.0/call.go:64        failed to dial 12D3KooWEweUswc6ZrQJACgGf13gmVBVVssK6LjCMENs6pu5yHth: all dials failed
Dec 29 17:04:28 ipfs ipfs-cluster-follow[108]:   * [/ip4/192.168.2.212/tcp/16587] dial tcp4 192.168.2.212:16587: connect: connection refused
Dec 29 17:04:28 ipfs ipfs-cluster-follow[108]:   * [/ip6/2001:470:1f11:90d::1/tcp/16587] dial tcp6 [2001:470:1f11:90d::1]:16587: connect: network is unreachable
Dec 29 17:04:28 ipfs ipfs-cluster-follow[108]:   * [/ip6/fd2c:853:d11f:0:e445:edff:fe20:38d3/tcp/16587] dial tcp6 [fd2c:853:d11f:0:e445:edff:fe20:38d3]:16587: connect: network is unreachable
Dec 29 17:04:28 ipfs ipfs-cluster-follow[108]:   * [/ip6/2600:8803:e600:18c:e445:edff:fe20:38d3/tcp/16587] dial tcp6 [2600:8803:e600:18c:e445:edff:fe20:38d3]:16587: connect: network is unreachable
Dec 29 17:04:28 ipfs ipfs-cluster-follow[108]:   * [/ip6/fc7c:867c:98e9:7667:f32:9091:65a1:3d65/tcp/16587] dial tcp6 [fc7c:867c:98e9:7667:f32:9091:65a1:3d65]:16587: connect: network is unreachable
Dec 29 17:04:28 ipfs ipfs-cluster-follow[108]:   * [/ip4/192.168.32.254/tcp/16587] dial tcp4 0.0.0.0:16587->192.168.32.254:16587: i/o timeout
Dec 29 17:04:28 ipfs ipfs-cluster-follow[108]:   * [/ip4/10.0.0.2/tcp/16587] dial tcp4 0.0.0.0:16587->10.0.0.2:16587: i/o timeout
Dec 29 17:04:28 ipfs ipfs-cluster-follow[108]:   * [/ip4/68.12.168.55/tcp/16587] dial tcp4 0.0.0.0:16587->68.12.168.55:16587: i/o timeout
Dec 29 17:15:59 ipfs ipfs-cluster-follow[108]: 2020-12-29T17:15:59.045+0100        INFO        cluster        ipfs-cluster@v0.13.0/cluster.go:487        reconnected to 12D3KooWK6dvqX7kXvJW8LkFDtT5zzTZLFF8PQAjHmR6Y9ych53C

The last line is then printed again and again a couple times per hour and nothing else.

If I try to ipfs swarm connect to any of the three peers listed in https://github.com/RubenKelevra/pacman.store/blob/master/collab-cluster-config/service.json, it times out. I only have an IPv4 internet connection, so I could only try the three IPv4 addresses.
The time this started happening seems to coincide with the shutdown of loki, at least judging by the commit date of a482b39. Maybe this has something to do with it.

ipfs-cluster-follow pkg.pacman.store list only prints

pinned               bafybeiai2lhrnb6v53jkt7wf5wd7uljaoko342x3oxswe3lqkmxuws45ei x86-64.archlinux.pkg.pacman.store@2020-12-23T21:10:40+00:00
pinned               bafykbzacecana3ogwm5n3ung7zit47e4tmjlozfvs7hjalp2tiacgilsbs3sq cluster-service-lowpower.json@64513e3
pinned               bafykbzacedqspngi4evtuy7axrvbsvrrhu2kpr4ndabiqhlkf4i5yhq5ner5a cluster-service.json@64513e3

Is this a problem with the cluster or with my setup?

[guysv]

noticed it too. last time i checked lastsync was 26 december.

[FireMasterK]

https://github.com/RubenKelevra/pacman.store/wiki/Status%2C-Announcements-%26-Maintenance#status

this is likely why

[teknomunk]

Should just roll back the ipfs version to a working version until the bugs get fixed.

[rpodgorny]

#40

[RubenKelevra]

Hey guys, thanks for your patience. It's running again. :)

@Luflosi wrote:

It worked ok except for high disk IO but that's not the problem here.

Yeah, I know. That's basically the second version of this cluster, the first one pinned each file individually. This works a lot better I/O wise, since not the whole tree has to be surveyed on each update, just the new data is added... but you cannot add a pin non-recusive. So I was unable to store the folders in the cluster pinset. (See ipfs-cluster/ipfs-cluster#1009)

In theory I could write a version 3 which starts to pin the files again recursively and the folders direct, but this puts quite some stress on the database of the ipfs-cluster - so not sure if this is the way to go.

For 400'000 generations in the cluster you end up with a database for the cluster with a size around 20 GB. I think that's pretty unacceptable. (see ipfs-cluster/ipfs-cluster#1180 (comment))

[RubenKelevra]

I forgot to mention that transactions is a feature on the road map - so once done we could take another look at this, as an update would still contain much data and thus a large database, but one update would be at least one generation - so the database shouldn't become so "cluttered".

The pin-list on the other hand would still be massive.

ipfs-cluster/ipfs-cluster#1008 and ipfs-cluster/ipfs-cluster#1018

[RubenKelevra]

Let's track the high IO usage issue here: #42