Good timing, I’ve literally just updated to 0.19.18 which should have the fix in!

No, no meaningful cpu spikes I could make out anywhere, although admittedly I was focusing on the lemmy server container mostly

I noticed in the logs before every timeout there were lots of “verify” words appearing, and in each iteration of that statement there were more and more verify words. Honestly had no idea what it meant at the point, only that I didn’t recognise it from looking at lemmy logs previously, it always appeared before a crash, and it felt suspicious.

Here’s an example from some logs before a crash:

2026-03-15T21:47:22.670586Z  INFO HTTP request{http.method=POST http.scheme="https" http.host=lemmy.zip http.target=/inbox otel.kind="server" request_id=2cc6dc65-571d-4a69-9733-5e80e455c00b}:receive:community:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:
verify:verify:verify:verify: activitypub_federation::fetch: 
Fetching remote object https://hexbear.net/comment/7004776

thread 'actix-server worker 18' has overflowed its stack

fatal runtime error: stack overflow

I pinged some logs over to Nutomic on matrix, who thought it might have been related to nested comments, and then I noticed Dessalines had made the linked thread, which matched pretty much with what I was seeing behaviour and logs-wise.

Usefully the logs link the object it’s fetching, and 9 times out of 10 its a deeply nested hexbear thread! Or someone from another instance commenting on a nested hexbear thread. Nutomic confirmed the behaviour based on the logs in the issue, and I’m pulling the logs when I get chance to see what other threads are causing it to crash, although hopefully the fix will make it’s way into 0.19.18 beta 3 so I can stop worrying about it!

It won’t be possible this time, it will pop up on game start like it did for S2, but I’m considering ways to do this while preventing unbalancing the teams too much

in all fairness to them, my post hasn’t federated over there for some reason

https://peachpie.theatl.social/c/thebutton@lemmy.zip

@Nusm@peachpie.theatl.social - you might want to see if you can pull this post into piefed: https://lemmy.zip/post/62493544

I’ve been enjoying Pete & Bas for a while. If you want to go further into the P&B universe, check out The Northern Boys

Likely related to this issue here: https://github.com/LemmyNet/lemmy/issues/6427

Something introduced in 0.19.16 is causing the server to restart. We can go 12+ hours with no problems, then 10 restarts in 3 hours. Seemingly random, but hopefully with the devs on it we can get a resolution.

It will be back!

This gives me some fun ideas to look at this concept, thank you :)

Did you manage to get past cloudflare in the end? It’s on full lockdown mode like the lemmy site is, but I can probably back it off for this game tbh

I managed to accidentally lock all the chats for a while (hopefully now resolved) - I am also rewriting the chat logic to try and iron out issues.

I am looking at rewriting chat entirely. I have now moved the pause button so it cannot be accidentally hit, but I’m still not sure exactly how it paused tbh. Like its pretty clear if i do press it 🤔 ah well.

Will add system react messages in too 👍

Thank you for the bug report, that does appear a terrible oversight on my part and I will do my upmost to fix it.

Yes slight bug with S1 as it worked differently, but I will manually patch that to 12 hours 36 minutes for S1, so S2 went longer!

May the 4th gives me lots of ideas for S3!

Started just over 24 hours ago! Sadly finished earlier on.

Thanks, have reported to the blorp dev and will report to voyager dev. They work on the platform directly but the apps don’t show the links.

Should be fixed now! :)

This one should be fixed now, was in relation to pre-signed up users and the chat not updating properly.

Thats what it is, but I’ll label it up so its clearer!