The title pretty much tells it all. A couple of weeks ago I tried to access the graf and found out that for some reason the php auth page didn’t work. The php service must be down, I thought before making a mental note to check it after having finished exams.
So, today, having new ideas for some projects I went and checked the server so I could have it up and ready to push the new versions to. journalctl -f, try to access the auth page and I am greeted with a red error message telling me that php-fpm is trying to read from an inode with errors or something like that. Darn it! Pull the SD card out, run fsck on it using my laptop and I find that the corruption wasn’t a punctual event but widespread. It would probably boot, but I am not trying to mantain a system with potentially a lot of libraries missing or services stopped because of missing files. Especially because I still don’t know what caused the corruption. I, personally, have not hot-unplugged the server, but I know it has been shut down a few times (long story, the server room became an office because of COVID-19), however given that some of the files affected were not supposed to be written to I don’t think it is only this. The login.php file was the first I found corrupted, and as should surprise nobody, the webpage should be read-only. I might end up creating a lot of different partitions to avoid issues like this in the future: logs, config, binaries, databases and pages in different partitions. Easy to backup and restore the ones that need it and with a few hooks and setuids the read-only partitions could be automatically unlocked and locked when needed. If done badly it could be dangerous, but if well done it also greatly reduces the damage an attack could do.
But this is still in the far future.