Grab Bag of Crackers

Hello all, hope you’ve been well and that you’ve had a good holiday season so far. It’s been a long time since I’ve posted something, and I’m sorry for that, as it’s been a combination of burn out and life putting a premium on my energy and time that’s stopped me from doing as much as I would have liked. I also would have liked to stick to a specific topic and show off some of the more impressive things I’ve worked on now and again but as it’s the end of the year, it’s probably better to do a sort of ‘year in review’  type post as well as a grab bag of news since the last post.

Well, so as to not duplicate effort, I’ll rattle off some of the things that have happened in the past few months first.

More anti-spam work has been done, refining both the filters and the circumstances under which they are triggered. It’s mostly cut down on false positives and kept the site free of some of the more persistent waves of spam. It’s a never-ending battle, however, and there are some areas of improvement that I think can help with those more complex cases of flooding and spam. Of course, there’s plenty of third party services that might help, but I’ve resisted using them because I fundamentally distrust how third parties handle data even when they promise privacy. Also, but no less importantly, it would also mean an ongoing cost as these services aren’t free.

It’s worth mentioning before moving on, that one innovation that I implemented earlier was the ability for moderators to temporarily restrict posting on the site by hitting a button. There’s some granularity here, with preset time limits as well as the option to only apply the restriction to new IPs. The idea here is to give the moderation team some breathing space if there’s dedicated spamming/flooding going on so bans can be applied and the board cleaned up. It’s something that is hopefully used very rarely but a good tool to have in the arsenal.

The last few months have also seen a load of optimizations as well as upgraded hardware. For the former, I’ve spent time wrangling some common processes like page generation and streamlining a few bits here and there as well as doing some web server software tweaks with caches, limiting some lookup that can be resource-intensive to specific circumstances etc etc. The site on the whole has been more responsive with fewer timeouts (none?) and moments where users are not getting a fast load and able to interact with the page’s contents. It’s likely most noticeable on the front page, story list and the like, but also should mean that posting is quicker and content gets served with fewer hiccups.

Some time after those changes went live, I also upgraded the instance where the site is hosted. Not to further improve performance, though that’s a nice bonus, but more to prepare for future changes to the site that are getting closer and closer to being finalized. Exciting stuff! I want to talk more about these things but it’s not yet time. I’m juggling a lot of things in the code at the moment and would prefer to wait until it’s almost ready to deploy.

The matrix server also got a performance bump, mostly out of necessity, as there were issues at times that made the software unresponsive. My attempts to troubleshoot and pinpoint things to an exact cause or specific bug were mostly fruitless and my optimization attempts did not offset the issues enough. I came to the conclusion that I would either have to switch to another homeserver software (conduit) or upgrade the hardware. I chose the latter because the former is stable enough theoretically but it is not treated by its developers as mature software.

I’ll likely update the funding levels on the patreon page to include these heightened running costs as they’ve exceeded the small cushion I already include in the figures. It’s not *too* bad, don’t worry, just a matter of $15-20 more for both servers. An added bonus is that it also means more storage space for images and the possibility of keeping archived images in their full resolution though I have to look into the feasibility and do some calculations before I commit.

There is one thing that I can share about the many things I’m doing—I’m overhauling how images are kept track of by the board software. Specifically, I’m continuing the slow efforts to normalize data in the database in order to reduce redundancy and make queries more efficient. I’ll likely talk about this in more detail in the future but the short of it is that a lot of data is useless stored in the posts table which isn’t always needed and so there’s a lot of extra columns and information that gets called, stored in memory and makes (human, debugging) readability an issue. A further upshot of the process is that by taking all those file information and storing it elsewhere, you can also create more complex relationships, such as having multiple entries have the same parent post … meaning that it makes it easier to eventually allow users to upload multiple images when posting.

I’ve mostly worked out the kinks in my test environment but there’s a crapton of unrelated functionality that expects data to be formatted just so that needs to be dealt with. You need to account for this new scheme when posting, deleting a post, regenerating the boards, on the front page, when viewing deleted posts in the moderation page, manipulating or calling them up in other moderation areas etc etc which means that testing each and every aspect to make sure nothing breaks is both necessary and time-consuming.

Other fixes and changes, like story tagging or the user panel that I’ve had to do also is subject to a lot of hurdles of that nature. I changed how some of the JSON data that’s used for finding stories or tags is generated and stored but I think it’s been worth the effort. Getting more regular and structured results is worth a bit more of complexity though there is some refactoring in the cards for the future as I think that there’s a lot of code that can be reused and centralized in some instances. It’s something relatively low-priority at the moment as I think that come the new year I’ll try to push some of the longer-term changes across the finish line.

I could go on as there’s a lot under-the-hood kinds of things that I haven’t mentioned like improving search engine visibility and stress testing but this post has gotten long enough and I haven’t even ‘reviewed’ the rest of the year!

I’ll be mercifully brief since you can always look at the other posts this year but here’s a quick recap:

• THP switched to a different server early on in the year and the migration was more or less smooth
• Older archived threads were updated for consistency of newer features and looks
• Tagging got a large overhaul with better tools for users to both add and look them up
• The story list got a fair amount of love such with things like pagination, better sorting, better tag filtering (excluding by tags also), better footer and presentation
• Tons and tons of changes to fight spam and new moderator-side tools and abilities
• Huge overhauls for the administration pages in general with a reworked ban list, board lockdowns, manually triggering regeneration and a lot more
• Fixes for mobile users and CSS issues to make using the site a more consistent experience
• Lots of minor quality-of-life fixes like more file formats, reminders when posting updates, smarted linking to story threads etc
• Matrix server improvements and fixes
• Heaps of bugfixes and optimizations to the code

There’s always more that can be said but I’ll leave it at that. I hope that I’ll be writing up another post sometime soon and that you’ll soon be able to see all the other hard work I’ve been putting in to make THP an even better place in 2024!

Have a very happy new year!