Blog

February 2022 work summary

On: By: Mark

February was a bit like January: plenty to do at the start and end of the month with not so much in the middle of the month. So during the middle of the month I released new versions of three of my extensions: Digests, Spam Remover and SCSS compiler.

I thought my development work for a major commercial client was done but a few new minor tasks were discovered. These tasks included:

  • Replicating a kill user feature that was overlooked. It gives an admin a one button press to remove a spam user and all their posts.
  • Changed an extension I wrote that emails to a mailing list all admin log entries to also send the list critical error log entries.

Other work for clients in February included:

  • WordPress work. Updated an old WordPress instance from WordPress5.3 to 5.9. I got errors after logging in and solved them by going into the database and changing the row in the wp_options table that enabled all the plugins. Installed a default theme. Moved a number of posts and pages into the trash. Rearranged the menu and added some pages. Then demonstrated how to add PDFs to the Media Library and how to use the Gallery extension, which I had to reinstall, but a gallery of smilie fingerprints was already set up. I just had to add the correct shortcode to the page so it would render. Set up another gallery for a set of other kinds of images and demonstrated uploading some of these. The client may be able to take it from here.
  • Installed the Topic Tags extension using a 3.3 version I found on a client’s test website. Client tried it out and had me install it in production.
  • Upgraded a board from phpBB 3.0.14 to 3.3.5. Tried to use Support Toolkit but had errors manifested by syntax errors in language pack files but eventually decided to give up on it. Set everyone to use the British English language pack. There were dozens of language packs installed. Three database errors had to be fixed to complete the upgrade. I stayed with default prosilver style. I disabled the contact page and set up reCaptcha V3 spambot countermeasure. I set up a home page link and installed the Advertisement management extension and placed sample ad.
  • Rehosting and conversion. This was a conversion from phpBB 2.0.16. I moved the files and database to temporary hosting for a subdomain. The board had 354,000 posts. I had to split extract of phpbb_post_text table into two extracts to get under the 50mb upload limit, as well as not carry over the search tables. Converted using phpBB 3.3.4 to avoid conversion bug, then updated to 3.3.5. I tried to install SiteSplat style but didn’t show an ad, so used ProLight style, then created a custom style based on it. The style needed to incorporate the site’s WordPress headers and footers, which took two hours of time. Installed the Advertisement Management, Site Map and Google Analytics extensions. I placed sample ad. User tested it then let me move it into place, which was a folder inside their new WordPress site done by someone else.
  • I was asked to check emailing from the board. No problems were found, but I couldn’t get to work with SMTP. I created an admin email address for domain and changed email address default for board to that. Provided guidance on subscribing to forums and topics. Later, attempted to integrate simpler WordPress theme HTML provided by their contractor but couldn’t get it to work. I moved over both the the files and database. The board came up without an issue after changes to config.php.
  • Upgraded a board from phpBB 3.2.9 to 3.3.5. I deactivated and removed a number of styles and made prosilver the default style for everyone. Upgraded the Pages extension from version 2.0.4 to 2.0.5. Originally I figured out that the reason the board did not come up was because BlueHost changed PHP to 8.0. Temporarily reverted the domain to PHP 7.3, then later after upgraded the board I enabled PHP 8.0 again.
  • Updated board from phpBB 3.3.4 to 3.3.5. Create a custom style inheriting from prosilver to simply updates next time. Since the Board3 Portal extension was installed, I changed the .htaccess file to load the portal page by default.

January 2022 work summary

On: By: Mark

Client work was busy at the start of the month and at the end of the month, but there was a weird two-week quiet period in the middle. During that time I mostly worked on extensions. I’m working on an update to my popular digests extension but it’s not quite ready to go, but contains new reporting features. I’m also working on a notify admin on error extension, which is getting me into phpBB’s notification system. The point of this extension is to send administrators notifications if new entries occur in phpBB’s error log. It’s surprising this isn’t built in. It needs to be.

My large commercial client still has not moved into a formal testing phase for all the changes we made over the last two years. However, I was asked to move this along by writing a test plan which I did over the last two days of January.

Other work in January included:

  • Cashing a nice holiday bonus from my large commercial client. They let me defer receiving payment until the new year, which works for tax purposes.
  • Provided guidance for another established commercial client for deleting and setting up new forums with the correct forum permissions. For the same client, I upgraded their board from phpBB 3.3.1 to 3.3.5. I also updated the Board rules, Digests, Tapatalk and Quick style extensions to their latest versions. I replaced Artodia SCSS compiler with my own SCSS compiler extension. I updated 12 language packs. I also updated all five styles and reapplied their customizations to them. I found and fixed bug in the Forum language extension.
  • Earlier in the month I went through a messy test phpBB 2.0.15 to 3.3.5 conversion. There were lots of issues. The main one turned out to be a bug in the phpBB 3.3.5 conversion software recently introduced where serialized config values were incorrectly escaped, causing a connection to the old database to fail. When I couldn’t get it to work on the client’s server, I moved a 4GB database and 60GB of files to my machine to do it there and encountered the same issue, leading me to figure out eventually where the problem lay. A conversion on my machine eventually worked after correcting the incorrect configuration values. It took about eight hours. There were some duplicate posts and HTML conversion errors, and it failed near the end when importing attachment types and attachment groups. I moved the database to the client’s server and it came up but there were a number of inconsistencies we had to work through, like missing or incorrectly placed avatars and smilies.
  • Later in the month I did a “for real” migration for the same client. The workaround is to install phpBB 3.3.4 and use its convertor program. I tried to run conversion on his web space but it didn’t work with phpBB 3.3.4 and returned a HTTP 500 error with nothing in the error log. So again I had to download the database and files to my machine and convert it there. The conversion was straightforward except for an issue at the end relating to attachments that I hacked but didn’t affect anything. Smilies, avatars and attachments all converted correctly this time. I then updated the software to phpBB 3.3.5. I then added the logo and installed the Advertisement Management extension and placed a sample ad. I was able to upload the database after moving it into an archive, but when I tried to make an archive of the files I found that I didn’t have enough space to create it! So this involved uploading about 335,000 files individually instead of uploading one large archive, a process that took at least 18 hours. I brought up board but first fixed a few directory permission issues. I started to create search index but stopped it when I realized it would take 16 hours. Board was placed in a temporary domain. Client pointed the domain to the new hosting space and re-created the search index.
  • I converted a board from phpBB 2.0.23 to phpBB 3.3.5. Due to bug in the conversion software discussed above, I installed phpBB 3.3.4 and afterward upgraded to phpBB 3.3.5, using lessons learned from last job. It all went fast and spiffy with no issues. About 400,000 posts were converted. I rebuilt the search index, disabled contact page, and set up a reCaptcha v3 spambot countermeasure.
  • Fixed a number of issues related to moving a board to GoDaddy. Reloading the modules table from reference phpBB 3.3.5 installation allowed me to get to the ACP forums tab. It also addressed an issue when registering. Three extensions needed to be updated: Google Analytics, Board Announcements and Stop Forum Spam. I changed spambot countermeasure to Q&A after reCaptcha V3 didn’t work because the host didn’t allow queries to other servers. I disabled the contact form. I installed the proflat style and integrated the customer’s logo. I had to manually add the modules for some active extensions as they got overwritten when I replaced the modules table with a reference. I tested the registration issue to make sure it was fixed. I was asked to hide some of the forums. I gave the unwanted forums the no access role for Anonymous user and registered users group to fix this issue.
  • I upgraded a board from phpBB 3.2.9 to 3.3.5. I installed the Nosebleed style and the American English pack plus the Cleantalk extension. I removed the site logo from Nosebleed style and integrated a new background image that fit the space exactly and was created by the client. I set the home page link and setup the reCaptcha V3 spambot countermeasure. I also removed some dead modules. Set up a redirect rule to redirect http traffic to https for the board and provided guidance on favicon usage. Collaboration on setting up forums and categories to come.

Conversions and phpBB 3.3.5

On: By: Mark

I ran into a major bug in phpBB doing a conversion for a client to phpBB 3.3.5 recently.

A conversion is when you migrate from a different forum solution to phpBB. It’s also used for phpBB 2, which was retired in 2007, as the architecture for phpBB 3 is completely different than phpBB 2. You would think after 15 years there would be no more phpBB 2 boards left. My clients would disagree. I get probably half a dozen of these a year, almost all from phpBB 2. It is amazing that there are so many of these boards still around. phpBB 2 was very simple, but very reliable.

The conversions used to be painful but the convertor software has become much more reliable. Now most conversions have minor issues at best. So I wasn’t expecting any trouble when I did a recent phpBB 2 conversion other than a lot of waiting around: this board had 3.9 million posts, 1 million private messages and 60GB of attachments.

But there was a problem that manifested as soon as I hit the submit button after entering the old database information. I got a HTTP 500 error and the PHP error log generated some really hard to understand error messages.

I spent the better part of two days puzzling it out, eventually assuming that it was environmental on the client’s server. So I moved a 4GB database and 60GB of files to my machine to try it there, but it happened again.

To shorten a long story, the convertor failed because configuration information for the conversion was serialized and stored in the database was not serialized properly. The configuration information contained escapes (\”, mostly). When PHP was told to unserialize these it couldn’t, returning NULLs where an array would have been expected. With a lot of trial and error I found if I removed most of these escapes it would unserialize properly and I could continue the conversion. But clearly something had changed in the conversion software.

I reported it but apparently the phpBB Group already knew of the error, and it will be fixed in phpBB 3.3.6. It was not reported in their support forum, which is where serious problems to a release are usually posted. Most likely it just got overlooked, but also after 15 years the phpBB Group probably figured hardly anyone runs conversions anymore, so why bother to highlight it?

There are two workarounds if you find yourself in this situation:

  • Wait for phpBB 3.3.6 or higher edition as it should be fixed
  • Install phpBB 3.3.4 instead of 3.3.5, run the conversion software and afterward update to phpBB 3.3.5

I tested the latter approach with another client yesterday and it worked fine.

Pricing update January 2022

On: By: Mark

I’m changing my pricing, which was last revised in a major way in February 2018. It’s been overdue for quite a while.

As I do more of this work, I’ve realized that that the pricing of certain work (generally the more complex work, such as large upgrades or conversions) doesn’t come close to reflecting the actual effort required, at least at my nominal hourly labor rate.

But also prices have been up in general the last four years, and inflation in particular has gone up in the last year or so. So I need to account for that.

I still offer both commercial and noncommercial pricing. Noncommercial pricing is discounted to 50% of the commercial rate. My base labor rate is now $70/hour commercial, $35/hour noncommercial.

One price was lowered: the cost of adding additional extensions, language packs, emoticon packs and rank images packs, at least if you want me to do three or more of these in the same job. These went from $30 to $15 each commercial, and $15 to $7.50 each noncommercial. That’s because when you do these in bulk, they don’t take much time. Most of the time is in finding the right versions. Uploading and installing them is generally simple.

Overall, prices were raised about 16%. For the more complex tasks like large upgrades and conversions, the pricing change is about 60%. This is because these tasks are time consuming and challenging, and typically involve a lot of close work with the client. There are usually plentiful questions and answers required, and careful attention to detail to satisfy the client. This results in a lot of time, so my pricing should reflect that. It may be that this time is still not fully captured in my pricing. I’ll reassess this in a year.

You can see my updated pricing summary here.