I ran into a major bug in phpBB doing a conversion for a client to phpBB 3.3.5 recently.
A conversion is when you migrate from a different forum solution to phpBB. It’s also used for phpBB 2, which was retired in 2007, as the architecture for phpBB 3 is completely different than phpBB 2. You would think after 15 years there would be no more phpBB 2 boards left. My clients would disagree. I get probably half a dozen of these a year, almost all from phpBB 2. It is amazing that there are so many of these boards still around. phpBB 2 was very simple, but very reliable.
The conversions used to be painful but the convertor software has become much more reliable. Now most conversions have minor issues at best. So I wasn’t expecting any trouble when I did a recent phpBB 2 conversion other than a lot of waiting around: this board had 3.9 million posts, 1 million private messages and 60GB of attachments.
But there was a problem that manifested as soon as I hit the submit button after entering the old database information. I got a HTTP 500 error and the PHP error log generated some really hard to understand error messages.
I spent the better part of two days puzzling it out, eventually assuming that it was environmental on the client’s server. So I moved a 4GB database and 60GB of files to my machine to try it there, but it happened again.
To shorten a long story, the convertor failed because configuration information for the conversion was serialized and stored in the database was not serialized properly. The configuration information contained escapes (\”, mostly). When PHP was told to unserialize these it couldn’t, returning NULLs where an array would have been expected. With a lot of trial and error I found if I removed most of these escapes it would unserialize properly and I could continue the conversion. But clearly something had changed in the conversion software.
I reported it but apparently the phpBB Group already knew of the error, and it will be fixed in phpBB 3.3.6. It was not reported in their support forum, which is where serious problems to a release are usually posted. Most likely it just got overlooked, but also after 15 years the phpBB Group probably figured hardly anyone runs conversions anymore, so why bother to highlight it?
There are two workarounds if you find yourself in this situation:
Wait for phpBB 3.3.6 or higher edition as it should be fixed
Install phpBB 3.3.4 instead of 3.3.5, run the conversion software and afterward update to phpBB 3.3.5
I tested the latter approach with another client yesterday and it worked fine.
Most of my work is pretty much the same: upgrades and updates to existing phpBB boards. Lately though I’ve been tackling some unusual projects. One of these is to move a Delphi board to phpBB.
Delphi forums are proprietary and are available only on their website. I understand it has no official way to export a Delphi forum. A board there has twenty years of conversations they did not want to lose. A clever user though has figured out how to export the board using YAML.
Example of YAML file for importing
After looking at it, I figured it was cost prohibitive to do. I did a proof of concept for them, demonstrating that I could read the YAML file and get basic user, forum, topic and post information from it, and display it neatly in HTML. I explained that this would be custom work and that it would take a lot of labor, even at my nonprofit rate. I estimated $2000 – $3000 and assumed it was cost prohibitive.
Proof of concept to display user information from YAML file
That was back in February but I was recently given the go ahead to start this project. Which raises the question: how on earth to do something like this?
What convertors from other forum solutions to phpBB that do exist tend to be for phpBB 3.0, and we’re now on phpBB 3.3. And no one has written a Delphi forums converter because it has no export capability. Since then phpBB has evolved, and due to the way post text is now encoded, it’s very challenging to write any new convertor.
Which is why I said if it can be done I would want to populate an old phpBB 2 forum with its data, and convert that to phpBB 3.3. There’s a convertor built into phpBB that does just this. But phpBB 2 became obsolete 13 years ago. However, its database is relatively simple so it should be straightforward to populate from data elsewhere.
But phpBB 2 was built for versions 4 and 5 of PHP, versions now obsolete and dangerous to use, or even acquire. It’s hard to even stand up a PHP 5 environment now. Most web hosting won’t support PHP 5, and mine doesn’t. So I’d have to roll my own development environment with PHP 5.
There are two ways to do this:
Via a virtual machine
Via a separate machine
I’ve used VitualBox on my Mac for years for standing up virtual machines. While it works, it doesn’t work well on a Mac. It’s a pain to use, particularly if you need to move files between the Mac and the VM.
Because I need PHP 5 for clients generally at least once a month, I decided a separate machine was more practical. So I bought a Raspberry Pi 4 (4GB) for less than $100. The actual machine is about $35 but I added a keyboard, mouse and a book. It took some help online to figure out how to install a web server and PHP 5 for it. The Pi is cheap because it uses RISC-based processors, so it’s not Intel compatible. Also, I could use FTP to move files to it, and using a free VNC viewer, I could work on it from my Mac desktop.
Processing the YAML turned out not to be easy. I tried various things, including PHP’s PECL YAML class, which I couldn’t get to work. I was able to eventually get Symfony’s YAML class to work.
Given the go ahead, I created a phpBB 2 board on my Pi with PHP 5.6 installed. While this worked great, my script wouldn’t run there. Symfony requires PHP 7.2. So I exported the phpBB 2 database and imported it into my Mac where I run PHP 7.4. I would have to populate the database from the YAML files I was given there, and eventually move it back to the Pi when it was time to try to upgrade the database.
Loading the phpbb_users table was not too challenging. A few fields needed to be escaped using mysqli_escape_string so the data could be loaded into the table. Some fields had to be trimmed to fit inside the maximum field length.
phpBB 2 places user data in a number of tables. Topics and posts haven’t been created yet, but there is a phpbb_user_group table. Each user had to belong to a group. Creating a test user on phpBB 2, I learned that it will create a new personal group. Then I had to write some SQL to put all 600+ users into this table. I also needed to populate a number of other columns in the phpbb_users_table. Each should have a new password, but since these users were added in bulk, they share the same password for now. All share the same bogus email address too, since I didn’t have that in the YAML. They will have to change both after the board is converted. Back in the old days, the password was encrypted with a simple md5 hash.
But the customer’s thread YAML wasn’t 100% clean. It wasn’t a major issue, but 22 threads (which amount to topics) wouldn’t parse. Even the good threads though had some issues. Some threads have no name to them. I will have to create a synthetic topic name for them. Some don’t have a forum name either, so those have to go into some sort of catch all forum.
There is a major issue is that the YAML’s attachment information provided a file upload name, but the name is not in the /files folder provided with the YAML. Files seem to have some sort of hash name. This may mean that attachments cannot be imported. If I want to import any attachments, I will have to install the old phpBB 2 attachment mod.
But at least this approach looks viable, if tedious. More learning experiences ahead.
