Blog

Upgrading and updating phpBB

On: By: Mark

(Post updated July 17, 2017 to reflect phpBB 3.2.)

In general, upgrading from a major version to another version (2.0 to 3.0, for example) or from a minor version to another minor version (3.0 to 3.1) is error prone and a hassle, particularly if you used shared hosting or are not technically inclined. In these cases you might want to contact me and have it done professionally.

The phpBB group has instructions for upgrading that are linked below:

Upgrades move you from a minor version of phpBB, such as 3.1, to another minor version, such as 3.2. Updates keep you on the same minor version but add patches and bug fixes, such as 3.2.0 to 3.2.1

There are many ways of updating phpBB 3.2 to a newer version of 3.2. The phpBB group suggests the automatic update package. I disagree. It is faster and simpler to use the changed files method. This is because the automatic update package involves the hassle of telling phpBB how to use FTP and setting various file permissions. In addition, I have found it error prone.

Here is how I upgrade clients easily with the changed files method:

  1. I back up the forum’s database locally using the Administration Control Panel, Maintenance tab.
  2. I make a file backup of the phpBB directory, usually using the web host control panel file manager. This is an extra precaution but is probably not necessary if you are careful with the remaining steps. Select all files and directories in the phpBB root folder and select the compress option. Give the file an easy to remember name like forum_files_backup_before_upgrade_to_3.2.1.gz. Store it wherever you want on your web host or download it, just make sure you don’t accidentally delete it.
  3. I download the changed files archive for the current version of phpBB.
  4. Once downloaded I expand (uncompress) the archive. Inside are more archives showing the changed file for each micro version. For example, when downloading the 3.2.1 archive you will find phpBB-3.2.0_to_3.2.1.zip archive. I’ll expand that. It will create a directory called phpBB-3.2.0_to_3.2.1. You should expand the archive for your current version of phpBB, which you can find on the main Administration Control Panel web page.
  5. I go into this folder. This contains all the files that have changed from that version to the newest version. Be careful not to accidentally delete the following:
    1. config.php
    2. files folder
    3. images folder
    4. store folder
  6. Carefully use FTP to upload these to your phpBB root folder using FTP. File folders should match. For example, if there is an adm folder in the archive there should be an adm folder in the directory you are uploading to, your phpBB root directory. You will overwrite the old files, which is fine because they have been improved to add features and fix bugs.
  7. Inside the main archive is an install directory. I upload that to the phpBB root folder.
  8. Inside the main archive is also a vendor directory. I upload that to the phpBB vendor folder, overwriting whatever is there.
  9. For upgrading to phpBB 3.2, change the URL to the install folder, ex: http://www.myforum.com/phpbb3/install. Click on the update tab. Select the option to update the database only.
  10. For upgrading to a newer version of phpBB 3.1, run the program install/database_update.php, ex: http://www.myforum.com/phpbb3/install/database_update.php.
  11. After completion, I delete the install folder using FTP.

That’s it, you are done.

If for some reason something is not working correctly, you just recover your files and database since they are backed up. You may need to manually clear the contents of the phpBB cache folder with FTP.

January 2016 work summary

On: By: Mark

In January, most of my work was concentrated with two established clients. Curiously few new clients came in through the email box.

For client #1,as part of a small team I am helping her to not only get her forum rehosted but also to solve a number of technical problems with her various websites, for which she lost permissions due to a tech that abandoned her. Basically she needed a lot of coaching and hand holding because she is not a technical person, but is happy to pay for services. It involved a lot of back and forth Skype sessions, and working with other techs to untangle her domains to do what she wanted done. Among the things I did for her:

  • I provided guidance on integrating Windows with a Mac (her primary computer) and suggested she purchase and install Parallels Desktop for the Mac so she could run a few Windows programs that she depends on without the expense of buying a new machine. I suggested she upgrade her Mac to 16GB of memory first (she had only 4GB).
  • She wants to use GMail to manage a number of email accounts on different domains. So I researched the GMail Pro service and the issues involved in pointing her MX records to Google so she could do this.
  • Discussed moving her static web site to a content management system, which will likely be WordPress. Found some WordPress plugins that might meet her need need for a directory solution.
  • Moved her forum from one host to another. Challenged due to some massively long posts that timed out when trying to load them into the database. Had to also fix some folder permissions.

Client #2 had a number of issues going on. He lost his support person and had to manage the forum by himself, and his group and group forum permissions were inconsistent and a mess. Some of the things I did for him:

  • Tutoring on how to put users into groups and answering general questions, like how to determine which permissions were used for a particular user
  • Answered questions on how to set up anonymous posting, and some of the potential downsides (including potential spam and the ability of the public to view his forums)
  • Created some offline reports in PHP to allow him to see the extent of her permission problems. Used some phpBB permission objects to delineate which groups could read forums. This was necessary because he had hundreds of forums and dozens of groups, making getting this information from the user interface difficult. Also created a report showing which users had forum permissions outside of group permissions. Eventually we figured out a procedure to “wipe” forum permissions for all groups, which I blogged about.
  • Patched phpBB so he could delete a forum. This was needed because his database is SQLServer and it is necessary to use the CAST function for numbers of a certain size, and phpBB doesn’t do this (it’s a bug). Put together an estimate to move their forum’s database from SQLServer to MySQL where these issues are unlikely to happen. Hope to migrate the data using MySQL Workbench.

Work for other clients:

  • Converted a forum from phpBB 3.0.7-PL1 to 3.1.6. Tried a number of styles, user settled on Platinum Red. Integrated their logo. Changed the PHP version to one supported by 3.1 (must be PHP 5.3.3+) and did the conversion on my own machine to get around PHP timeouts, which added considerable time and expense and made me rethink my pricing. I now have a $50 surcharge for these types of conversions for shared hosts, since the work must be done offsite. Restoring the search index also had to be done on my machine because of HTTP 504 timeouts on his shared host.
  • Assisted in a registration issue. Users got a false message saying that Javascript was not enabled. I had previously had the user install Cleantalk to address his spam problem. The problem turned out to be a Cleantalk issue that the vendor eventually fixed.

 

Smartfeed 3.0.3 RC2 Released

On: By: Mark

The main change in this release was to add a missing join that was causing a Cartesian product, but also changed some code to help the mod get approved. I am submitting this release candidate to the phpBB extension team for review and possible approval. It’s pretty unusual to get it approved on the first pass, but you never know.

It can be downloaded here.

How to reset group and user forum permissions for phpBB forums

On: By: Mark

phpBB has an awesome permissions system that is very fine grained and very comprehensive. While very functional it can often be hard to use too because there are so many possible permutations: group permissions, user permissions, role permissions, etc.

It’s not hard at all if you have a board with lots of forums and/or lots of groups to find yourself in forum permissions hell. A user might belong to multiple groups, for example. Which group permissions will override other group permissions? You might long for a button that just resets all the forum permissions, but nothing like this exists. You may want to comprehensively reset all your forum permissions instead.

Group forum permissions and user forum permissions allow you to finely tune these permissions. With lots of forums though the user interface can be overwhelming. It can be hard to click on the drop down controls to set them properly and to do this for all the forum permissions for one group. Worse, sometimes you’ll take the time to do this and you’ll find out not all the permissions took. (An aside: if this happens to you, it is probably because there are so many fields on the form you are submitting that PHP can’t handle it all, but won’t tell you about it. Upping your PHP max_input_vars setting may solve this problem. Or you can do it to a subset of forums at a time, rather than all.)

If you want to reset your forum permissions, try these steps:

  1. Backup the database first. You may make mistakes or you may find you don’t want to do this when you see the result.
  2. Create a forum called something like “No privileges forum”. Do not copy forum permissions. Note: if you do this and look at the permissions for a user or group for this forum, you’ll see “No role assigned”. This is not the same as no permissions at all.
  3. If you want to get rid of some of your user defined user groups, now is a good time to blow them away. It will make the rest of this easier.
  4. ACP > Permissions > Group forum permissions. For each predefined group, set the forum permission for the “No privileges forum” to “No access” and save.
  5. For each user defined group, do the same thing.
  6. Now copy these permissions to your other forums. ACP > Forums > Copy forum permissions. For the first field select the “No privileges forum” forum. For the Apply permissions to field, select all the other forums. Warning: at the end of this step, no one will be able to access any forums on the main index. In fact, no forums will show and you should see “This board has no forums.” This includes administrators. Don’t worry. The forums are still there just not accessible. The message is really incorrect, but presumably set this way to deter spammers.
  7. Now change the permissions for one forum (but not the “No privileges forum”) so that the group privileges are the way you want them for all your groups. This means selecting the forum role to apply for each group or (not advised) using the Advanced Permissions link if you need very granular permissions. (It is better to change the forum role permissions instead.) If you don’t set the privilege for a group they will have the No access role because of the previous step: the forum will not be seen on the index at all for that group (unless belonging to another group gives them that privilege). Start with the predefined groups then do any user defined groups. You will have to do this for each group of interest but always to the same forum.
  8. Now do step 6 again but in the first field select the forum whose privileges you just granted and apply it to all the other forums, except the “No privileges forum”.
  9. If you have user defined groups that should access only certain forums, set the forum privileges using ACP > Users and groups > Group forum permissions. Do this for each group. Again, it’s advisable to use a forum role.
  10. All user (non-group) privileges are now lost. If you need to add privileges for specific users outside of a group, do this now. It’s a best practice to not set individual privileges, but grant all privileges through groups.

Test. You can test as yourself or test using a user’s permissions, but all should be well.

Digests 3.0.0 (Alpha) released

On: By: Mark

Phew! What a journey. Obviously I got over the hurdle in my last post. The first version of this extension, 3.0.0 (Alpha) is complete.

You can find download and installation instructions on the phpBB topic.

Please do not use in a production environment. Provide any feedback you have on the phpBB topic, not here please.

My thanks to the phpBB Extensions Writers for their extensive help and coaching through this challenging process.

Running as cron issues with digests extension

On: By: Mark

The good news is that the class I created to override the messenger class is written and working correctly. The last issue is to get the mailer running as a phpBB cron task. I am currently stuck there and am seeking assistance to get past this last hurdle.

This latest ripple is discussed here.

It’s pretty mysterious and hard to figure out. TWIG is an open source replacement for phpBB’s templating system. phpBB calls it and TWIG generates the templates. The templates are needed to do the work of formatting the digests in the email. It works fine when I run it in manual mode but when I run it as a cron task it fails with the error described.

It may be that phpBB’s cron system assumes the templating system is not needed, which means I will have to add it. More likely it is something obscure about TWIG and I need a TWIG expert to suggest a fix/workaround.

Hard to know how long this will take. It may be something that is figured out soon, or not. Fixing this problem may expose others. It’s a critical component, however, so I can’t release without it.

Digests extension status report

On: By: Mark

In my last post on this topic, I outlined what I had left to do to put out a first release of a digests extension. Update below on each point:

  • Thoroughly testing the new way of sending digests via phpBB system cron. In creating a manual mailer and integrating it into the Administration Control Panel I have in effect created an easy to use test tool for mailing digests. So far though I haven’t actually emailed anything, but I can write individual digests to the digests’ cache folder. This has a number of advantages, the biggest one being that I can bring it up in a browser and carefully examine it for errors. Once inside a browser I can closely inspect it with Firebug and run HTML validation tests too. The digest extension will send HTML digests as HTML 5 instead of XHMTL 1.1, which is cleaner code. Anyhow, it’s much easier to test all the permutations with the manual mailer and I thoroughly went through testing all the features of the mailer except actually emailing the digest. I can’t get my local instance of XAMPP to send emails. To test that part I will do it on my website where that integration is built in. It’s the last step in development.
  • Completing the integration of all the features with the manual mailer and the digest cron job. Done.
  • Extending the messenger class to send HTML emails. Last piece of functionality to port and I’ll do that part on my web server (where I can send actual emails) and port those changes back to my local instance for packaging.
  • General code review and cleanup. I’ve cleaned up all the language variables and removed dozens of them that were no longer used. I’ve done a pass through all the major blocks of code but it will need a couple more passes as part of the code are unnecessarily complicated.
  • Updating the migration program to properly migrate digest settings from the 3.0 mod to the 3.1 extension. Still to do. This could potentially be time consuming. The goal is to support the mod from version 2.2.6 forward, but if that’s too complex I might have to require a later version, perhaps the last approved version of the mod.
  • Updating documentation including how to set up a system cron. To do as one of the last steps.

There are other things to do:

  • Verify that all the functionality for the file changes in the mod are incorporated in the extension. This is virtually done but there are a few controls and such that won’t be in the extension, mainly because the new architecture makes them moot. I had about a week of delay trying to get the interface where you can sign up for a digest upon registration working. I had to figure out how phpBB 3.1’s event architecture worked, find the right events and tweak the code to work with the event logic.
  • Thoroughly test the actual emailing part. This includes testing using a manual mailing and testing using a scheduled mailing by creating a system cron on my web server and making sure digests actually go out hourly.
  • Full system test. This involves creating a test instance and installing it with phpBB 3.1.6 and seeing if anything breaks. I won’t test all the permutations but try most of the major ones. There are usually a few things that need to be fixed and retested.
  • Publish the code to GitHub.
  • Package it and place it on this server, and link to it.
  • Place appropriate announcements on phpbb.com.

This will probably go out as an Alpha release, as it is well tested and having done so much work on the Smartfeed extension the code is much cleaner than my initial attempt for that extension.

I’m hoping the rest will take about two weeks or less but it depends in part on any business that comes in my mailbox.

December 2015 work summary

On: By: Mark

Here’s a summary of my work in December 2015, which was a bit lighter than normal. However, I still met my income goals for the year. When I was not fixing user problems I was mostly working on my digests extension.

  • A forum stopped functioning because the sessions table needed repair and the client had no idea how to do this. The more general problem was the storage engine for all the forum’s tables were MyISAM, so to keep the problem from recurring I changed the storage engine for all tables to INNODB, which is what phpBB does for new installations of phpBB. Later, a problem was reported that a newly registered user could not see any forums. After an investigation, I discovered there were no read or post permissions for members of this group for any forum.I tried to add them in phpBB but they didn’t take. An invalid form message occurred instead. Applying permissions for all forums did not work reliably. However, with perseverance I was able to get it to work for a category. I asked customer for guidance on what forums should be exposed to newly registered users and what their privileges should be. Also did an analysis of their spam problem and asked for authorization to fix it.
  • Client had issues when he cleared his forum’s cache. Images no longer appeared because images were no longer on his file system and he didn’t have copies of them. Client decided the best approach was to upgrade from 3.0.12 to 3.1.6. Client’s setup was peculiar because he had a dedicated host with a highly non-standard web host control panel. After a few hours of research I was able to puzzle through his hosting interface and I upgraded his forum from from 3.0.12 to 3.1.6 using the default prosilver style. I had to tweak his php.ini file to remove errors that showed up on the Administration Control Panel’s main screen. I provided client with rehosting advice since he was paying about $500 a month for a dedicated server he didn’t need. When he is ready to rehost I expect to help him.
  • I ghostwrote a technical article for a client.
  • I placed a new logo on the header of a forum, changed the styling to remove text appearing on the header and remove a blue vertical image. At client’s request, I also placed a logo on the login page below the login button. Later in the month I provided advice on her overall site architecture, which was not mobile-friendly. I suggested she move the non-forum content into WordPress, and suggested some shopping cart WordPress plugins that she could use for her electronic commerce. 
  • Client did his own rehosting but afterward the database kept disappearing and database was sometimes empty. His config.php file was pointing to the wrong database and the host was periodically recovering it, causing the weird symptoms. The database user attached to the forum’s database had disappeared, so I created a new database user, gave it needed permissions to the database and changed his config.php accordingly. I restored his database from a file backup. The program viewtopic.php was missing but I uploaded that from a phpBB 3.0.11 reference.
  • Fixed an issue with “Strict Standards: Non-static method utf_normalizer::nfkc() should not be called statically” error message using solution I found here. Forum was using phpBB 3.0.10. After discussion with the client about whether to upgrade to 3.1 or the latest version of 3.0, he decided to upgrade to 3.0.14. I upgraded him and reinstalled his 4 mods. I also upgraded my digests mod that he had installed from version 2.2.11 to 2.2.26. In addition, I tweaked the logo by putting it into the theme and placing an off-white color behind it. I provided guidance on properly using the Advertising Management mod. Later, at client’s request I made tweaks to the banner background image. A couple of days later client discovered his ads were not appearing. I had forgotten to make changes to his style for ad placement. Puzzled through them and had to tweak the style a bit to make them look acceptable.
  • Attended an hour-long Skype call (requirements discussion) with a client. At her request I did some minor editing of various web pages (outside of phpBB). I removed a Facebook icon, changed some link destinations and installed ACP Add User extension on her forum. This was because her forum is very sensitive and the client wanted her staff to create all users manually. I spent another 1.5 hours tutoring a staff member on how to do administration and moderation. Later in the month, I added the ACP Add Users extension to their sandbox forum.
  • I installed the Cleantalk extension on a forum, added the registration key, removed lots of inactive (spam) users and set up the Administration Control Panel for new users to confirm registration so admin didn’t have to approve these personally.
  • Client had changed the database password and updated his config.php but no one could access the forum. This was because the old password was actually embedded in PHP code in the forum’s cache folder. I cleared cache manually using FTP and the board was usable again. I provided instructions on how to do this manually should it recur in the future.

Digests extension update

On: By: Mark

I’m slogging through the conversion of mail_digests.php, which is now a run method on a mailer class in a cron folder of the extension. There are a couple of issues that are hard to puzzle through. First, I want the same program to be used both as a phpBB system cron (called once an hour) and be called manually from the Administration Control Panel when needed for troubleshooting purposes. (Previously you had to run mail_digests.php from a URL for initial testing. This won’t work with the new architecture.) Figuring out how to do this dual handshake though it pretty challenging.

Digests uses the messenger class to mail digests but the messenger class templating system is not smart enough to do loops, so I have to borrow from the templating system for the bulk of each digest’s content then copy and paste the results into the messenger class. I need loops in the template system to logically arrange posts into blocks of posts inside forums and topics. However, with the new architecture it’s not easy for an ACP program to use templates outside of an ACP environment. I eventually figured out that just copying these templates inside the ACP template area (ext/phpbbservices/digests/acp/style) worked when in an ACP mode.

Now I am focusing on how to save a digest to a file. So far I haven’t figured out how to do this, because the messenger class does not have an assign_display method like the template class has. I’m hoping the phpBB mod writers group can suggest something. It’s not a critical feature but I’d hate to remove it. I’ll most likely have to extend the messenger class to do this. Speaking of extending classes, I have to extend this class anyhow to allow it to send HTML formatted emails. I haven’t extended a class in any major way before, so that’s a learning curve I’ll be tackling soon.

The good news is that all the code from mail_digests.php is ported over. I am tweaking it so it can be used from the ACP. It will need extensive testing which will take time but to do this testing I have to send actual emails with digests out. And that’s a problem because I am doing development in a local environment. I can’t get XAMPP to send email. I tried configuring my development instance to send email via GMail and its SMTP server, but it won’t work. I think my iMac is blocking outgoing traffic and that’s the issue, but I can’t find anything on this with a web search. At some point I’m going to have to move it to my web host and finish development there and that will take some time. I can only go so far before I have to test actual emailing.

So it’s going about as I expected. My consulting work is light at the moment which helps. I’m guessing there is about a month more work before I have a development candidate to put out there for comment, but this may be optimistic. The major blocks of work remaining are:

  • Thoroughly testing the new way of sending digests via phpBB system cron
  • Completing the integration of all the features with the manual mailer and the digest cron job
  • Extending the messenger class to send HTML emails
  • General code review and cleanup
  • Updating the migration program to properly migrate digest settings from the 3.0 mod to the 3.1 extension
  • Updating documentation including how to set up a system cron

Digests extension update: new manually run the mailer interface

On: By: Mark

The new digests extension will include an ACP interface that will allow the admin to manually run digests. Previously this had to be done by specifying a URL. Mostly it is for initial testing of digests or for occasional troubleshooting. However, you will also use it to manually send digests for days and hours that might have been missed, such as when the cron job was down, using the option to create digests in the past. I have the front end done and am working on the backend code. Take a look:

Manually run digests mailer
Manually run digests mailer

This option will also let you write digests to files on your server to see them via a URL as HTML or text. They will hide in the extension’s cache folder. This potentially allows a static digest to be created showing, say, all public posts on the forum for a particular day or week. Such an interface might get more thoroughly thought through later. Right now this is mostly for testing and troubleshooting. Email clients can ignore HTML markup, but by writing digests to a file you can see it in a browser. Of course writing it to the extension’s cache folder also makes potentially private content publicly accessible, which is why the interface will have a way to clear this cache folder too.

On the backend, since mailing will be done via phpBB’s cron system, mail_digests.php goes away as a standalone program. Instead it will be invoked through a run method on a mailer class for the extension, which will hide in the extension’s cron/task folder.