Blog

Digests extension user control panel interface developed

I have replicated the digests modification user control panel interface for my digests extension under development. It all looks and behaves the same as it does for the modification. Fortunately a lot of the code could be copied and pasted with some changes for the new architecture. The hardest part was creating the interface in the first place, as the methods for doing so are not well documented. I spent a lot of time on issues like getting form keys to show as hidden fields. Fortunately, the people contributing to the phpBB extension writers forum were of great help. A snapshot of the Forums Selection screen is shown below.

Forums selection, Digests extension UCP interface
Forums selection, Digests extension UCP interface

With the Administration Control Panel digests extension user interface also complete, the next step is to port mail_digests.php, which as the name implies actually sends the digest. This will include challenges too including:

  • I will need to extend the mailer class to allow HTML formatted emails. I’ve never done this before.
  • It’s not advisable to print raw text to the screen anymore, so I’ll need to develop some sort of template that shows progress as digests are created and mailed.  Most of the time this won’t be seen as digests are run automatically.
  • Add logic to hook mail_digests into phpBB’s cron structure, as it will kick off digests.

So it’s fairly challenging work but a lot of the code can be ported. It’s the stuff above that will consume most of the time and puzzling.

Changing your phpBB tables’ storage engine

Note: this post was updated October 17, 2019.

Most phpBB forums use the MySQL database to hold the forum’s data. It’s an obvious choice because community editions are free and it comes bundled with your hosting package. If your database is MariaDB, don’t worry. These instructions work for MariaDB too. Why? It’s because MariaDB is a clone of an earlier community edition of MySQL. Functionally, it is virtually identical.

When MySQL was first created, there were few storage engines available. Storage engines contain the code (software) and internal data structure for storing data in tables. Consequently unless you have created a new forum it’s likely that your forum’s tables use the old MYISAM storage engine.

With phpBB 3.1, the default storage engine for phpBB tables using MySQL and MariaDB has changed from MYISAM to INNODB. However, the conversion program from 3.0 to 3.1 will not change the storage engine. INNODB is a different way of storing the data in a table. It’s generally more self sufficient than MYISAM and you are less likely to need to repair it. INNODB tables are best used for tables that are frequently read and written to. In fact, the REPAIR command does not work on tables using the INNODB engine.

I generally recommend to my clients to change their tables’ storage engines to INNODB because they are less likely to have issues. I see clients that struggle through issues like the sessions table needs to be repaired and they don’t know how to do it. Sometimes tables using MYISAM just get inconsistent leading to issues such as orphaned posts.

phpMyAdmin is a tool generally provided by your web host in its control panel. While you can change your storage engine in phpMyAdmin, it’s not intuitive. There is nothing in the software that allows you to do this in the user interface. You can do it on the SQL tab, however. You can use phpMyAdmin to change your storage engine but you can also do it from SSH with the right privileges by running the mysql command. These instructions assume phpMyAdmin is used.

A few caveats:

  • Backup your database first.
  • You should disable your forum during the process. Users might get error messages or timeouts while this is happening.
  • The more posts and users you have the longer it will take. Generally it takes only a few minutes. It’s possible that phpMyAdmin will time out. In this case go back in when you can, see which tables have been converted and convert any remaining tables. Simply look at the tables for your forum’s database and examine the TYPE column to see which tables remain to be done, which are those where the type still shows MyISAM.
  • If your sessions table uses a MEMORY storage engine, this is fine. You might want to use MEMORY for the sessions table only because the sessions table is accessed very frequently. The MEMORY storage engine very fast because nothing is stored to a file, but if the database server crashes or is rebooted all the rows in MEMORY tables are lost. This is okay in the case of the sessions table. It just means that people have to log in again.

These instructions assume you are using phpMyAdmin.

  1. Look at your config.php file for your forum. You can download it with FTP or you can view it in your web host control panel’s file manager. It should look something like this:
<?php
// phpBB 3.2.x auto-generated configuration file
// Do not change anything in this file!
$dbms = 'phpbb\\db\\driver\\mysqli';
$dbhost = 'localhost';
$dbport = '';
$dbname = 'forum_database';
$dbuser = 'forum_admin';
$dbpasswd = 'forum_password';
$table_prefix = 'phpbb_';
$phpbb_adm_relative_path = 'adm/';
$acm_type = 'phpbb\\cache\\driver\\file';

@define('PHPBB_INSTALLED', true);
// @define('PHPBB_DISPLAY_LOAD_TIME', true);
@define('PHPBB_ENVIRONMENT', 'production');
// @define('DEBUG_CONTAINER', true);
  1. Make a note of the value for $dbname
  2. Log into phpMyAdmin
  3. Select the database containing your forum, which is the same as the value of $dbname.
  4. You can see your tables’ storage engine in the TYPE column. If you are already using INNODB for all your tables there is no point in doing anything.
  5. Click on the SQL tab.
  6. Copy and paste the SQL below into the window. Replace the value in quotes for dbname with the value of the variable $dbname in your config.php file. The result of the query is a bit strange, because it renders SQL. You will use the generated SQL to actually change the storage engine.
SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' ENGINE=INNODB;') 
FROM information_schema.TABLES 
WHERE TABLE_SCHEMA = 'dbname' AND engine = 'MyISAM'
  1. It will render a list like this. In most cases the tables will start with “phpbb_” but if it doesn’t don’t worry. The tables should start with the value of $table_prefix in your config.php file. Note that what the previous query does is construct the SQL you need to change the storage engines, but only where the current engine is MYISAM. Only a few rows are shown for demonstration purposes.
...
ALTER TABLE phpbb_banlist ENGINE=INNODB; 
ALTER TABLE phpbb_bbcodes ENGINE=INNODB; 
ALTER TABLE phpbb_bookmarks ENGINE=INNODB; 
ALTER TABLE phpbb_bots ENGINE=INNODB; 
ALTER TABLE phpbb_config ENGINE=INNODB; 
ALTER TABLE phpbb_config_text ENGINE=INNODB; 
ALTER TABLE phpbb_confirm ENGINE=INNODB;
...
  1. Check the Show All checkbox if it is there to see all the rows.
  2. In phpMyAdmin at the bottom of the screen look for a link Copy to clipboard. Click on the link and all the generated SQL will be copied to your clipboard.
  3. Open a text editor and paste it into a new document. Remove any lines before the first ALTER statement. If there are any statements where the table name does not start with the value of $table_prefix, you might want to edit these out because these tables are probably not used by phpBB.
  4. Disable your forum while doing these changes: ACP > General > Board configuration > Board settings > Disable Board > Yes
  5. While you could just paste all these statements into the SQL window in phpMyAdmin again for your database, this is not recommended unless your forum is very small in size. Instead, it’s recommended that you do a block of these at a time, say ten statements per block. If there are no errors, then do the next block until all are done. For very large tables like the phpbb_posts table or the phpbb_search_wordmatch table, it is recommended that you do these individually. Each line that is executed should change the storage engine for the table from MyISAM to InnoDB.
  6. When all done, you can verify that the table storage engines have changed by clicking on the link for the database in phpMyAdmin and examining the TYPE column again. They should all report InnoDB.
  7. Reenable your forum: ACP > General > Board configuration > Board settings > Disable Board > No
  8. You don’t need to save the SQL in your editor.

That’s it! Enjoy!

November 2015 work summary

Consulting slowed a bit this month, which was good in a way. It allowed me to work on a beta version of my Smartfeed extension and toward the end of the month I started work on the Digests extension. As always, client information is anonymized. Here’s some of the phpBB work I did in November:

  • I continued ad hoc work for an existing client during the month. I enabled the Windows media extension group so various Windows media would play. First I had to fix another SQLServer CAST bug in phpBB that manifested when enabling the new extension group. The basic issue here is that the client uploads inhouse videos and attaches them to posts, but each video is very large (megabytes). phpBB keeps a configuration variable containing the size of all the items in the files directory. When this value is stored, SQLServer (the client’s database) can’t handle it unless the value is CAST as an integer using the SQL CAST function, which required a code patch that I added. I reported the additional code that is needed as an issue in the phpBB bug tracker. I also participated in an email discussion on the most efficient way to embed videos on the forum and recommended a HTML 5 approach (upload .mp4 files), which phpBB does not currently support. phpBB is behind on integrating HTML 5 videos smartly. I added this issue in the phpBB bug tracker. The template attachment.html needs to be updated to allow the browser to natively play .mp4 videos inside the new HTML 5 <video> tag so no plug in is required. Right now the Quicktime player is loaded, which means it has to be added as a browser plug in, and this is slow and doesn’t usually work on mobile devices. Also provided guidance on Google Analytics: getting an ID for tracking and setting it up. I answered a question about why a user was not able to download an attachment. I also monitored and replied to posts on their forum.
  • I converted a forum from 3.0.12 to 3.1.6 using the prosilver style. I installed the Tapatalk extension and installed and configured the Advertising Management extension, which required renaming a table to port ads from the 3.0 mod to a place where the extension could find them.
  • For one client I installed a new logo with an orange theme. Later in the month I placed a new ad for him and put it above posts in the view topic page. Later in the month I replaced the content of an ad. I investigated issue of a missing user. Apparently his account was deleted. I coached customer on how to find a good web host. The web host saw bad data in private message table in the message_text column. I repaired the table but when it recurred I then changed the storage engine for all tables except the sessions table set to memory to INNODB. So far this is working.
  • I upgraded a forum from 3.0.11 to 3.1.6. I installed the subsilver2 style and made it the default. I integrated the old logo. I provided guidance on using phpBB’s built in ATOM feed and suggested that the user use it to dynamically serve forum topics on his main web site. I provided a link to a phpBB knowledge base article on using phpBB’s feed and showed him some WordPress plug ins that had a feed widget that could consume the feed.
  • I upgraded forum from 3.0.9 to 3.1.6, using prosilver as the default style. Customer agreed to handle the styling issues. When I ran database_update.php, I encountered lots of errors that I had to address, the origins of which are unclear, which meant fixing data and trying again. To get a clean conversion I eventually emptied the migrations table. I still had to do some things manually. This included commenting out code to move the prune users module and emptying then recreating the data in the modules table because module names were not correct. Later in the month the user reported some more issues. After investigation I determined that an acl_option permission that the converter should have added must have missed. Without it a user could not change their profile. I added it and assigned it to 5 roles per my reference database and that solved that problem. A couple of days later another permission issue was discovered when the extensions tab would not appear in the Administration Control Panel. This was also due to a missing permission that I had to add manually. Then the tab reappeared. I double checked all the entries in the configuration table and all the permissions to make sure they were all there compared to a reference. They appear to be so I think there won’t be future problems like this.
  • I placed an additional ad for a client in the Advertising Management extension.
  • To address a serious spam problem, I installed the Cleantalk mod on phpBB 3.0.12. Client must pay $8 for 1 year of service.
  • A HTTP 500 error occurred only on registration. This was due to a bad configuration value for the CAPTCHA (wasn’t changed in the migration from 3.0 to 3.1 and needed the namespace). Post counts were also zeroed on the index page. I manually fixed index to show correct # of posts, last post time and last poster using various SQL queries. I added missing columns in forums table and missing rows in config table that phpBB 3.1 needs. I checked all the tables with a reference to make sure all columns were present for 3.1.
  • After rehosting mysterious HTTP 500 errors were generated on the new host and there was nothing in the error log to provide any clues about what went wrong. I eventually figured out that PHP was not compiled with an integration with the MySQL database. To connect PHP and MySQL, I used Unix yum command with the right integration package and the forum started working. Had to use SSH to do this work as the client had his own server and acted as its tech.
  • A user who upgraded his board for some reason was unable to create a new forum inside it. After investigating, I determined that a column in the forums table did not have a default value. I gave it a null value as a default. Once things were stable I upgraded the forum from 3.1.5 to 3.1.6.
  • I upgraded forum from 3.0.12 to 3.1.6. The old forum had lots of mods but none were critical. Changed style with upgrade from subsilver2 to prosilver.

Digests extension update

Yes, I have started work on the Digests extension at last. With the Smartfeed extension now in Beta, I felt I understood the 3.1 architecture well enough to tackle Digests, the more complex and popular of my two mods.

And it is moving forward rather predictably so far, thank goodness. I can copy and paste a lot of my code from the Digests mod, but much of it has to be retrofitted requiring lots of global find and replaces. But some code that used to work simply doesn’t anymore, and you discover these when you see error messages or when things don’t work right. So everything has to be carefully retested and given that there is a lot of business logic in the extension, it just takes time. Digests is definitely in the top 10% of extensions for code size and complexity. Anyhow, the screenshot shows the Edit Subscribers page in the Administration Control Panel, now moved to the Extensions tab. All the screens look pretty much the same. I removed a couple options that are now handled by the improved architecture.

Edit Subscribers
Edit Subscribers

(Note: the above image shows robots and search engines as subscribers. I changed their user status to member temporarily for my convenience of testing the pagination logic.)

It all looks so easy you are probably thinking the rest will be a breeze. But in some ways this is the easy part, which is why I started here. The User Control Panel interface has to be created, and this will be new since Smartfeed has no such interface. mail_digests.php will need to be recreated and it will have to be made to work with the new improved cron interface built into phpBB 3.1. However, a real cron job will still be needed to ensure digests go out hourly because if you dig into the internals of phpBB’s updated cron capability, it’s up to you to program a real cron job. Of course the documentation will need a major update as well.

I do expect that the migrator program will ingest all your digests settings from phpBB 3.0 so those will not have to be recreated.

So I am guessing I am 1-2 months from having a first cut of a development version for testing. If consulting picks up delays may get extended. Sorry it takes too long but just remember the price is right: it’s free. My consulting is not and when I have business I will prefer it doing this work for no charge.

Fixing phpBB spam registration problems

(Note: since this post is frequently read, I updated it for phpBB 3.2 on June 28, 2017 and August 1, 2019)

I am frequently sought out to help address issues with spam registrations on phpBB forums. The most typical spam problem I encounter is not spam posts, but spam registrations. The symptoms are lots of bogus usernames in your member list or inactive user list, often strings of random letters and numbers for the username, with a “website” listed in their profile that points to a spam site.

I am happy to fix these for you. You may wish to try some of the solutions below first. The solutions discussed apply principally to phpBB 3.0, 3.1 and 3.2.

Getting rid of existing spam registrations

Most of these “users” don’t bother to complete registration, simply want to leave a spam memberlist link and be gone. Consequently they show up as inactive users.

  • Inactive users can be deleted in the Administration Control Panel. ACP > Users and Groups > Inactive Users. Select the inactive users you believe to be spam registrations and in the little drop down box in the bottom right of the screen select Delete and press Submit. You may have to do this many times for many screens to get rid of them all. (Note the Mark All link at the bottom of the page. This can speed up things.) This is the safest approach.
  • You can also use the Prune Users function: ACP > Users and Groups > Prune users. A user that has not completed registration is not necessarily a spammer. Unfortunately, the Prune User function is not smart enough to be able to examine the profile website field to see if it contains data as this is often the key to filtering out likely spammers. However, if you prune a legitimate inactive user they can always come back to the board and register again. To remove these users it helps to look at your memberlist by date and see if you can figure out when the spam started. In the Prune users utility you can enter this date in the Joined field. Entering 0000-00-00 in the Last Active field essentially is the same as filtering inactive users only. You can select the option to delete posts for these users too but if they haven’t completed registration there should be no posts to delete.
  • You can globally delete these on the backend with SQL but beware: you may also delete legitimate new users that haven’t finished the registration process. Running any raw SQL statement is inherently risky, so backup the database first! Make sure you select the right table prefix (phpbb_ is shown in the examples) for the users table. Your config.php file contains your table prefix, which is usually “phpbb_” as well as the database you are using. If you have more than one application this will distinguish the one containing the data for your forum. You can use a tool (typically phpMyAdmin in your web host control panel) to issue SQL to delete these. Here’s an example of SQL for MySQL that will remove all inactive users with zero posts but with a website URL:
    • For phpBB 3.1 and 3.2:
delete from phpbb_users
 where user_id in
 (select user_id from (select u.user_id from phpbb_users u, phpbb_profile_fields_data p
 where u.user_id = p.user_id
 and user_posts = 0
 and user_type = 1
 and pf_phpbb_website <> '') as u)
    • For phpBB 3.0:
delete from phpbb_users
 where user_id in
 (select user_id from (select user_id from phpbb_users
 where user_posts = 0
 and user_type = 1
 and user_website <> '') as u)

Reducing spam registrations

Since most spam comes in the form of registrations and not spam posts, generally tightening up the registration process can reduce or eliminate these registrations. Many admins require administrator approval for each registration, but this becomes labor intensive. Most CAPTCHAs have been thoroughly hacked and are to be avoided. Here are some better alternatives:

  • Tighten up your spambot countermeasure to something more likely to work, like the question and answer one.
    • ACP > General > Board configuration > Spambot countermeasures
    • Under Available plug ins, select Q&A and press Configure
      • Create a question that is unique to the focus of your forum and won’t be guessed with a Google search. Create as many questions as you want. All must be successfully answered to complete registration.
      • Go back to Spambot countermeasures, select Q&A again and press Submit. This will change the countermeasure.
    • If this stops the spam, it’s probably safe to change it so the admin doesn’t have to approve every registration. An email verification is a good approach. ACP > Board configuration > User registration settings > Account activation > By user (email verification)
    • As a best practice or whenever you start to notice spam registrations again or quarterly change the registration question(s)
  • phpBB 3.2 supports Google’s new version of reCaptcha as a spambot countermeasure.
    • First go to the reCaptcha site and generate the public and private keys you will need for your domain.
    • ACP > General > Board configuration > Spambot countermeasures
    • Under Available plug ins, reCaptcha and press Configure
    • Enter the public and private keys from the reCaptcha site into the fields and press Submit.
    • Go back to Spambot countermeasures, select reCaptcha again and press Submit. This will change the countermeasure.

Some sites get hammered and even these steps are not enough. If this happens to you try these options:

  • phpBB 3.2 has an Akismet Anti-spam extension. Using it is not necessary free, but usually is for personal or club forums. The extension doesn’t cost anything but the Akismet service requests money depending on your usage. The extension also checks posts for spam.
  • If you don’t mind paying $8USD a year, my customers report 100% success with Cleantalk. You will have to download the modification or extension, create an account on Cleantalk and enter your registration key in phpBB. It’s worth the time and expense.
  • phpBB 3.1 and 3.2: Install the Stop Forum Spam extension. This checks the IP and some other information of the user registering or the guest poster against popular blacklists, and if there is a match they cannot post. It’s not 100% perfect so some legitimate people may not get through and it’s also possible some spam will get through.
  • phpBB 3.1: Install and use the Sortable Captcha extension
  • phpBB 3.0: Install the Advanced Block Mod. Warning: this is a very complex mod to install and configure correctly. You will need to point it to more updated blacklists. You may want to have me install it professionally.

I will cover how to remove spam posts in a future blog post.

Monthly work summaries

I removed the Recent Work page I created showing my monthly work summaries. Instead, I created posts after the fact for each month to show work for that month. As this information is more of a journal, using posts is  more appropriate. I’ll be posting these work summaries for the previous month near the start of the month.

All client information is removed to ensure their confidentiality.

 

Smartfeed extension version 3.0.1 (Beta) released, digest extension work to start

Smartfeed 3.0.1 User Interface

Business slowed a bit allowing me to update the Smartfeed extension under development. I’m labeling this version 3.0.1 as a Beta version as it seems to be stable and features seem to work as advertised. You can download it from my Smartfeed page. There are no new features but lots of code improvements have been made and some bugs found and addressed.

I feel I understand this well enough now to begin work on the Digests extension, which is good news for you suffering digests users.

October 2015 work summary

Less work than usual this month, in part because I was settling into a new house and that took precedence. Fortunately, work slowed down to give me time to do things I had to do anyhow, so no stress!

  • Painful lesson learned in incomplete backups. See this blog post. Geeky details: I tried to upgrade a forum with 1.4M posts from 3.0.11 to 3.1.6. Upon running database_update.php I encountered an error: Fatal error: Class ‘Symfony\Component\HttpKernel\DependencyInjection\Extension’ not found in /home/mattash/public_html/forum/phpbb/di/extension/core.php on line 25. I figured it was likely due to bad or inconsistent data in the client’s 3.0 database. So I installed the phpBB support toolkit and fixed issue with certain forums not appearing by running the left ID/right ID utility. I also removed some empty topics. I changed phpbb_users table to change language from “english” to “en” — it’s unclear how this got so nonstandard. I slowly upgraded from one micro version of 3.0 to another, from 3.0.11 to 3.0.14, hoping to mitigate risk. I tried again with a fresh install of 3.1.6 but same error occurred. I sought help on topic on phpbb.com but none of their suggestions worked. I tried a third time to do the conversion with a clean copy of 3.1.6 but same result. Could not restore database after 3.1 upgrade attempt. This was because the downloaded backups were not complete. I asked client to see if web host had a backup and to restore that. Eventually I recovered the database by patching together pieces of other databases and writing a program to create the missing topics. With a complete archive, for cost reasons the user decided to leave it as an archive and create a new 3.1 forum. Once it was created, I installed a bunch of extensions and moved his new forum into the web root. I installed the Board3 Portal extension on the new site and it is set to be the default for the site.
  • Ad hoc work for an existing client during October. Patched db.php to use a SQLServer CAST SQL statement because a SQL Server error occurred when trying to update a config_value for a row in the phpbb_config table due to a too small integer for the value that needed to be written. I made another patch to add a CAST statement when another function trigger a similar error. Enabling a new extension group caused an integer overflow again so I yet again added more code to use a CAST statement for SQLServer. I attended a Skype conference call with client and answered questions from his employee and on the forum. I documented a MCP screen overlay issue with a phpBB bug report. I put in an enhancement request on phpBB’s bug tracker to use the <video> tag for .mp4 files so plug ins would not be needed for these files. I fixed another SQLServer issue with integer overflow on Manage attachment link.
  • First I addressed an issue where phpBB could not connect to the database. The password in the config.php file apparently was no longer valid for the database and database username. So I created a new password in cPanel for database user and changed config.php to reflect the new password. Then I upgraded forum from 3.0.11 to 3.1.6, reinstalling the logo and using the ported prosilver_se style.
  • Upgraded a forum from 3.1.2 to 3.1.6.
  • Installed a beta version of the Advertising Management extension for a client. While I was in there I upgraded phpBB from 3.1.5 to 3.1.6. I later placed four ads in the beta Advertising Management extension since the user needed some help with HTML for the ads and added code to track clicks.
  • I ghost wrote an technical article.
  • I was paid for an hour of miscellaneous work over last few months plus placed a new ad in the top spot for new advertiser for the client.
  • Client was having a terrible spam problem. I first removed some spam users with zero posts and a listed website URL in profile, 444 users altogether using SQL. I later removed 6193 bogus topics by guest on the same forum, 976 bogus posts by guest and removed 33 orphaned posts, all with SQL. Everything is clean and shiny again, and spambot countermeasures and permissions were improved to reduce the likelihood of this happening again.
  • I upgraded plain prosilver forum from 3.0.x to 3.1.6. The database upgrade took a long time because of 300,000 users, most of them bogus. I also removed these inactive users from the database.
  • I upgraded my digests mod from 2.2.25 to 2.2.26 for a client to fix an error where an inactive user who had unsubscribed was getting a digest.
  • On a phpBB 2 forum, the unread and new private message counts were incorrect in the users table, triggering an error when trying to store a negative value for the number of saved messages when the column type did not allow it in MySQL. I did a number of queries, exports and imports to fix these values. I also added a real name statement next to username on the registration screen.

Perils of phpBB consulting

I’ve been doing this phpBB consulting work part time for more than nine years now. Lately I’ve learned a painful lesson that you can never be too careful.

A client with a 1.4M posts database wanted to upgrade from 3.0 to 3.1. I dutifully did what I always do before starting any work: backing up the files and the database.

The files are straightforward to backup. The database on the other hand…

Shared or VPS  hosting have some disadvantages: resource limitations that are not always known. It’s more problematic with shared hosting. phpBB admins suffer through these all the time. Database backups don’t finish because of HTTP 500 timeouts, for example. That’s what happened with this client. Not too surprising that with 1.4M posts, the phpBB backup would time out because shared hosting can’t create a backup in the number of CPU seconds that a program will get from the host, allocated in Apache and PHP’s configuration.

So I went into the web host control panel, kicked up phpMyAdmin and used its export utility. With 1.4M posts, the download took a long time, 15 minutes or more. But it downloaded, or at least appeared to download successfully. There it was in my downloads folder, and ungzipped it was 2.4G. I made a number of these over the several days I was working on the project as I tried a number of approaches to upgrade the forum. Kept getting mysterious errors documented here.

No problem if the conversion failed because I had both a file and a database backup. Of course, reloading 1.4M posts and associated data takes a while. First I had to drop the tables. I ended up using a staggered MySQL importer to import everything. It took hours.

But after all that effort when I tried to bring up the board, it complained about missing tables.

I finally looked at the bottom of the database export using the Unix tail command. The export was not complete. It had timed out, but there was no symptom that it had. All my other exports had failed for the same reason, so none of my backups were complete, and there was no clue to let me know something went wrong.

So I’ve spent the better part of a week trying to bring it back. We were at the mercy of the web host, who had some stuff backed up, but inconsistently. Some tables had newer tables than others. Fortunately the posts table looked complete and a copy of the users table looked complete. I wrote a program to recreate the missing topics. So things are back, sort of. Client is not happy but under the circumstances he was lucky to get it back at all. And I encountered an issue I never had before and learned a painful lesson.

It’s for these reasons that I’ve revised my pricing to up my fees on boards with lots of posts and users. This work becomes unprofitable at a “one size fits all” rate. There are too many minefields. Now I have to verify that my backups are actually complete. I can’t rely on software to do its job.