Blog

Smartfeed 3.0.0 Development available for testing

On: By: Mark

Finally, a very first version of my Smartfeed extension, obviously not ready for production, is out for testing and comment. More detail is available on my Smartfeed page. Comments should be directed to the phpBB topic. The GitHub project is here.

Sorry it took so long. My consulting work (as well as moving my household and buying a house) has taken a lot of time. It took a lot of time and help to chew over various technical issues.

There is obviously a long way to go before the extension is approved, but it’s a start.

September 2015 work summary

On: By: Mark
  • Created an updated version of rss.php originally written for phpBB 2 by some other developer that would work with phpBB 3.1. This allowed the client to seamlessly integrate recent topic titles into dynamic content areas on his other websites and elsewhere on the domain hosting the forum.
  • Upgraded forum from 3.0.7-PL1 to 3.1.5. Had to change some table columns collation types to UTF-8 and install en_us language pack to complete upgrade. Reinstalled logo. I then spent two hours systematically finding all BLOB columns in his database and changing these to the reference implementation for MySQL, mostly VARCHAR and TEXT. I also went through all tables and looked for columns where the collation was not UTF8-BIN and changed it to this. This should address any future issues with inserts failing due to no default values for a column. Note: there were subsequent issues with default values. Went through all the tables comparing schemas with a reference and adding SQL to give default values to columns where they were missing. Also checked indexes and fixed a few when there were concerns about performance. Changed all tables to use the INNODB engine.
  • Upgraded a forum to 3.1.5 from 3.0.12. Assisted with malware removal by removing two files identified by the SiteLock software ($15). Fixed an issue with getting into Administration Control Panel resulting in a HTTP 500 error. The problem was because the board start date was not formatted in the database as a Unix timestamp.
  • Added 600GB to a volume on a Linux dedicated server that had ran out of quota.
  • Digests cron was deleted resulting in no digests going out hourly. Made various attempts to create a new cron that successfully sent digests but nothing worked. Got it to work from my uptimerobot.com account. User then created their own account on uptimerobot.com. Created the monitor there and removed it from my monitor.
  • Worked to add the look of AcidTech Green (a style for phpBB 3.0) to a client’s phpBB 3.1 forum. Decided on using a 3.1 DVGFX style, which is prosilver based. Reused the old logo, but it took a while to figure how to center it. Had to change HTML in overall_header. Integrated the menu from the web site main page into the forum. Adjusted some colors for forum titles, topic titles and post titles. Waiting on FTP permissions for the main site so I can replicate the new forum logo on other static web pages.
  • Upgrade forum from 3.1.5 to 3.1.6.
  • People could not log into forum. Mangled front end splash page contributed to problem. Created 3.1.3 instance in a /forum folder and attached it to his database. Updated links on splash page for contact page, member list and member login. Changed permissions so guests could see memberlist to replicate old functionality that did not work. Previous developer developed a 3.1 style that had many customizations on it, based on Artodia Elegance Black, but seems to be lost. Client is trying to recover it so I can replicate the changes to the style in the new /forum folder. Only 3.0 styles were found in the recently recovered website folder.
  • Upgraded forum from 3.1.5 to 3.1.6. Still dealing with some registration issues of new members being treated as spam users. Need more info on that.
  • Client upgraded board to 3.0.14 and users could not register. Version looked old. Upgraded him to 3.0.14 and reapplied Board Portal 2.0.2 and the abandoned Dicemod 8.8.8b and it came up and I could register.
  • Work for September for an existing client. Worked to define requirements and find a developer for an extension that client would like developed. Participated in a Skype conference call with client to discuss various issues. Installed a larger 32×32 favicon at client’s request. Discussed viability of a phpBB access form with client. Upgraded forum from 3.1.5 to 3.1.6. Participated in discussion with client regarding concern about inability to drop files into phpBB. Discussed issue of potential bug with Topic Preview extension with client. Converted phpbb_config table to expand config_value column to varchar(8000) to address probable bug that occurred when a client added attachments to a post. That did not solve the problem so I did more debugging. The real problem was that an explicit SQL CAST statement was needed in phpBB for SQLServer. Created a patch to /phpbb/config/db.php to address phpBB bug and filed a bug report on phpbb.com.

Downtime

On: By: Mark

Because I will be moving on September 24 and won’t have internet service at my house until September 30, I expect that I will not be able to provide any phpBB services during this period. This may change if a friendly neighbor decides to let us use their Wifi. Blame it on Comcast, which thinks customer service is giving you an appointment only two weeks from when you call.

August 2015 work summary

On: By: Mark
  • Upgraded a new forum installed via a web host control panel which had version 3.0.12 to 3.1.5. Installed Share On extension. Cropped and reduced user’s logo by 25% and made it the new logo for the forum.
  • Fixed a problem a user was having uploading pictures to his forum. Upon investigation the problem turned out to be with PHP, not phpBB. PHP requires a folder with temporary space, and it was filled on his IIS web server. Deleting the 60,000+ files solved the problem.
  • Provided continued ad-hoc support for a client. Installed a favicon.ico file and linked it to their site. Provided guidance on how not to password protect a new forum. Answered a question about AirDrop and phpBB. Referred client to various developers skilled at creating extensions, as client wanted an extension built more quickly than I could accommodate him. Fixed an issue so that first name would appear in the list of members.
  • A client’s phpBB forum stopped working when GoDaddy changed the server name of his database server without informing him. Client made a few unsuccessful attempts to upgrade his phpBB 2 forum in hoping it would fix the problem. It made the problem worse and numerous databases were created with incomplete data. Worked with GoDaddy to recover the tables from his old database server, but their copy contained data from 2012. Client found a copy he made a few days earlier that looked intact. Successfully loaded it into the new database. Determined it was running phpBB 3.0.7-PL1. Installed that version of phpBB and got the forum back up. His export though marked numerous columns typically created as VARCHAR as BLOB and these BLOB columns did not have default values. This made it impossible to create new posts, post a new topic or update the user control panel. I fixed these individually by altering these columns in the database to match the schema in a standard MySQL instance of phpBB 3.0 which included default values on insert. Customer is now in stable mode and will let the forum run for a week or two to see if there are additional instances before upgrading to phpBB 3.1.
  • Client upgraded his forum to 3.0.14 and in doing so overwrote file changes for my Digests mod. I reapplied the file changes and upgraded his version of digests from 2.2.23 to 2.2.26.
  • Completed a long upgrade from phpBB 2.0.15 to phpBB 3.1.5 for a huge forum with close to a million posts. Tried various conversions by moving it to my local machine but there were always issues during the conversion program that even with my dedicated machine caused the conversion program to crash. Did this on OS/X using XAMPP, a virtualized instance of Windows also running XAMPP and on the client’s shared server hosted by Hostgator. I used all my tricks that had worked in the past and nothing worked at converting the posts. I investigated the issue on phpbb.com but there was no similar case. I wrote debug code in the conversion program. It always hung in a block of 20 posts after converting about 300,000 posts. I examined the data in these posts to see if there was a SQL injection issue in the post text but found no issues. Examining the Apache error log and my Mac console, the error appeared to occur because an Apache mod ran out of memory, but it was unclear what mod or data triggered the memory condition. I advised the client a highly qualified Linux system administrator might be able to figure out and solve the cause of the crash, but since it happened in various operating systems it would be expensive to solve. Client opted to make the old phpBB 2 database read only, and stand up a phpBB 3.1 instance instead. I was able to get the conversion program to complete if I emptied the posts and topics table. The user was glad to be able to carry over all the users, forum names, avatars, etc on the new instance. This is the first time I was unable to migrate all the posts from a phpBB 2 instance to a phpBB 3 instance.
  • Upgraded forum from 3.1.4 to 3.1.5. Added new style SE Square left, shrunk and placed logo, added an ad on the top that is linked to website.
  • Upgraded forum from 3.0.11 to 3.1.5. Installed subsilver2 style and made numerous changes to match old colors, font sizes, etc. Had to upgrade PHP to 5.4 first. Enabled a question on registration feature to reduce spam. Changed config.php file permissions to 640 per request. There was lots of back and forth with the client figuring out where the cPanel was and the proper authorization credentials.
  • Helped consultant integrate a WordPress wrapper around a phpBB 3.1.5 forum. we_universal style was installed. I partially installed the social login extension for phpBB 3.1. Integrated some ads. Made some tweaks based on customer feedback.
  • Upgraded forum with no special requirements and few posts from 3.0.12 to 3.1.5.
  • Upgraded forum from 3.0.12 to 3.1.5 and rehosted it as well on a server that supported PHP 5.3.3+. Added Cleantalk extension and added user’s Cleantalk key. Integrated their logo. Added a home link. Added shrunk logo when screen width dropped below 700px. Final conversion took 5 hours of effort due to issues associated with running database_update.php against a newest version of the database, previous version had worked fine, but it kept erroring out. Client had installed a clean copy of phpBB 3.1 and what appeared to be a clean database, but there were issues anyhow during the conversion. I had to work around some issues, fix bad module_basename values in modules table, add some rows to config table before it was stable and acting normally.
  • Eight hours of work to make numerous small styling changes that the client wanted and to make various dashboard images show based on various screen widths.
  • At an editor’s request, expanded a previously submitted article about data security to include topics they wanted discussed.

July 2015 work summary

On: By: Mark
  • Added a fancy HTML button to a web page for a client to take the user to their forum.
  • At a client’s request I looked into his virtual server to see if his mail server was sending spam. I could not find any evidence that it was, but I did determine that it was configured to not keep a log of outgoing emails.
  • Updated a forum from 3.1.4 to 3.1.5 and provided guidance to client so they could do these in the future.
  • Upgraded a forum from 3.0.7-PL1 to 3.1.5. Installed and configured a new style. However, certain email notifications were not going out and in other places when a user tried to submit a post or reply to the post, a white screen occurred while the post was actually saved. This also affected private messaging for certain users. Eventually I figured out that the old forum supported a number of language packs that were not installed in the upgrade, with the primary problem was that the en language pack is installed by default while most users were expecting the en_us language pack. But there were lots of other lesser used language packs as well. I solved the problem by making every user use the en language pack via a MySQL SQL UPDATE statement. I also filed a bug report for this deficiency on phpbb.com’s bug tracker.
  • Provided continued ad-hoc support for a client, mostly answering questions and providing troubleshooting when problems arose.
  • Ghost wrote an article for a client for a technical publication.
  • Placed more ads for a client on their forum. Installed Precise Similar Topics extension. At client’s request removed a lot of padding in various places to give the prosilver style a tighter look.

Smartfeed extension update

On: By: Mark

I’ve been slowed a bit by vacation but I am still progressing on the Smartfeed extension. The new extension architecture adds all sorts of twists, complications and tradeoffs for us extension developers that are not well documented or easily understood. In my case I had to try stuff to see if it worked and based on what I learned I have had to tweak things, sometimes resulting in major architectural changes. But it is coming along. I am more than halfway through the part that actually generates the feed, the last part of the actual development part. There is also an extensive code clean up and repackaging part and then releasing it for Development, Alpha and Beta testing. phpBB 3.0 code does not easily port to 3.1, although some of it does. There is a lot of trial and error, but hashing through it for Smartfeed will make doing Digests substantially easier.

Some development notes:

  • The extension will no longer support the PHP Secure Communications Library. This provided mcrypt-like support if the mcrypt module was not integrated into PHP. I tried to integrate it, but it was too much of a hassle and made the code bloaty. I also know from experience that the phpBB extension team will critically examine everything including 3rd party libraries and likely demand changes. The smaller I can keep the code the better because it will get approved faster. So you must have PHP’s mcrypt module installed if you want the ability to take advantage of features like the ability to see posts since your last login or the ability to get posts in protected forums. This won’t be a problem for most phpBB forums because it is typically compiled into PHP. If you don’t have it, users will just see a message saying these features are not supported.
  • Part of developing the extension will be to do things in a more modern way. One thing will be to put the code in a GitHub repository. I haven’t figured out that part yet, but I expect to do it as part of the release. This will make it easier for others to suggest changes.
  • I am learning that code that used to be in separate files can be collapsed into one file. This is not true of styles and language files, but it is true of most include files and constants. They can go into the main class instead. There used to be two separate programs: smartfeed.php and smartfeed_url.php. In the future they will all be handled in one executable.
  • I am moving to use phpBB’s templating system to generate RSS and ATOM markup for the feed. In the past I used the FeedCreator class. It turns out it’s not that difficult to do once you know the templating system.

Anyhow, it is coming along. Thanks for your patience.

On vacation July 18 – 25

On: By: Mark

I’ll be on vacation during the dates shown in the post title. Even worse, I don’t expect to be very internet accessible, so please don’t expect any correspondence from me during this period. I’ll start working on service requests again starting Monday, July 27.

If you need service during this period, it’s best to look elsewhere. phpbb.com itself is a useful place to find qualified consultants. Often those offering services will put notices in their signatures in forum posts. Of course, anyone you try to get help from over the Internet is inherently suspicious. See phpbb.com’s warning page.

Smartfeed extension update

On: By: Mark

I’ve spent a couple of weeks trying various approaches to the last big chunk of Smartfeed, generating the feed itself. For much of that time I was mired in confusion on how to integrate an external feed into the extensions framework. I think I finally figured out an approach that will work and it will simplify things. Meanwhile I was diverted with other issues, such as how to not use an include file for constants. The solution seems to be to build these into the extension’s main object when it is constructed. A lot of things require rethinking with the new architecture.

Anyhow, what I am likely to do is collapse smartfeed_url.php and smartfeed.php (as they were called in the modification) into a single program. Since the two programs share a lot of similar code, it gives me the opportunity to move the shared code into functions. I was using a 3rd party library (the Universal Feed Creator) to create the feed. Now it looks like I will simplify the approach and simply use phpBB’s template system, with a switch based on the feed type (Atom, RSS 1.0 and RSS 2.0). Now that I am over this major bottleneck, I think the rest of the development will proceed in a fairly straightforward manner.

June 2015 work summary

On: By: Mark
  • I embedded a total of 6 ads, 2 on index only and the rest on forum pages only on a phpBB 3.1 forum. I had to do this manually because there is currently no extension for placing ads for phpBB 3.1. I did it by embedding template switches in the appropriate templates. I made a search field behave and look similar to a vBulletin site. I also went into the stylesheets and removed some padding/margin around certain elements and blocks in a 3.1.5 prosilver style because the customer wanted a tighter look to the page.
  • Upgraded a forum from 3.0.7-PL1 to 3.1.5, ditching old style and using the default prosilver style.
  • Moved a 1 post forum with lots of dense of text and images in the post (integrated with phpBB Gallery) to a new domain. The database was very large. To move the tables over successfully, I removed rows in some unneeded tables in the export, since otherwise timeouts occurred when the database was loaded, making it impossible to complete the database migration. The data in these tables could be recreated during forum use. mbstring errors were encountered on the new domain due to new PHP configuration.
  • An administrator on a domain could not see new posts. Had to add the user to Administrators group and set group forum permissions for administrators to full access for some forums, so the search permission would exist for all forums.
  • A Smartfeed 2.2.27 installation did not work for customer that installed it himself. He wanted me to fix it. He missed some language file changes that needed to be applied to the French language pack. I also applied missing user interface changes to his default Supernova style. The PHP warnings he was getting was caused by a BOM marker at the bottom of /language/fr/acp/board.php. Once removed that issue went away.
  • To address a crushing amount of spam registrations, I installed the Advanced Block Mod 1.1.14 on phpBB 3.0.11 instance. Had to make template changes for adm/style/acp_log.html to accommodate 3.0.11.
  • Provided troubleshooting assistance for a client that was getting HTTP 500 errors. After looking at his web server error logs, I determined these were related to his using a shared server and resource limitations. Offered advice on upgrading to better hosting plan.
  • Client wanted to keep the size of his web server log down to a manageable size, as his popular forum was creating huge web logs. I changed his Apache configuration so that the web server log kept no more than three 3 months of web log statistics. Also upgraded his forum from 3.0.13-PL1 to 3.0.14, accommodating the Advanced Block Mod.
  • For one client: investigated an issue of IE11 and phpBB logging out when Quicktime is not installed. Can’t replicate since I don’t have Windows 8.1 to test, but it’s probably not a phpBB problem. It is more likely an infrastructure issue that could be addressed by Windows update or similar software. Researched and responded to question on how to require moderation in particular forums for particular groups. Researched extensions for client in preparation for call and responding to forum questions. Had a Skype call with client on phpBB upgrade requirements. 2 hours unsuccessfully trying to upgrade from phpBB 3.0.14 to phpBB 3.1.5 on their Windows servers. Eventually determined the upgrade stopped because Windows file permissions on folders that needed public write access were missing. 2 more hours upgrading phpBB to 3.1.5, installing numerous extensions and integrating their logo.
  • Previous customer accidentally deleted his forum’s database making the board inaccessible and did not have a copy of it. I still had my backup. I tried to recover it and upgrade it to phpBB 3.1.4 but every attempt to upgrade to 3.1.4 caused TWIG template errors. database_update.php ran very quickly as well, suggesting that a lot of stuff needed to be done that wasn’t. Test database would update to 3.1.4 using 5/23 database instance, but production would not. Stopped at a step during database_update.php and would not progress. Tried to replicate success with test database and experienced similar issues. Eventually I figured out the button at the bottom of the page wasn’t working, and I succeeded in getting it to continue by pressing the browser refresh button instead.

Smartfeed extension ACP update

On: By: Mark

After weeks of wrestling with issues mostly related to adding and deleting extensions in the new phpBB 3.1 architecture, I finally got past them with a lot of support from the extension writers forum on phpbb.com. The Administration Control Panel user interface is now pretty much done, and the actual user interface part got done in just a morning of work. Fortunately I was able to plug in a lot of code from the old version once I had a framework working, which I got from examining the ACP board settings program. See attached screenshots. As you can see, Smartfeed now appears on the Extensions tab, which is generally where extensions should appear.

With the user interface mostly done I can now begin rewriting the program that actually generates the feed, formerly smartfeed.php. Due to the extension architecture, it will have a different URL.

Primary performance throttles

Security settings

Additional settings and options