Hopefully the New Year will bring a new method of protecting your bulletin boards from spammers .

This entry has been cross-posted from my own personal blog. Those of you who are following the bbProtection project may also find this piece of news useful. I personally like the work that Dave is doing to ensure the continued success of phpBB2 and thus we will more than likely be releasing a bbProtection client for it when the beta period rolls around .

It’s been a grand old 8 years to the day since phpBB1 hit the software shelves of the Internet and started to gain the interest of users and developers alike. Since then we’ve had many changes; phpBB2 was launched, phpBB 2.2 became 3.0 “Olympus” before finally being sent out of the door last year.

Shortly after the joyous celebrations of Olympus going Gold came the announcement of the retirement of the Grandfather of bulletin board software, phpBB2. It’s been around for many years and the phpBB Group are slowly phasing out support and development time to concentrate on phpBB3 as well as phpBB 3.2 “Ascraeus”.

However, there is plenty of life left in the old dog yet! Therefore Dave, a hardcore phpBB2 advocate, has set up phpBB2Refugees.com; a place where phpBB2 users can congregate after phpBB.com closes its doors on phpBB2. The site is designed to be a continuation of what was typical over at phpBB.com; great support as well as MODs and styles available to download. It may be hard to believe but the site is based on phpBB2; Dave has used a number of modifications to illustrate the point that phpBB2 can and is still used as a fully functional forum system for many sites across the web today.

Although there may be a strong presence of current phpBB.com team members and other phpBB supporters the site is in no way officially endorsed by the phpBB Group, nor is it official in any sense. With that little caveat sorted, get over there and join the launch party.

Advertising Losses

If you are lucky enough to be able to find dedicated advertisers for your board then you probably need to ensure that your board content is held up to certain standards. Unless you’re trying to capture spammer data (a process Mark described quite well in a recent blog post) having spammers run rampant over your board is a problem. Advertisers are not likely to look favorably on a site that doesn’t look like it’s being managed well. I own a board that is earning over $200 monthly from various sources. Clearly I do not want to run the risk of losing that income, so I take measures to prevent spam. But to be honest, the monetary cost is secondary.

Boards Shutting Down

In my opinion, the bigger loss to the Internet as a global community is when board owners simply give up and shut down. I have seen more than one board owner take this approach because they simply can’t deal with the problems presented by spammers. Our family adopted a dog from a local “no kill” shelter a few years ago. Imagine my surprise (and pleasure) to learn that the shelter organization was running a phpBB2 board for adoptive dog owners. Granted this is a board that falls more into the “Local Interest” category than the “Global” concept I mentioned earlier. But the board was still providing a service that was valued by the members.

The board eventually shut down because the administrators felt like the time required to clean the board up every day (and it was literally getting spammed every day) was better spent elsewhere. I wish that I had been able to get involved but at the time I had not yet really started developing any techniques to combat spam, so I probably would not have had much to offer. Now I think I could restart the board with proper protection measures in place, but the site management won’t even consider it.

How Many Boards Have We Lost?

This is just one simple example that I have personal experience with. How many other boards out there have been shut down due to lack of knowledge, lack of time, or sheer frustration? People often run boards for their hobbies or special interests. They do it for the love of their subject and the desire to give something back to the community.

Spammers can suck all of the pleasure out of running a board. When they do that, we all lose.

So just what is a honeypot?

Wikipedia can explain this one better than I can:

In computer terminology, a honeypot is a trap set to detect, deflect, or in some manner counteract attempts at unauthorized use of information systems.

In our terms; we have phpBB installations that are stock, normal installs that you would find across the internet. They haven’t been modified to try and stop spammers, and they are open to attack.

So why run these honeypots then?

Put simply it is beneficial for us to collect as much data as we can get our hands on regarding spammers and their behaviour. This includes the raw registration and post data that we can use later on to refine and make the service better and more able to detect spam.

It also shows really interesting trends that we can compare between honeypots. For example, a massive increase in spam offering degrees and other educational qualifications around the start of September shows that spammers are using targeted dates such as the start of the academic year to attempt to promote their “services”.

How much have you caught?

My honeypot has caught around 10,000 spam posts and around 900 spam user accounts. Dave’s statistics are considerably higher at around 35,000 spam posts and 5,400 users at the time of writing.

Both boards are purely designed to collect data and make it clear that they are doing so, therefore I don’t expect many if any of these posts or registrations to be from anything else than automated bots who just target every forum they find across their warpath.

What have you caught?

Mainly generic spam; medicine, pills, general obscene content and suchlike. More interesting examples include spam hidden in jokes and funny anecdotes as well as the classic fake signature trick that Dave noticed a while back.

Can I run a honeypot?

Of course, providing you have basic web hosting and are able to install a bulletin board. However, be prepared for a serious amount of spam that isn’t necessarily pretty or nice (a lot of spam relates to X rated content). This especially holds true for domains that are more popular than others, for example Dave’s honeypot recieves more spam than mine does; probably related to the fact that his domain has been around a lot longer than mine has and is therefore on more spammer’s lists.

If you do decide to set one up let us know if you spot any interesting trends or patterns appearing

I noticed a post by him citing a new type of comment spam that he and others had started to see happening on blogs that they own and manage. This new style is working to try and hide the links that spammers are attempting to push onto sites, mostly by linking them via a single character on the comment, so that it isn’t obvious when just browsing through the comment itself. Todd’s post also goes into some detail about how spammers try and gain a sense of trust before starting to put links into their posts. I’d recommend reading the article, it’s certainly interesting to hear his thoughts on the spam issue and new trends that he and others are seeing.

Bots are relative easy to beat but if spammers are using real people to leave relevant comments linking to their spam sites this will be a bigger challenge.

This quote was of interest to me because it goes a long way in summing up what bbProtection is and just why it works. When checking and filtering spam you need to actually check what is being submitted rather than who is submitting it. Of course, certain checks on a particular spammers IP range or other details relating to the submission can prove useful when checking if a post is spam or not but if a system is checking the actual content of the post then spammers are going to find it increasingly difficult to get their posts and registrations through the system.

bbProtection is designed with the bulletin board in mind but we would of course welcome anyone to use our open API to design a system that works for different blog packages as well. However, more about that in a later post .

Also, look out for some new blog posts coming over the next few weeks relating to the project as well as the spam problem in general .

In the first post in this series I started talking about how the Wordpress service Akismet compares to bbProtection. While the concept behind both services is very similar, the target audience and data structures are quite different. The first major difference that I called out for consideration was the fact that Akismet does not, to the best of my knowledge, have a way to process user registration records. Since for many years phpBB2 board owners were faced with bots that would register and never post this would be a problem.

bbProtection Does Posts Too

The bbProtection service is designed by bulletin board users for bulletin board users. The team understands how boards work, and have direct experience with running boards of their own. I personally have been waging the war against spammers on my boards for many years and was a part of the pilot program for the first release. Why am I making a big deal about this?

It’s simple, really. Akismet was designed for blogs, not boards. bbProtection is designed for boards, not blogs. In my opinion that means that the specific services offered by bbProtection are going to provide superior post / text handling options compared to anything that could be provided by a non-board service like Akismet.

For example, boards typically use a markup syntax that is different from HTML. (Wordpress blog owners can set up limited tags that can be used in comments, but they’re HTML/XHTML tags instead.) For phpBB this is called BBCode and it includes tags like [b] for bold and [img] for embedding an image, and so on. These tags will be understood and processed by bbProtection in an appropriate fashion. For example, I would imagine that one possible indication of a spammer would be the number of links in the post. Without recognizing and parsing the [url] tag syntax that would be more difficult to do.

Before I took steps to address the onslaught of spam I was getting on one of my blogs, I would see a comments that included BBCode syntax as well as HTML syntax … for the same link! The bots appeared to be attempting to cover all their bases. (All your base are… oh, never mind )

Conclusion

The bottom line, as I see it, is that while Akismet may do a wonderful job for blogs and offer an API for other uses, there is still a very real market for board owners that want a solution that was created specifically for their needs. It is my hope that bbProtection will fill that need. To repeat something from earlier in this post: bbProtection is a custom service being created by board owners for board owners. If you are a board owner and want to help provide input as far as features and direction for the service, the door is open. Let us hear from you.

What is Akismet?

I will start with a brief overview for those that might not be familiar with Akismet. This package is an extremely popular plugin for Wordpress. It comes from the same minds that brought us Wordpress, so it was understandable that it should focus first on blog comment spam. To use it a blog owner has to register for a key. I assume that the key is part of the data submitted to the Akismet server. The Akismet process reviews the comment and determines if it looks like spam or not. If it determines that it’s not spam, the comment is approved. If it thinks it might be spam, the comment is held in a queue for later review. The specifics of the process used behind the scenes to tag a comment as spam are not available to the public, and for obvious reasons. If spammers knew what triggers to watch out for, the effectiveness of the service would be drastically reduced.

The name comes from a combination of Automatic and Kismet. Kismet is related to fate or karma… the idea is that spammers get what they deserve. If they hit enough different blogs with the same (or similar) content, then they are quickly identified and prevented from making further public comments. The real strength of the system is the fact that comments from blogs around the globe are all fed to Akismet for analysis… a true-life example of “strength in numbers” in action.

How is bbProtection Different from Akismet?

The bbProtection service (in its first rollout) provided a way for board owners to use a shared database to compare registration and post data with other boards. At first glance the general purpose of bbProtection appears to be very similar to Akisment. The target audience is different (bulletin boards versus blogs) but is that enough? Why not just use the Akismet API and write a bulletin board interface? I believe there are several advantages to a board-specific solution.

The first important distinction that I would like to call attention to is that Akismet does not process user registration attempts. For many years the bane of a phpBB2 board owner was the never-ending stream of users that would register just to get their web site listed in the member listing. Many of those users would never even bother to activate their account. At one point I verified that nearly 20% of my board members were created by registration bots rather than by real humans. I suspect without proper prevention features in place that percentage would be even higher today.

Since most bulletin boards are very similar the bbProtection service should be able to compare user registrations from just about any vendor. The fields that are targeted by spammers (website, location, any other similar fields) should be mapped by function rather than by specific fieldname in the database. By comparing registrations on boards from different owners, vendors, and subject matter across the Internet it becomes easier to identify and eliminate spammers.

Summary So Far

The key thing here is that many registration bots never tried to post. Without a posting it becomes a different sort of challenge, and one that I don’t think Akismet is equipped to handle. While that is one very important difference between Akismet and bbProtection, is it enough? I think that it is, but I have more thoughts on the subject to share in my next post.

The presentation is well worth a watch and is also downloadable from the Londonvasion page, under the [08] phpBB Spam header.

Dave explains a little more about the content of the presentation over at his blog, so go and check it out .

It is very exciting to work on a project like bbProtection, for a number of reasons: its purpose (fighting spam!), its scale (huge!) and the possibility of this taking really off and becoming a very successful project. This has a very motiving effect and will certainly keep my team and myself going. I personally learned a lot during the early beta days of bbProtection, but this will be covered in a future blog post by either Mark or yours truly.

Spam: We’re still on the lookout for new developers. We can only do “so much”, and we would really welcome some enthusiastic (and not to be forgotten, knowledgeable) people to join the development team. As per our abuse of Uncle Sam on the bbprotection.net placeholder page: we need you to help us, which eventually helps… you. If you are interested, please email to the relaunch email address found on the placeholder page, or drop me a line at IRC; my username is Vic on irc.freenode.net.