O'Reilly Network    
 Published on O'Reilly Network (http://www.oreillynet.com/)
 See this if you're having trouble printing code examples

Shining Light Into the Realtime Blackhole List

by David Strom

Related Content

Sound Out on the RBL

Securing Your Home Network With the Edge Firewall

Cell Phone Viruses: The New Frontier

I never thought the day would come when I would be considered a spammer. You see, I have run the Web Informant mailing list over the past five years, mostly for my own (and hopefully my readers') amusement. I thought I was in the clear and being a good Netizen. But apparently that isn't the case. More on why shortly.

I got interested in this issue when Dale Dougherty told me about the problems he had with being placed on the Mail Abuse Prevention System (MAPS) Real-time Blackhole List, or RBL. This list has over 3000 entries and is used by hundreds or thousands of servers around the world. It began as a personal project by long-time Internet veteran Paul Vixie. A dedicated crew who are determined to stop spammers now maintains the list, run by project manager Kelly Thompson. The idea behind the RBL is a laudable goal, to be sure. All of us get far too much spam. Even test accounts I have at Yahoo and Hotmail accounts get spam, and I haven't sent any mail to anyone (besides myself) from them -- ever.

Before I get into the issues, note that ISPs and others who maintain their Internet presence can use the RBL in one of three ways. Your e-mail provider can tag suspect messages as spam and pass them along to their ultimate recipients. Your e-mail provider can block any suspect e-mail from the listed spammers. Or your provider can block all IP traffic going towards the listed domains (actually, it is IP addresses of the abusers). There are a set of carefully worded descriptions on their site on how you get on the list, how you get off the list, and other information.

By and large, it is a good system. While spammers continue to escalate their arms race and stay ahead of the spam cops, the RBL has undoubtedly cut down on the amount of spam sent around the Internet. Actually, there are other operations, including the orbs.org (Open Relay Behaviour-modification System) folks. They have different practices and standards.

On the whole RBL is trying to fix two different problems:

First is the problem of an open mail relay. In the old days of the Internet, mail servers could easily exchange messages with each other with nary a care. However, this feature has been exploited by spammers to the point that most ISPs should and do shut down the relays, so that only certain computers can send mail using their mail servers. This means if you use one ISP (say Earthlink) for dial up access, you probably can't send mail from your host maintained by Verio unless you use a web-based mailer.

Second is the problem of junk mail proliferation by people who receive money to send out tons of e-mail.

In my opinion, RBL isn't completely successful for several reasons. First off is that they have a very restrictive definition of best e-mail practices, and this definition is somewhat unclear from their public materials. The point of contention has to do with how individuals verify when they are added to mailing lists. This seems like a minor point but isn't.

From the RBL web site: "A mailing list should include only those who have explicitly indicated an interest in receiving messages from the list. Prudent mailing list management mandates verification of all subscription requests before mailings commence." The issue is what constitutes verification. They say on their site: "there are numerous ways to confirm or verify an e-mail address."

However, when I spoke to RBL's manager Thompson, she said that all mailing list owners should include "a closed loop confirmation system, one which confirms any additions to protect people from having their friends sign them up unintentionally." I don't believe this is common practice. I certainly don't have a complete closed loop confirmation system for Web Informant, and of the tens of mailing lists that I have joined over the years can't remember more than one or two which used such a system.

Jed Lewison, who is director of eCommerce Product Marketing for Real Networks, also agrees that the closed-loop definition is too restrictive a definition. Real has been on the RBL for some time because it sends out monthly e-mails to more than 100 million of its customers. "Closed loop confirmations isn't really the industry standard," he says.

So according to the materials at mail-abuse.org, I should be considered a spammer and cease my noxious processes. But when I spoke to Thompson directly, she assured me the likelihood of any of my subscribers complaining would be low. So even though I don't actually confirm everyone that I add to my list by sending him or her an inquiring e-mail that they have to confirm and return to me, I am probably in the clear.

Second, RBL has an imperfect appeals process. Real's Lewison says, "We send out millions of e-mails a month. If one person complains, we get listed." Once you are listed, you can't easily be completely removed.

The O'Reilly Network was placed on the list when recipients complained about getting spam, largely because the e-mail wasn't properly identified, as Dougherty admits. "We did not do a good job of identifying the connection between the O'Reilly Network and perl.com," he said. "And so several people complained to mail-abuse.org." Then they were taken off the list (after some gnashing of electronic teeth and various e-mails and phone calls) and placed on "probation." What does this mean? It is a sort of spammer purgatory -- while their traffic isn't blocked and they aren't part of the RBL itself, it means that until they change their practice, they are under the microscope to behave themselves. And that includes using the closed loop confirmations above, which O'Reilly has stated they won't do. Thompson indicated that someone could stay in purgatory forever. However, there is no way to know you have THAT particular status, since unlike the RBL it isn't publicly available.

Third, there is an issue of how the RBL folks notify the abuser. "Even though the newsletter had several real e-mail addresses in it, no one at MAPS chose to contact us at those addresses," said Dougherty. Given who they are and how they do business, RBL would prefer to send e-mail to the postmaster and abuse accounts at the ISP who handles the ultimate spammer. As Thompson says, "I know the postmaster@earthlink account is read regularly and very responsive. I don't know if the postmaster account at every Earthlink-maintained domain customer is read regularly." Good point. What about including both the ISP and the spammer in the initial warning message? She didn't have a good answer to that, something that could easily be done and result in more timely corrections of problems.

Finally, most people don't understand that there are two different blocking actions, as mentioned earlier: the mail server and all IP traffic. Sometimes the folks at RBL will add to their list just the mail server, and sometimes all servers, of an abuser. Thompson told me that reasons varied for the different actions, depending on many different factors.

Some e-mail list owners say that these policies aren't uniformly applied. One has said, "Only people who aren't famous or who don't control large or popular sections of the Internet are put on RBL." When someone from O'Reilly also questioned this practice, "RBL indicated that they did not do this consistently. They would not block traffic to a high-profile site." This seems to me unfair to say the least.

Paul Hoffman, who runs the Internet Mail Consortium and has spent a great deal of time dealing with spam policy issues, says, "It is good to have someone in the world like the MAPS RBL even though I don't agree with all of their policies. For example, should a co-located customer be allowed to decide whether to use the RBL or not? In most cases, they can't decide -- once their hosting provider subscribes to it, they have to deal with the consequences. That means if I disagree with the sites listed on the RBL, I have to switch providers. I think the RBL should only apply to edge routers."

In the meantime, I welcome your e-mails and thoughts. Just as long as they aren't spam.

David Strom founded CMP's Network Computing magazine in 1990 and was its first editor-in-chief. He's now the president of David Strom, Inc.

Related Content

Sound Out on the RBL

Securing Your Home Network With the Edge Firewall

Cell Phone Viruses: The New Frontier

Discuss this article in the O'Reilly Network Forum.

Return to the O'Reilly Network Hub.

Copyright © 2009 O'Reilly Media, Inc.