I’m at the very early stages of writing a book about how to secure, monitor and un-hack WordPress. This book will be the culmination of everything I know about keeping WordPress hardened against attacks, how to keep an eye on your install so that you’re the first to know if something has happened, and how to handle the situation if you bought the book too late and got pwned anyway.

Can I absolutely guarantee that you’ll never get hacked if you do everything in this book? Of course not – new exploits emerge all the time. However I can promise you that you’ll be a lot less likely to get hacked if you follow these instructions, and if you do get hacked, you’ll be a far better place to recovery quickly and completely – and if that’s not worth $5 to you, you deserve what you get.

I can also promise that if there is enough interest in this book, I will keep it updated and release new revisions for free for those people who have already purchased it. I don’t believe in having to re-purchase a book just because a paragraph or two was added. It pisses me off when I have to do it, and I’m sure it pisses you off, too.

How much?
Right now, I’m toying with a $5-$10 price range, depending on how long and detailed it ends up being. It will not be more than $10, regardless of how long it is.

Are you doing advanced sales?
YES! If you’d like to pre-order, check out the page on GoFundMe. There are a few different options for pre-ordering, so it’s worth a look.

Why an e-book instead of a “real” book?
I might make this available in hardcopy through Lulu or some such service – but there are several reasons I went with e-book. First, I’ve written “real” books before. The process is frustrating, and there’s not a lot of money in it, in tech books anyway. By self-publishing, more of the money ends up in my pocket where it belongs. Second, as a recent convert to e-books (thanks to my iPad), I prefer to give folks the option of printing if they need to, but spare the trees for those that don’t. And third, stuff changes all the time in technology. Paper books have never seemed like a great way to cover tech topics, since half the book will be obsolete within a year or two, my own previous books included. Self-publishing via e-book means I can update the information as needed, and not feel like a jackass for making people buy a new edition of the book.

What qualifies you to write this book anyway?
I’ve been working with WordPress since 2005, and have spent a considerable amount of time over the past several years doing forensic and recovery work on hacked WordPress blogs. For some idea of the type of content you can expect to find here, check out this blog post from Jan 2010. I’ve helped folks like Chris Brogran, Scott Stratten, the Crave Network and others quickly recovery from a hack, and lock down their installs so they are less vulnerable to future attacks. Ask them – they’ll tell you.

How much swearing will there be?
Probably less than you expect. My goal is that this can be something some of you can recommend to your clients (the ones that won’t pay you to secure their sites), and as tempting as it is, that might not fly for the general public. I will, however, try to keep it entertaining in my own way.

What’s the title going to be?
NFC. Suggestions welcome.

How can I stay updated on the book’s progress?
I’ll be updating the official book website as new developments arise. Check it out here.

I have another question!
Tag me on Twitter at @snipeyhead or leave a comment below and ask away.

Also check out:

I’ll start with the obvious candidates that you probably already have installed if you’re a developer. I’ve also added a few that are useful for post-hack diagnostics and recovery.

General

Firebug – Firebug is great for web development in general, but the debugging tools can help track down calls to rogue javascript on external servers, among many other things.

Web Developer Toolbar – Another great web dev tool, the Web Developer Toolbar makes it easy to turn javascript and cookies on and off selectively, view form fields and disable restrictions and much, much more.

DNS Cache – simple addon that lets you clear or disable Firefox’s DNS cache. Not specifically for pen testing, but useful nonetheless.

Notable - love this addon, which lets you do a full-page screenshot with annotations over at notableapp.com. As you’re testing, there’s a good chance you’re going to need to show your other devs or account managers a screenshot so they can see the vulnerability being exploited. While something simple like Fireshot would work fine (or native screenshots), I like using Notable for complex situations that require explanations on multiple points on the page. Exports to annotated PDF.

Groundspeed – simple form toolkit that allows you to edit form fields (hidden to text, etc), remove length restrictions, change/remove javascript event handlers, and change form target so that it opens in a new tab.

Code Injection

SQL Inject Me – helps test for SQL injection vulnerabilities.

XSS Me – used to test for reflected Cross-Site Scripting (XSS). It does NOT currently test for stored XSS.

Access Me – used to test some access vulnerabilities related to web applications. The tool works by sending several versions of the last page request. A request with the session removed will be sent. A request using the HTTP HEAD verb and a request using a made up SECCOM verb will be sent. A combination of session and HEAD/SECCOM will also be sent.

JS Deobfuscator – many attacks inject obfuscated javascript into a page so that it becomes harder for you to simply grep the source for something obvious, like the domain name to which the bad script is redirecting the user. This addon helps deobfuscate the javascript so you can see what’s really going on.

Hackbar – helps with testing sql injections, XSS holes and site security. Ugly as sin, but it works well.

Header and URL Monitoring/Tampering

Note that some of these addons do similar things – try them and stick with whichever one you like best.

HttpFox – monitors and analyzes all incoming and outgoing HTTP traffic between the browser and the web servers.

Live HTTP Headers – view HTTP headers of a page and while browsing.

Modify Headers – add, modify and filter http request headers. You can modify the user-agent string, add headers to spoof a mobile request (e.g. x-up-calling-line-id) and much more.

Tamper Data – use tamperdata to view and modify HTTP/HTTPS headers and post parameters, trace and time http response/requests and security test web applications by modifying POST parameters.

User Agent Switcher – allows you to easily toggle between pre-set user agent strings, or set your own.

Environment Detection

Header Spy – lightweight addon that displays information about the website’s server in your statusbar. This is not as useful for pen testing as it is for impressing the crap out of clients who don’t know what server they’re running.

Host Spy – integrated shortcut to show you who a website’s IP neighbors are on shared hosting.

ShowIP – Small addons that shows the IP address of the website in your statusbar and a link to some additional tools.

URL Flipper – quickly and easily increment and decrement numbers and strings in URLs for navigating through URL sequences (for example, user ids or session info in the query string.)

Wappalyzer – uncovers the technologies used on websites. It detects CMS and e-commerce systems, message boards, JavaScript frameworks, hosting panels, analytics tools and several more.

Searching

Offensive Security Exploit Database – this adds the excellent database of exploits at exploit-db.com as one of your search engine options.

I’ve also just created a search plugin for XSSed.Com, but it’s pending approval at Mozilla, so not sure when that will be ready for you which can be downloaded here. It’s not exactly rocket science to add a new search site to your browser search bar, but I figured it was quick and easy to whip up. Feel free to check out the source code for the plugin here.

Too Many Addons Got You Down?

If you’re finding your plugins are slowing down Firefox too much, you might want to create a separate Firefox profile specifically for testing, and switch to that profile when you’re ready to start hammering away. Also bear in mind that you might need to tweak some settings on these, or only enable them right before you use them, as the toolbars and sidebars can be a bit bulky.

Also keep in mind that the Net option on the web developers toolbar, or any of the header analyzer addons can be very helpful in general testing between dev and live environments (load the page on live and make sure nothing is being pulled from the dev address) and also to make sure your SSL requests are being handled correctly.

Some Additional Thoughts…

When folks ask me how I do penetration testing – whether I use software, or do it by hand – the best way I can answer is “both”. Software will only ever get you so far, but it’s a critical tool in helping you figure out where the vulnerabilities are. It’s not unlike using a metal detector to find treasure. When the metal detector is doing its job, it finds, well, metal. Not necessarily treasure, although fancier metal detectors have additional software that helps try to identify the buried object by shape and size. You still have to physically dig up the item and rely on your knowledge and experience to determine whether or not it really is treasure, or just junk. The metal detector simply finds something that meets a basic set of requirements, to save you from having to dig up every square inch of the beach.

When testing web applications for vulnerabilities, software does very much the same thing. It simply automates tasks that you could do by hand but that would take an unreasonable amount of time, but ultimately when it finds something, you still need to know enough about what you’re looking at to determine how big a threat it actually is. Most of the time the software will attempt the to try the lowest-level exploit, for example, the ability to execute arbitrary javascript in a page. Your testing tools may demonstrate that you can create a javascript alertbox on the page, but it’s your knowledge and experience that will help you determine the full extent of the vulnerability, for example whether that arbitrary javascript could be used to redirect a user to a new page, hijack the user’s session data, etc.

The reason I ended this post with a long-winded ramble is because I wanted to make it clear that just having the tools isn’t enough. Actually using them, and knowing what to do with the results are important. Understanding the basic mechanics of how exploits work is the only way you can make sure your applications are written to mitigate them. Having the tools installed but never understanding or using them is like buying a metal detector and keeping in the closet and then wondering why you haven’t found anything valuable yet.

If you’re interested in learning more about web application penetration testing and security, check out the following books:

• The Web Application Hacker’s Handbook: Discovering and Exploiting Security Flaws
• Web Security Testing Cookbook: Systematic Techniques to Find Problems Fast
• Foundations of Security: What Every Programmer Needs to Know
• Ninja Hacking: Unconventional Penetration Testing Tactics and Techniques

How Do You XSS?

These addons are not obviously meant to be a replacement for more capable and thorough penetration testing tools such as metaploit, netsparker, etc. They’re just meant to be a convenient way for developers to test code during and after development.

There is a more comprehensive collection of addons listed here, but this is what I use. If you’ve got a favorite that I’ve missed, please be sure to share in the comments!

Also check out:

I had tossed that idea around a lot over the past year, since I run several websites that run on WordPress, but I had heard from enough people who ran into plugin/MU conflict issues that made things go ‘splody ‘splody that I opted not to. So instead, every time a new version of WordPress came out, I’d end up upgrading around 20 installs. Blech.

With version 3.0 of WordPress, the ability to create multiple sites using one install of WordPress is built right into the core, so no need to fool around with WPMU. The temptation was too great this time, so I decided to give it a whack. It was not what I would call a smooth process, but it wasn’t terrible either.

STOP: If you are already running WPMU and you just want to figure out how to upgrade your existing WPMU sites to WordPress 3.0, you’re reading the wrong article. Try this one instead.

Goals

What I wanted to get out of this was to have one main core install, but run multiple sites on their own domains that all pulled from that main core, so upgrading to later versions would mean upgrading one core instead of a dozen or two. These properties remaining at their current separate domain names (such as www.crankyhaiku.com, www.geekhaiku.com etc) was critical, both because of search engine optimization and for branding reasons.

Upgrading

The normal upgrade part was flawless, as WordPress upgrades tend to be these days. Automatic upgrade has never quite worked for me, so I always do a manual upgrade. It takes longer to upload the files, but it’s a pretty painless process. So to upgrade to 3.0, I did the usual:

• backup (which I didn’t actually have to do, since I automatically backup to the Amazon Cloud every night using Automatic WordPress Plugin) but I’m paranoid
• delete the wp-admin directory
• delete the wp-includes directory
• upload everything in the WordPress package – except for wp-content – to the web root
• hit the upgrade script to trigger the database updates

Flawless, as usual. Not so much as a hiccup. Now came the trickier part – adding the “Network” functionality previously available in WPMU to start to consolidate sites.

Creating a Multi-Site Network

I can’t speak for how easy or difficult this normally was with WPMU, so unfortunately I can’t tell you how this process compares to a normal WPMU setup. It wasn’t awful, but it was definitely buggy.

The WordPress documentation on Creating a Network walks through the basics well enough, so I suggest you start there so you know what to expect.

Note: You will not be able to go through the wizard in your WordPress admin until you deactivate ALL of your plugins. You can obviously re-enable them later, but I found that many of them did not keep their original settings.

I suspect this might be because I chose “network activate” instead of just plain “activate”. I had wanted to make those plugins available for all sites in the network, and didn’t realize that it would wipe out my existing snipe.net settings when I did so. Oh well. (Incidentally, that explains why you might see some weird stuff on the site until I have a chance to go through everything one by one. Double “related posts” bits at the end of the articles, Apture wasn’t working, etc.) All of the settings are fixable, but it may take you a little time to figure out what’s been lost, and what you have to do to set it back to the way it was before.

Editing Your wp-config.php

Beyond the setup in your WordPress admin, you’ll need to make a few changes to your wp-config.php file and your htaccess file. I hadn’t updated my wp-config for several versions, so I decided to use the wp-config-sample.php file and just pull my existing database variables over. Whether you use your old wp-config.php or start fresh with the stock WordPress sample, you’ll need to add the following to your wp-config.php, just above the comment that says “/* That’s all, stop editing! Happy blogging. */”

define( 'MULTISITE', true );
define( 'SUBDOMAIN_INSTALL', true );
$base = '/';
define( 'DOMAIN_CURRENT_SITE', 'www.yoursite.com' );
define( 'PATH_CURRENT_SITE', '/' );
define( 'SITE_ID_CURRENT_SITE', 1 );
define( 'BLOG_ID_CURRENT_SITE', 1 );

If you followed my suggestion and read the WordPress documentation on creating a network (you did read that, right?), you’ll see that you have two choices for how your network will be set up: sub-domain (blah1.yourdomain.com, blah2.yourdomain.com) or directory-based (yourdomain.com/blah1, yourdomain.com/blah2). Make sure you think this one through before you get started, since there doesn’t seem to be an easy way to switch between the two.

As I mentioned, I didn’t want my sites to live at subdomain.snipe.net, or snipe.net/blogname – I wanted them to live at their own urls. I also didn’t want a bunch of crap littering up my document root. The easiest way to do this on Rackspace Cloud Sites is through a combination of setting up a site alias, and using mod_rewrite to handle domains:

• Set up a domain alias, like secondblog.com, and point it to originalblog.com
• Modify the mod_rewrite rules in your htaccess access file
• In your site preferences, point the blog url to the aliased domain name

If you’re not on Rackspace Cloud Sites, you can just follow the directions in the WordPress documentation.

Tweaking Your .htaccess

You’ll need to make sure the bit below is in your htaccess file – but your WordPress Network Setup wizard will point that out to you anyway

RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^ - [L]
RewriteRule . index.php [L]

One thing to look out for besides having to reset your plugin preferences: when I created my Network, setting this site as the default, it automatically tried to set the url as snipe.net/blog. I’m not sure why it did this, and I’m certain I didn’t add it anywhere, but when I committed the changeover to Network, all of my urls were broken (since snipe.net/blog/ doesn’t exist). It was a quick change that you can handle via the Settings menu, but watch out for it and be sure to test your links once you’ve made the switch.

Importing Blogs

Now that you’ve got a Network set up, you have actually add them to the Network so that they’re using the same core. I expected this to be a much bigger pain in the ass than it ended up being. All I had to do was go to the original admin, go to TOOLS > EXPORT and download the XML file. Then go into my WordPress 3.0 admin, select the site I wanted to admin, and go to TOOLS > IMPORT > WORDPRESS, and upload the XML file. Worked perfectly, so far as I can tell.

Security Notes

Consolidating all of your WordPress sites into one multi-site install has many benefits, the most obvious one being that it’s easier to maintain one core install than updating every single instance of WordPress you run. That said, you may want to consider a few things:

While one install is probably more “secure” than multi-installs in the real world simply because you’re more likely to keep one site updated than dozens, there are a few things to consider.

If you run multiple WordPress blogs under the same user (the same account, in Rackspace Cloud Sites), all of the files are owned by the same linux user and group. This means that if one of your WordPress installs ends up compromised, either because you forgot to upgrade one of them, or because of a vulnerability in your hosting company, once an attacker has access to one of your blog installs, they have access to any other files owned by that user. Which means all of your other blogs, even the ones that are running current WordPress versions.

Along this same line of thought, if you’re running multiple WordPress installs under different users and you end up consolidating them to take advantage of the multi-site functionality, do so understanding that in this scenario, all of your blogs will be owned by the same user/group in the same webspace, so one vulnerability could easily turn into a much bigger problem.

Conversely, tracking down backdoors and maliciously modified files could potentially be easier, since you have fewer installs to search through.

WordPress has been much better about quickly patching holes, and being proactive about finding vulnerabilities. If your site ends up getting hacked, these days it’s more likely to be a vulnerable plugin, an outdated install you forgot all about, or a PC virus that added your FTP login to a botnet – not the core WordPress install itself. I say this with a certain amount of confidence, since I have restored at least two-dozen hacked WordPress sites (not mine) since the beginning of the year, and have therefore spent countless hours investigating the attack, identifying the vector, and writing up summaries to post to badwarebusters.org in an effort to help other people facing the same hack.

To be clear, running a multi-site install isn’t any riskier than running multiple blogs under the same user. But if you’re currently running your blogs under different users, you should at least be aware of how that could potentially impact you.

Final Thoughts

My thought is that it might have been smarter to install WPMU, and then upgrade to 3.0, since the upgrade process for a WPMU setup to 3.0 seems like it was a little less wonky, but I don’t really know.

I’ve really only just started playing with this during the fragment of free time I had today (work has been brutal for the past month or so). So far, pulling the theme in has been as simple as downloading them from their respective old WordPress installs and uploading them to the new 3.0 themes directory and activating them so that they’re available to the rest of the sites in the network.

And certainly, if you’ve found an easier way to get this done, please let me know in the comments.

Also check out:

Background

The MSWDS is one of only four significant annual events within the PHP community (others include tek, DPC and ZendCon), and this summit is a bit harder to get invited to. Unlike most other conferences, where all you need is the cash to pony up for a conference pass and a hotel room to crash in, invites are very limited and attendees are selected because they have had some interaction with the folks at Microsoft, and are believed to be leaders and influencers within the open source community. To be blunt, these summits cost Microsoft a lot of money, so they need to make sure they’re getting the best bang for their buck.

Keeping that in mind, it would be easy to assume that we were being brought out there so that Microsoft could pitch us on the latest and greatest Microsoft products, trying to get the movers and shakers of open source to drink the corporate kool-aid and switch to Microsoft products. While more acceptance of Microsoft products within the open source community is obviously a goal, they are making a concerted effort to learn from us – what we need, where they are falling short, and how we can move forward together.

Discussions and Format

The summit itself was a total of three days, with the last day being optional for those open source developers who were willing to sign an NDA to discuss some of Microsoft’s emerging technology. During the three days, different representatives from Microsoft’s product teams sat down with us and asked for our comments, thoughts and ideas about where they’re at, and where we think they should be going. We met with folks from the IIS Web Platform team, the SQL server team, as well as some representatives from Codeplex, Silverlight, Powershell, ASP.NET Ajax (which is not exclusive to ASP.NET, despite the name), and Bing maps.

We had a chance to air grievances, which was cathartic in some ways, but I think it was more important to us to be able to sit down with the actual teams who are working on this technology at Microsoft, and really get into the specific challenges we face. The approach was not generally pitchy, and with very few exceptions, a great deal of effort was put into making all of us from the open source community feel like respected authorities in our field whose opinions really matter.

Something they did this year which was apparently not done last year was to include representatives from well-known PHP-based projects who are not normally parts of the PHP community. I honestly hadn’t realized that the many of the folks over Joomla, WordPress and Drupal often don’t consider themselves as part of the greater PHP community, and getting a chance to discuss that with them brought up some interesting perspectives. I don’t think the guys representing these projects were there in an official capacity, but their point of view was one that had honestly not occurred to me before, so that was a really interesting and unexpected benefit. There was some debate on whether or not these types of projects should be a more involved part of the PHP community, with good points on both sides, but I think most walked away with some ideas on how to move forward in making those lines of communication more accessible and open.

My Perspective

Of course the ultimate question from Microsoft was “What would it take for you to switch to Microsoft products for your clients?” My smartass remark was, of course “A fucking miracle.” But everyone in the room knew I was joking. I hope. If we weren’t willing to work with Microsoft on improving their products to work with open source better, we wouldn’t have been there.

As often as Microsoft has been an easy target in the past, and as much bad blood as there may have been in the past, there are people at Microsoft that care about working with the open source community, and who are making progress to get there. It is our job as technology professionals to fairly evaluate technology and make recommendations based on what makes the most sense technologically and financially. It is NOT our job to make religious decisions based on zealotry.

That means that if and when Microsoft can meet my needs and/or the needs of my clients, it can and should be part of that evaluation or I’m not doing my job. Does that mean I’m ready to switch back? No. Not yet, anyway. But I believe they are listening, and I saw some things during this summit that make me far more likely to start including some parts of Microsoft’s products into the technology I suggest as being potentially viable for client projects, which is a far cry closer than I was last week. Specifically, some of the stuff I learned about Silverlight, Bing’s geolocation products and Windows Azure (Microsoft’s cloud hosting platform) was pretty impressive. As I get to play with these products a little more, I’ll be blogging about them with my fair evaluation of pros and cons, so stay tuned.

I’m also excited to see where the Microsoft Web Platform Installer product heads. Right now, the WebPI product is a very easy to use, slick solution for the less techy individual who wants to, for example, deploy a WordPress blog in 5 minutes or less and may not have the savvy to do the install themselves – basically a MS version of Cpanel/Fantastico, which we have had available to us as web administrators for over a decade. That product is less interesting to me right now, but some of the directions they could go in for more advanced users like us hold real potential. We had some suggestions that were well-received, and if they are actually implemented in the way I envision them, it could honestly turn the table and make some of the Microsoft web server products something that I could consider recommending, or even using myself. (I should also mention that Cpanel is the most horrific, insecure, hack-prone web control panel I’ve ever used, and I am NOT endorsing it as a solution.)

The reality is that competition inspires innovation, and Microsoft getting better means progress for everyone. I saw a post on Twitter that basically implied that open source representatives attending this conference were traitors or sellouts. I don’t see it that way at all. We have amazing open source products like Firefox because the open source community worked together to create a better product, and Microsoft responded by making vast improvements to Internet Explorer, building in more security and standards compliance. When we work together to innovate, everybody wins.

Another transition I’ve been seeing in Microsoft which was really made more obvious by this summit is that there is a less omnipresent feeling of “all or nothing” within many Microsoft departments. As open source advocates, we enjoy having choices. Previously with Microsoft, you’d get the most benefit from their products by committing to an entirely Microsoft development process (“drinking all of the kool-aid, since the best stuff is the sugary goop at the bottom”), with benefits sharply falling off if you opted to pick and choose. This philosophy has always been distinctly in opposition with the open source philosophy, and I believe was likely the cause for some of the distrust coming from the open source community. Seeing this transition into a paradigm of being able to cherry-pick what we like for some things and sticking with open source solutions we like better for others is a step in the right direction, in my opinion.

An additional unexpected benefit to sitting down with all these MS product people was that I got a chance to better understand some of the legal/licensing challenges Microsoft faces. I’m not making excuses for them, but I hadn’t considered some of the obstacles in the way of people at MS who care about working with us. Microsoft is a big target with deep pockets, and they have to cover their own asses. I was quicker to dismiss some of the corporate decisions as being “evil” prior to sitting down with some of them and understanding why they do what they do. Don’t get me wrong – some of their decisions (*cough*sudo*cough*) still don’t make sense to me and I believe they are wrong, but I think I have a better understanding of where they sit than I did before

Wrapping Up

Overall, I would consider this summit a great success, and I hope I get to participate again in the future. There are several people who really deserve a shout-out for all of the hard work that went into this and are directly responsible for it’s success. From the PHP community, Cal Evans was a co-host and an absolute rock star, always quick to make sure things ran smoothly and kick-start conversations and redirect us back when we went off on tangents. From Microsoft, Karri Dunn, Tonya Young, Josh Holmes, Peter Laudati, Lauren Cooney, and others were amazing. I may be forgetting a few – I am still a little wiped from the week and the traveling.

Was this an instant fix? Certainly not. Do we all have a lot more work to do before we’re “there”? Absolutely. But as Cal Evans put it on his own blog roundup, “The more people I get to know at Microsoft, the less I’m able to despise the company.” They took the time to find out what we think, even when it may not have been what they wanted to hear. Time will tell whether or not they actually act on it.

Other PHP Representatives Blog Post Roundups

I’ll be updating this list as more people finish their blog post roundups, so you can get take their on the summit. Many of them are far smarter than I am, so it’s worth reading what they have to say.

• Cal Evans (@CalEvans)
• Chris Cornutt (@enygma)
• Maarten Balliauw (@maartenballiauw)
• Rafael Dohms (@rdohms)
• Keith Casey (@CaseySoftware)
• Romain Bourdon (in French) (@le_vrai_roms)
• Marco Tabini (@mtabini)
• Sam Moffatt (@Pasamio)
• Helgi Þormar Þorbjörnsson (@h)

MS Representative Blogs

If you’d like to see more about the fabulous people at MS who are working hard to move the company forward in a way that works with open source, check out their blogs. I’m proud to call these guys friends, and as long as we continue to have people like this working for Microsoft, I think the lines of communication and cooperation between both sides of the aisle will keep moving forward.

• Dave Bost (@DaveBost) – Developer Evangelist
• Josh Holmes (@JoshHolmes) – UX Architect Evangelist
• Tobin Titus (@tobint) – MSDN Site Manager
• Peter Laudati (@jrzyshr) – Developer Evangelist
• Mark Brown (@MarkJBrown) – Product Manager for Microsoft Web Platform
• William Coleman (@will_coleman) – Developer Evangelist
• Lauren Cooney (@lcooney) – GPM for Web Platforms at Microsoft
• Jas Sandhu (@jassand) – Interop Strategy Evangelist
• Ruslan Yakushev (@ruslany) – Program Manager on IIS team in charge of FastCGI and PHP support
• Scott Hanselman (@shanselman) – Principal Program Manager Lead

Funnily, as I’m writing this, Futurama: Into the Wild Green Yonder has been on television, and the scene that just played is the one where Calculon says “I’d like to thank the academy, my agent, and most of all my operating system – Windows 7, for everything it –” at which point his OS locks up. Windows 7 is actually a great product, and I run it on my Mac using Bootcamp and VM Fusion, but I thought the timing was amusing.

Kool-aid photo taken in Belize by me – just thought it was funny. Lead post image by Eli White.

Also check out:

If you have to send HTML email, there are certainly lots of good libraries available for download to make your life easier. I opted to use PHPMailer by Worx International, since it’s one I have used in the past, and getting up and running with it takes about 3 minutes, tops.

What makes it particularly nice is that you don’t have to do anything special to format your HTML to be ready for mailing. If your workflow is anything like mine, you design an HTML email and code it into HTML, and then post it somewhere for the client to review. Since this HTML file already exists on your server, using PHPMailer is a breeze, since all you have to do to start sending HTML email is point the program to the HTML file you already created.

After you upload the PHPMailer library, all it takes is a few lines of code, and you’re done:

[sourcecode lang=php]require_once $_SERVER['DOCUMENT_ROOT'].’/phpMailer/class.phpmailer.php’;

$mail = new PHPMailer(); // defaults to using php “mail()”
$body = $mail->getFile(‘../invites/email.html’);
$mail->From = $from_email;
$mail->FromName = $from_name;

$mail->Subject = “Test email subject”;
$mail->AltBody = “To view the message, please use an HTML compatible email viewer!”;
$mail->MsgHTML($body);

$mail->AddAddress($to_email, $to_name);

if(!$mail->Send()) {
echo “

Also check out:

The code below should correctly autolink all of the autolinkables in your PHP script:

• links @username to the user’s Twitter profile page
• links regular links to wherever they should link to
• links hashtags to a Twitter search on that hashtag

[sourcecode language='php']function twitterify($ret) {
$ret = preg_replace(“#(^|[\n ])([\w]+?://[\w]+[^ \"\n\r\t< ]*)#”, “\\1\\2“, $ret);
$ret = preg_replace(“#(^|[\n ])((www|ftp)\.[^ \"\t\n\r< ]*)#”, “\\1\\2“, $ret);
$ret = preg_replace(“/@(\w+)/”, “@\\1“, $ret);
$ret = preg_replace(“/#(\w+)/”, “#\\1“, $ret);
return $ret;
}[/sourcecode]

Someday I’ll try to find the time to write a regex primer/tutorial. Regex is another one of the things, like SVN, that seems scary and incomprehensible to many people, but eventually it clicks and makes sense. In the meantime, if you’re interested in learning more about using Regex in PHP, check out the following resources:

More Regexy Goodness:

• Using Regular Expressions in PHP via regular-expressions.info
• Introduction to PHP Regex via phpro.org
• Using Regular Expressions in PHP via webcheatsheets.com
• 15 PHP Regular Expressions for Web Developers via catswhocode.com

Make sure you leave yourself some time to actually try out some examples, and dissect the examples they give so that you really grok what’s happening. Once it clicks, it seems so simple, you won’t believe you ever let it beat you up and take your lunch money.

Image by xkcd, via their awesome web store. The comic rocks my geeky face off. I buy my clothes there. So should you.

PS – still really, really hate posting code in WordPress. Even in HTML mode, it keeps converting my fscking special characters, which then get double/triple/etc converted.

Also check out:

Right then. Moving on.

OAuth is an open protocol to allow secure API authorization in a simple and standard method from desktop and web applications. In layman’s terms, it is a system by which you can allow a user to authenticate with an OAuth-enabled service without providing you with their credentials to that service.

In my Twitter anti-social media douchebag service, DoucheNuker.Com, we use Twitter’s OAuth to validate the user and make Twitter API requests on their behalf, specifically sending a DM to the douchebag they are nuking, another DM to @spam to report them to Twitter as a spammer, and then a block request to block the spammer’s account from being able to follow them in the future.

Why OAuth?

Using OAuth allows you to write applications that access the Twitter API but do not require your users to give you their Twitter username and password. This is important for a variety of reasons:

• If the user changes their Twitter login, they do not have to update that information with you for your application to continue working for them
• Using OAuth puts the user in control – if they ever wish to stop using your application, they can disable it through Twitter instead of trusting your application to stop using their login information. Once they disable it through Twitter, any requests by your application will require them to manually approve the connection again.
• Increased sense of trust, since the user doesn’t have to worry about your application stealing their Twitter credentials and using it for nefarious purposes. I personally wouldn’t trust any web-based application that asks for my Twitter username and password, and given Twitter’s recent history of bad press regarding their security, more and more users are following that lead.

Definitions

Before I show you how to use Abraham’s shmancy library to connect to Twitter’s OAuth, you should understand the basics of how OAuth works and what it’s doing. And before we get too caught up in that, it’s important that we establish some definitions that you’ll see if you do any additional research into OAuth:

User: The users of your application.
Consumer: Your application, which you have registered with Twitter
Service Provider: The third-party service the consumer (your application) is authenticating against – in this case, Twitter.

These terms are used in much of the OAuth documentation, so they’re worth remembering.

So now that you know the lingo, how does OAuth actually work? For a detailed technical view of what gets passed back and forth, check out the core spec documentation on OAuth. Included in that documentation is the detailed chart below.

As you can see, the documentation frequently uses the terms defined above.

If that flow diagram seems a little overwhelming, don’t sweat it. I have a simplified version just for you (featuring a stoner Twitter user and a Twitter bird with a Thyroid problem), specifically with respect to the bits you need to know to set up your first Twitter application with OAuth. The other things OAuth does are important, but this is the stuff that directly impacts you, and that you need to grok to get started with your app.

I was absurdly and inexplicably tempted to randomly throw a Boba Fett icon into that diagram, but was afraid it might confuse people. That said, I have poor impulse control, so here’s a random Boba Fett icon, so I can sleep tonight. As my friend Jason Ramboz says, “Step 4, Boba Fett freezes the key in carbonite for transport.”

Moving on.

Now that you’ve got a good idea of how the basics of OAuth work, you’re ready to get started with Abraham’s great Twitter OAuth library. He does provide an example script in the downloadable code, but it might be confusing for people just starting out.

Getting Started – Registering Your Application with Twitter

Before you even start mucking around in any code, you have to register your new application with Twitter. You’ll need a name and url for your application in order to register it, and you’ll need to define a callback url. The callback url is the full url of the page Twitter should send the user to after it’s done authenticating. This file can be named anything you want, but make sure the one you create on your server matches the one you register with Twitter. All of these details can be changed later if you change your mind or need to update something.

Once you’ve registered your application, Twitter will issue you a Consumer Key and a Consumer Secret for your new app. You’ll need these to get your sample code from the Twitter OAuth library working. As you can probably tell by the name, your Consumer Secret should remain private and you should never give it out to anyone. It’s used in your code so that Twitter can identify your application when you’re making API calls.

By forcing you to send your consumer key and secret with your API calls, Twitter is able to determine which application is sending the API calls, and can verify that the Twitter user you are attempting to send API requests on behalf of has actually authorized your application to access their account. If the user decides they no longer want to allow your application, they can edit their allowed application preferences and your application will no longer be able to make API calls on their behalf.

You can access a list of all of the applications you have registered with Twitter – and links to edit their details or view the consumer key and consumer secret – by going to your oauth clients page on Twitter.

The Twitter OAuth PHP Library Code

You’ve got your consumer keys from Twitter, so now you’re ready to download Abraham’s Twitter OAuth library code. You can pull the code from http://github.com/abraham/twitteroauth. As I mentioned, he does provide an example script, but there’s not a lot of explanation given to it, so some people might be a little confused by it if its their first foray into Twitter applications with OAuth. We’re going to whip up something a little more straightforward and simple, so you can easily modify it to suit your needs.

Unpack/unzip the archive you downloaded from github. You’ll see the two main files, OAuth.php and twitterOAuth.php are in the top level directory, and there is a directory called ‘example’, that has the included example script.

For our example, we’re going to put the two OAuth files into a directory called ‘twitterOAuth’, which is a sub-directory of where the index.php and callback.php files live. As you may have guessed, the callback.php file is the one we’ve registered with Twitter as being our callback url. We’ll keep common configuration options such as the consumer key and consumer secret, and database credentials in a config.php file.

[source lang='php']/* config.php */

/* Consumer key from twitter */
$consumer_key = ‘xxhjgxhjxhhjgxjhjxgjyx768678xx’;

/* Consumer Secret from twitter */
$consumer_secret = ‘jhgjdfgfgjhj76jgjgjhxxxjhxxx’;
[/source]

Now we create the index.php file, which will be used to generate the authentication link, inviting users to authorize and login using Twitter.

[source lang='php']/* index.php */

session_start();

/* Destroy the session if the user is logging out */
if ((isset($_GET['logout'])) && ($_GET['logout']==’true’)) {
session_destroy();
session_unset();
}

/* Include the config file */
require_once(‘config.php’);

/* include the twitter OAuth library files */
require_once(‘twitterOAuth/twitterOAuth.php’);
require_once(‘twitterOAuth/OAuth.php’);

/*
Create a new TwitterOAuth object, and then
get a request token. The request token will be used
to build the link the user will use to authorize the
application.

You should probably use a try/catch here to handle errors gracefully
*/
$to = new TwitterOAuth($consumer_key, $consumer_secret);
$tok = $to->getRequestToken();

$request_link = $to->getAuthorizeURL($tok);

/*
Save tokens for later – we need these on the callback page to ask for the
access tokens
*/
$_SESSION['oauth_request_token'] = $token = $tok['oauth_token'];
$_SESSION['oauth_request_token_secret'] = $tok['oauth_token_secret'];

echo ‘

login using twitter | ‘;
echo ‘Logout

‘;
[/source]

The callback.php file is the script that Twitter sends the user back to after authenticating. Here you’ll probably want to set some cookies, store some user data in the database, and start letting the user do whatever it is your application does.

[source lang='php']/* callback.php */

session_start();

/* Include the config file */
require_once(‘config.php’);

/* include the twitter OAuth library files */
require_once(‘twitterOAuth/twitterOAuth.php’);
require_once(‘twitterOAuth/OAuth.php’);

/* check for an auth access token. If there’s no auth token set, go ahead and fetch one from Twitter,
* using the API call. */
if ((!isset($_SESSION['oauth_access_token'])) || ($_SESSION['oauth_access_token'])==”) {

$to = new TwitterOAuth($consumer_key, $consumer_secret, $_SESSION['oauth_request_token'], $_SESSION['oauth_request_token_secret']);
$tok = $to->getAccessToken();

/* Save tokens for later – might be wise to
* store the oauth_token and secret in a database, and
* only store the oauth_token in a cookie or session for security purposes */
$_SESSION['oauth_access_token'] = $token = $tok['oauth_token'];
$_SESSION['oauth_access_token_secret'] = $tok['oauth_token_secret'];

}

/* Connect to the Twitter API */
$to = new TwitterOAuth($consumer_key, $consumer_secret, $_SESSION['oauth_access_token'], $_SESSION['oauth_access_token_secret']);
$content = $to->OAuthRequest(‘https://twitter.com/account/verify_credentials.xml’, array(), ‘GET’);
$user = simplexml_load_string($content);

if ($user->screen_name!=”) {
echo ‘

SimpleXML. Using SimpleXML, we can call up any node values within the XML using $user->field_name, as you can see above.

I’ve included a print_r($user) so that you can see the full details of the array being returned, but you’ll obviously want to comment that out in your live code.

The output array will contain the following fields:

[source lang='html']SimpleXMLElement Object
(
[id] => 14246782
[name] => snipe
[screen_name] => snipeyhead
[location] => New York
[description] => Codemonkey, designer, author, speaker, blogger, swordfighter, Warcrafter, sarcasticgeek, scuba diver, blacksmith, crimefighter, Mentat, MBTI: ENTP, Totally NSFW
[profile_image_url] => http://s3.amazonaws.com/twitter_production/profile_images/303658881/Photo_4-rcrop2_normal.jpg
[url] => http://www.snipe.net
[protected] => false
[followers_count] => 4224
[profile_background_color] => 340100
[profile_text_color] => 3C3940
[profile_link_color] => 6C2125
[profile_sidebar_fill_color] => AEA797
[profile_sidebar_border_color] => 943A39
[friends_count] => 3756
[created_at] => Fri Mar 28 20:37:35 +0000 2008
[favourites_count] => 314
[utc_offset] => 12600
[time_zone] => Tehran
[profile_background_image_url] => http://s3.amazonaws.com/twitter_production/profile_background_images/22127710/twitterback2.jpg
[profile_background_tile] => false
[statuses_count] => 20570
[notifications] => false
[verified] => false
[following] => false
[status] => SimpleXMLElement Object
(
[created_at] => Mon Jul 27 01:50:36 +0000 2009
[id] => 2862508774
[text] => @elazar In case a name gets blocked/banned – when its reinstated (by someone claiming it, not spamming), it has a new ID#
[source] => Tweetie
[truncated] => false
[in_reply_to_status_id] => 2860170987
[in_reply_to_user_id] => 9105122
[favorited] => false
[in_reply_to_screen_name] => elazar
)

)[/source]

We’re not actually doing anything magical here yet, since that information is all available publicly via a user’s RSS feed, but the key line of code you want to look at in callback.php is this one:

[source lang='php']$content = $to->OAuthRequest(‘https://twitter.com/account/verify_credentials.xml’, array(), ‘GET’);[/source]

The OAuthRequest function is what actually sends the requests to the API, so you’ll be using this a lot. In the example above, all we were doing was getting the access tokens, but you’ll use OAuthRequest for just about everything else, too. For example, to send a Direct Message in Twitter, you’d use:

[source lang='php']
$params = array(‘user’ => ‘username’, ‘text’ => ‘this is a test message’);
$do_dm = simplexml_load_string($to->OAuthRequest(‘http://twitter.com/direct_messages/new.xml’, $params, ‘POST’));[/source]

To block a user, you’d do:

[source lang='php']$doblock = simplexml_load_string($to->OAuthRequest(‘http://twitter.com/blocks/create/username.xml’, array(), ‘POST’));[/source]

To send a status update:
[source lang='php']$content = simplexml_load_string($to->OAuthRequest(‘https://twitter.com/statuses/update.xml’, array(‘status’ => ‘Test OAuth update. #testoauth’), ‘POST’));[/source]

Important! Storing user IDs

Whenever you’re storing Twitter IDs in a database, be sure to store the Twitter ID number in addition to (or instead of) the Twitter username. While it may seem obvious to use a numeric value over a mixed alphanumeric, Twitter doesn’t expose user’s ID numbers without a little digging, so it might be easy to forget that they exist.

There are two main reasons why using the numeric ID is critical:

• Users can change their Twitter usernames. If they did this, your entire database could potentially be screwed up, since username key you’re looking for won’t match any longer.
• If an account has been suspended due to spam or imposters, it can potentially be available for registration again after a grace period. If a spammer had a username before, and then a legitimate user reclaimed it, your records could potentially have old data from the previous user’s account.

The second point above became crystal clear while working on DoucheNuker.Com. If a user account was suspended due to spamming, and then a legitimate user took it over, that new, legitimate user could potentially be considered a spammer in our database if we didn’t store (and query against) the ID number, too. When a username is reissued or reclaimed, it gets a new user ID number, so as long as you store and use the Twitter user’s ID number, your database can remain agnostic to name changes and reissues.

You’ll note in the Twitter REST API documentation that almost all API requests allow the option of using the username or the user ID, and some actually require the user ID and cannot be used with just a username.

Important! Error Messages and Throttling

You do not want to authenticate against Twitter every single time you load the page, but will instead want to store the request tokens in a database or session so that you don’t keep hammering Twitter’s API each time the page loads.

Remember that the although the Request Token you used to generate the authorization link will change often, a user’s Access Token and Access Secret Token do not, so you can safely store those in a database and use those instead of re-validating every time.

As of right now, Twitter is throttling validation requests to 15 per Twitter account per hour. This was implemented to improve Twitter’s security and make it harder for bad guys to brute force their way into someone else’s Twitter account. There is discussion about rolling this change back, or only throttling to 15 failed attempts per hour, but as of this moment, if you attempt to authenticate more than 15 times in an hour, you’ll get a message that says “Too many requests in this time period. Try again later.” There is no way around this message for now, so plan your application accordingly.

This limit is entirely separate from the Twitter Rate Limit that throttles the number of times you can hit the API. Whitelisting your account and IP address with Twitter will NOT circumvent this rate limit, so make sure you design your app in a smart way that will not attempt to authenticate more than absolutely necessary.

The default rate limit for calls to the REST API is 150 requests per hour. The REST API does account- and IP-based rate limiting. Authenticated API calls are charged to the authenticating user’s limit while unauthenticated API calls are deducted from the calling IP address’ allotment.

You’ll notice in all of API requests, we’re using SimpleXML to capture the value of the XML that’s returned. We need to do this in order to make sure we’re capturing any error messages that Twitter returns to us. Without error messages, when stuff doesn’t work as expected, we’re flying completely blind. Always make sure to plan your application in a way that handles errors intelligently. Let’s take a look at the API call to send a Direct Message again:

[source lang='php']$params = array(‘user’ => ‘username’, ‘text’ => ‘this is a test message’);
$do_dm = simplexml_load_string($to->OAuthRequest(‘http://twitter.com/direct_messages/new.xml’, $params, ‘POST’));

/* Check for an error response from Twitter */
if ($do_dm->error!=”) {
echo ‘

ERROR: ‘.$do_dm->error.’

‘;
}[/source]

Now we’re capturing the error returned from Twitter, and can handle this appropriately with our users. The error might be indicating that the user cannot send a Direct Message to someone they’re not following. Or there might be something else amiss – so you’ll want to make provisions in your script to help the user understand why something might not be working.

And that’s honestly all there is to it. Now that you’ve got the OAuthRequest function sussed, you just need to check with the Twitter API Wiki to determine the correct urls and parameters to send, based on what you’re trying to do.

I have to say, having worked with a LOT of APIs, including Facebook, Amazon, and at least a half-dozen others, Twitter’s API is actually the most well-documented and simplest to use. Surprising, really, since Facebook and Amazon have actual business models, so you’d think they’d invest just an iota of time into documenting their shit. I’ve gone into long tirades here on my blog about how miserably awful the Facebook API documentation is, and Amazon’s API is probably 10x worse. Twitter’s API is, overall, pretty accurate and up to date. If its your first foray into writing an application with an API, I think Twitter is actually a good place to start – before you graduate to Facebook and wish you were dead.

Recap – Important Links

• Register your application with Twitter
• List of all of your registered apps on Twitter
• Twitter API Documentation
• Twitter API Rate Limiting Documentation
• Download & Docs for Abraham’s OAuth PHP library
• OAuth official website

And that’s all there is to it. Please use your new powers for good and not evil. No annoying games, no “increase your followers” services, etc. If you have any questions, leave ‘em in the comments.

Also check out:

The nuts and bolts of IP-based geolocation is as follows:

1. User lands on the web page
2. User’s IP is captured and posted to a third-party geolocation service, which returns a latitude and longitude for the IP address
3. Website performs a radius database search using the latitude and longitude provided by the third party service and returns content relative to that location

Bear in mind, your database must contain data that is stored with a latitude and longitude for this to work. You cannot compare a user’s latitude and longitude to the lat/long of a piece of data if you have no lat/long associated to your content.

Also, please note: the services mentioned here – and the code provided – is intended for US-based geolocation. The code will remain basically the same for non-US lookups, but you may need to use alternate third-party services if your country is not covered by the ones mentioned here.

1. Getting Latitude/Longitude for Your Database Data

The first part of this process starts with you tagging your existing or newly added database data with the correct latitude and longitude. Each item you wish to include in the radius search should have a valid lat/long value in the latitude and longitude fields. If you are modifying an existing database, you would execute an alter table command to add the new lat/long fields. For this example, we’ll be using a table called stores:

[sourcecode language='sql']CREATE TABLE `stores` (
`store_id` INT NOT NULL AUTO_INCREMENT ,
`store_address` VARCHAR( 40 ) NULL ,
`store_city` VARCHAR( 40 ) NULL ,
`store_state` VARCHAR( 2 ) NULL ,
`store_country` VARCHAR( 2 ) NULL ,
`store_phone` VARCHAR( 15 ) NULL ,
UNIQUE (`store_id`));[/sourcecode]

And then we add the new latitude and longitude fields to our stores table, using the DOUBLE datatype:

[sourcecode language='sql']ALTER TABLE `stores` ADD `latitude` DOUBLE NULL ,
ADD `longitude` DOUBLE NULL ;[/sourcecode]

Now our table is set up to store the latitude and longitude data, but from where do we actually get the data, short of manually looking it up for each row of data in our stores table? Easy. There is a fabulous free service available at rpc.geocoder.us that lets you post an address to their API, to which it responds with the latitude and longitude values for that address. The easiest way to handle this process is to set up a cURL request in the admin area where you’re managing your data – otherwise you’ll need to write a script to cycle through the rows of data, fetching the latitude and longitude.(You will, of course, need to have PHP configured with cURL support for this to work.) Using this service, free lookups are throttled by your IP address to one request every 15 seconds – this may cause issues when you’re initially trying to get the lat/long data into your database for existing records, but shouldn’t be much of an issue afterwards.

First let’s set up a small function to handle the cURL request:

[sourcecode language='php']function curl_string ($url){
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_HEADER, 0);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($ch, CURLOPT_TIMEOUT, 120);
$result = curl_exec ($ch);
curl_close($ch);
return $result;
[/sourcecode]

The format you will be sending the query to the rpc.geocoder.us API is: http://rpc.geocoder.us/service/csv?address=1600+Pennsylvania+Ave%2C+Washington+DC

[sourcecode language='php']
// set the url of the API using the address, city and state we want to query

$url_page = “http://”.”rpc.”.”.geocoder”.”.us/service/csv”;
$url_page .=”?address=”.urlencode($address).”,”.urlencode($city).”,”.$state;

// execute the cURL request
$string = curl_string($url_page);

// turn the comma separated csv data turned into an array,
// so we can easily see that a match was found and use the pieces
$address_pieces= explode(“,”, $string);

// make sure the array contains data
if (count($address_pieces) > 0) {

// update the database
$sql = “update stores set latitude=’”.$address_pieces[0].”‘, “;
$sql .=”longitude=’”.$address_pieces[1].”‘, “;
$sql .=”zip=’”.$address_pieces[5].”‘ where id=’”.$store_id.”‘”;

if ($update_latlong = mysql_query($sql)) {
echo ‘Lat/long updated!’;
} else {
// if the query failed, print out an error
echo mysql_error();
}

// if the array does not contain data, no match was found in geocoder.us, so suggest using google maps to find the lat/long manually
} else {
echo ‘No geolocation match.’;

}[/sourcecode]

The code above will help you update your existing data if you were to use it in a script that cycles through your database records, or you can use it as part of your store administration area, doing the cURL request every time a new store is saved to the database.

2. Getting the Latitude and Longitude of the User

Now that you have base lat/long data for the data in your database, you have to obtain the lat/long for the user visiting the site. For this next part, you will again need to access a third-party service, this time to get the latitude and longitude based on the user’s IP address. I use a commercial service available from MaxMind.Com. It’s not free, but their prices are very reasonable ($20 per 50,000 queries) – and by using a cookie to store whether or not the user’s lat/long has already been returned, you can save on the number of accesses you use up on a busy site.

[sourcecode language='php']// begin the session
session_start();

$expireTime = 60*60*24*30; // 30 days
session_set_cookie_params($expireTime);

if ((!isset($_SESSION['geo_country'])) || ($_SESSION['geo_country']==”)) {

// enter your MaxMind license key here
$license_key=’XXXXXXXXXXXXXXX’;
$ip = $_SERVER['REMOTE_ADDR'];

$query = “http://”.”geoip1.”.”maxmind”.”.com/b?l=” . $license_key . “&amp;amp;amp;amp;amp;amp;amp;i=” . $ip;
$url = parse_url($query);
$host = $url["host"];
$path = $url["path"] . “?” . $url["query"];
$timeout = 1;
$fp = fsockopen ($host, 80, $errno, $errstr, $timeout)
or die(‘Can not open connection to server.’);

if ($fp) {
fputs ($fp, “GET $path HTTP/1.0\nHost: ” . $host . “\n\n”);

while (!feof($fp)) {
$buf .= fgets($fp, 128);
} // endwhile

// split the output into an array
$lines = split(“\n”, $buf);
$data = $lines[count($lines)-1];
fclose($fp);

$geo = explode(“,”,$data);
$user_geo_country = $geo[0];
$user_geo_state = $geo[1];
$user_geo_city = $geo[2];
$user_geo_lat = $geo[3];
$user_geo_lon = $geo[4];

$_SESSION['geo_country'] = $user_geo_country;
$_SESSION['geo_state'] = $user_geo_state;
$_SESSION['geo_city'] = $user_geo_city;
$_SESSION['geo_lat'] = $user_geo_lat;
$_SESSION['geo_lon'] = $user_geo_lon;

setcookie(“geolocCookieCountry”, $user_geo_country, time()+$expireTime, “/”);
setcookie(“geolocCookieCity”, $user_geo_city, time()+$expireTime, “/”);
setcookie(“geolocCookieState”, $user_geo_state, time()+$expireTime, “/”);
setcookie(“geolocCookieLat”, $user_geo_lat, time()+$expireTime, “/”);
setcookie(“geolocCookieLon”, $user_geo_lon, time()+$expireTime, “/”);

} // endif $fp

} // endif session set
[/sourcecode]

Now, you’ll only be querying MaxMind if the user hasn’t already accessed the site before and had their latitude and longtude stored in the cookie. Note the $license variable in the code above. When you sign up for the MaxMind Web Service, you will be given a license number, which you’ll insert there.

3. Putting the Two Together to Return Results Within X Miles

To tie the two together and query the database, returning only results that are within a specified number of miles of the user’s IP address, we’ll need two classes:

[sourcecode language='php']class RadiusCheck {

var $maxLat;
var $minLat;
var $maxLong;
var $minLong;

function RadiusCheck($Latitude, $Longitude, $Miles) {
global $maxLat,$minLat,$maxLong,$minLong;
$EQUATOR_LAT_MILE = 69.172;
$maxLat = $Latitude + $Miles / $EQUATOR_LAT_MILE;
$minLat = $Latitude – ($maxLat – $Latitude);
$maxLong = $Longitude + $Miles / (cos($minLat * M_PI / 180) * $EQUATOR_LAT_MILE);
$minLong = $Longitude – ($maxLong – $Longitude);
}

function MaxLatitude() {
return $GLOBALS["maxLat"];
}
function MinLatitude() {
return $GLOBALS["minLat"];
}
function MaxLongitude() {
return $GLOBALS["maxLong"];
}
function MinLongitude() {
return $GLOBALS["minLong"];
}

}[/sourcecode]

and

[sourcecode language='php']class DistanceCheck {

function DistanceCheck() {
}

function Calculate(
$dblLat1,
$dblLong1,
$dblLat2,
$dblLong2
) {
$EARTH_RADIUS_MILES = 3963;
$dist = 0;

//convert degrees to radians
$dblLat1 = $dblLat1 * M_PI / 180;
$dblLong1 = $dblLong1 * M_PI / 180;
$dblLat2 = $dblLat2 * M_PI / 180;
$dblLong2 = $dblLong2 * M_PI / 180;

if ($dblLat1 != $dblLat2 || $dblLong1 != $dblLong2)
{
//the two points are not the same
$dist =
sin($dblLat1) * sin($dblLat2)
+ cos($dblLat1) * cos($dblLat2)
* cos($dblLong2 – $dblLong1);

$dist =
$EARTH_RADIUS_MILES
* (-1 * atan($dist / sqrt(1 – $dist * $dist)) + M_PI / 2);
}
return $dist;
}

}[/sourcecode]

And then, to perform the actual query:

[sourcecode language='php']
// set a default number of miles to search within
$Miles = ’50′;

// set the user’s latitude and longitude as the one to search against
$Latitude = $user_geo_lat;
$Longitude = $user_geo_lon;

$zcdRadius = new RadiusCheck($Latitude,$Longitude,$Miles);
$minLat = $zcdRadius->MinLatitude();
$maxLat = $zcdRadius->MaxLatitude();
$minLong = $zcdRadius->MinLongitude();
$maxLong = $zcdRadius->MaxLongitude();

$sql = “SELECT store_address, store_city, store_state, store_phone, “;
$sql .= “SQRT((((69.1*(latitude-$Latitude))*(69.1*(latitude-$Latitude)))+((53*(longitude-$Longitude))*(53*(longitude-$Longitude))))) “;
$sql .= “AS calc FROM stores where “;
$sql .= “latitude >= ‘$minLat’ “;
$sql .= “AND latitude <= '$maxLat' ";
$sql .= "AND longitude >= ‘$minLong’ “;
$sql .= “AND longitude <= '$maxLong' ";
$get_data = mysql_query($sql);

// loop through the matching database results
while($storedata = mysql_fetch_assoc($get_data)) {

// calculate the number of miles away the result is
$zcdDistance = new DistanceCheck;
$Distance = $zcdDistance->Calculate($Latitude,$Longitude,$storedata['latitude'],$storedata['longitude']);

// and for the non-US people, here’s the km calculation
$calc_km = round(($Distance * 1.609344),2);

echo ‘

‘.$storedata['store_address'].’
‘.$storedata['store_city'].’, ‘;
echo $storedata['store_state'].’ ‘.$storedata['store_country'].’
‘;
echo $storedata['store_phone'].’
‘;
echo ‘Distance: ‘.$Distance.’ (‘.$calc_km.’ km)’;
}[/sourcecode]

And that’s really all there is to it.

Store Locator Only

If you want to create a store-locator style script without automagically getting the user’s current location, you’d use the code above, almost verbatim. The difference is that you’d need an additional table of zipcodes with associated lat/long, available for purchase (again, not very expensive) from ZipCodeDownload.Com. The process would go as follows:

1. User arrives at your site, and enters a zip code and mile radius they wish to search using your search form – clicks submit
2. The script queries the zip code table to find the latitude and longitude associated with the postal code the user has entered, and uses THAT latitude and longitude as the values for $Latitude and $Longitude in the code above, and uses the user-entered values for $Miles from the search form.

Everything else stays exactly the same. Naturally, you’ll probably want to add some sanity checks in your own code (gracefully displaying default content if no latitude and longitude is returned, etc) but I decided to skip that here so as not to confuse anyone.

Enjoy, and if you end up using this anywhere, let me know how it works out.

Also check out:

]]> http://www.snipe.net/2008/12/ip-geolocation-radius-search-in-php/feed/ 10 http://www.snipe.net/2008/12/fixing-curly-quotes-and-em-dashes-in-php/ http://www.snipe.net/2008/12/fixing-curly-quotes-and-em-dashes-in-php/#comments Thu, 11 Dec 2008 15:23:31 +0000 snipe http://www.snipe.net/?p=476 The curly quotes, or “smart quotes” generated by Microsoft Word and other applications can be a real headache to developers. If you’ve built an administration area for your content publishers, and the publishers frequently compose their posts in Word and then copy+paste into your form to publish to the web, you may run into the situation where the curly quotes are replaced by your browser’s version of an unrecognized symbol, often a question mark. This can be particularly frustrating when Word-generated characters such as these curly quotes or em dashes break content-generated XML feeds, even after you’ve been careful enough to convert “normal” HTML special characters so that your XML would be valid. Fortunately, there is an easy workaround.

Rather than try to convince your publishers to stop using Word to compose their content, the easier (and more effective) solution will be to replace the curly quotes with “normal” quotes before the data is inserted into the database.

The function below will convert curly quotes and em dashes into standard quotes and dashes “-”. If you’ve got a handful of classes or functions that you routinely use as part of your data scrubbing process (to clean data before it gets sent to the server), you may want to include this function in that group, that way you don’t ever have to think about it again.

[sourcecode language='php']
function convert_smart_quotes($string)
{
$search = array(chr(145),
chr(146),
chr(147),
chr(148),
chr(151));

$replace = array(“‘”,
“‘”,
‘”‘,
‘”‘,
‘-’);

return str_replace($search, $replace, $string);
}
[/sourcecode]

Also check out:

]]> http://www.snipe.net/2008/12/fixing-curly-quotes-and-em-dashes-in-php/feed/ 17 http://www.snipe.net/2008/12/planning-a-facebook-application-part-two/ http://www.snipe.net/2008/12/planning-a-facebook-application-part-two/#comments Fri, 05 Dec 2008 06:42:25 +0000 snipe http://www.snipe.net/?p=430 I know I promised you that we’d get into some code in the next part in this series, but the article is coming out much longer than I anticipated, as I expect it to be one of the most thorough articles out there regarding Facebook application design. Part two of this series will walk through designing a real-life Facebook application – one I wrote specifically for this article series.

The first article discussed what features and screens you have available to you in Facebook apps in general – and in this article, we’re going to go through the process of actually planning a real one. You’ll see some pitfalls and hopefully learn to think about your applications in a new, more complete way, harnessing all of the aspects of Facebook’s platform.

Although we won’t be delving into code in this article either, I strongly encourage you to read this article (and the previous article, if you haven’t yet) for your own sanity’s sake. For all the frustrations that come with writing Facebook applications, I can say from experience that at least 75% of my frustration was a direct result of devoting enough time towards the planning phase of each specific application. Plus, the application we spec out in this article is going to be the base that we use in the next part – the one where we actually do start touching code. That article is about 60% written, but breaking it up makes far more sense, since you SHOULD be planning your applications before you start touching code anyway. I promise you, getting this part right will save you days, if not weeks, of re-writing code and fixing issues. I’ve done it. It sucks.

Determine the Social Action Points Before You Start Development

One thing I have learned with regard to Facebook application development is that its very easy to rush into development and inadvertently miss some key integration and social action points.  It’s easy to do, since they are not “visible” as you’re developing the app. When the user interacts with your application, you may forget to consider all of the potential notifications that can be triggered, because you can’t physically see them. Out of site, out of mind means that you’re throwing them in haphazardly after the app is built, or worse yet, not using them at all.

“Invisible” (but very important) things to consider are:

• Newsfeeds – when a user directly interacts with your application, you can post that action to the user’s newsfeed.
• Email notifications – if a user has permitted your application to send them email, you can send plain text or (limited) HTML emails to notify them of action items

Newsfeeds are an absolutely critical part of your Facebook application, so spending a little time planning what actions a user can take on each screen of your app is a great start. The actions the user takes that are entered as newsfeed items are prominently visible on the user’s profile page, and will be visible to their friends on the Facebook “home” page that shows recent actions friends have taken.

Sample newsfeed application items

The main goals of your newsfeed items should be to highlight worthy user actions – the benefit of this will be that the user’s friends see these newsfeed items and may be interested in checking out your application.

Emails are important, but since your users must opt-in to receive them, you may not reach as many people with them. Still, its important to plan them out the same way you plan newsfeeds.

As we go through the sample application, we’ll figure out where newsfeed items would be appropriate. Each application is allowed x number of newsfeed posts per user per day, so you want to choose them wisely and make sure they are truly relevant and not spammy. Each application’s cap is different, and that number changes based on the way people interact with it, but applications start off with around 20 max newsfeed posts per user per day. (This can make testing a bitch, which is another reason its important to get them into the application early in development.)

Also bear in mind that Facebook’s newsfeeds MUST be triggered by an action the app user has taken. You cannot have your application send newsfeed entries without the user clicking on something or interacting in some way. The format must be “<user> has done xyz”, so the newsfeed reflects on the action that specific user took.

Three Sizes of Newsfeeds

Facebook allows you to create three versions of a newsfeed item – a headline, a short story and a fullsize story. Headlines are the default view, so they should be well-crafted. If a user likes the headline, they can click to see the short story or fullsize, but your headline version has to be great to make them click.

Our Sample Application

Because no application should be started without a game plan, we need a plan for what the sample application we’re discussing here will do. For the sake of this tutorial, our application will behave similarly to the “poke” feature of Facebook, only we’ll be blowing kisses. (I realize this is a cheesy idea for an application, but it will keep it simple, and will allow us to demonstrate actions that can only be invoked when a user takes action on another user.) You can see this app live in action here.

In the Planning Your Facebook Application article, we discussed planning out each of the boxes that will be available to you in the application. So for this application, we’re going to plan for this:

1. Application canvas page – listing of recent incoming and outgoing kisses for the viewing application user
2. Profile box - display of recent received kisses for the profile owner, with action item for viewer to send the profile owner a kiss
3. Boxes tab: Wide -  listing of recent incoming and outgoing kisses for the profile owner user, tailored for non-application users, since Facebook users other than the profile owner can see this page
4. Boxes tab: Narrow – Smaller version of the wide display, for users who opt to use the narrow box column
5. Fan page – Although I opted not to create a fan page view of this app in the real app, for the sake of argument we could include a list of most recent back-and-forth kiss activity, or a leaderboard of the most active kissers – with an action item inviting users to start blowing kisses. Remember that not everyone viewing the fan page that your app gets added to will have added the application, so the view should be tailored towards viewers who are not application users. Since it will be on a fan page, and could get decent exposure depending on the popularity of the page to which it’s added, you’ll want to make sure this view encourages sharing and competition to compel new people to add it.
6. Application tab – same display as the canvas page

I encourage you to actually draw these screens and boxes out on paper, or in your favorite graphics program. It won’t take long, and will be a huge help in thinking through the interface and making sure you’re writing the app in a way that will make sense to the user. The mockups (often called wireframes) don’t have to be complicated or pretty, but they should roughly represent all of the key interface elements in position and placement.

Our application is very simple, so we only have two main screens: the homepage that will display the incoming and outgoing kisses, and the page that allows the user to pick from a list of their friends and blow kisses to them. A more complicated application could require a dozen or more screens, with each screen representing a page of functionality within the app.

Canvas page (displays recent kisses, sent and received)

Homepage mockup

Send kisses page

Blow kisses page

As we’re creating these wireframes, it becomes easier to figure out where newsfeeds and app-generated emails will be most useful.

One the canvas page, the page listing incoming and outgoing smooches, the user isn’t taking any direct action, so there is no direct newsfeed trigger for newsfeeds or emails. The blow kisses page is a good place for both.

When the user blows someone (or several someones) a kiss, it should trigger a newsfeed item: “<user> has blown <recipient> a kiss!” Because of the rule that the user has to directly trigger the newsfeed item, it would NOT be permitted to also add a newsfeed item into the recipient’s newsfeed such as “<recipient> has received a kiss from <user>”. You could trigger that second newsfeed item when the user picks up their kiss though, since the user (in this case, the recipient) took direct action with the application. This may seem like nuance, but its important to understand the difference in order to remain compliant with Facebook’s terms of service.

The recipient should then receive an email notification letting them know someone has blown them a kiss. If the user has not yet allowed the application and agreed to let the application send them emails, they will not receve this notification, however, so its not a form of communication you should rely on.

Another Example

As I have mentioned, this application is extremely simple. It was designed to be simple to keep the coding part (coming soon, I swear) simple – but it doesn’t give us the opportunity to really demonstrate places where newsfeeds will serve you best in your application.

Another application I wrote (which is a work in progress) might be a better example. The WoW Toons application, which is a simple application that allows users to display their World of Warcraft characters on their profile, has a few more potential interaction points.

Canvas Page: Displays the user’s current WoW characters (also known as ‘toons’) with level, race, class, etc. It also contains the functionality for the user to force an update to their character, so if they have gained a level, the app will reach out to the WoW database and fetch new data. It then compares the newly fetched character level with the one stored in the database, and if the new level is higher, it updates the database AND triggers a newsfeed item: “<user> has reached level <level> on <pronoun> <race> <class> – Ding!”

When a user adds a new character to their profile, a newsfeed item gets inserted: “<user> has added their level <level> <race> <class> to their Warcraft Toons profile”

Newsfeeds also allow you to add an action item link (although its small and arguably not that noticeable.) For the Blow Kisses app, a “Blow <user> a Kiss Now” link with a link to the WoW app would be appropriate. For the WoW Toons app, something like “Add your own Warcraft toons now” link makes sense.

WoW Toons canvas page

The WoW database also provides data on specific gear (armor, weapons, pvp stats, etc), and although I don’t currently track that data (might tho), when a user refreshes their gear, it would be appropriate to add a newsfeed item for new weapons, stats, etc. “<user> has reached 1045 pvp kills”.

Your Friends Toons Page: In the WoW Toons application, you can see a listing of the toons for your friends who have added the app.

Friends page

One possible feature that could be added would be to show the “last updated” date on this page, and allow users to “nudge” their friends if the friend in question hasn’t updated their characters recently. “<user> has nudged <recipient> to update <pronoun> Warcraft toon.”

In the Mr. Right application mentioned in the first article, we wanted to use newsfeeds to show the gifts that Mr. Right (a fictional boyfriend avatar) has sent the user. Because it is not allowed to automatically generate newsfeed items that a user has not triggered, we had to change the app to let the user click on a button labeled “Send me a Gift”. Since the user took direct action by clicking the button, we were able to insert a newsfeed item: “<user> has received a <gift name> from Mr. Right.” The action text was then “Get your own Mr. Right now”, linking to the application.

Mr. Right gift newsfeed

The above is an example of the short story newsfeed format. “Alison received a puppy from Mr. Right” is the headline, and then the short story body contain the image and additional text.

Conclusion

I’m sure I sound like a broken record by now, encouraging you to take the extra time to create wireframes and plan out every part of your application before you start coding. I promise you that it will be worth it in the end – you will end up with a better application that users will find easier to use and that more effectively uses the social parts of the social networking platform. I have personally cut corners and skipped this part too often – its so tempting. “Oh, I know what its supposed to do. Its a simple application.” And it never, ever is as simple as it first seemed.

Coding comes next, so be sure to subscribe to our RSS feed to make sure you don’t miss the next part in this series, where we walk through creating a simple Facebook application based on the dicussion in this article.

Also check out:

]]> http://www.snipe.net/2008/12/planning-a-facebook-application-part-two/feed/ 40