I’ll start with the obvious candidates that you probably already have installed if you’re a developer. I’ve also added a few that are useful for post-hack diagnostics and recovery.

General

Firebug – Firebug is great for web development in general, but the debugging tools can help track down calls to rogue javascript on external servers, among many other things.

Web Developer Toolbar – Another great web dev tool, the Web Developer Toolbar makes it easy to turn javascript and cookies on and off selectively, view form fields and disable restrictions and much, much more.

DNS Cache – simple addon that lets you clear or disable Firefox’s DNS cache. Not specifically for pen testing, but useful nonetheless.

Notable - love this addon, which lets you do a full-page screenshot with annotations over at notableapp.com. As you’re testing, there’s a good chance you’re going to need to show your other devs or account managers a screenshot so they can see the vulnerability being exploited. While something simple like Fireshot would work fine (or native screenshots), I like using Notable for complex situations that require explanations on multiple points on the page. Exports to annotated PDF.

Groundspeed – simple form toolkit that allows you to edit form fields (hidden to text, etc), remove length restrictions, change/remove javascript event handlers, and change form target so that it opens in a new tab.

Code Injection

SQL Inject Me – helps test for SQL injection vulnerabilities.

XSS Me – used to test for reflected Cross-Site Scripting (XSS). It does NOT currently test for stored XSS.

Access Me – used to test some access vulnerabilities related to web applications. The tool works by sending several versions of the last page request. A request with the session removed will be sent. A request using the HTTP HEAD verb and a request using a made up SECCOM verb will be sent. A combination of session and HEAD/SECCOM will also be sent.

JS Deobfuscator – many attacks inject obfuscated javascript into a page so that it becomes harder for you to simply grep the source for something obvious, like the domain name to which the bad script is redirecting the user. This addon helps deobfuscate the javascript so you can see what’s really going on.

Hackbar – helps with testing sql injections, XSS holes and site security. Ugly as sin, but it works well.

Header and URL Monitoring/Tampering

Note that some of these addons do similar things – try them and stick with whichever one you like best.

HttpFox – monitors and analyzes all incoming and outgoing HTTP traffic between the browser and the web servers.

Live HTTP Headers – view HTTP headers of a page and while browsing.

Modify Headers – add, modify and filter http request headers. You can modify the user-agent string, add headers to spoof a mobile request (e.g. x-up-calling-line-id) and much more.

Tamper Data – use tamperdata to view and modify HTTP/HTTPS headers and post parameters, trace and time http response/requests and security test web applications by modifying POST parameters.

User Agent Switcher – allows you to easily toggle between pre-set user agent strings, or set your own.

Environment Detection

Header Spy – lightweight addon that displays information about the website’s server in your statusbar. This is not as useful for pen testing as it is for impressing the crap out of clients who don’t know what server they’re running.

Host Spy – integrated shortcut to show you who a website’s IP neighbors are on shared hosting.

ShowIP – Small addons that shows the IP address of the website in your statusbar and a link to some additional tools.

URL Flipper – quickly and easily increment and decrement numbers and strings in URLs for navigating through URL sequences (for example, user ids or session info in the query string.)

Wappalyzer – uncovers the technologies used on websites. It detects CMS and e-commerce systems, message boards, JavaScript frameworks, hosting panels, analytics tools and several more.

Searching

Offensive Security Exploit Database – this adds the excellent database of exploits at exploit-db.com as one of your search engine options.

I’ve also just created a search plugin for XSSed.Com, but it’s pending approval at Mozilla, so not sure when that will be ready for you which can be downloaded here. It’s not exactly rocket science to add a new search site to your browser search bar, but I figured it was quick and easy to whip up. Feel free to check out the source code for the plugin here.

Too Many Addons Got You Down?

If you’re finding your plugins are slowing down Firefox too much, you might want to create a separate Firefox profile specifically for testing, and switch to that profile when you’re ready to start hammering away. Also bear in mind that you might need to tweak some settings on these, or only enable them right before you use them, as the toolbars and sidebars can be a bit bulky.

Also keep in mind that the Net option on the web developers toolbar, or any of the header analyzer addons can be very helpful in general testing between dev and live environments (load the page on live and make sure nothing is being pulled from the dev address) and also to make sure your SSL requests are being handled correctly.

Some Additional Thoughts…

When folks ask me how I do penetration testing – whether I use software, or do it by hand – the best way I can answer is “both”. Software will only ever get you so far, but it’s a critical tool in helping you figure out where the vulnerabilities are. It’s not unlike using a metal detector to find treasure. When the metal detector is doing its job, it finds, well, metal. Not necessarily treasure, although fancier metal detectors have additional software that helps try to identify the buried object by shape and size. You still have to physically dig up the item and rely on your knowledge and experience to determine whether or not it really is treasure, or just junk. The metal detector simply finds something that meets a basic set of requirements, to save you from having to dig up every square inch of the beach.

When testing web applications for vulnerabilities, software does very much the same thing. It simply automates tasks that you could do by hand but that would take an unreasonable amount of time, but ultimately when it finds something, you still need to know enough about what you’re looking at to determine how big a threat it actually is. Most of the time the software will attempt the to try the lowest-level exploit, for example, the ability to execute arbitrary javascript in a page. Your testing tools may demonstrate that you can create a javascript alertbox on the page, but it’s your knowledge and experience that will help you determine the full extent of the vulnerability, for example whether that arbitrary javascript could be used to redirect a user to a new page, hijack the user’s session data, etc.

The reason I ended this post with a long-winded ramble is because I wanted to make it clear that just having the tools isn’t enough. Actually using them, and knowing what to do with the results are important. Understanding the basic mechanics of how exploits work is the only way you can make sure your applications are written to mitigate them. Having the tools installed but never understanding or using them is like buying a metal detector and keeping in the closet and then wondering why you haven’t found anything valuable yet.

If you’re interested in learning more about web application penetration testing and security, check out the following books:

• The Web Application Hacker’s Handbook: Discovering and Exploiting Security Flaws
• Web Security Testing Cookbook: Systematic Techniques to Find Problems Fast
• Foundations of Security: What Every Programmer Needs to Know
• Ninja Hacking: Unconventional Penetration Testing Tactics and Techniques

How Do You XSS?

These addons are not obviously meant to be a replacement for more capable and thorough penetration testing tools such as metaploit, netsparker, etc. They’re just meant to be a convenient way for developers to test code during and after development.

There is a more comprehensive collection of addons listed here, but this is what I use. If you’ve got a favorite that I’ve missed, please be sure to share in the comments!

Also check out:

Background

The MSWDS is one of only four significant annual events within the PHP community (others include tek, DPC and ZendCon), and this summit is a bit harder to get invited to. Unlike most other conferences, where all you need is the cash to pony up for a conference pass and a hotel room to crash in, invites are very limited and attendees are selected because they have had some interaction with the folks at Microsoft, and are believed to be leaders and influencers within the open source community. To be blunt, these summits cost Microsoft a lot of money, so they need to make sure they’re getting the best bang for their buck.

Keeping that in mind, it would be easy to assume that we were being brought out there so that Microsoft could pitch us on the latest and greatest Microsoft products, trying to get the movers and shakers of open source to drink the corporate kool-aid and switch to Microsoft products. While more acceptance of Microsoft products within the open source community is obviously a goal, they are making a concerted effort to learn from us – what we need, where they are falling short, and how we can move forward together.

Discussions and Format

The summit itself was a total of three days, with the last day being optional for those open source developers who were willing to sign an NDA to discuss some of Microsoft’s emerging technology. During the three days, different representatives from Microsoft’s product teams sat down with us and asked for our comments, thoughts and ideas about where they’re at, and where we think they should be going. We met with folks from the IIS Web Platform team, the SQL server team, as well as some representatives from Codeplex, Silverlight, Powershell, ASP.NET Ajax (which is not exclusive to ASP.NET, despite the name), and Bing maps.

We had a chance to air grievances, which was cathartic in some ways, but I think it was more important to us to be able to sit down with the actual teams who are working on this technology at Microsoft, and really get into the specific challenges we face. The approach was not generally pitchy, and with very few exceptions, a great deal of effort was put into making all of us from the open source community feel like respected authorities in our field whose opinions really matter.

Something they did this year which was apparently not done last year was to include representatives from well-known PHP-based projects who are not normally parts of the PHP community. I honestly hadn’t realized that the many of the folks over Joomla, WordPress and Drupal often don’t consider themselves as part of the greater PHP community, and getting a chance to discuss that with them brought up some interesting perspectives. I don’t think the guys representing these projects were there in an official capacity, but their point of view was one that had honestly not occurred to me before, so that was a really interesting and unexpected benefit. There was some debate on whether or not these types of projects should be a more involved part of the PHP community, with good points on both sides, but I think most walked away with some ideas on how to move forward in making those lines of communication more accessible and open.

My Perspective

Of course the ultimate question from Microsoft was “What would it take for you to switch to Microsoft products for your clients?” My smartass remark was, of course “A fucking miracle.” But everyone in the room knew I was joking. I hope. If we weren’t willing to work with Microsoft on improving their products to work with open source better, we wouldn’t have been there.

As often as Microsoft has been an easy target in the past, and as much bad blood as there may have been in the past, there are people at Microsoft that care about working with the open source community, and who are making progress to get there. It is our job as technology professionals to fairly evaluate technology and make recommendations based on what makes the most sense technologically and financially. It is NOT our job to make religious decisions based on zealotry.

That means that if and when Microsoft can meet my needs and/or the needs of my clients, it can and should be part of that evaluation or I’m not doing my job. Does that mean I’m ready to switch back? No. Not yet, anyway. But I believe they are listening, and I saw some things during this summit that make me far more likely to start including some parts of Microsoft’s products into the technology I suggest as being potentially viable for client projects, which is a far cry closer than I was last week. Specifically, some of the stuff I learned about Silverlight, Bing’s geolocation products and Windows Azure (Microsoft’s cloud hosting platform) was pretty impressive. As I get to play with these products a little more, I’ll be blogging about them with my fair evaluation of pros and cons, so stay tuned.

I’m also excited to see where the Microsoft Web Platform Installer product heads. Right now, the WebPI product is a very easy to use, slick solution for the less techy individual who wants to, for example, deploy a WordPress blog in 5 minutes or less and may not have the savvy to do the install themselves – basically a MS version of Cpanel/Fantastico, which we have had available to us as web administrators for over a decade. That product is less interesting to me right now, but some of the directions they could go in for more advanced users like us hold real potential. We had some suggestions that were well-received, and if they are actually implemented in the way I envision them, it could honestly turn the table and make some of the Microsoft web server products something that I could consider recommending, or even using myself. (I should also mention that Cpanel is the most horrific, insecure, hack-prone web control panel I’ve ever used, and I am NOT endorsing it as a solution.)

The reality is that competition inspires innovation, and Microsoft getting better means progress for everyone. I saw a post on Twitter that basically implied that open source representatives attending this conference were traitors or sellouts. I don’t see it that way at all. We have amazing open source products like Firefox because the open source community worked together to create a better product, and Microsoft responded by making vast improvements to Internet Explorer, building in more security and standards compliance. When we work together to innovate, everybody wins.

Another transition I’ve been seeing in Microsoft which was really made more obvious by this summit is that there is a less omnipresent feeling of “all or nothing” within many Microsoft departments. As open source advocates, we enjoy having choices. Previously with Microsoft, you’d get the most benefit from their products by committing to an entirely Microsoft development process (“drinking all of the kool-aid, since the best stuff is the sugary goop at the bottom”), with benefits sharply falling off if you opted to pick and choose. This philosophy has always been distinctly in opposition with the open source philosophy, and I believe was likely the cause for some of the distrust coming from the open source community. Seeing this transition into a paradigm of being able to cherry-pick what we like for some things and sticking with open source solutions we like better for others is a step in the right direction, in my opinion.

An additional unexpected benefit to sitting down with all these MS product people was that I got a chance to better understand some of the legal/licensing challenges Microsoft faces. I’m not making excuses for them, but I hadn’t considered some of the obstacles in the way of people at MS who care about working with us. Microsoft is a big target with deep pockets, and they have to cover their own asses. I was quicker to dismiss some of the corporate decisions as being “evil” prior to sitting down with some of them and understanding why they do what they do. Don’t get me wrong – some of their decisions (*cough*sudo*cough*) still don’t make sense to me and I believe they are wrong, but I think I have a better understanding of where they sit than I did before

Wrapping Up

Overall, I would consider this summit a great success, and I hope I get to participate again in the future. There are several people who really deserve a shout-out for all of the hard work that went into this and are directly responsible for it’s success. From the PHP community, Cal Evans was a co-host and an absolute rock star, always quick to make sure things ran smoothly and kick-start conversations and redirect us back when we went off on tangents. From Microsoft, Karri Dunn, Tonya Young, Josh Holmes, Peter Laudati, Lauren Cooney, and others were amazing. I may be forgetting a few – I am still a little wiped from the week and the traveling.

Was this an instant fix? Certainly not. Do we all have a lot more work to do before we’re “there”? Absolutely. But as Cal Evans put it on his own blog roundup, “The more people I get to know at Microsoft, the less I’m able to despise the company.” They took the time to find out what we think, even when it may not have been what they wanted to hear. Time will tell whether or not they actually act on it.

Other PHP Representatives Blog Post Roundups

I’ll be updating this list as more people finish their blog post roundups, so you can get take their on the summit. Many of them are far smarter than I am, so it’s worth reading what they have to say.

• Cal Evans (@CalEvans)
• Chris Cornutt (@enygma)
• Maarten Balliauw (@maartenballiauw)
• Rafael Dohms (@rdohms)
• Keith Casey (@CaseySoftware)
• Romain Bourdon (in French) (@le_vrai_roms)
• Marco Tabini (@mtabini)
• Sam Moffatt (@Pasamio)
• Helgi Þormar Þorbjörnsson (@h)

MS Representative Blogs

If you’d like to see more about the fabulous people at MS who are working hard to move the company forward in a way that works with open source, check out their blogs. I’m proud to call these guys friends, and as long as we continue to have people like this working for Microsoft, I think the lines of communication and cooperation between both sides of the aisle will keep moving forward.

• Dave Bost (@DaveBost) – Developer Evangelist
• Josh Holmes (@JoshHolmes) – UX Architect Evangelist
• Tobin Titus (@tobint) – MSDN Site Manager
• Peter Laudati (@jrzyshr) – Developer Evangelist
• Mark Brown (@MarkJBrown) – Product Manager for Microsoft Web Platform
• William Coleman (@will_coleman) – Developer Evangelist
• Lauren Cooney (@lcooney) – GPM for Web Platforms at Microsoft
• Jas Sandhu (@jassand) – Interop Strategy Evangelist
• Ruslan Yakushev (@ruslany) – Program Manager on IIS team in charge of FastCGI and PHP support
• Scott Hanselman (@shanselman) – Principal Program Manager Lead

Funnily, as I’m writing this, Futurama: Into the Wild Green Yonder has been on television, and the scene that just played is the one where Calculon says “I’d like to thank the academy, my agent, and most of all my operating system – Windows 7, for everything it –” at which point his OS locks up. Windows 7 is actually a great product, and I run it on my Mac using Bootcamp and VM Fusion, but I thought the timing was amusing.

Kool-aid photo taken in Belize by me – just thought it was funny. Lead post image by Eli White.

Also check out:

My friend Peter Bukowinski first brought the exploit to my attention, posting a link to Apple.Com’s iTunes affiliate search interface. The link he sent me led to a page that looked like this:

Notice that the url in the browser bar is actually apple.com – this was not a parody site.

Evidently, Apple’s developers had neglected to validate the data being sent through the query string. The actual url was:

http://www.apple.com/itunes/affiliates/download/?artistName=your+mom&thumbnailUrl=http://www.moneysavingmom.com/money_saving_mom/images/2008/09/02/joblogo.gif&itmsUrl=http://www.bjs.com/&albumName=a+better+blowjob

So by editing the variables passed through the url, you could have a little harmless fun at Apple’s expense:

As you can see, by editing the query string and changing the variables for artistName, thumbnailUrl and itmsUrl, we could make the page hosted on Apple.Com’s server display whatever mischief we want. The variables were being echoed out directly on the page without any validation, filling in the blanks in their iTunes affiliate page template: [image] Looking for [blank] by [blank]?

But What is XSS?

Honestly, if you’re a web developer and you don’t know what XSS is by now, you suck at your job and should probably go back to spanking it to porn in your mom’s basement and leave the coding to the grownups. It’s been around long enough that you forfeit your right to call yourself a web-anything if you don’t know what it is by now. That said…

From the Cross-Site Scripting FAQ on cgisecurity.com:

Cross site scripting (also known as XSS) occurs when a web application gathers malicious data from a user. The data is usually gathered in the form of a hyperlink which contains malicious content within it. The user will most likely click on this link from another website, instant message, or simply just reading a web board or email message. Usually the attacker will encode the malicious portion of the link to the site in HEX (or other encoding methods) so the request is less suspicious looking to the user when clicked on. After the data is collected by the web application, it creates an output page for the user containing the malicious data that was originally sent to it, but in a manner to make it appear as valid content from the website. Many popular guestbook and forum programs allow users to submit posts with html and javascript embedded in them. If for example I was logged in as “john” and read a message by “joe” that contained malicious javascript in it, then it may be possible for “joe” to hijack my session just by reading his bulletin board post.

Cross-Site Scripting is nothing new, not even on large, popular websites. While this example on Apple.Com resulted only in a humorous page being available under a large company’s domain, many XSS attacks can be far more sinister – and the attack had far more potential than our harmless prank, as users on Reddit.Com noticed that the exploit did allow malicious scripting including JavaScript injection and IFrame injection (thanks to @shocm for bringing the Reddit thread to my attention).

It looks as though Apple’s server sanitized the <script></script> tag, but there are at least a half-dozen ways to inject javascript without using a <script> tag, many of which are outlined on the XSS Cheat Sheet.

This type of exploit is a DOM-based exploit. Wikipedia does a good job of summing it up:

Traditionally, cross-site scripting vulnerabilities would occur in server-side code responsible for preparing the HTML response to be served to the user. With the advent of web 2.0 applications, a new class of XSS flaws emerged, however: DOM-based vulnerabilities come to be during the content processing stages delegated to the client, typically in client-side JavaScript. The name refers to the standard model for representing HTML or XML contents, called the Document Object Model or DOM for short. The model is the primary way for JavaScript programs to manipulate the state of a web page, and populate it with dynamically computed data.

A typical example is a piece of JavaScript accessing and extracting data from the URL via the location.* DOM, or receiving raw non-HTML data from the server via XMLHttpRequest, and then using this information to write dynamic HTML without proper escaping, entirely on client side.

Netcraft featured an XSS vulnerability on PayPal’s website discoverd by a Finnish security researcher in May 2008:

An exploit was reported in March 2007 on YouTube’s website, allowing a similar type of JavaScript attack:

And an XSS exploit of eBay was documented in this YouTube video, also in 2007, and Twitter has suffered several XSS exploit attacks as recently as this year. But those are just a small handful of examples from a really long list.

In fact, just today, an article came out in The Register, detailing an XSS cookie hijacking attack that affects many large websites, including Google and Facebook. From the article:

A security researcher has discovered a weakness in a core browser protocol that compromises the security of Google, Facebook, and other websites by allowing an attacker to tamper with the cookies they set.

The weakness stems from RFC 2965, which dictates that browsers must allow subdomains (think www.google.com) to set and read cookies for their parent (google.com). The specification also states that if a cookie for a subdomain doesn’t already exist, the browser should use the cookie belonging to the parent instead.

The arrangement makes it possible for attackers to steal or even alter the cookies that websites use to authenticate their users. Attackers would first have to identify an XSS, or cross-site scripting, bug in some part of the site they are targeting. But because virtually any subdomain will suffice, the scenario isn’t unrealistic, two web security experts said.

Apple’s development team responded quickly to the exploit on their site – a little too quickly in my opinion, since I was preparing to have a little more fun with it, but they had patched it by the time I got home. It should still serve as a reminder to developers of just how important data scrubbing and validation is, no matter whether your site is big or small, with 2 hits a day or 2 million.

XSS Vulnerabilities Compromise User Data – And Your Reputation

As the eBay exploit video shows (and as anyone on Twitter saw this year), XSS attacks are not just embarrassing – they can be used for phishing scams, tricking users to login to a fake site, exposing their login credentials or worse. PayPal and bank phishing schemes often prompt the user to “confirm” their bank account information or credit card information “for security purposes”.

Other XSS exploits may trick users into thinking their computer has been infected by a virus, prompting them to download “free software” to clean their system – meanwhile the software the panicking user is downloading actually is the virus. And even though you might never fall for something so blatantly obviously, lots and lots of people do, every day.

Often attackers will inject JavaScript, VBScript, ActiveX, HTML, or Flash into a vulnerable application to fool a user in order to gather data from them. Everything from account hijacking, changing user settings, cookie theft/poisoning, or false advertising is possible. New malicious uses are being found every day for XSS attacks. XSS exploits can even be used to facilitate “Denial Of Service” attacks (or DoS attacks), and potential “auto-attacking” of hosts if a user simply reads a post on an infected message board.

While XSS attacks themselves cannot compromise files on your server, XSS holes can allow Javascript insertion, which may allow for limited execution. If an attacker were to exploit a browser flaw (browser hole) it could then be possible to execute commands on the client’s side. If command execution were possible it would only be possible on the client side. In simple terms XSS holes can be used to help exploit other holes that may exist in your browser or server.

User-Submitted Data and Your Database

Trusting user variables without cleaning or validating them opens you up to a whole host of problems if your application is powered by a database.

For example, the following SQL command is used to validate user login requests:

[sourcecode language="sql"]$sql_query = “select * from users where user=’$user’ and password=’$pass’”[/sourcecode]

If the user-submitted data is not properly validated, an attacker can exploit this query and pass the login screen by simply submitting specially crafted variables.

For example, attacker can submit the following data as a $user variable: admin’ or ’1′=’1 . When this $user variable is glued together with the query, it will look as followed:

[sourcecode language="sql"]$sql_query = “select * from users where user=’admin’ or ’1′=’1′ and password=’$pass’”[/sourcecode]

Now, the attacker can safely pass the login screen because or ’1′=’1′ causes the query to always return a “true” value while ignoring the password value.

Using similar techniques, an attacker can retrieve database records, pass login screens, and change database contents, for example by creating new administrative users. Using similar techniques, a malicious attack will be able to execute arbitrary shell commands, read or write arbitrary commands, and more.

It is our responsibility to protect our users (and the trust they put in us, deserved or not), and XSS vulnerabilities open the doors to all manner of mischief. At their most benign, they can result in a site defacement. At their worst, they compromise the very safety and livelihood of the people that fall for them – not to mention the impact on your company’s reputation.

If you think your site is too small for hackers to bother with, think again. There are plenty of script kiddies out there that will happily run their exploit toolkit scripts and crawl page after page, testing for exploits. They are able to find common exploits in sites they have never physically even visited through this method. And once they find a vulnerability, word spreads fast.

It should also serve as a reminder to us as a internet users to be a little less trusting. It seems obvious, but we are so trained to respond to visual cues and prompts for activities we do every day – logging into a website, checking out with PayPal, etc – that we can sometimes become careless. We sometimes trust the big guys a little too much, assuming that because they’re that big, they’ve got to have their shit together. Even when we know better, our interactions online have become somewhat mechanical.

While browsers are getting better at helping us realize if we’re entering data into a site that is suspect, ultimately the responsibility falls back upon us to pay attention to what we’re doing and to whom we give our valuable information.

So as a developer, what can you do to protect your software and sites from XSS attacks?

Here are a few good places to start.

Stuff to Read:

• Learn more about XSS and how it works on the Cross-Site Scripting FAQ on cgisecurity.com.
• Learn more about Backdooring a Webserver using MySQL, which details how a user could read/write files to your server and execute commands using mySQL.
• Check out the OWASP (Open Web Application Security) website and stay up to date

Stuff to Do:

Always clean and validate ANY data you receive from the user

Use the appropriate escaping for the programming languages and databases you use

Educate yourself on as many examples of XSS (both theoretical and in-the-wild) as you can, so you know what to look for. Wikipedia details a few common methods worth checking out on their Cross-Site Scripting page. The only real defense you have against attacks is to keep yourself informed and current on what the bad guys are up to. This isn’t one of those things that you can read about once and rest on your laurels. You must be vigilant and aggressive about staying on top of what’s going on. It’s your job.

Bookmark and test your scripts against the code samples on the XSS Cheatsheet on ha.ckers.org. This cheatsheet is specifically geared towards exploits that can potentially get past standard filtering that developers might do on their data, such as strip_tags. Also check out the MySQL Injection Cheat Sheet and the SQL Injection Cheat Sheet.

Periodically check your webserver access logs and error logs and look for anything that looks like someone might be trying to find a backdoor. Look for people trying to pass data that doesn’t belong, sending variables that are common configuration file names, and so on.

Turn OFF error reporting displayed to the browser on production environments, and instead log errors to a file. Error messages can expose information about your file structure and your database structure.

Pay attention! Keep your ear to the ground on sites like ha.ckers.org and other (mostly) whitehat exploit blogs and communities. This is your livelihood. Do your job.

If you’re using open source software, make sure you keep up to date with new releases. Many popular open source projects (such as WordPress, phpNuke, phpBB, etc) are frequent targets for malicious scripting. Be sure to hide references to your software version numbers from the public, since certain versions may have exploits that are well known, and attackers will know exactly how to target your site if they know what version you’re running.

Shell out the $39 for the 300 page e-Book Detecting Malice, written by Robert Hansen (aka RSnake, on Twitter at @RSnake) and actually read it. I can’t believe I’m actually endorsing a freaking e-Book, but its really that good. I don’t know Robert personally, I’m not endorsing it as a favor or because I like him as a person. For all I know he eats puppies for breakfast. But his book is fantastic.

And finally, Test test test test test!

There’s even an interesting Twitter account that highlights high-profile XSS exploits – it’s low-volume, but it’s surprising how many turn up that never make the news.

Incidentally, something I discovered while having a little fun on Apple.Com’s site – if you do a Google Images search for ‘hot tar enema”, only four images come up, and one of them is a photo of Rush Limbaugh. I’m not even making that up. Also funny, since I tweeted about that this afternoon, my tweet is now the number one Google search result for “hot tar enema”. Aren’t you jealous?

Also check out:

If you have to send HTML email, there are certainly lots of good libraries available for download to make your life easier. I opted to use PHPMailer by Worx International, since it’s one I have used in the past, and getting up and running with it takes about 3 minutes, tops.

What makes it particularly nice is that you don’t have to do anything special to format your HTML to be ready for mailing. If your workflow is anything like mine, you design an HTML email and code it into HTML, and then post it somewhere for the client to review. Since this HTML file already exists on your server, using PHPMailer is a breeze, since all you have to do to start sending HTML email is point the program to the HTML file you already created.

After you upload the PHPMailer library, all it takes is a few lines of code, and you’re done:

[sourcecode lang=php]require_once $_SERVER['DOCUMENT_ROOT'].’/phpMailer/class.phpmailer.php’;

$mail = new PHPMailer(); // defaults to using php “mail()”
$body = $mail->getFile(‘../invites/email.html’);
$mail->From = $from_email;
$mail->FromName = $from_name;

$mail->Subject = “Test email subject”;
$mail->AltBody = “To view the message, please use an HTML compatible email viewer!”;
$mail->MsgHTML($body);

$mail->AddAddress($to_email, $to_name);

if(!$mail->Send()) {
echo “

Also check out:

The code below should correctly autolink all of the autolinkables in your PHP script:

• links @username to the user’s Twitter profile page
• links regular links to wherever they should link to
• links hashtags to a Twitter search on that hashtag

[sourcecode language='php']function twitterify($ret) {
$ret = preg_replace(“#(^|[\n ])([\w]+?://[\w]+[^ \"\n\r\t< ]*)#”, “\\1\\2“, $ret);
$ret = preg_replace(“#(^|[\n ])((www|ftp)\.[^ \"\t\n\r< ]*)#”, “\\1\\2“, $ret);
$ret = preg_replace(“/@(\w+)/”, “@\\1“, $ret);
$ret = preg_replace(“/#(\w+)/”, “#\\1“, $ret);
return $ret;
}[/sourcecode]

Someday I’ll try to find the time to write a regex primer/tutorial. Regex is another one of the things, like SVN, that seems scary and incomprehensible to many people, but eventually it clicks and makes sense. In the meantime, if you’re interested in learning more about using Regex in PHP, check out the following resources:

More Regexy Goodness:

• Using Regular Expressions in PHP via regular-expressions.info
• Introduction to PHP Regex via phpro.org
• Using Regular Expressions in PHP via webcheatsheets.com
• 15 PHP Regular Expressions for Web Developers via catswhocode.com

Make sure you leave yourself some time to actually try out some examples, and dissect the examples they give so that you really grok what’s happening. Once it clicks, it seems so simple, you won’t believe you ever let it beat you up and take your lunch money.

Image by xkcd, via their awesome web store. The comic rocks my geeky face off. I buy my clothes there. So should you.

PS – still really, really hate posting code in WordPress. Even in HTML mode, it keeps converting my fscking special characters, which then get double/triple/etc converted.

Also check out:

Right then. Moving on.

OAuth is an open protocol to allow secure API authorization in a simple and standard method from desktop and web applications. In layman’s terms, it is a system by which you can allow a user to authenticate with an OAuth-enabled service without providing you with their credentials to that service.

In my Twitter anti-social media douchebag service, DoucheNuker.Com, we use Twitter’s OAuth to validate the user and make Twitter API requests on their behalf, specifically sending a DM to the douchebag they are nuking, another DM to @spam to report them to Twitter as a spammer, and then a block request to block the spammer’s account from being able to follow them in the future.

Why OAuth?

Using OAuth allows you to write applications that access the Twitter API but do not require your users to give you their Twitter username and password. This is important for a variety of reasons:

• If the user changes their Twitter login, they do not have to update that information with you for your application to continue working for them
• Using OAuth puts the user in control – if they ever wish to stop using your application, they can disable it through Twitter instead of trusting your application to stop using their login information. Once they disable it through Twitter, any requests by your application will require them to manually approve the connection again.
• Increased sense of trust, since the user doesn’t have to worry about your application stealing their Twitter credentials and using it for nefarious purposes. I personally wouldn’t trust any web-based application that asks for my Twitter username and password, and given Twitter’s recent history of bad press regarding their security, more and more users are following that lead.

Definitions

Before I show you how to use Abraham’s shmancy library to connect to Twitter’s OAuth, you should understand the basics of how OAuth works and what it’s doing. And before we get too caught up in that, it’s important that we establish some definitions that you’ll see if you do any additional research into OAuth:

User: The users of your application.
Consumer: Your application, which you have registered with Twitter
Service Provider: The third-party service the consumer (your application) is authenticating against – in this case, Twitter.

These terms are used in much of the OAuth documentation, so they’re worth remembering.

So now that you know the lingo, how does OAuth actually work? For a detailed technical view of what gets passed back and forth, check out the core spec documentation on OAuth. Included in that documentation is the detailed chart below.

As you can see, the documentation frequently uses the terms defined above.

If that flow diagram seems a little overwhelming, don’t sweat it. I have a simplified version just for you (featuring a stoner Twitter user and a Twitter bird with a Thyroid problem), specifically with respect to the bits you need to know to set up your first Twitter application with OAuth. The other things OAuth does are important, but this is the stuff that directly impacts you, and that you need to grok to get started with your app.

I was absurdly and inexplicably tempted to randomly throw a Boba Fett icon into that diagram, but was afraid it might confuse people. That said, I have poor impulse control, so here’s a random Boba Fett icon, so I can sleep tonight. As my friend Jason Ramboz says, “Step 4, Boba Fett freezes the key in carbonite for transport.”

Moving on.

Now that you’ve got a good idea of how the basics of OAuth work, you’re ready to get started with Abraham’s great Twitter OAuth library. He does provide an example script in the downloadable code, but it might be confusing for people just starting out.

Getting Started – Registering Your Application with Twitter

Before you even start mucking around in any code, you have to register your new application with Twitter. You’ll need a name and url for your application in order to register it, and you’ll need to define a callback url. The callback url is the full url of the page Twitter should send the user to after it’s done authenticating. This file can be named anything you want, but make sure the one you create on your server matches the one you register with Twitter. All of these details can be changed later if you change your mind or need to update something.

Once you’ve registered your application, Twitter will issue you a Consumer Key and a Consumer Secret for your new app. You’ll need these to get your sample code from the Twitter OAuth library working. As you can probably tell by the name, your Consumer Secret should remain private and you should never give it out to anyone. It’s used in your code so that Twitter can identify your application when you’re making API calls.

By forcing you to send your consumer key and secret with your API calls, Twitter is able to determine which application is sending the API calls, and can verify that the Twitter user you are attempting to send API requests on behalf of has actually authorized your application to access their account. If the user decides they no longer want to allow your application, they can edit their allowed application preferences and your application will no longer be able to make API calls on their behalf.

You can access a list of all of the applications you have registered with Twitter – and links to edit their details or view the consumer key and consumer secret – by going to your oauth clients page on Twitter.

The Twitter OAuth PHP Library Code

You’ve got your consumer keys from Twitter, so now you’re ready to download Abraham’s Twitter OAuth library code. You can pull the code from http://github.com/abraham/twitteroauth. As I mentioned, he does provide an example script, but there’s not a lot of explanation given to it, so some people might be a little confused by it if its their first foray into Twitter applications with OAuth. We’re going to whip up something a little more straightforward and simple, so you can easily modify it to suit your needs.

Unpack/unzip the archive you downloaded from github. You’ll see the two main files, OAuth.php and twitterOAuth.php are in the top level directory, and there is a directory called ‘example’, that has the included example script.

For our example, we’re going to put the two OAuth files into a directory called ‘twitterOAuth’, which is a sub-directory of where the index.php and callback.php files live. As you may have guessed, the callback.php file is the one we’ve registered with Twitter as being our callback url. We’ll keep common configuration options such as the consumer key and consumer secret, and database credentials in a config.php file.

[source lang='php']/* config.php */

/* Consumer key from twitter */
$consumer_key = ‘xxhjgxhjxhhjgxjhjxgjyx768678xx’;

/* Consumer Secret from twitter */
$consumer_secret = ‘jhgjdfgfgjhj76jgjgjhxxxjhxxx’;
[/source]

Now we create the index.php file, which will be used to generate the authentication link, inviting users to authorize and login using Twitter.

[source lang='php']/* index.php */

session_start();

/* Destroy the session if the user is logging out */
if ((isset($_GET['logout'])) && ($_GET['logout']==’true’)) {
session_destroy();
session_unset();
}

/* Include the config file */
require_once(‘config.php’);

/* include the twitter OAuth library files */
require_once(‘twitterOAuth/twitterOAuth.php’);
require_once(‘twitterOAuth/OAuth.php’);

/*
Create a new TwitterOAuth object, and then
get a request token. The request token will be used
to build the link the user will use to authorize the
application.

You should probably use a try/catch here to handle errors gracefully
*/
$to = new TwitterOAuth($consumer_key, $consumer_secret);
$tok = $to->getRequestToken();

$request_link = $to->getAuthorizeURL($tok);

/*
Save tokens for later – we need these on the callback page to ask for the
access tokens
*/
$_SESSION['oauth_request_token'] = $token = $tok['oauth_token'];
$_SESSION['oauth_request_token_secret'] = $tok['oauth_token_secret'];

echo ‘

login using twitter | ‘;
echo ‘Logout

‘;
[/source]

The callback.php file is the script that Twitter sends the user back to after authenticating. Here you’ll probably want to set some cookies, store some user data in the database, and start letting the user do whatever it is your application does.

[source lang='php']/* callback.php */

session_start();

/* Include the config file */
require_once(‘config.php’);

/* include the twitter OAuth library files */
require_once(‘twitterOAuth/twitterOAuth.php’);
require_once(‘twitterOAuth/OAuth.php’);

/* check for an auth access token. If there’s no auth token set, go ahead and fetch one from Twitter,
* using the API call. */
if ((!isset($_SESSION['oauth_access_token'])) || ($_SESSION['oauth_access_token'])==”) {

$to = new TwitterOAuth($consumer_key, $consumer_secret, $_SESSION['oauth_request_token'], $_SESSION['oauth_request_token_secret']);
$tok = $to->getAccessToken();

/* Save tokens for later – might be wise to
* store the oauth_token and secret in a database, and
* only store the oauth_token in a cookie or session for security purposes */
$_SESSION['oauth_access_token'] = $token = $tok['oauth_token'];
$_SESSION['oauth_access_token_secret'] = $tok['oauth_token_secret'];

}

/* Connect to the Twitter API */
$to = new TwitterOAuth($consumer_key, $consumer_secret, $_SESSION['oauth_access_token'], $_SESSION['oauth_access_token_secret']);
$content = $to->OAuthRequest(‘https://twitter.com/account/verify_credentials.xml’, array(), ‘GET’);
$user = simplexml_load_string($content);

if ($user->screen_name!=”) {
echo ‘

SimpleXML. Using SimpleXML, we can call up any node values within the XML using $user->field_name, as you can see above.

I’ve included a print_r($user) so that you can see the full details of the array being returned, but you’ll obviously want to comment that out in your live code.

The output array will contain the following fields:

[source lang='html']SimpleXMLElement Object
(
[id] => 14246782
[name] => snipe
[screen_name] => snipeyhead
[location] => New York
[description] => Codemonkey, designer, author, speaker, blogger, swordfighter, Warcrafter, sarcasticgeek, scuba diver, blacksmith, crimefighter, Mentat, MBTI: ENTP, Totally NSFW
[profile_image_url] => http://s3.amazonaws.com/twitter_production/profile_images/303658881/Photo_4-rcrop2_normal.jpg
[url] => http://www.snipe.net
[protected] => false
[followers_count] => 4224
[profile_background_color] => 340100
[profile_text_color] => 3C3940
[profile_link_color] => 6C2125
[profile_sidebar_fill_color] => AEA797
[profile_sidebar_border_color] => 943A39
[friends_count] => 3756
[created_at] => Fri Mar 28 20:37:35 +0000 2008
[favourites_count] => 314
[utc_offset] => 12600
[time_zone] => Tehran
[profile_background_image_url] => http://s3.amazonaws.com/twitter_production/profile_background_images/22127710/twitterback2.jpg
[profile_background_tile] => false
[statuses_count] => 20570
[notifications] => false
[verified] => false
[following] => false
[status] => SimpleXMLElement Object
(
[created_at] => Mon Jul 27 01:50:36 +0000 2009
[id] => 2862508774
[text] => @elazar In case a name gets blocked/banned – when its reinstated (by someone claiming it, not spamming), it has a new ID#
[source] => Tweetie
[truncated] => false
[in_reply_to_status_id] => 2860170987
[in_reply_to_user_id] => 9105122
[favorited] => false
[in_reply_to_screen_name] => elazar
)

)[/source]

We’re not actually doing anything magical here yet, since that information is all available publicly via a user’s RSS feed, but the key line of code you want to look at in callback.php is this one:

[source lang='php']$content = $to->OAuthRequest(‘https://twitter.com/account/verify_credentials.xml’, array(), ‘GET’);[/source]

The OAuthRequest function is what actually sends the requests to the API, so you’ll be using this a lot. In the example above, all we were doing was getting the access tokens, but you’ll use OAuthRequest for just about everything else, too. For example, to send a Direct Message in Twitter, you’d use:

[source lang='php']
$params = array(‘user’ => ‘username’, ‘text’ => ‘this is a test message’);
$do_dm = simplexml_load_string($to->OAuthRequest(‘http://twitter.com/direct_messages/new.xml’, $params, ‘POST’));[/source]

To block a user, you’d do:

[source lang='php']$doblock = simplexml_load_string($to->OAuthRequest(‘http://twitter.com/blocks/create/username.xml’, array(), ‘POST’));[/source]

To send a status update:
[source lang='php']$content = simplexml_load_string($to->OAuthRequest(‘https://twitter.com/statuses/update.xml’, array(‘status’ => ‘Test OAuth update. #testoauth’), ‘POST’));[/source]

Important! Storing user IDs

Whenever you’re storing Twitter IDs in a database, be sure to store the Twitter ID number in addition to (or instead of) the Twitter username. While it may seem obvious to use a numeric value over a mixed alphanumeric, Twitter doesn’t expose user’s ID numbers without a little digging, so it might be easy to forget that they exist.

There are two main reasons why using the numeric ID is critical:

• Users can change their Twitter usernames. If they did this, your entire database could potentially be screwed up, since username key you’re looking for won’t match any longer.
• If an account has been suspended due to spam or imposters, it can potentially be available for registration again after a grace period. If a spammer had a username before, and then a legitimate user reclaimed it, your records could potentially have old data from the previous user’s account.

The second point above became crystal clear while working on DoucheNuker.Com. If a user account was suspended due to spamming, and then a legitimate user took it over, that new, legitimate user could potentially be considered a spammer in our database if we didn’t store (and query against) the ID number, too. When a username is reissued or reclaimed, it gets a new user ID number, so as long as you store and use the Twitter user’s ID number, your database can remain agnostic to name changes and reissues.

You’ll note in the Twitter REST API documentation that almost all API requests allow the option of using the username or the user ID, and some actually require the user ID and cannot be used with just a username.

Important! Error Messages and Throttling

You do not want to authenticate against Twitter every single time you load the page, but will instead want to store the request tokens in a database or session so that you don’t keep hammering Twitter’s API each time the page loads.

Remember that the although the Request Token you used to generate the authorization link will change often, a user’s Access Token and Access Secret Token do not, so you can safely store those in a database and use those instead of re-validating every time.

As of right now, Twitter is throttling validation requests to 15 per Twitter account per hour. This was implemented to improve Twitter’s security and make it harder for bad guys to brute force their way into someone else’s Twitter account. There is discussion about rolling this change back, or only throttling to 15 failed attempts per hour, but as of this moment, if you attempt to authenticate more than 15 times in an hour, you’ll get a message that says “Too many requests in this time period. Try again later.” There is no way around this message for now, so plan your application accordingly.

This limit is entirely separate from the Twitter Rate Limit that throttles the number of times you can hit the API. Whitelisting your account and IP address with Twitter will NOT circumvent this rate limit, so make sure you design your app in a smart way that will not attempt to authenticate more than absolutely necessary.

The default rate limit for calls to the REST API is 150 requests per hour. The REST API does account- and IP-based rate limiting. Authenticated API calls are charged to the authenticating user’s limit while unauthenticated API calls are deducted from the calling IP address’ allotment.

You’ll notice in all of API requests, we’re using SimpleXML to capture the value of the XML that’s returned. We need to do this in order to make sure we’re capturing any error messages that Twitter returns to us. Without error messages, when stuff doesn’t work as expected, we’re flying completely blind. Always make sure to plan your application in a way that handles errors intelligently. Let’s take a look at the API call to send a Direct Message again:

[source lang='php']$params = array(‘user’ => ‘username’, ‘text’ => ‘this is a test message’);
$do_dm = simplexml_load_string($to->OAuthRequest(‘http://twitter.com/direct_messages/new.xml’, $params, ‘POST’));

/* Check for an error response from Twitter */
if ($do_dm->error!=”) {
echo ‘

ERROR: ‘.$do_dm->error.’

‘;
}[/source]

Now we’re capturing the error returned from Twitter, and can handle this appropriately with our users. The error might be indicating that the user cannot send a Direct Message to someone they’re not following. Or there might be something else amiss – so you’ll want to make provisions in your script to help the user understand why something might not be working.

And that’s honestly all there is to it. Now that you’ve got the OAuthRequest function sussed, you just need to check with the Twitter API Wiki to determine the correct urls and parameters to send, based on what you’re trying to do.

I have to say, having worked with a LOT of APIs, including Facebook, Amazon, and at least a half-dozen others, Twitter’s API is actually the most well-documented and simplest to use. Surprising, really, since Facebook and Amazon have actual business models, so you’d think they’d invest just an iota of time into documenting their shit. I’ve gone into long tirades here on my blog about how miserably awful the Facebook API documentation is, and Amazon’s API is probably 10x worse. Twitter’s API is, overall, pretty accurate and up to date. If its your first foray into writing an application with an API, I think Twitter is actually a good place to start – before you graduate to Facebook and wish you were dead.

Recap – Important Links

• Register your application with Twitter
• List of all of your registered apps on Twitter
• Twitter API Documentation
• Twitter API Rate Limiting Documentation
• Download & Docs for Abraham’s OAuth PHP library
• OAuth official website

And that’s all there is to it. Please use your new powers for good and not evil. No annoying games, no “increase your followers” services, etc. If you have any questions, leave ‘em in the comments.

Also check out:

Something like this would do the trick:

[source='php']$template_bundle_id = 123456789;
$tokens = array();

try {
$facebook->api_client->feed_publishUserAction($template_bundle_id, json_encode($tokens), $array_of_friends_id, ”);
} catch(Exception $e) {
// error trapping goes here
}[/source]

The code above would insert a one-line story into the user’s profile wall (as seen below), and into their friends’ /home.php newsfeed.

While the profile item is nice, the truly viral aspects come from the friends’ newsfeed items. Most people don’t spend a lot of time on profile pages – they rely on their /home.php page for the overview of what their friends are doing by way of posted items, status updates and application messaging.

With the latest change in the Facebook API, suddenly feeds stopped appearing in the the newly redesigned friends’ newsfeed, now referred to as the “stream”. They were still being pushed to the user’s profile, but as we already mentioned, the usefulness of putting feed items there is somewhat limited.

With these new updates, publishUserAction no longer publishes anything to the users’ friends’ stream, and now only publishes one-line stories to the user’s own profile wall.

Enter Feed Forms and Facebook.showFeedDialog

Using showFeedDialog, the user is prompted whether or not they want to publish the action to their stream. While this does mean changing some application code, switching oer to the new feed system is actually very easy for the most part. (I cannot speak for how one would do it in a Flash-based application, as I haven’t had to tackle that yet.)

Assuming you’ve already created your feed template bundle in the feed template console, you’d do something like this:

[source='php']$feed_template = “{‘filmname’:'”.$film_name.”‘, ‘images’:[{'src':'".$film_cover."', 'href':'http://apps.facebook.com/snagfilms'}]}”;

echo ‘‘;[/source]

As long as you have a short-story version in your feed template, that’s pretty much all you have to do. If your previous feed templates only used one-line stories, you WILL have to create new feed template bundles that use short stories, as one-line stories are NEVER published to the stream. Otherwise, just stick that line of PHP/JavaScript into your form handler (or whatever script completes the action the user is initiating) in your application and you’re back in business.

To further complicate things, Facebook just announced the beta launch of their Open Stream API using Stream.publish, which looks like it might offer a simpler-but-different method by which you can publish feeds. Fortunately for this article – and for the time being, my sanity – the Open Stream API is in beta, and only application developers can publish to their stream from their apps. As more information is available about Stream.publish, I’ll try to keep you updated.

Also check out:

For me, the impetus to switch to a caching method for one of my database driven sites was sparked by Mosso, since they bill by cpu cycle, and I have one site that is, well, humongous (60k+ pages), and it happens to the highest traffic site on the account. While the database queries were all very efficient, and each page had, on average, no more than 6 queries, performance and cpu cycles would both be helped quite a lot by implementing a cache. This caching solution was a temporary fix, while we switched to a new CMS that was already using a robust caching system. It’s quick, it’s dirty, but it got the job done for the interim.

We’ll walk through how to execute a simple PHP cache, and then I’m going to explain how doing so without a little forethought will screw you right in the ear. Note that this is called a Quick and Dirty solution for a reason. There are more complex, more efficient methods available, but this covers some basics.

Using output buffering, caching pages is incredibly easy. Simply put, output buffering allows you to control when output is sent from the script. This is particularly handy if you’re using cookies or sessions or some other process that sends headers to the browser before the page loads (as anyone who has gotten those pesky “headers already sent” errors can tell you.)

Please note that this article assumes your cache files will be created in a directory called ‘cache’ – and that this cache directory must be writable by the webserver.

Please also note: the syntax highlighter was made of fail for this article and was double, sometimes triple converting HTML entities. I have fixed it a dozen times, and then every time I edit the post, I have to fix it all over again. So if you notice any funky characters that don’t look like they belong in the script snippets, they probably don’t. Let me know and I’ll fix them, yet *again*.

The basic stuff

In all its 6-lines of glory, this is actual, working caching code.

[source=php]// TOP of your script
ob_start(); // start the output buffer
$cachefile =”cache/cachefile.html”;
// Your normal PHP script and HTML content here
// BOTTOM of your script
$fp = fopen($cachefile, ‘w’); // open the cache file for writing
fwrite($fp, ob_get_contents()); // save the contents of output buffer to the file
fclose($fp); // close the file
ob_end_flush(); // Send the output to the browser[/source]

There are, of course, two major flaws with just using the script above. First, we’re always writing to cachefile.html file, which would only be useful to you if your website was only one page. And second, notice that the script writes to the cache, but never actually retrieves the cache file – it’s still running through the whole script every time. But, this is just the beginning. That’s all there is to the actual caching part – the rest of this article will deal with the when/where of caching, but the how is that right there.

Which brings us to the next step… adding the ability to check whether or not a cache file exists, and use that instead of running through the normal script. We’re going to keep using the one-page website model for now, but I’ll get into creating cache files for different pages later.

Checking for a cache file

Creating the cache file from database-driven content is easy, as we’ve seen – but it’s only useful if we actually check if a cache file exists and serve that instead of live database output. Using he modification below, we are checking to see if a cache file already exists and if it does, include it and exit instead of running through the normal PHP script.

[source=php]// TOP of your script
ob_start(); // start the output buffer
$cachefile = ‘cache/cachefile.html’;
if (file_exists($cachefile)) {
// the page has been cached from an earlier request
include($cachefile); // include the cache file
exit; // exit the script, so that the rest isn’t executed
} [/source]

This is marginally more useful, since it actually prevents the script from executing if a cache file exists, however the way this is currently written, it will include that file for an indefinite time, never actually executing your full script again. Normally, in a cache situation, we want the ability to “expire” content after a certain time, so an updated version will be displayed and cached. You could automatically force a new page cache file to be generated by setting a cron job to automatically delete your cache files every hour/day/week/whatever – or you could handle this on the script level.

Setting cache urls

In our examples, we’ve been using cache/cachefile.html as the filename for the cache file that is generated. As I mentioned, this is great if your site is only one page, but otherwise every page this script is run on will create the same cache file, so you’ll end up serving the same cached file as content for every page on your site. Not awesome.

The easiest way to create individual cache files for each specific page is to do something like this:

[source=php]$cachefile = basename($_SERVER['SCRIPT_URI']);[/source]

This takes the unique url of the page requested and and uses that as the cached filename.

But, there’s a gotcha. If your site uses pages that pass GET requests, such as a search page, etc – the SCRIPT_URI won’t see that as part of the url, so once someone does a search, all subsequent search requests will serve that same cached file unless you make the file name unique to each GET request.

In other words, if your search is located at yoursite.com/search.php, and when someone performs a search, the url looks something like yoursite.com/search.php?q=foo, PHP sees that url as search.php, regardless of the query string. So basically, it will break your search, big time.

NOTE: It may not be worth caching every GET request if your site doesn’t get a lot of traffic to files that use this. Or if disk space is a concern. Since there are a potentially unlimited number of GET strings that could be passed to your script (even bogus ones that don’t return valid results on your site), you may want to evaluate whether or not caching search pages is appropriate. In my case, it was – but it may not be for everyone. At the very least, if you opt to do this, make sure you’ve got some sanity checking in there so some asshole with a grudge can’t just sit there creating new, bogus query strings to eat up your disk space.

If you decide to cache query string data, you could do something like this:

[source=php]$cachefile = basename($_SERVER['SCRIPT_URI']);
if ($_SERVER['QUERY_STRING']!=”) {
$cachefile .= ‘_’.base64_encode($_SERVER['QUERY_STRING']);
}[/source]

This basically just grabs the file name, checks to see if there is any GET data passed and if there it, it generates a url-safe base64-encoded sting that you can use as your cache file name.

Setting an expiration

You have three basic options for expiring your cache:

1. Set up a cron job to automatically delete all of your cache files at specified intervals
2. Check the data source file for modification, and expire it if the source file is newer than the cache file
3. Check the timestamp of the cache file and delete+regenerate if it is older than x

Cron Job: Setting up a cron job to delete your entire cache at specific intervals is arguably the easiest solution, but not really the most efficient, especially with very large websites. Rather than just deleting the page that’s been determined to be expired, you’re deleting (and then subsequently regenerating) a large number of files in one shot.

Data Source: Checking the data source file for modification is potentially the smartest way to handle caching, since it means the cache would never be expired if the data didn’t change. That certainly makes sense to do, since a page that hasn’t been updated doesn’t need to be regenerated, so you’re really getting the most bang for your caching buck.

The problem arises when you’re caching pages that are dynamically generated based on database records. The actual script that generates the data may not have been changed for quite some time, but the data records you’re fetching from the database may have been changed, so just checking the cache file date against the date the script was last modified will not give you what you need.

A workaround there would be that you could do a quick db query at the top of every page to find out when the record was last modified and compare that to the modification time on the cache file, but that means that every page, even your cached pages, will be performing a database hit on every page load. This may be perfectly acceptable to you, but it’s something to consider. Perhaps a better way of handling this would be to modify the content management system by which you publish content, so that the cache file is only deleted when you publish edits. This method would be the most thorough and efficient way, since your cache file would only be updated when you update something, and would be left to be served statically unless the data has changed. Although that’s outside the scope of this quick and dirty article, extending the code below to accommodate that wouldn’t take much work.

Cache Timestamp: We’re going to address the third option, since it’s the most commonly used and would serve as the foundation for the second option anyway.

[source=php]// TOP of your script
$cachefile = basename($_SERVER['SCRIPT_URI']);
$cachetime = 120 * 60; // 2 hours
// Serve from the cache if it is younger than $cachetime
if (file_exists($cachefile) && (time() – $cachetime < filemtime($cachefile))) {
include($cachefile);
echo "“;
exit;
}
ob_start(); // start the output buffer [/source]

This script gets the file name, sets a cache time, checks to see if the cache file exists, and if it does, it checks if the cache file is younger than the cachetime. If the cache is still valid, it includes the file and exists the script. If not, it will continue on to execute the script and create a new cache file. It also tacks on a comment at the very end of the cache file that tells you when the file was cached. This can be helpful in debugging, and helping you verify that the page you’re seeing is in fact a cached version, not a live version. (You can see this in action if you view the source of this page and look down at the very bottom of the source code.)

The script, the whole script and nothing but the script

Put all together, this is what our caching script looks like:

[source=php]// TOP of your script
$cachefile = ‘cache/’.basename($_SERVER['SCRIPT_URI']);
$cachetime = 120 * 60; // 2 hours
// Serve from the cache if it is younger than $cachetime
if (file_exists($cachefile) && (time() – $cachetime < filemtime($cachefile))) {
include($cachefile);
echo "“;
exit;
}
ob_start(); // start the output buffer
// Your normal PHP script and HTML content here
// BOTTOM of your script
$fp = fopen($cachefile, ‘w’); // open the cache file for writing
fwrite($fp, ob_get_contents()); // save the contents of output buffer to the file
fclose($fp); // close the file
ob_end_flush(); // Send the output to the browser[/source]

Gee… Oh… Cache challenges

I know. Going to hell for that awful joke. Moving on…

Caching is a great way to speed things up on dynamic sites and save on server resources – however if your site has any kind of more advanced features, you need to be selective about where you apply it. The cache is not smart, so you have to be. Ideally, you’ll be building your caching system into the site as you develop the site and the content administration system – but if you end up having to add caching later, you really have to think everything through.

Examples of things that WILL break if you use caching unless you specifically work around them:

User login: “Welcome, user” logged in functionality (the first user who logs in will create the cache, and everyone else logging in will see their name instead of their own!

Voting: If you have any kind of voting functionality built into your pages, new votes will not be captured and old ratings will be displayed

Anything requiring a POST request: Same as above the first person submitting the form will get correct results, but anyone submitting it after them will get the first user’s cached results.

Geo-IP lookup: If you’re displaying geographically relevant information to the user based on their IP address, the same rules apply. The first user hitting your site will create the cache file and everyone else accessing it will see their geographic results instead of their own.

And so on…

That said, all hope is not lost. Depending on the situation and what functionality I’m trying to preserve, I usually handle this one of two ways:

Only serve cached files to users who are NOT logged in. This takes care of a lot of the issues right there – if a user has a profile preferences page, email preferences page, or whatever – all of these will be cached by the first user accessing them. The easy way around this is simply to serve live data to the user if they are logged in, cached pages if they are not. This will reduce the effectiveness of your caching system to some degree, but many users never both logging in, so you’re still getting a significant savings. (If 90% of your site’s content is only available to logged-in users, you may need to rethink your caching system though.)

Use AJAX. This is one of the few situations where AJAX really can be 100% appropriate. Since AJAX requests are asynchronous and are not cached, this is a great solution for your voting script situations. Mind you, you should make sure your solution degrades gracefully for users who have javascript turned off.

Only cache parts of your page instead of the whole thing. With a little more work, you can set up your caching system to only cache parts of your page, and not the entire page. This may reduce the effectiveness of the caching system, but may be necessary depending on your situation.

One final gotcha

You should consider a graceful way of handling database failures as well. Say you have your cache time set for 3 days – a long time by some standards, but not at all unreasonable if you have disk space to spare and your content doesn’t update that often. If your database throws an error when your cache file is being regenerated, that error will continue to be displayed for 3 days, even if the database error has been corrected. You should consider how to handle that gracefully, even if its a cheap and dirty method. For example, you could set up a website monitoring service that notifies you when your content has changed. If your page isn’t loading properly, you’ll be notified by text or email, and that will give you the opportunity to fix the error and manually blow out your cache so it can regenerate.

A note for WordPress users

If you’re using WordPress and are looking for a way to reduce server load and speed your blog up, you’re in luck. WP-Super Cache is an unparalleled caching solution for WordPress that is basically plug-and-play, no coding required.

Caching Libraries

Thanks to the fabulous comments to this article (and I genuinely do mean that), I am reminded to remind you that this method is exactly what it says it is - quick and dirty – and it makes NO attempt to be the best solution to your caching needs. It is as much an exercise in considering where caching is appropriate (and inappropriate) as much as it is anything else.

For more sophisticated (and certainly more elegant) solutions, check out PEAR’s Cache_Lite , xCache (lighthttpd), eAccelerator and Zend_Cache, and read up on APC and memcached.

Also check out:

But… but you said….

It appears a few of my concerns from my previous article were addressed – at least in part. Unlike a month ago, it seems that Facebook has improved their security, so that if someone has their privacy locked down tightly, their name no longer appears on the site. When I tested with my own Facebook login, my picture was the default Facebook user icon and my name was listed as Facebook User. This is a big improvement in my eyes.

My main argument against using Facebook Connect on a site is using it as the only way to login, giving your users the choice of Facebook Connect, or not commenting. On Snipe.Net, we do not require a login of any kind, so this is less of an issue. If a user doesn’t want to use Facebook Connect but wants to comment, they are still free to do so. They can opt to use Facebook Connect if they want comment noted in their Facebook newsfeed. If they don’t, that’s fine too.

Goals

I am a firm believer in having specific goals for implementing new technology – not simply using it because it exists *cough*ajax*cough*. My goals here are simple – to encourage more Facebook users to visit the site. Generally speaking, regular readers of Snipe.Net tend to be of the somewhat geeky persuasion, with the exception of the random person who found the site by way of Google because they were having a specific problem that we’ve addressed here. Geeky people tend to have other geeky people as their friends – so this is an opportunity to share the joy and light that is Snipe.Net with more geeks.

It could be argued that most “real” geeks wouldn’t be caught dead on Facebook – but if that’s the case, no harm no foul. Nothing has been comprimised by adding it, even if nothing has been gained.

Making it happen

Making a website using Facebook Connect from scratch requires a little programming know-how. Making a WordPress blog Facebook Connect-enabled doesn’t, since there is a handy little plugin for it already. It appears the plugin is actually Facebook sanctioned, as the WordPress plugin documentation is available right from the Facebook developers wiki.

All you have to do is insert:

[source='php']< ? php do_action('fbc_display_login_button')  ?>[/source]

into your comments.php file. Couldn’t be easier.

As a simple example, the comments.php snippet would look something like this, noting that the new line of code appears outside the else/if loop that checks if the user is logged in:

[source='php']<? php if ( $user_ID ) : ?>
…
<? php else : ?>
….
<? php endif; ?>
< ? php do_action('fbc_display_login_button')  ?> [/source]

The installation was a breeze – and although I’m still testing things out, all I had to do was add a line of code to the comments file in my WordPress theme. I opted to be a little more creative with it, and stack the “normal” WordPress comment form next to the Facebook Connect prompt, so as not to make the comment form area any longer or more unwieldy than it already is.

D’oh! Something’s borked!

One gotcha – and I don’t know if this is a bug on my end, or a plugin conflict, or what yet – but after activating the Facebook Connect plugin, my edit post functionality in the admin seems to be borked. When I try to edit a specific post, the page stops loading after the:

[source='html']

[/source]

Still looking into this issue, and once I figure out what the cause is, I’ll update this post. As it stands now,  I have to deactivate the plugin in order to edit posts, and then re-activate it. A pain in the ass, and if I don’t find a solution soon, my Facebook Connect experiment is going to go away real quick. I’ll start by disabling some of my admin plugins and see if that helps. More to come.

Update: I disabled the Fluency Admin plugin and everything seems to be working fine. Pity, I like that admin skin. But the improved admin in WordPress 2.7 is certainly usable enough. Problem solved.

Also check out:

Creating your own WordPress theme will allow you to break out of free (or commercial) templates that mean your blog invariably looks just like hundreds (if not thousands) of other blogs out there using the same theme. Plus, once you’ve created a few and feel pretty comfortable, you can potentially take on paid work customizing WordPress templates, create commercial templates to sell to people less brave than you, or offer free templates on your website as a way to draw traffic to your blog.

Getting Started

First things first, if you have something designed already, I strongly urge you to code it out into (X)HTML before even looking at a WordPress theme tutorial. Trying to wrangle style sheets while trying to grok the theme structure might be a bit much, so you’ll be ahead of the game if you’ve already got your design and (X)HTML coding done. (If you’re unsure about how to code a table-free layout, using only CSS, check out our article, Making the Leap to All-CSS Layout.)

One thing to keep in mind as you’re slicing and dicing your design and beginning your layout coding, there are some commonly used CSS element names in most WordPress themes that you may want to use in your code. Chance are, you’re not going to be creating every single template file from scratch, but rather, reusing a sample or tutorial template. Keeping the element names consistent will make this much easier in the long run. The template classes often used are:

1. #wrapper (holds the entire layout except the footer)
2. #header (header part, including top page navigation)
3. #content (container that holds your main page content and sidebar)
4. #left-col (for the posting area, comment section and respond section)
5. #right-col (your sidebar)
6. #footer (footer)

Quick Note on CSS

Notice that we’re using element ids instead of classes in the list above. This is because classes (such as li.foo or .foo) are meant to be reusable. CSS id elements (such as #foo) are only used once in a page, and since these primary elements are never re-used on the page, we use ids instead of classes. Using ids instead of classes will make inheritance much easier and will speed up your development time.

Just as a reminder to CSS-newbies, if you have an element defined with an id (as opposed to a class), its very easy to control large groups of child class elements. For example, something like this:

[source lang='html']

[/source]

To apply a style to all of the list item elements inside of the div with the id of “nav”, you simply use this:
[source lang='css']#nav li {
color: white;
}[/source]

This would turn all of the text in your list items elements white. You can further define specifics by combining ids and classes, like:

[source lang='css']#nav li.widget {
color: green;
}[/source]

That will cause the text in all of your list items with the class of “widget” within the “nav” div to turn green. Easy, right? I bring this up here (and my apologizes to anyone reading this who already understands the relationship between id and class) is because your WordPress HTML/CSS will be much cleaner and easier to work with if you take this approach.

Moving On…

While I could spend the time writing my own tutorial from scratch here, a few other people have already written ones that kick the crap out of anything I could come up with, so I’ll leave the actual nuts and bolts to them. A few tutorials really stand out from the rest. Some of these tutorials are pretty complicated and involved, while others work with only the basics to avoid confusing the reader with too much information.

How to Create a WordPress Theme from Scratch, brought to you by NetTuts, does exactly this, taking a very simple approach that may be particularly helpful with people still perfecting their CSS-fu. At the end of the tutorial, you end up with a plain, but functional, place to start that includes all of the basic functionality your blog will need, without overloading it with design elements that might be confusing.

How to Create a WordPress Theme from Scratch, brought to you by ThemeTation, is an incredibly comprehensive tutorial that starts with actually designing, slicing and coding the PSD file. It might be a little much for someone who already knows how to slice and dice their PSDs into submission, but part 3a starts at actual implementation into WordPress. While this tutorial is a bit long, it gives you a start-to-finish walkthrough that may be very helpful to some.

So You Want to Create WordPress Themes, Huh?, brought to you by WPDesigner, is a bit of a middle ground of the first two tutorials listed. It’s very comprehensive, but it assumes that you’ve already got the layout slicing mostly covered, so the real focus is on WordPress theme structure and functionality.

There is even a three-part video screencast tutorial available on the CSS-Tricks website. Pack a lunch, as the complete series is over two hours long, but Chris does a great job addressing theme creation from start to finish. They also provide the demo theme created in the video as a download, so you can play along.

A Word on Widgets

Plugins are arguably the best part of WordPress. Chances are, if you want your blog to have some special feature or function, someone has already created a plugin that does it. Not all plugins are widget-ready, but the ones that are usually use the same CSS containers, so that they will fit seamessly into your page design. (Not all widgets do this, which can be a pain in the ass, but you should plan for the standard and fix the ones that don’t comply.

The standard WordPress sidebar widget CSS looks something like this:

[source lang='html']

[/source]

Because this is a semi-standard format, you might want to consider coding your HTML in this way, so you have a better chance of widgets fitting into your site’s style right out of the box. It won’t work every time, but it often does, and I wish I had realized that when I first got started.

Next Steps

As usual, I tend to recommend that you actually follow the steps in these tutorials, one by one as you go. I’m a hands-on learner, so nothing makes information stick in my brain better than actually getting into the muck of it.

Personally, when I learned to make WordPress themes, I started with the simplest tutorials listed above, and then opened the default theme that comes with WordPress, copied it, installed my new copy under a new name, and used that as the place to start. Once I felt pretty confident there, I poked around in some of the more complicated themes, to see how they did what they do. Start simple – don’t try to conquer the world of WordPress your first time. The more comfortable you get with the structure and functions, the more great ideas you’ll have on what you can do with your blog.

When you’re ready to get more advanced, check out the WordPress Codex, that explains what each function within WordPress does, usually with detailed examples and documentation. If you start to get fancy with your WordPress queries, specifically with regard to specifying posts from only one (or more) categories, or all posts from one (or more) categories except the one you specify, the Template Tags – Query page in the codex will be your new best friend. The forums are also a great place to get answers to a specific question.

Also check out: