Accessing AWS SimpleDB from PHP

This week, as I built part of my App Server for Distributed Systems Design, I hit another stumbling block. The library that Amazon provides in PHP for accessing SimpleDB requires PHP 5.2. I should have known that I need to use the latest version.

Not only did Amazon’s library not work for me, but it was huge and complicated. I found another library at: Google Code, but as fate would have it, that library didn’t work either. The code was pretty ugly imho, but at least it was straighforward enough for me to understand how accessing SimpleDB worked, which led me to make my own SimpleDB client.

The script will work with any PHP 5, and doesn’t depend on anything that isn’t built in by default. I hope it is helpful to someone else. It would be really easy to add the SimpleDB requests I haven’t implemented yet.

<?php
/**
 * AWS_SimpleDB_Client v0.1 by Andy VanWagoner, distributed under the ISC licence.
 * Provides simple access to Amazon's SimpleDB from PHP 5.
 *
 * Copyright (c) 2009, Andy VanWagoner
 *
 * Permission to use, copy, modify, and/or distribute this software for any
 * purpose with or without fee is hereby granted, provided that the above
 * copyright notice and this permission notice appear in all copies.
 *
 * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
 * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
 * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
 * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
 * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
 * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
 * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 */

class AWS_SimpleDB_Client {

	// AWS SimpleDB API Constants
	private static $service_endpoint	= "sdb.amazonaws.com";
	private static $api_version			= "2007-11-07";
	private static $timestamp_format	= "Y-m-d\TH:i:s.\\\\\Z";
	private static $signature_version	= 1;

	private static $user_agent = "AWS_SimpleDB_Client 0.1 - Andy VanWagoner";

	/**
	* Constructor
	*
	* @param string $access			// your AWS "Access Key ID"
	* @param string $secret			// your AWS "Seceret Access Key"
	*/
	function AWS_SimpleDB_Client($access, $secret) {
		$this->access_key = $access;
		$this->secret_key = $secret;
	}

	/**
	* AWS SimpleDB API - CreateDomain
	* NOTE: This call will take a while (AWS says 10 seconds)
	*
	* @param string $domain			// the domain to create
	*
	* @return array('status'=>array('code'=>, 'message'=>), 'RequestId'=>, 'BoxUsage'=>)
	*/
	function create_domain($domain) {
		$params = array(
			'Action' => 'CreateDomain',
			'DomainName' => $domain
		);

		return $this->post($params);
	}

	/**
	* AWS SimpleDB API - DeleteDomain
	* NOTE: This call will take a while (AWS says 10 seconds)
	*
	* @param string $domain			// the domain to delete
	*
	* @return array('status'=>array('code'=>, 'message'=>), 'RequestId'=>, 'BoxUsage'=>)
	*/
	function delete_domain($domain) {
		$params = array(
			'Action' => 'DeleteDomain',
			'DomainName' => $domain
		);

		return $this->post($params);
	}

	/**
	* AWS SimpleDB API - ListDomains
	*
	* @param string $next = ''		// Optional - Sent as NextToken parameter
	* @param string $max = 100		// Optional - Sent as MaxNumberOfDomains
	*
	* @return array('status'=>array('code'=>, 'message'=>), 'RequestId'=>, 'BoxUsage'=>,
	* 				'DomainName'=>array('...', ...) [, 'NextToken'=>])
	*/
	function list_domains($next = '', $max = 0) {
		$params = array('Action' => 'ListDomains');

		if ($max > 0 && $max post($params);
	}

	/**
	* AWS SimpleDB API - PutAttributes
	*
	* @param string $domain			// The domain the item is in
	* @param string $item			// The name of the item
	* @param array  $attributes		// array(array('Name'=>, 'Value'=> [, 'Replace'=>]), ...)
	*
	* @return array('status'=>array('code'=>, 'message'=>), 'RequestId'=>, 'BoxUsage'=>)
	*/
	function put_attributes($domain, $item, $attributes) {
		$params = array(
			'Action' => 'PutAttributes',
			'DomainName' => $domain,
			'ItemName' => $item
		);

		foreach($attributes as $i => $value) {
			$params["Attribute.$i.Name"] = $value['Name'];
			$params["Attribute.$i.Value"] = $value['Value'];
			if (isset($value['Replace']))
				$params["Attribute.$i.Replace"] = $value['Replace'];
		}

		return $this->post($params);
	}

	/**
	* AWS SimpleDB API - DeleteAttributes
	*
	* @param string $domain			// The domain the item is in
	* @param string $item			// The name of the item
	* @param array  $attributes		// array(array('Name'=>, 'Value'=>), ...)
	*
	* @return array('status'=>array('code'=>, 'message'=>), 'RequestId'=>, 'BoxUsage'=>)
	*/
	function delete_attributes($domain, $item, $attributes) {
		$params = array(
			'Action' => 'DeleteAttributes',
			'DomainName' => $domain,
			'ItemName' => $item
		);

		foreach($attributes as $i => $value) {
			$params["Attribute.$i.Name"] = $value['Name'];
			$params["Attribute.$i.Value"] = $value['Value'];
		}

		return $this->post($params);
	}

	/**
	* AWS SimpleDB API - GetAttributes
	*
	* @param string $domain			// the domain name
	* @param string $item			// the item's name
	* @param string $attribute		// Optional - If specified, only this attribute's values are retrieved.
	*
	* @return array('status'=>array('code'=>, 'message'=>), 'RequestId'=>, 'BoxUsage'=>,
	* 				'Attribute'=>array(array('Name'=>,'Value'=>), ...))
	*/
	function get_attributes($domain, $item, $attribute = '') {
		$params = array(
			'Action' => 'GetAttributes',
			'DomainName' => $domain,
			'ItemName' => $item
		);

		if ($attribute)
			$params['AttributeName'] = $attribute;

		return $this->post($params);
	}

	/**
	* AWS SimpleDB API - Query
	*
	* @param string  $domain		// The domain name
	* @param string  $query			// The query to run on this domain
	* @param string  $next = ''		// OPTIONAL - token supplied on last paged call
	* @param integer $max = 100		// OPTIONAL - max items you want returned 1-250, default = 100
	*
	* @return array('status'=>array('code'=>, 'message'=>), 'RequestId'=>, 'BoxUsage'=>,
	* 				'ItemName'=>array('...', ...))
	*/
	function query($domain, $query, $next = '', $max = 0) {
		$params = array(
			'Action' => 'Query',
			'DomainName' => $domain,
			'QueryExpression' => $query
		);

		if ($max > 250) $max = 250;
		if ($max > 0)
			$params['MaxNumberOfItems'] = $max;
		if ($next)
			$params['NextToken'] = $next;

		return $this->post($params);
	}

	/**
	 * Sign the parameters, following AWS version 1 signing
	 *
	 * @param array $params			// array of all (except for the signiture) params to be passed to amazon
	 *
	 * @return string				// signature string
	 */
	private function sign($params) {
		uksort($params, 'strnatcasecmp');

		$data = '';
		foreach ($params as $key=>$value) {
			$data .= $key . $value;
		}

		return base64_encode (	pack("H*", sha1((str_pad($this->secret_key, 64, chr(0x00)) ^ (str_repeat(chr(0x5c), 64))) .
								pack("H*", sha1((str_pad($this->secret_key, 64, chr(0x00)) ^ (str_repeat(chr(0x36), 64))) .
								$data)))) );
	}

	/**
	 * POST to AWS SimpleDB and then parse the response.
	 *
	 * @param array $params			// all params to pass on the post
	 *
	 * @return array('status'=>array('code'=>, 'message'=>), 'RequestId'=>, 'BoxUsage'=>, ...)
	 */
	private function post($params) {

		// Add all of the common parameters needed by AWS SimpleDB
		$params['AWSAccessKeyId']	= $this->access_key;
		$params['Timestamp'] 		= gmdate(self::$timestamp_format, time());
		$params['Version'] 			= self::$api_version;
		$params['SignatureVersion']	= self::$signature_version;
		$params['Signature'] 		= $this->sign($params);

		// Generate the POST request
		$content = http_build_query($params);

		$post  = 'POST / HTTP/1.0'															. "\r\n";
		$post .= 'Host: ' 			. self::$service_endpoint 								. "\r\n";
		$post .= 'Content-Type: ' 	. 'application/x-www-form-urlencoded; charset=utf-8'	. "\r\n";
		$post .= 'Content-Length: ' . strlen($content)										. "\r\n";
		$post .= 'User-Agent: ' 	. self::$user_agent 									. "\r\n";
		$post .= 																			  "\r\n";
		$post .= $content;

		$socket = @fsockopen(self::$service_endpoint, 80, $errno, $errstr, 10);
  		if ($socket) {
			fwrite($socket, $post);

			$response = stream_get_contents($socket);
			fclose($socket);

			// Parse the response
			return $this->format_result($response);
		}

		// Return a fail result
		return array('status' => array('code' => 404, 'message' => 'Not Found'),
			'Error' => array('Code' => $errno, 'Message' =>
				'Could not connect to ' . $this->$service_endpoint . " ($errstr)"
			)
		);
	}

	/**
	 * Take the XML document returned by AWS SimpleDB, and transform it into a hash
	 *
	 * @param string $result		// the full http response string from SimpleDB
	 */
	private function format_result($result) {
		list($http_headers, $content) = explode("\r\n\r\n", $result, 2);
		$header_lines = explode("\r\n", $http_headers);
		list($protocol, $code, $message) = explode(" ", $header_lines[0], 3);

		// record the http status
		$formatted = array('status' => array('code' => $code, 'message' => $message));

		$xml = simplexml_load_string($content);

		// Look for Errors
		if (isset($xml->Errors)) {
			$formatted['RequestId'] = (string)$xml->RequestId;
			$formatted['Error'] = array();
			foreach($xml->Errors->Error as $error) {
				array_push($formatted['Error'], array(
					'Code' => (string)$error->Code,
					'Message' => (string)$error->Message
				));
			}
			return $formatted;
		}

		// Get the metadata for this request
		$metadata = $xml->ResponseMetadata;
		$formatted['RequestId'] = (string)$metadata->RequestId;
		$formatted['BoxUsage'] = (string)$metadata->BoxUsage;

		// GetAttributes Response
		if (isset($xml->GetAttributesResult)) {
			$formatted['Attribute'] = array();
			foreach($xml->GetAttributesResult->Attribute as $attribute) {
				array_push($formatted['Attribute'], array(
					'Name' => (string)$attribute->Name,
					'Value' => (string)$attribute->Value
				));
			}
		}

		// ListDomains Response
		if (isset($xml->ListDomainsResult)) {
			$formatted['DomainName'] = array();
			foreach($xml->ListDomainsResult->DomainName as $domain) {
				array_push($formatted['DomainName'], (string)$domain);
			}
			if (isset($xml->ListDomainsResult->NextToken)) {
				$formatted['NextToken'] = (string)$xml->ListDomainsResult->NextToken;
			}
		}

		// Query Response
		if (isset($xml->QueryResult)) {
			$formatted['ItemName'] = array();
			foreach($xml->QueryResult->ItemName as $item) {
				array_push($formatted['ItemName'], (string)$item);
			}
			if (isset($xml->QueryResult->NextToken)) {
				$formatted['NextToken'] = (string)$xml->QueryResult->NextToken;
			}
		}

		return $formatted;
	}
}

?>

JSON and POST in PHP

As I’ve been trying to do Lab 2 without having to modify my ami or change my apache configuration, I’ve found some nice helpers.

First, trying to encode and decode JSON in PHP 5.2 is easy… you just use the built in functions json_encode() and json_decode(). However, my Fedora ami is only running PHP 5.03. So, how do use JSON without recompiling my php installation, or downloading 5 billion files? Michal Migurski created a php-json library that is now a part of PEAR, but he still has a copy of his original encoder/decoder at http://mike.teczno.com/JSON/JSON.phps. It’s licenced BSD-style so have at it.

Next, I wanted to sent http requests by POST, including file uploads, again without downloading 5 billion files or messing with my ami. My solution was to actually learn the ‘application/x-www-form-urlencoded’ format and ‘multipart/form-data’ format and send the HTTP request across a socket.

A resource that helped me with the ‘application/x-www-form-urlencoded’ format is on http://www.wellho.net/resources/ex.php4?item=h110/getpost.php. For the ‘multipart/form-data’ format http://chxo.com/be2/20050724_93bf.html was very helpful. One gotcha to remember though is that PHP heredoc strings usually use \n line endings. While this may not cause any problems, to be safe and consistent with HTTP, you should use \r\n line endings.

Putting the two together into one function gave me the following:

function http_post($host, $path, $data_hash, $file = '', $file_param_name = '') {
	$boundary = md5(uniqid());
	if ($file && $file_param_name) {
		$binary = file_get_contents($file['tmp_name']);

		$content_type = "multipart/form-data; boundary=$boundary";

		$items = array();
		foreach (array_keys($data_hash) as $key) {
			array_push($items, "--$boundary\r\nContent-Disposition: form-data; name=\"$key\"\r\n\r\n{$data_hash[$key]}\r\n");
		}
		array_push($items, "--$boundary\r\nContent-Disposition: form-data; name=\"$file_param_name\"; filename=\"{$file['name']}\"\r\n");
		array_push($items, "Content-Type: {$file['type']}\r\nContent-Transfer-Encoding: binary\r\n\r\n$binary\r\n--$boundary--\r\n");
		$data = implode('', $items);
	} else {
		$content_type = 'application/x-www-form-urlencoded; charset=UTF-8';

		$items = array();
		foreach (array_keys($data_hash) as $key) {
			array_push($items, urlencode($key) . '=' . urlencode($data_hash[$key]));
		}
		$data = implode('&', $items);
	}

	$content_length = strlen($data);
	$fp = fsockopen($host, 80);
	fputs($fp, "POST $path HTTP/1.1\r\n");
	fputs($fp, "Host: $host\r\n");
	fputs($fp, "Content-Type: $content_type\r\n");
	fputs($fp, "Content-Length: $content_length\r\n");
	fputs($fp, "Connection: close\r\n\r\n");
	fputs($fp, $data, $content_length);

	$http_response = stream_get_contents($fp);
	fclose($fp);

	list($headers, $body) = explode("\r\n\r\n", $http_response, 2);
	return $body;
}

Note that the $file parameter would be $_FILES['your-form-input-name'], and $file_param_name would be ‘your-form-input-name’. $data_hash, I assume would be obvious. It’s an array with key => value pairs to send. The upload file would not appear in $data_hash.

Distributed System Design

This is my last semester at BYU, and I am taking Distributes System Design. As I go through the preocess of creating (with my classmates) a distributed web application, I will be logging my progress and experiences here.

P.S. – Happy Civil Rights Day! I’m using this holiday to catch up on my homework.

Discontinuance

I hope I am not disappointing too many people, but I don’t feel strongly enough about this blog and I don’t have enough I want to say to blog on a regular basis.

If you are interested in reading things my wife and I write, we have a blog at vanwagoner.wordpress.com.  It will likely be mostly her postings, but when I do write, it will go there first.

Barefoot (Revisited)

As I thought more on the topic of Internet security and privacy, I decided to revisit the image of going barefoot on the net. This time, however, in the form of a poem. Here it is:

Barefoot

I stand barefoot in the dark
On the brink of the unknown
I see not to the left or to the right
But where I stand alone

Voices call beyond my view
And I smell nearby perfume
Promising treasures to be obtained
And pleasures to consume

Also far off another’s call
Warns of glass littering the ground
Thistles and briars to snare my feet
And beasts that devour without sound

There is much I need to do
And knowledge that I hope to gain
If I can just put my foot forward
And risk the misfortune and pain

Do I dare venture forward
Into the darkness I fear
Or shall I wait here forever
For a light that may never appear

Copyright © 2008 Andrew VanWagoner

Welcome to the Great Northwest

Julie, Abby, and I have left Provo for the summer. I have started an internship as a Software Developer with Amazon.com. We are living in downtown Seattle, and are still recovering from culture shock. Hopefully, I will soon get back to writing regularly.

Growing Software

Agile methods have proven time and again to produce better software quicker. Extreme programming, iterative development, and release early release often provide the designers and developers with more and better feedback from customers.

I recently helped develop a custom software package, and I learned very quickly into the project that knowing upfront what the product needed to do was near impossible. Incrementally adding functions as they we discovered they were needed turned out to be fairly quick and easy, since we kept the architecture simple adding only what we knew was necessary, and removing what we found to not work, or not be necessary.

Not only are these approaches better for the end result, but they are certainly more motivating for the developers themselves. At each stage of the process there is a prototype that compiles and runs, whether or not all the functionality is there. This helps avoid propagating early mistakes. Also, they can focus on one aspect of the product at a time, and receive verification from testing, and validation from the customer. In my limited experience the program ends up more efficient and maintainable in this process.

In the now dead waterfall approach to programming, where specifications are known upfront and the entire development is done together, and then all testing is done at the end, growing software one piece at a time deals with the fundamental issue that usually specs won’t be correct or complete at the beginning. The waterfall model also leaves testing for the end, so the code where bugs are found have long since left the developers memory. Finding bugs, usually the hardest part, is greatly simplified when only one module was added since the last successful prototype.

Though strictly speaking programs are deterministic and lifeless, I often see a personality (inadvertently instilled by the programmers) in software. When software is grown piece by piece rather than built from bottom to top, the software behaves better.

@mbience a Cappella

In high school some friends and I made an a cappella group called Ambience. My brother Jacob was our 1st tenor, I sang 2nd tenor, Jacob Morris sang baritone, and Trevor Watts sang bass. We traded parts often, too. We weren’t that good, but we did have a lot of fun. Below are a few of our songs. (The only ones I dare let the public hear.)

• 16 Tons
• In the Still of the Night
• Kiss the Girl

P.S. – If the links aren’t working, or it is too slow, please let me know about it.

Copyright and the Right to Copy

The battle for control over mass-reproducible media is still far from over. The record companies are still not ready to embrace the way music can be distributed in our connected world. Now though, most people are tired of the debate. Back in 2003, Orson Scott Card wrote an interesting article about the competitors in the fight, and I found some points worth talking about.

1.) The difficulty of technologically enforcing copyright

Using encrypted DVDss as one case, it is extremely easy to install DeCSS or similar libraries to circumvent the encryption. It is very annoying that I can’t legally play DVDs at all in Linux. (Unless I get a particular distro OEM.) Aggravating customers is a bad business move.

2.) The money makers are producers, not artists

Most musicians make their big bucks on live concerts. Easily distributed mp3s have only made musicians more popular, and the crowds at their concerts bigger. Changing the copyright law will only hurt the big companies which are necessarily made obsolete. We don’t keep carriages around for fear of putting the drivers out of a job. Big record companies are an artifact.

3.) The technology is getting faster

The technology that initially incited the debate is only going to improve over time. It will not stop because some want to still make money the way they used to. It should be their loss, not everyone else’s.

Final thoughts
I only use software, music, and movies that I got legally. Like myself, most people are willing to pay a reasonable amount for things they like, and make donations to developers and artists who improve our quality of life. When laws outlive their purpose, it’s time for a change.

Why We Are Hooked

As people have slowly discovered the Internet, many have found they are spending more time online than they have available. Along with the overwhelming wealth of knowledge on the Internet, is the ability to connect with other people. This connection, in my opinion, is what hooks us. If you look in gaming, it is the MMORPGs, and other games that allow interaction with other players that keep loyal players for extended periods of time. If you see other successful Internet applications like email, IM, social networks, and blogs, these all appeal to our social nature. The paradox is that often when we are spending too much time being social online, we neglect our face-to-face relationships.