Experiments with Amazon Web Services

In OpenClimateGIS we experimented with Amazon Web Services (AWS) to determine its suitability for future work and proposals. The potential to scale both in storage capacity (http://aws.amazon.com/s3/) and computation/parallelism (http://aws.amazon.com/elasticmapreduce/) were of interest.

The OpenClimateGIS web service was built using the Django web framework running under a standard Ubuntu Linux server. The Apache web server software provided the HTTP interface. Rapid cloud deployment was important for replicability of the software installation. Using Amazon Web Services (AWS) as the cloud infrastructure, the Python libraries fabric and boto were used for application deployment and system administration on the cloud-hosted, remote server. Essentially, this resulted in the execution of single deployment script to start a fresh AWS instance, configure the new machine, and install OpenClimateGIS. The scripting framework made it easy to alter the created machine’s specifications (e.g. available RAM, instance size, number of cores) -- upgrading or downgrading these specifications was simplified greatly in practice.

The experience with AWS was positive. It was easy, if not trivial, to deploy a new server to match site traffic demands. Load balancing or large-batch processing scaling seemed straightforward, though our problem size did not test this.

From a technical perspective, AWS worked well. Cost considerations of using a fee-based service (e.g. AWS) versus a permanent hosting solution will require consideration of potential problem sizes and an institution’s infrastructure capacity. If the cost-benefit were to work out, then there is potential for shared data storage to facilitate cross-institutional usage of consolidated data archives with ready availability for cloud computation and the possibility of cost sharing.

Ben Koziol, Tyler Erickson, Richard B. Rood
March 8, 2013