Archive for the ‘Distributed Computing’ Category

Your API is a feature, give it real resource management

January 14, 2013

So much these days is about distributed resource management. That’s anything that can be created and destroyed in the cloud[0]. Proper management is especially important when the resource’s existence is tied to a real economy, e.g. your user’s credit card[1].

EC2 instance creation without idempotent RunInstance

EC2 instance creation without idempotent RunInstance

Above is a state machine required to ensure that resources created in AWS EC2 are not lost, i.e. do not have to be manually cleaned up. The green arrows represent error free flow. The rest is about error handling or external state changes, e.g. user terminated operation. This is from before EC2 supported idempotent instance creation.

The state machine rewritten to use idempotent instance creation,

EC2 instance creation with idempotent RunInstance

EC2 instance creation with idempotent RunInstance

What’s going on here? Handling failure during resource creation.

The important failure to consider as a client is what happens if you ask your resource provider to create something and you never hear back. This is a distributed system, there are numerous reasons why you may not hear back. For simplicity, consider the client code crashed between sending the request and receiving the response.

The solution is to construct a transaction for resource creation[2]. To construct a transaction, you need to atomically associate a piece of information with the resource at creation time. We’ll call that piece of information an anima.

In the old EC2 API, the only way to construct an anima was through controlling a security group or keypair. Since neither is tied to a real economy, both are reasonable options. The non-idempotent state machine above uses the keypair as it is less resource intensive for EC2.

On creation failure and with the anima in hand[3], the client must search the remote system for the anima before reattempting creation. This is handled by the GM_CHECK_VM state above.

Unfortunately, without explicit support in the API, i.e. lookup by anima, the search can be unnatural and expensive. For example, EC2 instances are not indexed on keypair. Searching requires a client side scan of all instances.

With the introduction of idempotent RunInstances, the portion of the state machine for constructing and locating the anima is reduced to the GM_SAVE_CLIENT_TOKEN state, an entirely local operation. The reduction in complexity is clear.

After two years, EC2 appears to be the only API providing idempotent instance creation[4]. Though APIs are starting to provide atomic anima association, often through metadata or instance attributes, and some even provide lookup.

You should provide an idempotent resource creation operation in your API too!

[0] “in the cloud” – really anywhere in any distributed system!
[1] Making money from forgotten or misplaced resources is a short term play.
[2] Alternatively, you can choose an architecture with a janitor process, which will bring its own complexities.
[3] “in hand” – so long as your hand is reliable storage.
[4] After a quick survey, I’m looking at you Rackspace, RimuHosting, vCloud, OpenNebula, OpenStack, Eucalyptus, GoGrid, Deltacloud, Google Compute Engine and Gandi.

Best practices: batch application’s input

March 8, 2012

For application developers and their users.

An application that is going to be run in batch mode — non-interactive, maybe scheduled to remote resources — is going to need input[0]. That input might be a few numbers or numerous data files. As an application developer, the case of data files can be difficult to get right. There are many ways to handle data files and one big pitfall: hard-coded paths.

Below are a few options for application developers and the resulting work for a Condor user of the application.

The best approach is to read input from stdin or have paths to data files passed via arguments. Doing so shows the developer has batch processing in mind, and provides the applications user with clear options.

A submit file for an application that reads from stdin –

executable = batch_app
input = input.dat
queue

A submit file for an application that takes data files as arguments –

executable = batch_app
arguments = --input=input.dat
transfer_input_files = input.dat
queue

A middle ground approach may be necessary if the set of input files is large or their relationships are complex. In such a case, a meta-data file can be used, or the input files can be laid out in a well-defined pattern in the filesystem. Note: “well-defined pattern in the filesystem” is often a myth.

Of these approaches, the meta-data file is preferred. It makes the input files and their relationships explicit. However, it can be more difficult for the application’s user from a Condor perspective. When the files are laid out in the filesystem the tendency is for the application to not have a well-defined layout, or a definition maintained independently of the application.

A submit file for an application that takes a meta-data file –

executable = batch_app
arguments = --input=input.meta
transfer_input_files = input.meta[,all the files listed in input.meta]
queue

The difficulty comes in listing all the files from the input.meta. This is often mitigated by providing URIs, or paths, in input.meta that may point into a shared filesystem. The files in a shared filesystem need not be transferred by Condor and need not be listed in transfer_input_files.

A submit file for an application that takes a hopefully-well-defined filesystem layout,

executable = batch_app
arguments = --input=data_dir
transfer_input_files = data_dir
queue

This is simpler because Condor will transfer everything under data_dir into the job’s scratch space and keep it under a directory called data_dir. Often, the data_dir will even exist on a shared filesystem and will not need to be transferred (remove transfer_input_files = data_dir and provide full path with --input).

Note: transfer_input_files = data_dir/ will not replicate the directory tree in the job’s scratch space. It will be collapsed.

These two approaches can be combined to get the best of both.

The worst approach is really a non-approach and involves hard-coding paths into the application. Arguably the application does not have a batch mode. It will fail when not run in its expected environment, which may simply mean by a user different from the developer, or on a new shared filesystem, or an old shared filesystem with new mounts. These application should be avoided or modified to provide a batch mode.

Developers beware, you can turn near success with a meta-data file into a failure by hard-coding its path.

Takeaway –

For developers, an application that has a batch processing will parametrize all its inputs[1].

For users, beware of applications that operate on data that you have not provided.

[0] Even if it is just a random seed.
[1] Database or URI connections to get data also matter.

Amazon S3 – Object Expiration, what about Instance Expiration

December 28, 2011

AWS is providing APIs that take distributed computing concerns into account. One could call them cloud concerns these days. Unfortunately, not all cloud providers are doing the same.

Idempotent instance creation showed up in Sept 2010, providing the ability to simplify interactions with EC2. Idempotent resource allocation is critical for distributed systems.

S3 object expiration appeared in Dec 2011, allowing for service-side managed deallocation of S3 resources.

Next up? It would be great to have an EC2 instance expiration feature. One that could be (0) assigned per instance and (1) adjusted while the instance exists. Bonus if can also be (2) adjusted from within the instance without credentials. Think leases.


%d bloggers like this: