[Windows Azure] How to use the Queue Storage Service

How to use the Queue Storage Service

version 1.7 version 2.0

This guide will show you how to perform common scenarios using the Windows Azure Queue storage service. The samples are written in C# code and use the Windows Azure Storage Client for .NET. The scenarios covered include inserting, peeking,getting, and deleting queue messages, as well as creating and deleting queues. For more information on queues, refer to the Next steps section.

Table of contents

• What is Queue Storage
• Concepts
• Create a Windows Azure Storage Account
• Setup a Windows Azure Storage connection string
• How to: Programmatically access queues using .NET
• How to: Create a queue
• How to: Insert a message into a queue
• How to: Peek at the next message
• How to: Change the contents of a queued message
• How to: Dequeue the next message
• How to: Leverage additional options for dequeuing messages
• How to: Get the queue length
• How to: Delete a queue
• Next steps

What is Queue Storage

Windows Azure Queue storage is a service for storing large numbers of messages that can be accessed from anywhere in the world via authenticated calls using HTTP or HTTPS. A single queue message can be up to 64KB in size, a queue can contain millions of messages, up to the 100TB total capacity limit of a storage account. Common uses of Queue storage include:

• Creating a backlog of work to process asynchronously
• Passing messages from a Windows Azure Web role to a Windows Azure Worker role

Concepts

The Queue service contains the following components:

• URL format: Queues are addressable using the following URL format:
  http://<storage account>.queue.core.windows.net/<queue>

  The following URL addresses one of the queues in the diagram:
  http://myaccount.queue.core.windows.net/imagesToDownload

• Storage Account: All access to Windows Azure Storage is done through a storage account. A storage account is the highest level of the namespace for accessing queues. The total size of blob, table, and queue contents in a storage account cannot exceed 100TB.

• Queue: A queue contains a set of messages. All messages must be in a queue.

• Message: A message, in any format, of up to 64KB.

Create a Windows Azure Storage account

To use storage operations, you need a Windows Azure storage account. You can create a storage account by following these steps. (You can also create a storage account using the REST API.)

1. Log into the Windows Azure Management Portal.

2. At the bottom of the navigation pane, click NEW.

   

3. Click DATA SERVICES, then STORAGE, and then click QUICK CREATE.

   

4. In URL, type a subdomain name to use in the URI for the storage account. The entry can contain from 3-24 lowercase letters and numbers. This value becomes the host name within the URI that is used to address Blob, Queue, or Table resources for the subscription.

5. Choose a Region/Affinity Group in which to locate the storage. If you will be using storage from your Windows Azure application, select the same region where you will deploy your application.

6. Optionally, you can enable geo-replication.

7. Click CREATE STORAGE ACCOUNT.

Setup a Windows Azure Storage Connection String

The Windows Azure Storage Client Library for .NET supports using a storage connection string to configure endpoints and credentials for accessing storage services. You can put your storage connection string in a configuration file, rather than hard-coding it in code:

• When using Windows Azure Cloud Services, it is recommended you store your connection string using the Windows Azure service configuration system (*.csdef and *.cscfg files).
• When using Windows Azure Web Sites, Windows Azure Virtual Machines, or building .NET applications that are intentded to run outside of Windows Azure, it is recommended you store your connection string using the .NET configuration system (e.g. web.config or app.config file).

In both cases, you can retrieve your connection string using the CloudConfigurationManager.GetSetting method, as shown later in this guide.

Configuring your connection string when using Cloud Services

The service configuration mechanism is unique to Windows Azure Cloud Services projects and enables you to dynamically change configuration settings from the Windows Azure Management Portal without redeploying your application.

To configure your connection string in the Windows Azure service configuration:

1. Within the Solution Explorer of Visual Studio, in the Roles folder of your Windows Azure Deployment Project, right-click your web role or worker role and click Properties.
   

2. Click the Settings tab and press the Add Setting button.
   

   A new Setting1 entry will then show up in the settings grid.

3. In the Type drop-down of the new Setting1 entry, choose Connection String.
   

4. Click the ... button at the right end of the Setting1 entry. The Storage Account Connection String dialog will open.

5. Choose whether you want to target the storage emulator (the Windows Azure storage simulated on your local machine) or an actual storage account in the cloud. The code in this guide works with either option. Enter the Primary Access Key value copied from the earlier step in this tutorial if you wish to store queue data in the storage account we created earlier on Windows Azure.
   

6. Change the entry Name from Setting1 to a "friendlier" name like StorageConnectionString. You will reference this connection string later in the code in this guide.
   

Configuring your connection string using .NET configuration

If you are writing an application that is not a Windows Azure cloud service, (see previous section), it is recommended you use the .NET configuration system (e.g. web.config or app.config). This includes Windows Azure Web Sites or Windows Azure Virtual Machines, as well as applications designed to run outside of Windows Azure. You store the connection string using the<appSettings> element as follows:

<configuration><appSettings><addkey="StorageConnectionString"value="DefaultEndpointsProtocol=https;AccountName=[AccountName];AccountKey=[AccountKey"/></appSettings></configuration>

Read Configuring Connection Strings for more information on storage connection strings.

You are now ready to perform the how-to tasks in this guide.

How to: Programmatically access queues using .NET

Obtaining the assembly

You can use NuGet to obtain the Microsoft.WindowsAzure.Storage.dll assembly. Right-click your project in Solution Explorer and choose Manage NuGet Packages. Search online for "WindowsAzure.Storage" and click Install to install the Windows Azure Storage package and dependencies.

Namespace declarations

Add the following code namespace declarations to the top of any C# file in which you wish to programmatically access Windows Azure Storage:

usingMicrosoft.WindowsAzure.Storage;usingMicrosoft.WindowsAzure.Storage.Auth;usingMicrosoft.WindowsAzure.Storage.Queue;

Make sure you reference the Microsoft.WindowsAzure.Storage.dll assembly.

Retrieving your connection string

You can use the CloudStorageAccount type to represent your Storage Account information. If you are using a Windows Azure project template and/or have a reference to Microsoft.WindowsAzure.CloudConfigurationManager, you can you use theCloudConfigurationManager type to retrieve your storage connection string and storage account information from the Windows Azure service configuration:

CloudStorageAccount storageAccount =CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));

If you are creating an application with no reference to Microsoft.WindowsAzure.CloudConfigurationManager and your connection string is located in the web.config or app.config as show above, then you can use ConfigurationManager to retrieve the connection string. You will need to add a reference to System.Configuration.dll to your project, and add another namespace declaration for it:

usingSystem.Configuration;...CloudStorageAccount storageAccount =CloudStorageAccount.Parse(ConfigurationManager.ConnectionStrings["StorageConnectionString"].ConnectionString);

ODataLib dependencies

ODataLib dependencies in the Storage Client Library for .NET are resolved through the ODataLib (version 5.0.2) packages available through NuGet and not WCF Data Services. The ODataLib libraries can be downloaded directly or referenced by your code project through NuGet. The specific ODataLib packages are OData, Edm, and Spatial.

How to: Create a queue

A CloudQueueClient object lets you get reference objects for queues. The following code creates a CloudQueueClient object. All code in this guide uses a storage connection string stored in the Windows Azure application's service configuration. There are also other ways to create a CloudStorageAccount object. See CloudStorageAccount documentation for details.

// Retrieve storage account from connection stringCloudStorageAccount storageAccount =CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));// Create the queue clientCloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();

Use the queueClient object to get a reference to the queue you want to use. You can create the queue if it doesn't exist.

// Retrieve a reference to a queueCloudQueue queue = queueClient.GetQueueReference("myqueue");// Create the queue if it doesn't already exist
queue.CreateIfNotExists();

How to: Insert a message into a queue

To insert a message into an existing queue, first create a new CloudQueueMessage. Next, call the AddMessage method. ACloudQueueMessage can be created from either a string (in UTF-8 format) or a byte array. Here is code which creates a queue (if it doesn't exist) and inserts the message 'Hello, World':

// Retrieve storage account from connection string.CloudStorageAccount storageAccount =CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));// Create the queue client.CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();// Retrieve a reference to a queue.CloudQueue queue = queueClient.GetQueueReference("myqueue");// Create the queue if it doesn't already exist.
queue.CreateIfNotExists();// Create a message and add it to the queue.CloudQueueMessage message =newCloudQueueMessage("Hello, World");
queue.AddMessage(message);

How to: Peek at the next message

You can peek at the message in the front of a queue without removing it from the queue by calling the PeekMessage method.

// Retrieve storage account from connection stringCloudStorageAccount storageAccount =CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));// Create the queue clientCloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();// Retrieve a reference to a queueCloudQueue queue = queueClient.GetQueueReference("myqueue");// Peek at the next messageCloudQueueMessage peekedMessage = queue.PeekMessage();// Display message.Console.WriteLine(peekedMessage.AsString);

How to: Change the contents of a queued message

You can change the contents of a message in-place in the queue. If the message represents a work task, you could use this feature to update the status of the work task. The following code updates the queue message with new contents, and sets the visibility timeout to extend another 60 seconds. This saves the state of work associated with the message, and gives the client another minute to continue working on the message. You could use this technique to track multi-step workflows on queue messages, without having to start over from the beginning if a processing step fails due to hardware or software failure. Typically, you would keep a retry count as well, and if the message is retried more than n times, you would delete it. This protects against a message that triggers an application error each time it is processed.

// Retrieve storage account from connection string.CloudStorageAccount storageAccount =CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));// Create the queue client.CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();// Retrieve a reference to a queue.CloudQueue queue = queueClient.GetQueueReference("myqueue");// Get the message from the queue and update the message contents.CloudQueueMessage message = queue.GetMessage();
message.SetMessageContent("Updated contents.");
queue.UpdateMessage(message,TimeSpan.FromSeconds(0.0),// Make it visible immediately.MessageUpdateFields.Content|MessageUpdateFields.Visibility);

How to: De-queue the next message

Your code de-queues a message from a queue in two steps. When you call GetMessage, you get the next message in a queue. A message returned from GetMessage becomes invisible to any other code reading messages from this queue. By default, this message stays invisible for 30 seconds. To finish removing the message from the queue, you must also call DeleteMessage. This two-step process of removing a message assures that if your code fails to process a message due to hardware or software failure, another instance of your code can get the same message and try again. Your code calls DeleteMessage right after the message has been processed.

// Retrieve storage account from connection stringCloudStorageAccount storageAccount =CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));// Create the queue clientCloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();// Retrieve a reference to a queueCloudQueue queue = queueClient.GetQueueReference("myqueue");// Get the next messageCloudQueueMessage retrievedMessage = queue.GetMessage();//Process the message in less than 30 seconds, and then delete the message
queue.DeleteMessage(retrievedMessage);

How to: Leverage additional options for de-queuing messages

There are two ways you can customize message retrieval from a queue. First, you can get a batch of messages (up to 32). Second, you can set a longer or shorter invisibility timeout, allowing your code more or less time to fully process each message. The following code example uses the GetMessages method to get 20 messages in one call. Then it processes each message using a foreach loop. It also sets the invisibility timeout to five minutes for each message. Note that the 5 minutes starts for all messages at the same time, so after 5 minutes have passed since the call to GetMessages, any messages which have not been deleted will become visible again.

// Retrieve storage account from connection string.CloudStorageAccount storageAccount =CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));// Create the queue client.CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();// Retrieve a reference to a queue.CloudQueue queue = queueClient.GetQueueReference("myqueue");foreach(CloudQueueMessage message in queue.GetMessages(20,TimeSpan.FromMinutes(5))){// Process all messages in less than 5 minutes, deleting each message after processing.
    queue.DeleteMessage(message);}

How to: Get the queue length

You can get an estimate of the number of messages in a queue. The FetchAttributes method asks the Queue service to retrieve the queue attributes, including the message count. The ApproximateMethodCount property returns the last value retrieved by the FetchAttributes method, without calling the Queue service.

// Retrieve storage account from connection string.CloudStorageAccount storageAccount =CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));// Create the queue client.CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();// Retrieve a reference to a queue.CloudQueue queue = queueClient.GetQueueReference("myqueue");// Fetch the queue attributes.
queue.FetchAttributes();// Retrieve the cached approximate message count.int? cachedMessageCount = queue.ApproximateMessageCount;// Display number of messages.Console.WriteLine("Number of messages in queue: "+ cachedMessageCount);

How to: Delete a queue

To delete a queue and all the messages contained in it, call the Delete method on the queue object.

// Retrieve storage account from connection string.CloudStorageAccount storageAccount =CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));// Create the queue client.CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();// Retrieve a reference to a queue.CloudQueue queue = queueClient.GetQueueReference("myqueue");// Delete the queue.
queue.Delete();

Next steps

Now that you've learned the basics of queue storage, follow these links to learn how to do more complex storage tasks.

• View the queue service reference documentation for complete details about available APIs:
  • Storage Client Library for .NET reference
  • REST API reference
• Learn about more advanced tasks you can perform with Windows Azure Storage at Storing and Accessing Data in Windows Azure.
• View more feature guides to learn about additional options for storing data in Windows Azure.
  • Use Table Storage to store structured data.
  • Use Blob Storage to store unstructured data.
  • Use SQL Database to store relational data.