Setting up Event Store on Azure Windows VM

After looking around a while for information on setting up Greg Young's Event Store on Azure, I decided to take a crack at it from scratch to learn a little about the new Azure Portal and setting up VMs and disk systems. I decided to keep track of everything I did and blog about it. In this post I will walk you through what I did to get everything set up.  I don't go into a lot of detail about optimization nor do I talk about clustering configurations.

Required Software and Accounts

To follow along you will need a Microsoft Azure account and access to the Event Store version 3 download available at http://geteventstore.com/downloads/.

Setting up Azure Storage Accounts

There is a specific order you have to go through to get to the point of installing the Event Store software itself. So let's get started with our first step, setting up storage accounts.  Since Azure has some very specific limitations around disk IO based on the VM size and the number and type of disks set up, I chose to create a separate storage account for each attached disk to allow for maximum I/O in regards to account throttling. The screenshots used in this post show the new Azure portal interface. It was new to me and took some time to get used to it. Since different VM sizes allow for a verying number of attached Data Disks, the VM size I use here allows for 4 Data Disks so you will understand why I chose 4 later on.

To set up a Storage Account log into the Azure portal at http://portal.azure.com and choose Browse to list the set of Azure features.  You want to locate the feature called Storage.

The Storage area allows you to set up Storage accounts.  If you have any storage accounts configured, they will be listed for you when you open this area.  There are some things to get used to here that are a little confusing if this is your first time using the new portal. You would think that making the 'Storage' selection would set the context for the actions you want to take, however, you will not find a context based 'new' option. Instead,just to the lower left of the storage account listing there is a button with the + sign and the word New on it. This brings up the a pane that they call a 'blade' that contains a predetermined list of items which at the time of this writing does not contain the option for a Storage related item. You actually have to click the Everything button to see all the other options. You want to locate the one for Storage. To get to it you must locate and click the option labeled 'Storage, cache, + backup' as shown below:

Now you should see the area where you can actually set up a storage account.

I numbered the steps in the screenshot according to the order you make your selections.

1. Click the Create button.
2. Type in the name you want to call the storage account. It must be all lowercase with no spaces or special characters. I chose to use a name that would be obvious as to what it is with a number suffix to identify the disk that we will later create.
3. Choose Local Redundant storage so the account resides in the same data center for each disk. Notice the MAX IOPS per disk I highlighted with a red box.
4. Choose a Resource Group from the ones listed or create a new one.  I created a new one called Default-Storage-EastUS2 since I am going to place all these in the EastUS2 region/location.
5. Choose the Location for the storage account.  We will be doing this four times and all of them need to be identical with the exception of the name. There is premium storage(SSD) that is available in certain regions so find our more about this ane make the choice that fits what you want.
6. Click the Create button and the Azure Fabric will create your storage account.

Since I will be choosing a VM size that allows up to 4 Data Disks, I created 4 separate storage accounts.  Each Azure subscription can have up to 100 storage accounts at the time of this writing. This is double what it was not too long ago. Next, we will set up a container in the storage account to host our disk we will mount later in the VM.

Setting up Storage Account Containers

For this part of the process we will need to navigate back to the storage account listing.  You should now have at least 4 storage accounts.  When you select one from the list it opens a blade that shows some details about the storage account. I showed what mine looks like below along with highlighting some things of interest.

1. Click on the Containers item to bring up the Containers blade.
2. Click the Add button at the top of the containers area to add a new container.  We do this because later on, when creating a disk for the VM it will ask us which storage account and what container we want our .vhd disk created in.
3. Enter the name 'hdd'.  I chose this so it would be obvious that this will be for a hdd.  You could use vhd instead if you want since it is a virtual hard disk.
4. Leave Access Type set to Private and click OK to create the container.
5. Repeat this for EACH storage account. Note: Do not create 4 containers in this one storage account...create a single hdd container in EACH storage account.

In the next section we will create our VM and the disks that go in these storage accounts.

Creating the Windows Virtual Machine

Now we have our prerequisites set up for the storage accounts, we can set up the virtual machine. On the far left side of the portal, click the Browse button to bring up the list of Azure Functional Areas where we can find the option for Virtual Machines.  See below for where to find this option:

Click the Virtual Machines menu item to list the virtual machines configured in the account. This shows a listing of the virtual machines with the name you give them, the location, and the current status. Once again you don't have a context sensitive 'New' option. so you will have to click the + New button in the lower left of the portal window.  Choose the 'Everything' button again and locate the 'Virtual Machines' item shown below:

Once you are in the Virtual Machine area, locate the tile for the Windows Server VM options. It should be royal blue in color.  I get feeling that over time these will move around and may be a way for you to have to sift through the options to show off the plethora of choices. Either way you want to locate the Windows Server options.  I found that the only one that seemed to make sense was the Windows Server 2012 R2 Data Center Edition which is what I chose.  Server 2012 has a feature that we can take advantage of called Shared Spaces.  This is an alternative to using something like striped RAID 0. I had seen this in Rinat Abdullin's article here. Storage Spaces allow you to take a JBOD setup and overlay a disk volume across them to take advantage of higher I/O. You can read more about Storage Spaces here.

So select Windows Server 2012 R2 Datacenter edition shown below and click Create:

This will bring up the blades for creating the VM. The screenshot below shows the steps in the order needed.

1. Enter a host name. This will be part of the Url needed to access the event store after it gets set up.
2. Enter a user name
3. Enter a password. These will be used to log in to the server via RDP so keep track of it.
4. Click pricing tier to select the VM size.  Choose A2 Standard.  I chose this because of the 4 disks @ 500 IOPS.
5. Click the Select button
6. Choose the Resource Group that you used to set up the storage accounts with
7. Choose the Location/Region where the storage accounts are configured.
8. Click the Create button to start the process of creating the Virtual Machine. It will take a while to create and will start upon creation.

Creating VM Disks

We now have our storage accounts with hdd containers set up and have our virtual machine set up. The next step is to go to the VM configuration and create and attach some disks to the virtual machine. Navigate to the Virtual Machine area where it will list out the virtual machines that have been created under the subscription. You should see the VM you created earlier. Select your VM from the list to bring up the blade that shows details about the VM. It will look something like this:

Scroll down in the blade to locate the area where Disks are displayed:

Mine shows 5 disks because I have already created the 4 disks plus the 5th is the OS disk where the server OS itself resides. Click on the Disks tile and it will bring up the Disk specific blade. The only disk you should see listed is the OS disk like below:

Click the Attach New button just to the top left of the disk listing.

1. Choose the Storage Container by selecting the storage account from the list which should contain the four storage accounts you created earlier.
2. Select the 'hdd' container created for the storage account.
3. Click OK
4. Enter the data file name. Again I chose to number these so I can keep them straight with the storage account they belong to.
5. Enter 1000 GB as the size.  This is the MAX size for the disks.
6. For Host Caching, I selected Read/Write. Now this is debatable as to the performance characteristics, however this post is not meant to be a fully optimized deployment at this time. Testing with your setup and read/write may need to be changed to READ ONLY later.
7. Click OK to create the disk.

You will repeat these steps for the other 3 disks so that when you are done, there will be a total of 5 disks that show up in the Disks blade with a total storage capacity of 4128 GB.  The 128 GB is from the default OS disk created when the A2 Standard VM was created.

Setting up Shared Storage Spaces in Server 2012 R2

Now that our disks are all set up and connected to the VM, we can use Remote Desktop to connect to the VM using the credentials you entered when setting up the VM. To RDP to the server navigate to the Virtual Machine listing and select the virtual machine you created. At the top of the blade there should be a Connect button:

If your VM is currently Running, please Stop the VM and we will set up some IP address configurations so we will be able to connect from the outside in a later portion of the post. In the VM configuration blade near the Disks tile, there is a tile labeled IP Addresses. Click this Tile to open the IP Addresses blade.

Under the 'Private IP address' section,

1. Choose 'STATIC'
2. Enter 10.0.0.4 and the IP Address. I chose this because that is what it originally was when the VM was created.  This is important because the Event Store configuration will bind to this private internal IP address.
3. Click Save at the top.

Re-start the VM so it can pick up the newly attached disks and the adjusted IP configuration. Click the Connect button to download the pre-configured RDP session file and log in to your VM. After logging in, the Server Manager should start automatically and contains an area to manage File and Storage Services.  Click the File and Storage Services menu item to open the listing and configuration area.

This view shows you the server listing, Volumes, Disks, and Storage Pools. Since we are interested in setting up a Storage Pool, click the Storage Pools option on the left side, and select New Storage Pool under the Tasks list like shown below:

This will open up the Storage Pool Wizard and will ask you to enter the name you want to call the pool and a description.  I chose to call mine 'esdata'.  After entering the name and description click Next and it should list the four disks we attached to the VM earlier in the process. What you are doing here is associating the 4 attached disks (storage account .vhds contained in the hdd containers) into a single pool.  We then can create a virtual disk volume of a given size when we mount and associate a drive letter. Just follow the wizard and select the four disks to place in the pool. When you are done you should have something that looks like the screenshot above that indicates a Storage Pool with 3.90 TB available.

When you select the newly created Storage Pool it should show that you have 4 disks in the pool like shown below:

Next, we create the Virtual Disk that we can mount so the OS can see it as a lettered drive. To do that, locate the Task options for the Virtual Disk and choose 'New Virtual Disk' and follow the steps described below:

1. Select the storage pool we just created and click Next.
2. Enter the name for the virtual disk and a description. I named mine 'vddesdata' to indicate it is a virtual disk drive for ES data, and click Next.
3. For Storage Layout choose Simple.  This is a striped set across the 4 disks for performance reasons.  Remember, the 4 disks are blobs that are replicated to 3 locations for reliability purposes. We don't need to be redundant with our redundancy :)
4. For provisioning, I chose Thin so that it doesn't allocate all of it ahead of time.  The ES system will pre-allocate what it needs and this will grow in accordance with that.
5. For size, this is really up to you and your needs. I chose a 40GB volume size since that is quite a bit and it is primarily for learning, development, testing, and validation.
6. When you have made and verified all of your selections, click Create.

Now you almost have everything ready to go for the install of ES.  The final thing is to decide on what drive letter you want to map it to.  I chose to map it to E.  It makes sense right?  E for Event Store?  Setting the drive letter is an exercise left to you.  Setting this stuff up for ES was the first time I ever did this kind of thing in Server 2012 so you should be able to get that mapped and ready.

Setting up Event Store

The virtual hardware is ready to go so now we can set up Event Store.  On the VM, using Internet Explorer, download the ES version 3+ (I used 3.0.1 for this) from here and place the zip file to a temporary location.  I just left it in the default Downloads folder.

1. Create a new folder in the root of the C: drive and call it 'eventstore'.
2. Unzip the contents of zip to the 'eventstore' folder.
3. Create two folders on the new drive you mounted. Create an 'esdata' folder and a 'logs' folder on the new drive. Note: I believe the logs can go to a transient disk such as the D: drive but for now we put the logs on the same disk as the data.

Since the Event Store is delivered as a command line app for the OSS version, at this time we will only be setting it up to run using command line parameters. I just created a batch file in the c:\eventstore folder called 'start-es.bat'.  My command line looks like this:

EventStore.ClusterNode.Exe --db e:\esdata --log e:\logs --ext-ip=10.0.0.4

The --db parameter sets the location of the data files.  The --log parameter sets the location of the log files (a fully optimized setup would probably have these on different disk subsystems with different read/write caching settings.  The --ext-ip parameter tells it what IP address to bind to for the clients to connect to for the administrative web UI and the client API's.  The 10.0.0.4 is used since that is the private IP address for the VM.  The public IP for the VM is dynamic and you will be using the [vmname].cloudapp.net address to get to this from the outside. Once you get the command line all set up, you can start the ES system.  Go ahead and fire it up.  Now you won;t be able to connect from the outside just yet. That requires you to set up and endpoint and open the firewall which we will do next.

If you want to test that the ES is up and running, then from IE on the VM, go to, http://10.0.0.4:2113

It should be up and running and provide you with a login screen.  The default username is: 'admin' and the default password is: 'changeit'.   Log in and change the password now!

Setting up Endpoints

To be able to get to anything from outside of Azure you may want to set up endpoints.  If you will only be connecting to the ES from inside Azure then you may not need to do this and can use the internal Azure network configurations to accomplish this. However, for my purposes, I wanted to connect from the outside.  To do this you must set up at least one endpoint and that is the API endpoint on port 1113.  You can choose different ports if you want but the default for this is 1113.

To set up End Points you need to go open the blade for your VM.  Oddly enough, while I was typing this out an update was applied to the Portal UI.  The way you get to the End Points area changed.  On the VM blade, look for a button that says "All Settings".  This will open another blade that lists all the available settings for the VM.  Locate the one that says End Points and click it.  See below:

If you want to be able to get to the Admin UI from the outside you must set up an end point for port 2113. For the client API's it is port 1113.  To set these up follow the instructions below:

1. Click +Add to open the Add an endpoint blade
2. Enter the name for the endpoint. I chose ESAdmin and ESTCP (for API tcp connections)
3. Leave protocol as TCP.
4. Enter 2113
5. Enter 2113
6. Click OK

Do this for which ever of the two ports you want to be able to access from the outside.

We are almost done. One thing left before we can test out external connectivity and run a WRFL test.

Opening Firewall Ports

When we set up endpoints, we actually did two things. First we allowed external access to the public port.  Secondly it sets up port forwarding to the private port.  They just happen to be the same.  If you wanted the external interfaces to be on different ports, this is the easiest place to do that. Just change the public port to whatever you want and it will be forwarded to the private port where ES is listening.  To tell Windows Server to allow outside connections to those ports, we must set up two firewall rules, one for each external port.  In many cases you may not want to open the admin interfaces to the outside. I did because this is my playground and testing area. Let's set up the firewall ports.

I will not be showing all the steps required to open firewall ports since that is covered elsewhere, but I will tell you that you need to create an Inbound Rule that allows TCP connections on ports 1113,2113 (if you so choose). Once you do this then you should be able to open a web browser to:

http://[yourvmname].cloudapp.net:2113 and see the login for your ES Administration area.

Run some Tests

Now that we have everything all ready to go lets see how well it works.  To do this, I chose not to run the Tests from an external client for this quick test but instead ran it from the VM in a local command prompt.  There is a built in TestClient that can be used to run some different tests. I don't know what all of them do but one I have seen people use as a quick benchmark is WRFL which is a write flood test. To run this, RDP to the VM and make sure the ES is up and running using the command line we used before. 

1. Open a new command prompt in the c:\eventstore folder.
2. Run the following command line.

EventStore.TestClient
>>> WRFL 10 1000000

This will start up a quick benchmark test that will use 10 test clients and run 1 million request messages through.

Back when Rinat Abdullin ran this using version 2 (I think), he got around 5576 reqs per second using an 8 Core, 14GB memory and 16 5GB disks in a RAID 0 configuration. Here are my results using the A2 VM which is 2 Core, 3.5 GB Memory with 4 1GB disks using Storage Spaces.

[02212,06,21:06:58.122] 1000000 requests completed in 183366ms (5453.57 reqs per

 sec).

Now, this number is very close to what Rinat got with his setup.  I decided to run the test again with just 5 clients:

WRFL 5 1000000

and this time I got:

[02640,06,21:17:05.035] 1000000 requests completed in 133909ms (7467.76 reqs per

 sec).

Keep in mind that my tests were run on the same machine as the ES so they were competing with the ES. I guess the point I am making is that the story has gotten better with regard to performance and this can be improved much more with some different configurations.  I went with a smaller cheaper configuration and achieved basically the same result.  Using a cluster of these with some other modifications would yield much better results but this is not the point of the post.

So what does this get you and what don't you get?

You get a good performing system to test with, not optimal compared to using local storage on a regular server. You get some basic fault tolerance with the data drives being replicated by Azure and you get quite a bit of cheap storage.  So what is missing?  Scalability, elasticity, super high performance.  You cannot just fire up another VM and load balance, but you could move this to a bigger VM with more disks in the Storage Spaces pool and use the Premium Storage using SSDs. You can also grow into the clustered configuration without too much trouble.

One other thing this is missing is the fact that you are running this as a command-line process.  It is not responding to start, stop events from the OS.  It doesn't start up when the VM starts nor would it shut down gracefully when the VM shuts down.  Oh and the fact that it is running under the user account is another major thing.  Log out the user and the process goes down with it.  These are all things you can overcome. After all, this is open source software so you can wire in the things needed to run it as a windows service and to handle the appropriate events properly.  That is the subject of another post I suppose.  Anybody already written the code to do that?