URBACKUP TO THE CLOUD
I thought I would share this with the community in case anyone else is interested in using UrBackup to back your serves to Amazon Web Services.
I work for a national construction company and backing up field offices and job sites has always been an issue. Plus cost is big factor so we cannot use expensive commercial cloud backup solutions. This is a real good cost effective solution for us.
Concept:
AWS has an appliance called Amazon Storage Gateway that allows local storage to be backed up to AWS S3 storage. The following is a short overview of AWS Storage Gateway.
Amazon Storage Gateway site: https://aws.amazon.com/storagegateway
The storage Gateway is a virtual appliance you install on your hypervisor, either VMware ESX or Hyper-V.
The storage gateway has 3 operational modes
Gateway Cached Volume:
You can store your primary data in Amazon S3, and retain your frequently accessed data locally. Gateway-Cached volumes provide substantial cost savings on primary storage, minimize the need to scale your storage on-premises, and retain low-latency access to your frequently accessed data.
Gateway-Stored Volumes:
In the event you need low-latency access to your entire data set, you can configure your on-premises data gateway to store your primary data locally, and asynchronously back up point-in-time snapshots of this data to Amazon S3. Gateway-Stored volumes provide durable and inexpensive off-site backups that you can recover locally or from Amazon EC2 if, for example, you need replacement capacity for disaster recovery.
Gateway-Virtual Tape Library (Gateway-VTL):
With Gateway-VTL you can have a limitless collection of virtual tapes. Each virtual tape can be stored in a Virtual Tape Library backed by Amazon S3 or a Virtual Tape Shelf backed by Amazon Glacier. The Virtual Tape Library exposes an industry standard iSCSI interface which provides your backup application with on-line access to the virtual tapes. When you no longer require immediate or frequent access to data contained on a virtual tape, you can use your backup application to move it from its Virtual Tape Library to your Virtual Tape Shelf in order to further reduce your storage costs. (Note: UrBackup is not supported)
Cost:
Gateway Appliance: $125.00 per month
Volume Storage: $0.030 per GB Per month
Snapshots: $0.095 per GB per month
More information is available on the site.
Using AWS Storage Gateway with UrBackup
This procedure uses VMware ESXi as the hypervisor. I am no fan of Hyper-V and I am not sure how you would set up the networking part in Hyper-V
We will be using a standalone ESXi server but a VCenter setup would be quite similar.
Planning the AWS Storage Gateway:
We will be using the Gateway Cached Volume configuration because that gives the most storage in AWS to house backups without needing a huge amount of local storage.
Local server requirements:
Depending on the size of your full backups and the amount of backups you want to keep local will determine how much local storage space you will need on your server.
The storage gateway uses a minimum of 2 volumes for operation. 1 upload buffer volume and 1 or more local cache volumes.
The upload buffer should be at least the size of your largest full backup + some additional space for overhead. If your largest full backup is 100 GB then your upload buffer should be say 125GB. If you run all your full backups on the same day and that total is 250GB then your upload buffer should be say 275-300 GB.
Your cache volume is where you store your local data for quick retrieval and recovery. The gateway stores recently accessed files local and retrieves files from AWS S3 as needed.
If your full backups are 250 GB and your rate of change is say 20% / day then a good estimate for local backup storage would be 500-750 GB.
As you can see this is a judgement call. The most important thing is not to run out of upload buffer space while writing backups.
Installing the Gateway Appliance:
On your ESXi server click on the host and select the configuration tab and select Networking
Click Add Networking and add a new vSwitch. Do not connect any external NICs to this switch, it will be used internally by ESXi. Name it “Internal”
Log on to your AWS account and select AWS Storage Gateway from the console.
Once the console comes up select Deploy as new Gateway
Follow the prompts and download the gateway appliance.
Deploy the ova appliance to your ESXi server per the instructions in the AWS console.
Once the wizard is asking for the IP of your gateway stop.
Once installed and before you start the Gateway Appliance right click the appliance VM and select Edit Settings and add a 2nd Network Adapter, use E1000 and connect it to the Internal vSwitch.
This NIC will be used for iSCSI traffic only.
Add 2 new drives to the VM.
First one will be your upload buffer drive. Assign to SCSI(2,0)
Second one will be your cache volume drive. Assign to SCSI(2,1)
A new SCSI controller will be added to your VM. Change the type to Paravirtual (important step!).
Power on the appliance.
Go to the console of the appliance in VMware and log on. The username and password are on the screen.
Select 2. Configure Static IP.
Select eth0 and assign a static IP, gateway and DNS. This should be on your LAN with internet access.
Select 2. Again and select eth1 and assign a private IP address to this interface. This is separate from your LAN so use something like 192.168.200.X. The gateway and DNS are irrelevant so just enter something there, the gateway should be in the subnet your using so use .1 or .254.
Now select 5. Set Default Route Adapter
Enter eth0.
Log out of the console.
Now on the AWS console enter the IP address of eth0 in the prompt and select Activate Gateway.
On the console click on your Gateway name on the left
Click the Gateway tab…
Click the Configure Local Storage.
Assign your buffer drive to Upload Buffer and your cache drive to Cache Storage.
On the Volume tab select Create New Volume.
Accept the default notification you are prompted to create. These are very helpful to monitor your gateway.
Set the capacity. Set this to how much backup storage you want on AWS. This is the volume size your backup server will see. This will be larger than your cache drive size so you can pick a size such as 10TB or so.
(Note: you only pay for what you use, not the selected size of the volume)
Give the volume an iSCSI target name.
Select the private IP address you assigned to eth1.
Click Create Volume.
Wait for the status to go from CREATING to AVAILABLE.
Your gateway is now ready.
Configuring your UrBackup Server:
We are going to use Linux in this example but Windows is similar and possibly even easier.
Create your VM on the ESXi server and install the OS. I used CentOS 7.
Add a second NIC and assign to the “Internal” vSwitch.
Power up the VM and log in.
Assign static IP addresses. NIC 1 to the LAN, NIC 2 to the Private Subnet.
Update the server.
Install iscsi initiator. (already installed in windows)
‘# yum install iscsi-initiator-utils’
start the iscsi daemon:
‘# systemctl enable iscsid’
‘# systemctl start iscsid’
Discover targets.
‘# iscsiadm –m discover –t sendtargets –p xxx.xxx.xxx.xxx’ <-IP assigned to gateway volume
Make a note of the returned target name starting with iqn.
Login to the target.
If you only have one volume on your gateway you can just type:
‘# iscsiadm –m node –l’
in the output of the command look for “successful” at the end and you are good.
Now you should have a new device on your system such as /dev/sdb.
Partition and format the new volume:
(My recommendation) Do not attach the volume using LVM. This will make it much more complicated if you need to recover your backup server after a full site failure or attach your volume from a snapshot to an EC2 instance. Doing it wrong could erase the entire volume.
Make sure you have the following installed:
‘# yum install xfsprogs’
Find the name of your new device:
‘# fdisk –l’
Make note of the device: I.e. /dev/sdb
Partition the device.
‘# parted /dev/sdb mklabel gpt’
‘# parted – /dev/sdb mkpart primary xfs 1 -1’
Format the partition.
‘# mkfs.xfs –L somelabel /dev/sdb1’
“somelabel” can be anything you want.
Create a mount point i.e /backup
mount the partition.
‘# mount /deb/sdb1 /backup’
Make sure everything looks good.
‘# df –H’
Set permissions.
Since no one uses this except the backup process I did.
‘# chmod 0777 /backup’
You can be more granular if you like.
Edit /etc/fstab.
Add line
/dev/sdb1 /backup xfs _netdev 0 0
Important: if you don’t add _netdev you server will hang at boot!!! This says wait for the network before mounting this volume.
Install and configure UrBackup and you are done!!!
Optional Snapshot Schedule
On the Snapshot Schedule tab in AWS you can create a schedule to take snapshots of your volume. A snapshot is a point in time backup of your volume, This can be used to recover your volume in case it is lost or you can attach this snapshot to an EC2 server instance. The gateway itself takes periodic recovery snapshots that can be used if your local VM is lost or crashes and you need to rebuild it. So taking these snapshots is purely optional.
There is no facility to automatically delete old snapshots in the AWS console. So if your not managing these snapshots it could get costly. This can be done with AWS CLI or PowerShell in a scheduled task. I have PowerShell script that does this and will post it in another post.
I hope this was informative.
Thanks,
Cliff