Amazon EBS Software Raid-0

So I’ve been doing alot of NFS and MySQL servers in AWS lately and they’ve all wanted a nice and fast raid-0 array for obvious performance reasons, and the Amazon Doc on the subject is good, but not 100%. Like say for example you reboot and now your mdadm device has somehow changed it’s name to /dev/md127 instead of the /dev/md0 you created and nothing’s mounted. This is actually a common occurrence and my post will addresses that problem as well as a few others.

I’m assuming you are comfortable with the aws console and have already created and attached the number of volumes you want for your array, and they are all of the same size, and if using provisioned iops, they all have the same performance settings as well.

Of course as any other how-to states, ymmv. Don’t just copy and paste, use this as a guideline.

Handy commands to know going into this:

lsblk will list the devices attached to your system. Good to verify the volumes have successfully attached, and later to view mount points etc.

Seriously, you should know that df shows drives and available space, as well as mount points, the -h makes it more human readable.

mdstat is the output of your mdadm array(‘s) and will give you the information on what drives are attached to what array’s and more.

So now we have our system, let’s say 3, 500gb volumes have been attached to our instance, /dev/xvdg /dev/xvdh /dev/xvdi that we wish to put into a new array. First, strictly for performance sake, we are going to pre-warm our drives, that is, fill them up with a bunch of zero’s. The reason for this is because virtual drives are exactly that, virtual. Data-centers will allocate an amount of space, but until it’s actually been written to, it’s all virtual so you can suffer a 30-50% decrease in speed the first time a sector is written to, so we fill them all the way up with zeros!

I will start a ‘screen’ session for each drive then start the filling process. Do be careful when choosing what drive your doing this to. I literally had a client do this to his root volume (/dev/xvda) and completely borked their server while I was writing this!

To exit a screen session, depress and hold down the “Ctrl” key, type “a” then “d” and it will detach from the session. Do the same for the other two drives.

Ctrl+a,d to detach again.

Then it’s a matter of hurry up and wait. You will be waiting for hours, seriously. I usually will exit to another screen session and run top (or prefereably htop) and watch the dd command until they disappear which usually means they’ve finished.

To rejoin your screen session, just enter in the same screen command as above.

The resulting sessions should look something like so:

Which is exactly what we wanted, a completely filled drive. Nice!

Now we can actually build the array! This is exactly the same as in the aws doc. All you need to change is the number for –raid-devices and of course tell it which devices to use. Common sense should tell you that the number of volumes you add, need to total the number set in the –raid-devices section, in my case, 3.

The build process should be pretty quick and look something like this.

You can check out your new raid0 array with mdstat

And get something that looks like this:

You’ll also need to give this array a filesystem. Generally it’s going to be ext4 so

Note we use the md0 device, NOT one of the /dev/xvd? devices. We want the one that is reported by the mdstat we ran above which is our new array.

Create your mount point wherever you want this beastly storage array, in my case it’s often /data so that’s what we’ll use in this example.

The -m 000 option makes it an obvious mount point so you don’t have something or someone accidentally writing to this location when it’s not mounted.

Now is where I go a little more away from the script.

I like to make sure that my fstab works, because you really don’t want errors in your fstab before you reboot your server, so I first create the entries in /etc/fstab, but as I said you might run into problems when you reboot so first let’s get the uuid of this array.

This should give you the uuid of all the volumes attached to your server, of course we just want the /dev/md0 one, so copy the UUID=”xxx…” section to your clipboard or text editor.

Now we can edit our /etc/fstab.

Note! You need to remove the quotation marks from the UUID line if pasting! It should look something like this:

Note! the nofail option. This is important! if for some reason the array isn’t able to come up your instance will continue to boot, so you can still login and fix the problem. Without it you’ll have to do alot more work to fix it (but it is fixable). Also if your using a debian or ubuntu based instance, you can, and should, add the nobootwait option, which is the same thing just in debian land.

Now by doing it in this order, we will not only test our fstab, but mount the volume as well. To do that simply:

It should just return without errors, if so congrats! Your array is now ready to be used! If you did get an error though, do fix it at this point!

Check it out with lsblk and df -h and write some data if you like. If you’re looking to test performance I like fio, but that’s outside the scope of this post. Of course most clients seem to prefer to do real world testing and just start using it!

I’d also test rebooting, just to make sure everything mounts and is available better to know now!

There are some other performance related tweaks you can make, like adding noatime,nodiratime to the fstab mount, and increasing file handlers but that too is outside the scope of this post, this is written to get you up and running solid, if you want to tweak it feel free!

Hope this helped!

 

Psudo SSL for free the easy way.

So I was down at my local Vape Shop and the owner was telling me he was releasing his own line of e-liquid and if I could help him build a website. No sweat!

But he’s a very new shop, and of course doesn’t have a lot of money to pour into this project, as well as isn’t tech savvy so it’s been all pretty much put in my lap to do what I think is the best for cheap.

Well hosting I’ve taken care of by using Profitbricks, to keep the cost’s down (and I’ve had pretty good luck with them so far!) and installing a LAMP stack on Ubuntu is stupid easy these days:

Tack on phpmyadmin and you’re up and running.

But what about buying an SSL cert? He’s not going to want to pay for that, but being he will probably be selling from his site (currencyelixirs.com if your interested) SSL is a MUST! Plus being I will pretty much be handing this off to him as a turn key WordPress site, he (and I) won’t want to renew those certs every year. What to do?

Well I did the standard self signed cert (good tutorial here.) for now, and then it hit me. Cloudflare offer’s their flexible ssl for free!

For a quick rundown for those unaware, Cloudflare sit’s between standard web-traffic and your web-server. It does this by becoming the name server for your domain, and serving out addresses to web-servers that hold cached content your site.

They also offer what they call a flexible ssl option (still on the free plan which rocks!) that will provide ssl from the end user to Cloudflare’s cached copy. For increasing your rank in google searches, this will be enough, and the cert that is provided for your site IS valid!

However the connection from Cloudflare to your web-server is still plaintext which, if your thinking of doing transactions online is probably the biggest no-no you could make.

BUT! If you followed the instructions above and enabled a self signed cert and ssl from your web-server, you can enable the Full SSL mode in Cloudflare. Cloudflare will still serve out it’s legitimate and approved certificate, just like with the flexible SSL while at the same time ignoring any warning’s about your certificate being self signed, and still use ssl to connect to your web-server!

2015-02-28 18_04_46-CloudFlare Settings _ CloudFlare _ The web performance & security company

This is true end to end encryption, and I’ll argue it’s as secure as using signed certificates, as long as you take the proper precautions and set the right permissions on the key’s used.

Once that’s done, just create a page rule in Cloudflare to force SSL and you’re done!

2015-02-28 18_19_53-Page Rules for currencyelixirs.com _ CloudFlare _ The web performance & security

That’s it! You’re visitors get a nice valid cert on all of the content to your site which is now fully https, even though your using a self signed cert on your web-server!

Very Cool!

Now my site will take some more thinking and work, and ultimately I think I’ll be forced to use a valid cert as I’m using Amazon’s ELB’s and I’m not sure how well they would like a self signed cert, but I’m curious to try!

(If my site’s offline next weekend, now you know why!)

Debian Minecraft Server using ram disk

Alright time for a more involved post, in this one I’ll be taking a vanilla Debian server, installing java-7 setting up the ramdisk, and then sharing the /etc/init.d file that will not only start the server as soon as the system is up and running, but also allow you to control it like you would standard Linux applications via init.d.

It sounds like a lot, but it’s not so bad, especially because several other people have done most of the work I’m just going to combine it all into one post with the changes I’ve found helpful along the way.

I’ll also get out of the way that in this example I’m assuming you have a server setup that you can login to and have sudo access to at the least. And also that I’ll be using a heavily modified and absolutely AWESOME Minecraft server called Icehenge by The MindCrafters in this example, but the steps would be the same for vanilla Minecraft, just change the names as you see fit.

First let’s get a needed app called “screen” and also the pesky Java7 installed. If your going to be running older Minecraft versions, DON’T go up to Java8 or you will get errors! The instructions below are based off of this Web UpD8 post and uses their repo’s to make the install easier and updated.

Run each line below by itself.

Accept the Java license if you agree to their terms (like your going to read them…) and your done! Java should be installed! Check it with this command:

You should get output that looks like this:

See! Told you it was easy! On to the ramdisk!

You can place your ramdisk anywhere or even just use /dev/shm but I like putting it in good ‘ol /mnt/ so that’s what we’re going to do I’ll also be making another folder for backups of the world because if you don’t know I guess I should say this just to avoid someone getting pissed when they lose their world on their first reboot, Anything and Everything stored in your ramdisk will NOT survive a reboot or power loss! So backups are CRITICAL if your going to be using a ramdisk setup. There you’ve been warned. (and we’ll cover this as well a bit later in this post.)

Next let’s edit our fstab so that on boot, this ramdisk will be mounted to /mnt/ramdisk automatically, and since we’ll be in there now’s as good a time as any to make a mount for your backups as well if you need to. In my case I do as I’ll be using an NFS mount for my backups.

Also Yes I use nano, and I have some good reason’s I’ll defend in a different post, but it’s not required, use your text editor of choice.
Add the following to the bottom of /etc/fstab, omitting or changing the line’s for the NFS mount as it suits your needs.

As for the ramdisk, make sure it’s large enough that you can fit your whole Minecraft folder inside while still allowing ram for the server to run! You can change it’s upper limit using the size option. In my example above it’s 2048m and more then enough. Now we can test the fstab by running:

if you get errors, fix them before rebooting your server, or make sure to add the “nofail” option to the ramdisk as well or it might not reboot!

If everything worked you should be able to list the directory and you’ll see that the ramdisk is now chmod’d to 777 and both mount’s are working and mounted

Chalk another step off the board, your ramdisk is complete! Go ahead and reboot if you don’t believe me I’ll wait!

I’m also going to assume you already have the flavor of Minecraft server you plan on hosting downloaded and extracted to your /home/username/ folder as most guides from the downloads page will have already told you how to do this, I’ll skip it.

Now on to the init script. The basis of which can be found on this post by Aimless Bits. There’s also one that adds the command option with the ramdisk changes by an unknown poster here. I’ve changed that code a bit to work for my setup using Icehenge below.

First let’s get it created and opened, it will live in /etc/init.d/ and in my case it’s called Icehenge.

The lines that you need to change to work for you are 18-25. I store my server at /home/daniel/icehenge as referenced by line 20, and the world name on 25 needs to be set to the same name as your Minecraft server.properties file is set to.

Yea, there’s a lot there.

Before this script will run we need to make it executable (by normal users as well), and tell Debian to register it as part of the startup/shutdown routine.

The main commands you’ll want to know for now, are start, stop, backup and the ever important disksaverun. If your not running vanilla Minecraft, DON’T use the update function!

You can invoke it 2 different ways like so:

OK! We’re almost ready to play! At this point you should be able to start and stop your server, but we still need to address the ever important backups! Don’t skip this step! Again, you’ve been warned!

As the user that you have defined in your init script, edit that user’s crontab

and add the following 2 lines to the bottom of the file.

I chose these time’s as I figured I could live with losing up to 10 minutes of minecrafting without flipping out. I also staggered the time’s so that the backups don’t run over the disksaverun call’s to minimize any lag caused during the compression of the backups.

That’s it! your server should now be complete!

A few things to note. Once the Minecraft server is running, you can login to the Minecraft console by joining it’s screen session like so

disconnect from the server console screen session by holding down “ctrl+A” and typing “D” and you will be back on the cli.

Enjoy your Minecraft/Icehenge server and get to digging!

 

Script to start opentracker and unreal at boot time

So being I’m now parsing out my services after breaking down my hosting environment, I’m left with some services that I run having to be manually started every time I reboot a server. While that’s not a huge deal as they are all running Debian Linux, reboot’s are few, however, why take the chance of forgetting or having to manually do that when we can script all the things?!

(I also want to check out the new syntax highlighter plugin!)

So here is a startup script that I have tied to my rc.local file. It will start my opentracker bit torrent tracker (add it to your torrents! udp://tracker.blackunicorn.xyz:6969) as well as start up Unreal IRCD in this example a leaf server for my own irc network. I’ve also used my name, but I recommend creating a user with as few priv’s as possible to actually run this as. This is for example only.

You will also note that opentracker is run in a screen session that the user can rejoin using the command:

Set it as executable and your off to the races!

Because of the checking you can also run this anytime and it will report if either of these services are already running. Enjoy!

New Environment Speed Test

So being I’ve built a completely new hosting environment I of course wanted to see how it was performing, so I used the pingdom tools to do a speedtest on my page loading.

Granted, it’s a very basic site without a lot of content yet, but still:

speed-test

This site is loading faster then 71% of all sites ever tested at pingdom! Why is that exciting to me? Well, for starters, this site is running off of a t2.micro aws instance!

That’s 1 (small) core for compute, and 1 gig of ram! Very tiny box, with amazing results! If only the aws elb would decide on an ip range so that dns resolved faster and I bet it would load even faster!

For reference, this is what makes up this site so far, (not counting the RDS doing the mysql load, but it too is the smallest instance size they offer.):

aws-reference

I’ll provide some networking diagram’s on how everything is setup, and isolated from the public to increase security later this week hopefully!

 

*Update*

I’ve now fixed the ip issue by moving over to route 53 for my nameservers as they use a special alias “A” record that allows you to point a domain at their elb dns name and have it work as you would expect.

Looks like google think’s I’m doing alright too!

Selection_051

2015 Kicks off a brand new start!

So many Changes!

First off you’ll notice that all of my old posts are gone. Yes I still have them and maybe someday I’ll get around to importing them, but I’m kind of liking the idea of a fresh start. You see, that old blog, while it did have some good info in there, was a younger, less cautious person, so I can’t safely say I still feel the same way about all of my posts, (a lot can happen in 8 years!)

Also as some of you know, I’m no longer doing hosting, I have moved on from that part of my life as well, I need to focus more on myself, and let’s be honest, I was spending too much money keeping that setup running then I ever made off of it. You just can’t compete with the big guys these days!

I’m also much more adept on setting up servers and networks now then I was back then, and I’m able to implement the things I’ve learned into this new environment, and sometimes, it’s just easier to start fresh then patch a band-aid on some duct-tape on some gum being held tight by some bailing wire. (OK, so the last server wasn’t in THAT bad of shape, but you get the drift) Plus after being a physical server, going through a p2v conversion, uplifted into vm-ware, then a few years later, converted again into an amazon ami, small random errors started happening that I just didn’t want to bother troubleshooting.

I also plan on keeping the content more relevant to a tech theme instead of a random collection of randomness.

So yea, enough of the excuses, long story short, I have my reason’s!

So have a happy 2015, and I hope this year treats you all well!