Lessons learned using Freenas

Well I have  had my freenas machine  up and running a good solid year at this point. I am sure I have said this in the past, but during the planning phase before even purchasing take the following into considering :

1. Make Sure you buy the amount of drives needed to start off with
2. Consider type you array you are using ( Z1, Z2,and Z3 )
3. How much space you currently need to backup all data
4. Have backups already in place if data needs to be moved
5. Plan for future need of space and over estimate
6. Don’t cheap out on hard drives you will regret it later
7. Also dont’ cheap on hard ware either ( there is a reason there is hardware recommend guides out there)
8.  Plan out your data sets and sharing  structure before you start as well

I made a few mistakes along the way. I choose to start with 6 4tb drives and 6 3tbs drives in two separate 6 hard drive Z2 arrays. From the start that is a bad thing to do since that makes your arrays lop sides and can cause performance issues down the road. Also from the start don’t use used drives that don’t have a a lot of life span left.  I made this mistake and had instant regret. I don’t think I made it more than a few weeks before all of my 3tb hard drives started to  die very quickly. I was then forced into rushing out buying brand new hard drives which I wasn’t ready to purchase. Also I had no backups in place at this point since I was taking to raid arrays and combining them into one storage machine. I was short on funds and should have waited until I had all needed parts first. Another very large mistake I made looking back. I was then forced with buying 6 brand new 4tb hard drives and hoping that i don’t have more than two hard drives die ( sweated bullets )  for about week. Each resilver took about 5-10 hours doing one at a time. Soon as i resilvered one failed drive another one died. Somehow the storage gods smiled upon me and  I made it through that giant mess.

Now lets roll forward to present time.  I am sitting here with two 6 hard drive Z2 arrays  ( meaning I loose two drives per pool to parity) I am getting email alerts about going over my capacity. There is a rule of thumb in ZFS and Freenas to not go over 80% capacity of your pool. This isnt a set in stone rule, but more of  rule of them to go buy. In all honesty you could probably going to about 85% before you really start to see your performance start to drop off.  I am sitting here at this point wondering what to do. I really cant afford to go out and buy 12 brand new hard  drives that would hurt wallet way to much. I then made the drastic decision to nuke my pools and recreate a 12 hard drive Z2 ( I can hear the community screaming in horror at this point). I understand completely that having that wide of a array is somewhat risky.  I then considered I have two complete backups of all my data . I felt pretty safe considering one backup was in a separate array in a completely different system and one was sitting in a mostly offline cold storage usb enclosure. Well at this point I pretty much blew a weekend killing my array, recreating the reconfigured array, and copying back roughly 24tbs in data from backups.

I  hope that sharing my over a experience is a lesson learned to others thinking about getting into freenas as their main storage system. Hopefully those digging in wont make the same mistakes I did.