Well I have had my freenas machine up and running a good solid year at this point. I am sure I have said this in the past, but during the planning phase before even purchasing take the following into considering :
- Make Sure you buy the amount of drives needed to start off with
- Consider type you array you are using ( Z1, Z2,and Z3 )
- How much space you currently need to backup all data
- Have backups already in place if data needs to be moved
- Plan for future need of space and over estimate
- Don’t cheap out on hard drives you will regret it later
- Also dont’ cheap on hard ware either ( there is a reason there is hardware recommend guides out there)
- Plan out your data sets and sharing structure before you start as well
I made a few mistakes along the way. I choose to start with 6 4tb drives and 6 3tbs drives in two separate 6 hard drive Z2 arrays. From the start that is a bad thing to do since that makes your arrays lop sides and can cause performance issues down the road. Also from the start don’t use used drives that don’t have a a lot of life span left. I made this mistake and had instant regret. I don’t think I made it more than a few weeks before all of my 3tb hard drives started to die very quickly. I was then forced into rushing out buying brand new hard drives which I wasn’t ready to purchase. Also I had no backups in place at this point since I was taking to raid arrays and combining them into one storage machine. I was short on funds and should have waited until I had all needed parts first. Another very large mistake I made looking back. I was then forced with buying 6 brand new 4tb hard drives and hoping that i don’t have more than two hard drives die ( sweated bullets ) for about week. Each resilver took about 5-10 hours doing one at a time. Soon as i resilvered one failed drive another one died. Somehow the storage gods smiled upon me and I made it through that giant mess.
Now lets roll forward to present time. I am sitting here with two 6 hard drive Z2 arrays ( meaning I loose two drives per pool to parity) I am getting email alerts about going over my capacity. There is a rule of thumb in ZFS and Freenas to not go over 80% capacity of your pool. This isnt a set in stone rule, but more of rule of them to go buy. In all honesty you could probably going to about 85% before you really start to see your performance start to drop off. I am sitting here at this point wondering what to do. I really cant afford to go out and buy 12 brand new hard drives that would hurt wallet way to much. I then made the drastic decision to nuke my pools and recreate a 12 hard drive Z2 ( I can hear the community screaming in horror at this point). I understand completely that having that wide of a array is somewhat risky. I then considered I have two complete backups of all my data . I felt pretty safe considering one backup was in a separate array in a completely different system and one was sitting in a mostly offline cold storage usb enclosure. Well at this point I pretty much blew a weekend killing my array, recreating the reconfigured array, and copying back roughly 24tbs in data from backups.
I hope that sharing my over a experience is a lesson learned to others thinking about getting into freenas as their main storage system. Hopefully those digging in wont make the same mistakes I did.