I posted earlier of my love for ZFS Dedup, and how I was experimenting with OpenSolaris as a new base-unix instead of FreeBSD, because FreeBSD is so far behind in the ZFS ports.
Well, I'm switching back. It's been a few months, and we still don't see any new OpenSolaris code. Will Oracle release a new OpenSolaris? There's talk about it being dumped. There's too many questions, so I have decided to side-line and wait - so I'm back to FreeBSD until I see an OpenSolaris 2010 - But it's quite possible we'll have FreeBSD Dedup before we have another solid OpenSolaris release.
Tuesday, August 24, 2010
Quick and Dirty: FreeBSD 8.1 AHCI vs ATA (CAM)
While building and testing what will be a 24 drive SAN running FreeBSD 8.1, ZFS, and NFS/iSCSI I discovered a problem with hot-swapping the SATA drives.
It seems FreeBSD 8.1 AMD64 was running the SATA drives in ATA mode, so they really didn't hot-swap. If I pulled a drive and reinserted, I couldn't make it understand that the drive was back, even if I fooled with atacontrol's attach and detach. I had to reboot, which is no way to run a SAN, no mater how ghetto it may be.
A bit of poking showed that while my BIOS was set to AHCI (Intel ICH7R chipset), FreeBSD was still running in ATA mode. (type 'atacontrol list' and if you see drives, you are too). camcontrol is the program you use once in CAM mode.
The answer was to put ahci_enable="YES" in /boot/loader.conf.
This changes your drives to ada's from ad's which causes a boot problem, but that's easily fixed.
I did a quick a dirty test with 'raidtest' to show the speed difference. Here's my raidtest creation command:
raidtest genfile -s 128 -S 512 -n 50000
Here's the results:
With CAM: (ahci_enable="YES" in /boot/loader.conf)
iscsi# raidtest test -d /dev/zvol/tank/vol
Read 50000 requests from raidtest.data.
Number of READ requests: 24831.
Number of WRITE requests: 25169.
Number of bytes to transmit: 3286713344.
Number of processes: 1.
Bytes per second: 37162958
Requests per second: 565
Without CAM
iscsi# raidtest test -d /dev/zvol/tank/vol
Read 50000 requests from raidtest.data.
Number of READ requests: 24831.
Number of WRITE requests: 25169.
Number of bytes to transmit: 3286713344.
Number of processes: 1.
Bytes per second: 6069384
Requests per second: 92
Big difference eh? It's night and day. It then lead to some ZFS issues when I turned CAM on, because ZFS likes to hit write/read stalls when you move a lot of data, but I cleared that up with some further ZFS tweaks that I will detail another day.
If you're not running with ahci_enable="yes" in your loader.conf, you may want to look at enabling it.
I"ll be doing more tests over the new few days on a few different FreeBSD machines, and we'll see what the general results are.
It seems FreeBSD 8.1 AMD64 was running the SATA drives in ATA mode, so they really didn't hot-swap. If I pulled a drive and reinserted, I couldn't make it understand that the drive was back, even if I fooled with atacontrol's attach and detach. I had to reboot, which is no way to run a SAN, no mater how ghetto it may be.
A bit of poking showed that while my BIOS was set to AHCI (Intel ICH7R chipset), FreeBSD was still running in ATA mode. (type 'atacontrol list' and if you see drives, you are too). camcontrol is the program you use once in CAM mode.
The answer was to put ahci_enable="YES" in /boot/loader.conf.
This changes your drives to ada's from ad's which causes a boot problem, but that's easily fixed.
I did a quick a dirty test with 'raidtest' to show the speed difference. Here's my raidtest creation command:
raidtest genfile -s 128 -S 512 -n 50000
Here's the results:
With CAM: (ahci_enable="YES" in /boot/loader.conf)
iscsi# raidtest test -d /dev/zvol/tank/vol
Read 50000 requests from raidtest.data.
Number of READ requests: 24831.
Number of WRITE requests: 25169.
Number of bytes to transmit: 3286713344.
Number of processes: 1.
Bytes per second: 37162958
Requests per second: 565
Without CAM
iscsi# raidtest test -d /dev/zvol/tank/vol
Read 50000 requests from raidtest.data.
Number of READ requests: 24831.
Number of WRITE requests: 25169.
Number of bytes to transmit: 3286713344.
Number of processes: 1.
Bytes per second: 6069384
Requests per second: 92
Big difference eh? It's night and day. It then lead to some ZFS issues when I turned CAM on, because ZFS likes to hit write/read stalls when you move a lot of data, but I cleared that up with some further ZFS tweaks that I will detail another day.
If you're not running with ahci_enable="yes" in your loader.conf, you may want to look at enabling it.
I"ll be doing more tests over the new few days on a few different FreeBSD machines, and we'll see what the general results are.
Monday, August 2, 2010
ZFS on FreeBSD 8.1
I've been using ZFS on my FreeBSD box for some time now, and it's been serving out gigs and gigs of data faithfully for some time.
I am looking into OpenSolaris, but with the slow (or never) release of 2010.x I'm still fully a FreeBSD shop at this stage for Unix.
I've just upgraded to 8.1 Release a few days ago, and I'm liking the new release - I took the opportunity to also upgrade my Samba to 3.4.8 at the same time, so the system seems perkier. I didn't have the time to do any real speed tests, so take that for what it's worth.
BUT - I've been getting some "kmem_map to small" panics when I tried to copy down a 160 gig file. I can't recall if I've ever moved such a large file before, so I don't think this is an 8.1 issue.
Some digging around and I found this information on the FreeBSD mailing list that I wanted to pass along.
From: Steve Polyack
Subject: Re: Freebsd 8.0 kmem map too small
Date: Wednesday, May 5, 2010 - 7:46 am
On a system here with 8GB of RAM and a very large zpool consisting of
multiple zdevs, little tuning was needed. The system is running
8-STABLE(amd64) as of a month or two ago. The only things I have set in
/boot/loader.conf are:
vm.kmem_size="12G"
vfs.zfs.arc_max="4G"
Setting kmem_size to 12G came from a combination of recommendations I
saw in various mailing list posts (you can probably dig them up).
Setting it to the physical memory size was the initial recommendation,
while others recommended 1.5x physical memory size to help prevent
fragmentation/wasted space in kmem.
Regardless, this has served us quite well for the ~6 months the system
has been in use. It has never crashed, even under intensive
multi-threaded benchmarking.
Other people recommend vm.kmem_size to be 1/2 of available RAM, and then vsf.zfs.arc_max to be 512M less than that, but that doesn't make sense to me - What's the point of RAM if you can't use it? This box only runs ZFS and Samba, let these things have some memory.
I've now set those same values as Steve on my FreeBSD system in question is running 8 gigs as well. I'm going to start my 160 gig copy again (which BTW, I get ~34 Meg/Sec according to Windows 7)
If you don't hear back from me, it's working well for me. :-)
I am looking into OpenSolaris, but with the slow (or never) release of 2010.x I'm still fully a FreeBSD shop at this stage for Unix.
I've just upgraded to 8.1 Release a few days ago, and I'm liking the new release - I took the opportunity to also upgrade my Samba to 3.4.8 at the same time, so the system seems perkier. I didn't have the time to do any real speed tests, so take that for what it's worth.
BUT - I've been getting some "kmem_map to small" panics when I tried to copy down a 160 gig file. I can't recall if I've ever moved such a large file before, so I don't think this is an 8.1 issue.
Some digging around and I found this information on the FreeBSD mailing list that I wanted to pass along.
From: Steve Polyack
Subject: Re: Freebsd 8.0 kmem map too small
Date: Wednesday, May 5, 2010 - 7:46 am
On a system here with 8GB of RAM and a very large zpool consisting of
multiple zdevs, little tuning was needed. The system is running
8-STABLE(amd64) as of a month or two ago. The only things I have set in
/boot/loader.conf are:
vm.kmem_size="12G"
vfs.zfs.arc_max="4G"
Setting kmem_size to 12G came from a combination of recommendations I
saw in various mailing list posts (you can probably dig them up).
Setting it to the physical memory size was the initial recommendation,
while others recommended 1.5x physical memory size to help prevent
fragmentation/wasted space in kmem.
Regardless, this has served us quite well for the ~6 months the system
has been in use. It has never crashed, even under intensive
multi-threaded benchmarking.
Other people recommend vm.kmem_size to be 1/2 of available RAM, and then vsf.zfs.arc_max to be 512M less than that, but that doesn't make sense to me - What's the point of RAM if you can't use it? This box only runs ZFS and Samba, let these things have some memory.
I've now set those same values as Steve on my FreeBSD system in question is running 8 gigs as well. I'm going to start my 160 gig copy again (which BTW, I get ~34 Meg/Sec according to Windows 7)
If you don't hear back from me, it's working well for me. :-)
Wednesday, June 9, 2010
ZFS Dedeuplication
Oh my god - Is anyone else as excited about this as I am?
Are you thinking of a SAN running a massive ZFS raidz2 in deduplication mode?
I've already built a 6tb raidz2 array last year, and I'm finding I need more space - next shot will be for 12 or 16 tb, and I'd love to use dedup to help me out.
I'm so curious about it, I'm considering switching to Solaris as my disk host. Normally I'd use FreeBSD, but their ZFS is too far behind (version 13, Sun's got Version 22) to support dedup in the next 2 years.
I'll be posting some test results soon.. just experimenting with some 4 gig .OST files under a ZFS dedup scenario to get a feel for compression.
Early tests with straight MP3's showed 23% compression. These are very, very promising times.
Here's a few links you can drool over until you get the time to do something, or I post my results:
http://blogs.sun.com/jsavit/entry/deduplication_now_in_zfs
http://blogs.sun.com/bonwick/entry/zfs_dedup
http://en.wikipedia.org/wiki/Data_deduplication
Are you thinking of a SAN running a massive ZFS raidz2 in deduplication mode?
I've already built a 6tb raidz2 array last year, and I'm finding I need more space - next shot will be for 12 or 16 tb, and I'd love to use dedup to help me out.
I'm so curious about it, I'm considering switching to Solaris as my disk host. Normally I'd use FreeBSD, but their ZFS is too far behind (version 13, Sun's got Version 22) to support dedup in the next 2 years.
I'll be posting some test results soon.. just experimenting with some 4 gig .OST files under a ZFS dedup scenario to get a feel for compression.
Early tests with straight MP3's showed 23% compression. These are very, very promising times.
Here's a few links you can drool over until you get the time to do something, or I post my results:
http://blogs.sun.com/jsavit/entry/deduplication_now_in_zfs
http://blogs.sun.com/bonwick/entry/zfs_dedup
http://en.wikipedia.org/wiki/Data_deduplication
Sunday, January 10, 2010
ConnectWise - Taking the Plunge
Okay, after a little more than a year on CommitCRM as our main CRM/PSA package, I've taken the plunge and bought ConnectWise.
CW was never really considered when I was looking at CommitCRM and other packages last year - it was just too much (My current 4 user license with my features that I want is $15k USD). CommitCRM was just $3k, and it did everything that I needed last year.
CommitCRM is still a great little package - but that's just it - it's a little package. It will continue to grow as it's developed, but I need more (like project management) and I need it now, so it's time to switch.
I was considering TigerPaw for a while, and almost bought - but CW has more abilities that I needed - and you pay for them. TigerPaw was cheaper by nearly half.
In the end, the ability to work efficiently is worth more to me than holding on to the $$. We can make so much more if we have focused and properly utilized staff.
It's taking some getting used to - CommitCRM is a small, tight, well written program. It doesn't require much in the way of power to run, and it's client auto-installs when you run - All very easy.
ConnectWise is the exact opposite - It's a full Microsoft .NET development program - Requiring full MS SQL server, a dedicated Windows server, and the client runs via a .NET install (means you better have some workstation power).
I added a $14k Dell server to my CW cost, so I'm now up to nearly $30k for this package. You need a honking dedicated server, because they won't let you run it on anything else.
I'll write more about ConnectWise as I learn - I'm still in the initial training and getting used to the program - It's very complex, and it's taken me nearly a week to get to the stage where I can dribble out an invoice or two.
CW was never really considered when I was looking at CommitCRM and other packages last year - it was just too much (My current 4 user license with my features that I want is $15k USD). CommitCRM was just $3k, and it did everything that I needed last year.
CommitCRM is still a great little package - but that's just it - it's a little package. It will continue to grow as it's developed, but I need more (like project management) and I need it now, so it's time to switch.
I was considering TigerPaw for a while, and almost bought - but CW has more abilities that I needed - and you pay for them. TigerPaw was cheaper by nearly half.
In the end, the ability to work efficiently is worth more to me than holding on to the $$. We can make so much more if we have focused and properly utilized staff.
It's taking some getting used to - CommitCRM is a small, tight, well written program. It doesn't require much in the way of power to run, and it's client auto-installs when you run - All very easy.
ConnectWise is the exact opposite - It's a full Microsoft .NET development program - Requiring full MS SQL server, a dedicated Windows server, and the client runs via a .NET install (means you better have some workstation power).
I added a $14k Dell server to my CW cost, so I'm now up to nearly $30k for this package. You need a honking dedicated server, because they won't let you run it on anything else.
I'll write more about ConnectWise as I learn - I'm still in the initial training and getting used to the program - It's very complex, and it's taken me nearly a week to get to the stage where I can dribble out an invoice or two.
Thursday, December 10, 2009
FreeBSD 8.0
I've been waiting with baited breath for this to come out of it's RC cycle, and now it's finally here.
I deployed a few RC1, 2 and 3 copies as test platforms doing minor tasks, and was impressed with the snappiness of the OS in general. I really feel there is a better networking stack underneath the system now.
With 8.0 out, it's just a simple matter to use
freebsd-update -r 8.0-RELEASE upgrade
follow the instructions, two reboots, and you're done.
I'm liking it more and more as the years go on. Glad I found BDSi years ago, which started me on the BSD route instead of being pulled into one of the many Linux camps.
I deployed a few RC1, 2 and 3 copies as test platforms doing minor tasks, and was impressed with the snappiness of the OS in general. I really feel there is a better networking stack underneath the system now.
With 8.0 out, it's just a simple matter to use
freebsd-update -r 8.0-RELEASE upgrade
follow the instructions, two reboots, and you're done.
I'm liking it more and more as the years go on. Glad I found BDSi years ago, which started me on the BSD route instead of being pulled into one of the many Linux camps.
Friday, March 20, 2009
Benchmarks? FreeBSD ZFS vs GEOM vs Hardware Speed
I'm currently running through some disk benchmarks to determine the speed of various ZFS arrays in FreeBSD 7.1 compared to GEOM or Hardware arrays.
I'm selecting bonnie and dbench for my testing at the moment - any other benchmarks come to mind that would be useful?
I'm selecting bonnie and dbench for my testing at the moment - any other benchmarks come to mind that would be useful?
Wednesday, March 11, 2009
ZFS on FreeBSD 7.1
I've been flirting with ZFS on FreeBSD 7.1-RELEASE AMD64, and I must say I like it so far.
I'm hoping and dreaming of using it as my main filestore in a 'raidz' array of 1.5 TB drives to give me a massive temporary file store for my ever increasing Video and Music files.
Why ZFS?
I'm excited about ZFS mostly for being transactional, and for the raidz support - This is an enhanced version of RAID-5. I could have ran a GENOM RAID-5 array, but when you add in the other features of ZFS (and it has many - go check) it's very tempting to go with ZFS instead.
Unfortunately it's still experimental in FreeBSD, which may have some sticking with more tested technologies.
I should give a few tips if you're trying out ZFS
1) Don't bother with i386
You can run ZFS under i386 with some tweaks, but why? It's going to be limited, and I think you're asking for trouble. ZFS needs a lot of memory address space. Give it 64 Bits and you just have to put
zfs_enable="YES"
in /etc/rc.conf to make it start up.
2) Give it some power
I love using old hardware to run FreeBSD servers, but ZFS needs some oomph to run proprly. Sure people are running it in i386 with 384 Meg, but why? You can grab a decent Socket 775 board and 64 bit chip for $120, throw in 2 gigs of RAM for $60, and you're still under $200
3) Give it respect
This isn't UFS. It's still experimental, don't put your only copy of critical data on it. I'm going to put things that I really don't want to lose - MP3's and Videos, but not accounting data or similar. Will the world end if I lost it? No, but I'd be pretty pissed, so you know I'm doing a lot of testing on my hardware so I know how to recover from faults. Learn how to use it if you're going to want any dependability from it.
4) Don't try and boot from it
It won't. There are tricks to make the /usr and other partitions ZFS and the main boot on UFS, but why frig around and play with fire? We all have an old 80 or 40 gig HD around. Make that your UFS boot, and setup your other drives as dedicated ZFS.
5) Speed
In a very slap-dash test, my GEOM stripe was delivering data via samba at the same speed as my ZFS raidz pool. In some cases, even faster. Of course I'd have to do proper tests to make the playing field level, so this is just a quickie test to see if a ZFS raidz could hold up in speed to my GEOM stripe on my box for samba network use, and it does that just fine. It may suck for a MySQL db, but I haven't tested that yet.
6) Moving to different hardware
- If you build your pool on /dev/ad0, /dev/ad2, ad4 (etc) and then move to a new controller that is discovered as da0, da1, etc. You simply do this;
zpool tank export
zpool tank import
presto. My ad* based zpool is now showing up on my da* based controller.
7) Failure Redundancy
I setup my raidz, transferred a few gig of data to it, then pulled a drive while it was running. A few complaints on the console, but my data was still there. Speed seemed okay, and data was intact without a lot of rebuilding necessary. I later on rebooted and reconnect the drive, added it back into the pool, and life went on.
GEOM is annoying after a crash as it has to check the entire array before it lets you back in. On a 1.5 TB array, that takes some time. With ZFS, you can still fire up and access data.
You can also run a background check by typing
zfs scrub
No more off-line fscks! Schedule a low-priority scrub to run in the background, and you're ensuring your data is always clean and matching.
This simple crash test proved ZFS can do two main things for me ; 1) Notify me when something fails, and 2) accept the drive back after it's been fixed.
I now need to see if I pull a drive, format it, and put it back if it accepts it properly and rebuilds the array on it. One would hope, but I'd like to manually test it before putting all that media on it.
Conclusions:
Get FreeBSD 7.1-RELEASE AMD64 and grab 2 or more drives and play with it. Check the links below for setup info - it's really quite simple. I'm loving it, and I guess I will until it screws me over. :-)
Links for more info:
http://blog.thefrog.net/2008/04/zfs-on-freebsd.html
(Top link references the two bottoms ones which I used to setup my ZFS)
http://wiki.freebsd.org/ZFSQuickStartGuide
http://wiki.freebsd.org/ZFSTuningGuide
I also recommend
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide
http://kerneltrap.org/FreeBSD/BSDCan_2008_ZFS_Internals
http://opensolaris.org/os/project/sdosug/past_meetings/ZFS-general.20070919s.pdf
http://www.tech-recipes.com/rx/1405/zfs-how-to-fsck-or-check-filesystem-integrity-with-scrub/
I'm hoping and dreaming of using it as my main filestore in a 'raidz' array of 1.5 TB drives to give me a massive temporary file store for my ever increasing Video and Music files.
Why ZFS?
I'm excited about ZFS mostly for being transactional, and for the raidz support - This is an enhanced version of RAID-5. I could have ran a GENOM RAID-5 array, but when you add in the other features of ZFS (and it has many - go check) it's very tempting to go with ZFS instead.
Unfortunately it's still experimental in FreeBSD, which may have some sticking with more tested technologies.
I should give a few tips if you're trying out ZFS
1) Don't bother with i386
You can run ZFS under i386 with some tweaks, but why? It's going to be limited, and I think you're asking for trouble. ZFS needs a lot of memory address space. Give it 64 Bits and you just have to put
zfs_enable="YES"
in /etc/rc.conf to make it start up.
2) Give it some power
I love using old hardware to run FreeBSD servers, but ZFS needs some oomph to run proprly. Sure people are running it in i386 with 384 Meg, but why? You can grab a decent Socket 775 board and 64 bit chip for $120, throw in 2 gigs of RAM for $60, and you're still under $200
3) Give it respect
This isn't UFS. It's still experimental, don't put your only copy of critical data on it. I'm going to put things that I really don't want to lose - MP3's and Videos, but not accounting data or similar. Will the world end if I lost it? No, but I'd be pretty pissed, so you know I'm doing a lot of testing on my hardware so I know how to recover from faults. Learn how to use it if you're going to want any dependability from it.
4) Don't try and boot from it
It won't. There are tricks to make the /usr and other partitions ZFS and the main boot on UFS, but why frig around and play with fire? We all have an old 80 or 40 gig HD around. Make that your UFS boot, and setup your other drives as dedicated ZFS.
5) Speed
In a very slap-dash test, my GEOM stripe was delivering data via samba at the same speed as my ZFS raidz pool. In some cases, even faster. Of course I'd have to do proper tests to make the playing field level, so this is just a quickie test to see if a ZFS raidz could hold up in speed to my GEOM stripe on my box for samba network use, and it does that just fine. It may suck for a MySQL db, but I haven't tested that yet.
6) Moving to different hardware
- If you build your pool on /dev/ad0, /dev/ad2, ad4 (etc) and then move to a new controller that is discovered as da0, da1, etc. You simply do this;
zpool tank export
zpool tank import
presto. My ad* based zpool is now showing up on my da* based controller.
7) Failure Redundancy
I setup my raidz, transferred a few gig of data to it, then pulled a drive while it was running. A few complaints on the console, but my data was still there. Speed seemed okay, and data was intact without a lot of rebuilding necessary. I later on rebooted and reconnect the drive, added it back into the pool, and life went on.
GEOM is annoying after a crash as it has to check the entire array before it lets you back in. On a 1.5 TB array, that takes some time. With ZFS, you can still fire up and access data.
You can also run a background check by typing
zfs scrub
No more off-line fscks! Schedule a low-priority scrub to run in the background, and you're ensuring your data is always clean and matching.
This simple crash test proved ZFS can do two main things for me ; 1) Notify me when something fails, and 2) accept the drive back after it's been fixed.
I now need to see if I pull a drive, format it, and put it back if it accepts it properly and rebuilds the array on it. One would hope, but I'd like to manually test it before putting all that media on it.
Conclusions:
Get FreeBSD 7.1-RELEASE AMD64 and grab 2 or more drives and play with it. Check the links below for setup info - it's really quite simple. I'm loving it, and I guess I will until it screws me over. :-)
Links for more info:
http://blog.thefrog.net/2008/04/zfs-on-freebsd.html
(Top link references the two bottoms ones which I used to setup my ZFS)
http://wiki.freebsd.org/ZFSQuickStartGuide
http://wiki.freebsd.org/ZFSTuningGuide
I also recommend
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide
http://kerneltrap.org/FreeBSD/BSDCan_2008_ZFS_Internals
http://opensolaris.org/os/project/sdosug/past_meetings/ZFS-general.20070919s.pdf
http://www.tech-recipes.com/rx/1405/zfs-how-to-fsck-or-check-filesystem-integrity-with-scrub/
Sunday, March 8, 2009
MSP Offerings: Zenith Infotech in more detail
I signed up for Zenith's trial over a month ago, and only had a few hours to play with it before running out of research time. My life is pretty tightly scheduled so I had to shelve my MSP demo for almost a month.
I'm back to it now, but my trial expired. No big deal. Zenith is dirt cheap to use. Check these prices:
http://www.saaztraining.com/zenith deliverables 2007.pdf
So I just signed up and am now paying a few bucks a month to continue to try out Zenith. I'm also now seeing Zenith from the side of a "real user", not a "prospective user" so I should be seeing the true system, not the salesman-enhanced side of things.
I like the fact that I'm not pressured to buy anything big upfront. It's all SAAS, so there is next to no requirement on my end, other than using IE (Firefox doesn't work) for their management console.
True, with SAAS you're always paying for it. But remember; software is always evolving. I have a few products I can keep running for a few years without upgrading, but not too many. Usually you're always in need of the new features or the bug fixes, or the new platform support that the new version brings.
I'm starting to become very aware of the internal costs we pay for maintaining our our software - The time we spend frigging with our servers, setting them up, maintaining them, etc. That time is far more productive being spent on customer's needs, which can be billed. Part of me is a bit apprehensive about the lack of control, but most of me is happy that I don't have to deal with putting it all back together if it crashes.
So what do I like so far about Zenith?
Well, I like the reports - Very similar to what I generate for my clients, except it's all there. I can output nearly the same thing with 5 clicks, and I'm done. This task is simple enough that it can now be given to the admin staff, not needing a technician to build the reports.
The dashboard/overview of what is going on with my servers is rather simple, but you know, it tells me the main points I need. I'm sure this is customizable, I'm still in the process of figuring that out.
I will keep learning this package and then write a better report on it soon. Once that's done, I'm off to check out Kaseya, Bigfix, and probably Level in the same detail. It's going to be a lot of time, but this is a big decision. These are complex packages, and they need proper attention to compare.
I'm back to it now, but my trial expired. No big deal. Zenith is dirt cheap to use. Check these prices:
http://www.saaztraining.com/zenith deliverables 2007.pdf
So I just signed up and am now paying a few bucks a month to continue to try out Zenith. I'm also now seeing Zenith from the side of a "real user", not a "prospective user" so I should be seeing the true system, not the salesman-enhanced side of things.
I like the fact that I'm not pressured to buy anything big upfront. It's all SAAS, so there is next to no requirement on my end, other than using IE (Firefox doesn't work) for their management console.
True, with SAAS you're always paying for it. But remember; software is always evolving. I have a few products I can keep running for a few years without upgrading, but not too many. Usually you're always in need of the new features or the bug fixes, or the new platform support that the new version brings.
I'm starting to become very aware of the internal costs we pay for maintaining our our software - The time we spend frigging with our servers, setting them up, maintaining them, etc. That time is far more productive being spent on customer's needs, which can be billed. Part of me is a bit apprehensive about the lack of control, but most of me is happy that I don't have to deal with putting it all back together if it crashes.
So what do I like so far about Zenith?
Well, I like the reports - Very similar to what I generate for my clients, except it's all there. I can output nearly the same thing with 5 clicks, and I'm done. This task is simple enough that it can now be given to the admin staff, not needing a technician to build the reports.
The dashboard/overview of what is going on with my servers is rather simple, but you know, it tells me the main points I need. I'm sure this is customizable, I'm still in the process of figuring that out.
I will keep learning this package and then write a better report on it soon. Once that's done, I'm off to check out Kaseya, Bigfix, and probably Level in the same detail. It's going to be a lot of time, but this is a big decision. These are complex packages, and they need proper attention to compare.
Tuesday, March 3, 2009
Switching Firewalls from ipf to pf on FreeBSD
It's time to deploy a new FreeBSD firewall, and I thought I'd check out pf instead of my standard ipf package.
pf is written by the OpenBSD team, and was designed to replace ipf because of the licensing issues with that code not being a proper BSD license. There's also some political issues, but I'll leave those along.
The Background:
Since the early days my office firewalls have been FreeBSD platforms. I've always loved the BSD platform, back from the days where I built an ISP around the BSDi package in the early 90's.
For as long as I can remember, I've been using ipf (ipfilter) by Darren Reed. It became so standard that FreeBSD started including it in their releases which makes life much easier.
Unix based firewalls always had my vote - They are infinitely configurable and extensible. They are not overly hard to setup, and they don't need a lot of hardware to run. In the day my options were a very expensive Cisco-class router, or Unix.
But as NAT routers kept dropping in price and improving in quality, my Unix based router/firewalls started slowly disappearing from sites. Most small offices don't need much - basic NAT with one port for remote access from us, and they are happy. A simple D-Link DI-604 goes for $50. That's a far cry cheaper than a decent computer and the charge for my time to setup a custom firewall.
So the Unix-Based firewalls only exist in some of my more complex, larger, or secure customer sites. The rest have Linksys or D-Link routers.
A few months ago my aged FreeBSD firewall finally died. Heck, it was a Pentium-233Mhz - I think I got my money's worth out of it. I knew it's days were numbered so I had a spare DI-604 ready to go, and popped it into place when needed.
I didn't really notice much of a difference in speed of the network (my main concern against the little routers) and it was handy to be able to use a web interface to configure it. I know you can do this with Unix, but it was one of the things I didn't bother updating as I upgraded the boxes over the years, relying on SSH instead. I don't pay a lot of attention to my firewall, and I didn't want to worry about security issues in the web interface screwing me up.
Life's been fine on the little D-Link until I started getting heavier into uTorrent. You put a few Windows 2003 machines on a network running uTorrent, and you can bring a little NAT router to it's knees. They just don't have enough memory and processor power.
The answer was to go back into some sort of Unix based firewall. Wanting something easier and FreeBSD based, I found pfSense. That didn't work out well for me, but it's another blog.
I made some compromises on how I use uTorrent, and my little DI-604 was still useful...
..until I had some heavy Spammer hacking going on. Talk about dragging your network into the ground. The DI-604 can't handle large amounts of traffic, even if it's rejecting it, and I was getting a lot of traffic. One old Windows 2000 IIS server (ticking time bomb really) was compromised, and the scum of the world was lining up at my router to use my bandwidth.
Back to Today - Trying pf over ipf:
So I have to build a new FreeBSD firewall. A quick download of 7.1-RELEASE i386 Disc1 (no need for 64 bit on a fw as far as I can see), and I'm on my way.
(btw msk support in 7.1 seems to be much better than 7.0 - no watchdog timeouts for me so far)
I decided to make this one a pf test platform as pf is also included by default with FreeBSD 7.1. I've heard mixed reviews of which is better, but the thing that intrigued me the most was that pf was a superset of ipf - my knowledge of rules for ipf wouldn't' be wasted, and I'd have more things to play with. ipf bugs me at times, as it doesn't seem to be evolving (like qmail). There is something to be said for a stable codebase, but I also don't think we've reached the pinacle of these packages.
Setup is simple;
in rc.conf;
pf_enable="YES"
pflog_enable="YES"
and your config file is /etc/pf.conf
pfctl is your control program for start, stop, and status.
If you want to be lazy, I'm quite sure a standard ipf.conf would work if you wanted to play with this.
So far I like pf. It's got a bit of a learning curve, but it has additional features that ipf lacks.
I like that the NAT portion is in the pf.conf - just looks cleaner to me to have one file for it. I like that I have more NAT rules. I have a "no nat" rule that blocks any outgoing SMTP traffic that doesn't go through one of our main mailservers. Just a different way of doing it, but nice.
pf also has the ability to do QoS routing via the ALTQ interface.
I'm currently checking out fwanalog so I can get a better look at pf's logs (also works for ipf) and I think I'm going to have to move into a Snort setup so I can detect these attacks and prevent them in the future.
This link has some simple info for monitoring pf firewalls;
http://prefetch.net/articles/monitoringpf.html
I'll post more as I find them.
pf is written by the OpenBSD team, and was designed to replace ipf because of the licensing issues with that code not being a proper BSD license. There's also some political issues, but I'll leave those along.
The Background:
Since the early days my office firewalls have been FreeBSD platforms. I've always loved the BSD platform, back from the days where I built an ISP around the BSDi package in the early 90's.
For as long as I can remember, I've been using ipf (ipfilter) by Darren Reed. It became so standard that FreeBSD started including it in their releases which makes life much easier.
Unix based firewalls always had my vote - They are infinitely configurable and extensible. They are not overly hard to setup, and they don't need a lot of hardware to run. In the day my options were a very expensive Cisco-class router, or Unix.
But as NAT routers kept dropping in price and improving in quality, my Unix based router/firewalls started slowly disappearing from sites. Most small offices don't need much - basic NAT with one port for remote access from us, and they are happy. A simple D-Link DI-604 goes for $50. That's a far cry cheaper than a decent computer and the charge for my time to setup a custom firewall.
So the Unix-Based firewalls only exist in some of my more complex, larger, or secure customer sites. The rest have Linksys or D-Link routers.
A few months ago my aged FreeBSD firewall finally died. Heck, it was a Pentium-233Mhz - I think I got my money's worth out of it. I knew it's days were numbered so I had a spare DI-604 ready to go, and popped it into place when needed.
I didn't really notice much of a difference in speed of the network (my main concern against the little routers) and it was handy to be able to use a web interface to configure it. I know you can do this with Unix, but it was one of the things I didn't bother updating as I upgraded the boxes over the years, relying on SSH instead. I don't pay a lot of attention to my firewall, and I didn't want to worry about security issues in the web interface screwing me up.
Life's been fine on the little D-Link until I started getting heavier into uTorrent. You put a few Windows 2003 machines on a network running uTorrent, and you can bring a little NAT router to it's knees. They just don't have enough memory and processor power.
The answer was to go back into some sort of Unix based firewall. Wanting something easier and FreeBSD based, I found pfSense. That didn't work out well for me, but it's another blog.
I made some compromises on how I use uTorrent, and my little DI-604 was still useful...
..until I had some heavy Spammer hacking going on. Talk about dragging your network into the ground. The DI-604 can't handle large amounts of traffic, even if it's rejecting it, and I was getting a lot of traffic. One old Windows 2000 IIS server (ticking time bomb really) was compromised, and the scum of the world was lining up at my router to use my bandwidth.
Back to Today - Trying pf over ipf:
So I have to build a new FreeBSD firewall. A quick download of 7.1-RELEASE i386 Disc1 (no need for 64 bit on a fw as far as I can see), and I'm on my way.
(btw msk support in 7.1 seems to be much better than 7.0 - no watchdog timeouts for me so far)
I decided to make this one a pf test platform as pf is also included by default with FreeBSD 7.1. I've heard mixed reviews of which is better, but the thing that intrigued me the most was that pf was a superset of ipf - my knowledge of rules for ipf wouldn't' be wasted, and I'd have more things to play with. ipf bugs me at times, as it doesn't seem to be evolving (like qmail). There is something to be said for a stable codebase, but I also don't think we've reached the pinacle of these packages.
Setup is simple;
in rc.conf;
pf_enable="YES"
pflog_enable="YES"
and your config file is /etc/pf.conf
pfctl is your control program for start, stop, and status.
If you want to be lazy, I'm quite sure a standard ipf.conf would work if you wanted to play with this.
So far I like pf. It's got a bit of a learning curve, but it has additional features that ipf lacks.
I like that the NAT portion is in the pf.conf - just looks cleaner to me to have one file for it. I like that I have more NAT rules. I have a "no nat" rule that blocks any outgoing SMTP traffic that doesn't go through one of our main mailservers. Just a different way of doing it, but nice.
pf also has the ability to do QoS routing via the ALTQ interface.
I'm currently checking out fwanalog so I can get a better look at pf's logs (also works for ipf) and I think I'm going to have to move into a Snort setup so I can detect these attacks and prevent them in the future.
This link has some simple info for monitoring pf firewalls;
http://prefetch.net/articles/monitoringpf.html
I'll post more as I find them.
Subscribe to:
Posts (Atom)