Is there anyone here using ZFS on top of a GELI-encrypted provider on
hardware which could be considered "slow" by today's standards? What
are the performance implications of doing this? The reason I am asking
is that I am in the process of building a small home NAS/webserver,
starting with a single disk (intending to expand as the need arises)
on the following hardware:
http://www.tranquilpc-shop.co.uk/acatalog/BAREBONE-SERVERS.html This
is essentially: Intel Arom 330 1.6 Ghz dualcore on an Intel
D945GCLF2-based board with 2GB Ram, the first disk I am going to use
is a 1.5TB Western Digital Caviar Green.
I had someone run a few openssl crypto benchmarks (to unscientifically
assess the maximum possible GELI performance) on a machine running
FreeBSD on nearly the same hardware and it seems the CPU would become
the bottleneck at roughly 200 MB/s throughput when using 128 bit
Blowfish, 70 MB/s when using AES128 and 55 MB/s when using AES256.
This, on it's own is definately enough for my neeeds (especially in
the case of using Blowfish), but what are the performance implications
of using ZFS on top of a GELI-encrypted provider?
Also, free free to criticize my planned filesystem layout for the
first disk of this system, the idea behind /mnt/sysbackup is to take a
snapshot of the FreeBSD installation and it's settings before doing
potentially hazardous things like upgrading to a new -RELEASE:
ad1s1 (freebsd system slice)
ad1s1a => 128bit Blowfish ad1s1a.eli 4GB swap
ad1s1b 128GB ufs2+s /
ad1s1c 128GB ufs2+s noauto /mnt/sysbackup
ad1s2 => 128bit Blowfish ad1s2.eli
zpool
/home
/mnt/data1
Thanks for your input.
- Dan Naumov
Re: ZFS on top of GELI / Intel Atom 330 system by Pete French on
2009-05-29T09:10:56+00:00
> Is there anyone here using ZFS on top of a GELI-encrypted provider on
> hardware which could be considered "slow" by today's standards? What
I run a mirrored zpool on top of a pair of 1TB SATA drives - they are
only 7200 rpm so pretty dog slow as far as I'm concerned. The
CPOU is a dual core Athlon 6400, and I am running amd64. The performance
is not brilliant - about 25 meg/second writing a file, and about
53 meg/second reading it.
It's a bit dissapointing really - thats a lot slower that I expected
when I built it, especially the write speed.
-pete.
Re: ZFS on top of GELI / Intel Atom 330 system by Dan Naumov on
2009-05-29T09:19:01+00:00
Ouch, that does indeed sounds quite slow, especially considering that
a dual core Athlon 6400 is pretty fast CPU. Have you done any
comparison benchmarks between UFS2 with Softupdates and ZFS on the
same system? What are the read/write numbers like? Have you done any
investigating regarding possible causes of ZFS working so slow on your
system? Just wondering if its an ATA chipset problem, a drive problem,
a ZFS problem or what...
- Dan Naumov
On Fri, May 29, 2009 at 12:10 PM, Pete French
<petefrench@ticketswitch.com> wrote:
>> Is there anyone here using ZFS on top of a GELI-encrypted provider on
>> hardware which could be considered "slow" by today's standards? What
>
> I run a mirrored zpool on top of a pair of 1TB SATA drives - they are
> only 7200 rpm so pretty dog slow as far as I'm concerned. The
> CPOU is a dual core Athlon 6400, and I am running amd64. The performance
> is not brilliant - about 25 meg/second writing a file, and about
> 53 meg/second reading it.
>
> It's a bit dissapointing really - thats a lot slower that I expected
> when I built it, especially the write speed.
>
> -pete.
>
Re: ZFS on top of GELI / Intel Atom 330 system by Philipp Wuensche on
2009-05-29T09:31:58+00:00
Dan Naumov wrote:
> Is there anyone here using ZFS on top of a GELI-encrypted provider on
> hardware which could be considered "slow" by today's standards? What
> are the performance implications of doing this? The reason I am asking
> is that I am in the process of building a small home NAS/webserver,
> starting with a single disk (intending to expand as the need arises)
> on the following hardware:
> http://www.tranquilpc-shop.co.uk/acatalog/BAREBONE-SERVERS.html This
> is essentially: Intel Arom 330 1.6 Ghz dualcore on an Intel
> D945GCLF2-based board with 2GB Ram, the first disk I am going to use
> is a 1.5TB Western Digital Caviar Green.
>
> I had someone run a few openssl crypto benchmarks (to unscientifically
> assess the maximum possible GELI performance) on a machine running
> FreeBSD on nearly the same hardware and it seems the CPU would become
> the bottleneck at roughly 200 MB/s throughput when using 128 bit
> Blowfish, 70 MB/s when using AES128 and 55 MB/s when using AES256.
> This, on it's own is definately enough for my neeeds (especially in
> the case of using Blowfish), but what are the performance implications
> of using ZFS on top of a GELI-encrypted provider?
I have a zpool mirror on top of two 128bit GELI blowfish devices with
Sectorsize 4096, my system is a D945GCLF2 with 2GB RAM and a Intel Arom
330 1.6 Ghz dualcore. The two disks are a WDC WD10EADS and a WD10EACS
(5400rpm). The system is running 8.0-CURRENT amd64. I have set
kern.geom.eli.threads=3.
This is far from a real benchmarks but:
Using dd with bs=4m I get 35 MByte/s writing to the mirror (writing 35
MByte/s to each disk) and 48 MByte/s reading from the mirror (reading
with 24 MByte/s from each disk).
My experience is that ZFS is not much of an overhead and will not
degrade the performance as much as the encryption, so GELI is the
limiting factor. Using ZFS without GELI on this system gives way higher
read and write numbers, like reading with 70 MByte/s per disk etc.
greetings,
philipp
Re: ZFS on top of GELI / Intel Atom 330 system by Pete French on
2009-05-29T09:41:36+00:00
> Ouch, that does indeed sounds quite slow, especially considering that
> a dual core Athlon 6400 is pretty fast CPU. Have you done any
> comparison benchmarks between UFS2 with Softupdates and ZFS on the
Not at all - but, now you have got me curious, I just went to
a completely different system (four core opteron box, no ecnryption,
four 15k SCSI drives and a zpool of 2 mirrored pairs), and that
also gave me about 25 meg/second!
I am using the wildly unscientific "how long to copy a file"
method to benchmark here, with the file residing on a different
drive, which can provided it at 80 meg/second.
> same system? What are the read/write numbers like? Have you done any
> investigating regarding possible causes of ZFS working so slow on your
> system? Just wondering if its an ATA chipset problem, a drive problem,
> a ZFS problem or what...
I have no idea, and now I think I need to look into it! certainly
I should be getting better than 25 meg/sec out of the 15K SCSI's.
-pete.
Re: ZFS on top of GELI / Intel Atom 330 system by Dan Naumov on
2009-05-29T10:13:25+00:00
Thank you for your numbers, now I know what to expect when I get my
new machine, since our system specs look identical.
So basically on this system:
unencrypted ZFS read: ~70 MB/s per disk
128bit Blowfish GELI/ZFS write: 35 MB/s per disk
128bit Blowfish GELI/ZFS read: 24 MB/s per disk
I am curious what part of GELI is so inefficient to cause such a
dramatic slowdown. In comparison, my home desktop is a
C2D E6600 2,4 Ghz, 4GB RAM, Intel DP35DP, 1 x 1,5TB Seagate Barracuda
- Windows Vista x64 SP1
Read/Write on an unencrypted NTFS partition: ~85 MB/s
Read/Write on a Truecrypt AES-encrypted NTFS partition: ~65 MB/s
As you can see, the performance drop is noticeable, but not anywhere
nearly as dramatic.
- Dan Naumov
> I have a zpool mirror on top of two 128bit GELI blowfish devices with
> Sectorsize 4096, my system is a D945GCLF2 with 2GB RAM and a Intel Arom
> 330 1.6 Ghz dualcore. The two disks are a WDC WD10EADS and a WD10EACS
> (5400rpm). The system is running 8.0-CURRENT amd64. I have set
> kern.geom.eli.threads=3.
>
> This is far from a real benchmarks but:
>
> Using dd with bs=4m I get 35 MByte/s writing to the mirror (writing 35
> MByte/s to each disk) and 48 MByte/s reading from the mirror (reading
> with 24 MByte/s from each disk).
>
> My experience is that ZFS is not much of an overhead and will not
> degrade the performance as much as the encryption, so GELI is the
> limiting factor. Using ZFS without GELI on this system gives way higher
> read and write numbers, like reading with 70 MByte/s per disk etc.
>
> greetings,
> philipp
Re: ZFS on top of GELI / Intel Atom 330 system by Morgan Wesström on
2009-05-29T11:11:54+00:00
Dan Naumov wrote:
> Thank you for your numbers, now I know what to expect when I get my
> new machine, since our system specs look identical.
>
> So basically on this system:
>
> unencrypted ZFS read: ~70 MB/s per disk
>
> 128bit Blowfish GELI/ZFS write: 35 MB/s per disk
> 128bit Blowfish GELI/ZFS read: 24 MB/s per disk
>
> I am curious what part of GELI is so inefficient to cause such a
> dramatic slowdown. In comparison, my home desktop is a
>
You can benchmark the encryption subsytem only, like this:
# kldload geom-zero
# geli onetime -s 4096 -l 256 gzero
# sysctl kern.geom.zero.clear=0
# dd if=/dev/gzero.eli of=/dev/null bs=1M count=512
512+0 records in
512+0 records out
536870912 bytes transferred in 11.861871 secs (45260222 bytes/sec)
The benchmark will use 256-bit AES and the numbers are from my Core2 Duo
Celeron E1200 1,6GHz. My old trusty Pentium III 933MHz performs at
13MB/s on that test. Both machines are recompiled with CPUTYPE=core2 and
CPUTYPE=pentium3 respectively but unfortunately I have no benchmarks on
how they perform without the CPU optimizations.
I'm in the same spot as you, planning to build a home NAS. I have
settled for graid5/geli but haven't yet decided if I would benefit most
from a dual core CPU at 3+ GHz or a quad core at 2.6. Budget is a concern...
Regards
Morgan
Re: ZFS on top of GELI / Intel Atom 330 system by Dan Naumov on
2009-05-29T11:36:47+00:00
Now that I have evaluated the numbers and my needs a bit, I am really
confused about what appropriate course of action for me would be.
1) Use ZFS without GELI and hope that zfs-crypto get implemented in
Solaris and ported to FreeBSD "soon" and that when it does, it won't
come with such a dramatic performance decrease as GELI/ZFS seems to
result in.
2) Go ahead with the original plan of using GELI/ZFS and grind my
teeth at the 24 MB/s read speed off a single disk.
>> So basically on this system:
>>
>> unencrypted ZFS read: ~70 MB/s per disk
>>
>> 128bit Blowfish GELI/ZFS write: 35 MB/s per disk
>> 128bit Blowfish GELI/ZFS read: 24 MB/s per disk
> I'm in the same spot as you, planning to build a home NAS. I have
> settled for graid5/geli but haven't yet decided if I would benefit most
> from a dual core CPU at 3+ GHz or a quad core at 2.6. Budget is a concern...
Our difference is that my hardware is already ordered and Intel Atom
330 + D945GCLF2 + 2GB ram is what it's going to have :)
- Dan Naumov
Re: ZFS on top of GELI / Intel Atom 330 system by Emil Mikulic on
2009-05-29T11:45:33+00:00
On Fri, May 29, 2009 at 12:47:38PM +0200, Morgan Wesstr?m wrote:
> You can benchmark the encryption subsytem only, like this:
>
> # kldload geom-zero
> # geli onetime -s 4096 -l 256 gzero
> # sysctl kern.geom.zero.clear=0
> # dd if=/dev/gzero.eli of=/dev/null bs=1M count=512
I don't mean to take this off-topic wrt -stable but just
for fun, I built a -current kernel with dtrace and did:
geli onetime gzero
./hotkernel &
dd if=/dev/zero of=/dev/gzero.eli bs=1m count=1024
killall dtrace
geli detach gzero
The hot spots:
[snip stuff under 0.3%]
kernel`g-eli-crypto-run 50 0.3%
kernel`-mtx-assert 56 0.3%
kernel`SHA256-Final 58 0.3%
kernel`rijndael-encrypt 72 0.4%
kernel`-mtx-unlock-flags 74 0.4%
kernel`rijndael128-encrypt 74 0.4%
kernel`copyout 92 0.5%
kernel`-mtx-lock-flags 93 0.5%
kernel`bzero 114 0.6%
kernel`spinlock-exit 240 1.3%
kernel`bcopy 325 1.7%
kernel`sched-idletd 810 4.3%
kernel`swcr-process 1126 6.0%
kernel`SHA256-Transform 1178 6.3%
kernel`rijndaelEncrypt 5574 29.7%
kernel`acpi-cpu-c1 8383 44.6%
I had to build crypto and geom-eli into the kernel to get proper
symbols.
References:
http://wiki.freebsd.org/DTrace
http://www.brendangregg.com/DTrace/hotkernel
Re: ZFS on top of GELI / Intel Atom 330 system by Ivan Voras on
2009-05-29T11:50:26+00:00
Emil Mikulic wrote:
> On Fri, May 29, 2009 at 12:47:38PM +0200, Morgan Wesstr?m wrote:
>> You can benchmark the encryption subsytem only, like this:
>>
>> # kldload geom-zero
>> # geli onetime -s 4096 -l 256 gzero
>> # sysctl kern.geom.zero.clear=3D0
>> # dd if=3D/dev/gzero.eli of=3D/dev/null bs=3D1M count=3D512
>=20
> I don't mean to take this off-topic wrt -stable but just
> for fun, I built a -current kernel with dtrace and did:
>=20
> geli onetime gzero
> ./hotkernel &
> dd if=3D/dev/zero of=3D/dev/gzero.eli bs=3D1m count=3D1024
> killall dtrace
> geli detach gzero
>=20
> The hot spots:
> [snip stuff under 0.3%]
> kernel`g-eli-crypto-run 50 0.3%
> kernel`-mtx-assert 56 0.3%
> kernel`SHA256-Final 58 0.3%
> kernel`rijndael-encrypt 72 0.4%
> kernel`-mtx-unlock-flags 74 0.4%
> kernel`rijndael128-encrypt 74 0.4%
> kernel`copyout 92 0.5%
> kernel`-mtx-lock-flags 93 0.5%
> kernel`bzero 114 0.6%
> kernel`spinlock-exit 240 1.3%
> kernel`bcopy 325 1.7%
> kernel`sched-idletd 810 4.3%
> kernel`swcr-process 1126 6.0%
> kernel`SHA256-Transform 1178 6.3%
> kernel`rijndaelEncrypt 5574 29.7%
> kernel`acpi-cpu-c1 8383 44.6%
Hi,
What is the meaning of counts? Number of calls made or time?
Re: ZFS on top of GELI / Intel Atom 330 system by Vlad Galu on
2009-05-29T12:01:10+00:00
On Fri, May 29, 2009 at 2:49 PM, Ivan Voras <ivoras@freebsd.org> wrote:
>
> Hi,
>
> What is the meaning of counts? Number of calls made or time?
>
>
The former.
Re: ZFS on top of GELI / Intel Atom 330 system by Emil Mikulic on
2009-05-29T12:06:01+00:00
On Fri, May 29, 2009 at 01:49:54PM +0200, Ivan Voras wrote:
> Emil Mikulic wrote:
[...]
> > kernel`SHA256-Transform 1178 6.3%
> > kernel`rijndaelEncrypt 5574 29.7%
> > kernel`acpi-cpu-c1 8383 44.6%
>
> Hi,
>
> What is the meaning of counts? Number of calls made or time?
Time.
Sorry, I inadvertently cut off the headings: function, count, percent
As I understand it, hotkernel uses statistical sampling at 1001 Hz, so
the percentage is an approximation of how much time is spent in each
function, based on how many profiler samples ended up in each function.
Re: ZFS on top of GELI / Intel Atom 330 system by Dan Naumov on
2009-05-29T12:13:42+00:00
Pardon my ignorance, but what do these numbers mean and what
information is deductible from them?
- Dan Naumov
> I don't mean to take this off-topic wrt -stable but just
> for fun, I built a -current kernel with dtrace and did:
>
>
Re: ZFS on top of GELI / Intel Atom 330 system by Chris Dillon on
2009-05-30T01:17:05+00:00
Quoting Dan Naumov <dan.naumov@gmail.com>:
> Ouch, that does indeed sounds quite slow, especially considering that
> a dual core Athlon 6400 is pretty fast CPU. Have you done any
> comparison benchmarks between UFS2 with Softupdates and ZFS on the
> same system? What are the read/write numbers like? Have you done any
> investigating regarding possible causes of ZFS working so slow on your
> system? Just wondering if its an ATA chipset problem, a drive problem,
> a ZFS problem or what...
I recently built a home NAS box on an Intel Atom 330 system (MSI Wind
Nettop 100) with 2GB RAM and two WD Green 1TB (WD10EADS) drives in a
mirrored ZFS pool using a FreeNAS 0.7 64-bit daily build. I only see
25-50MB/sec via Samba from my XP64 client, but in my experience SMB
always seems to have horrible performance no matter what kind of
servers and clients are used. However, dd shows a different set of
figures:
nas:/mnt/tank/scratch# dd if=/dev/zero of=zero.file bs=1M count=4000
4000+0 records in
4000+0 records out
4194304000 bytes transferred in 61.532492 secs (68164052 bytes/sec)
nas:/mnt/tank/scratch# dd if=zero.file of=/dev/null bs=1M
4000+0 records in
4000+0 records out
4194304000 bytes transferred in 33.347020 secs (125777476 bytes/sec)
68MB/sec writes and 125MB/sec reads... very impressive for such a
low-powered box, I think, and yes the drives are mirrored, not striped!