I'm curious if anyone is doing anything like this at your job/site? I am attending a schmooze session tomorrow with these guys and the reseller in Philly...it looks good on paper but it's almost too good to be true.

feh, well I'd post the ppt but it's too big. =\

it looks good on paper but it's almost too good to be true.

Prepare to be screwed!

This short story seems oddly appropriate!

Ms fnd in a Lbry (http://home.comcast.net/~bcleere/texts/draper.html)

Good idea in theory.

As cheap as storage is, I can't see myself trusting my data to that theory in practice though.

unfortunately when you have to replicate TB of data to multiple sites and you're using San volume controllers in the mix that $140 dollar drive now costs you thousands in licensing.

We are going to do it, especially for our large databases 1tb and larger and our file servers. In theory 1tb will become 250gb and will get smaller and smaller with each iteration. The device itself is also replicated so you're protected there too.

The appliances are fairly cheap but I imagine support us not. Oracle, Hershey, wachovia, bank of America, etc use this.. Oracle databases get about 58% compression whereas vmware gets 120% crazy...

I'm eager to see it in practice. If you're interested pm me and I will mail you the 2mb ppt.

But essentially thisigrates right into your existing framework an uses the same bkp software you have in place it acts as a VTL.

Data deduplication, or single instance storage as some call it, is the "next big thing" in distributed storage. Everyone with any real data volume is doing it or moving to it in some form or another.

Obviously mileage with individual products may vary. Most of them use a strong hash function to determine uniqueness but some use reversible functions. Implementations that de-duplicate across larger pools of data are also more efficient than those that keep things separate in per-machine data sets.

Our VTL plan involves using a tape system to make vaulted copies of certain backups before it enters the repository for processing and then mirrors the backend storage across a WAN to an standby site. You have massive amounts of near-line backup retention and the safety of stolid old tapes just in case.

Malse, I do enjoy reading your input on these things as it seems you're doing similar things at your site that we're close to implementing ourselves.


Data Domain, the folks who gave this presentation do their crunching during the actual backup or inline dedup whereas many others are doing it post backup. Their appliance sits between the backup manager and the target disks.

So our plan will be this:

At our PDC we currently have two IBM 3584s w/LTO3 drives in each. That we are pumping almost 6TB a night. We are going to move one of those TL to our SDC b/c right now we don't even have a TL to restore from if we had a disaster... /giggle

We will put one of the data domain appliances in our PDC, the DD565which has a logical cap of 320TB to 810TB w/ usable of 16.2TB. And place another one possibly even a smaller one at one of our other hospitals and not even our SDC so we have the replication of the deduped database.

We plan to move all the backups that are currently going to that second TL, they are huge but are few and they are all databases/images. So that 3TB a night should in theory shrink to GB and we'll have a copy online and not offsite like we're doing now. Our storage for these servers is so premium b/c of the SVC and metro mirror that we can't keep 2 copies on disk but instead do hotbackups every night and send them offsite.

We also have been supeonaed (sp) b/c of pending law suits that we cannot purge any data for our exchange or file servers so in the next phase we will be moving these backups there. B/C right now it's a frigging nightmare how we're doing them.... backup to disk, then to tape, oh and then another copy to tape. One copy stays in the TL and one goes offsite to the never delete pool This started in June, we're up to almost 1000 tapes offsite, of course that's b/c when we caught wind of it we finagled our backups this way before we were ordered to do it, but /sigh our backup infrastructure is insane, it has grown from a couple hundred GB a night to several TB in only a few years and our clinical database is growing at a very alarming rate. We went live on AIX in May of 2007 we were at 850GB used, as of today we're sitting at 2.2TB used. And, we have 4 copies of that database :) So yeah, we're excited about dedupe, we're very excited....

We just got estimates for the new IBM DS5000 that has 32GB cache in their controllers and can do 1TB drives... entry level for 80TB (list price) is over $300k, that is before our umpteen gazillion SVC and metro mirror and TPC licenses. This shit gets pricey. But, we also recently heard of XIV and their new array that can metro mirror, flash copy, and outperforms DS5000, DS4800 on SATA drives and the savings is about $40k just for hardware....you don't have to buy licenses for any of the copy features and it can metro mirror like the SVC does. So who knows where we will be at the end of this fiscal year.

I just know our SAN has exploded in the past 1.5yrs. I love it though :)

We have:
3 IBM DS6800s
1 IBM DS4100
6 IBM DS4800s

At our PDC we have 2 switches, Brocade SanDirectors with 48 port blades of which we have 2 blades almost full at the moment...it can hold 10 blades @4GB.

Our SDC has has 2 Brocade SanDirectors with only 48 ports total on each switch also @4GB. We host our own SDC and do not use a service, we control the fiber too. The biggest drawback is that as the crow flies it's only 3miles away which is why we'll replicate the dedupe to a different hospital.

The two sites have dual dark fiber paths that form a triangle between us, our primary hospital, and the SDC. 1st path is straight thru. The 2nd path goes thru ONS gear. We are replicating in the neighborhod of 5TB currently with that number increasing with each project so deduping our backup data makes us happy.

Reversible functions? Wouldn't that just be file compression?

Reversible functions? Wouldn't that just be file compression?

There is an algorithm that does the crunch and it's less of a crunch and more of a pointer. When you back up a 1TB database it crunches it down (so yes compression there) but, it goes beyond and as each bit/chunk comes in it says... have I seen this before? Oh yeah, it's here, it's there, I don't need this again. Data Domain evaluates at the 4k, 8k, and 12k chunks. So it's a variable have I seen this before. It could literally crunch a whole doc or part of the doc, if the crunching gets to over 12k and it can't find a match it will back the whole thing up. The real beauty comes when you do the next backup and say the "structure" of something hasn't changed, the dedupe database knows the changes and crunches even more. The examples are like 1TB to 250GB to 10GB.

Here is a pic from the presentation...



So, as to file compression and reversal.... uhhh I don't know :(

Reversible functions? Wouldn't that just be file compression?

In a technical sense compression is de-duplication since you're replacing a predictable extent of data with a known reference to it elsewhere.

The difference I was referring to was that many systems use fast hashes to identify any unique set while depending on the vast keyspace to protect them from collisions (resulting in two pieces of different data being identified as the same), whereas more conservative ones spend much more time to verify any given set is already known and accounted for before throwing it away.

File compression is a deduplicative effort at a file level, obviously, you can get much better orthogonality if you run it over more and more of your amalgamated data, to the point that you effectively store only minutes deltas after a few iterations of storage. It's conceptually similar to zipping up two of the same file at once as opposed to compressing them separately, if you're thinking in terms of LZW-style dictionaries imagine applying the combined dictionary to every block of every hard disk of every computer in your system.


Something that kind of hurt my head when I was looking at one vendor's implementation was their recommendation to do traditional stream compression before passing it off to the deduplicator instead of after. My initial thought was that you would want to preserve the original input to avoid small changes resulting in larger downstream changes due to diverging dictionaries or curves resulting in less overall efficiency, but their practical test had found the converse was true; makes sense if you think about it but isn't intuitive.

Since hash functions are not reversible, you are in the strictest sense throwing information away when you use them. For staggering large data volumes, the effect loss of data-space will bite you, though I'm not sure there is enough information on earth to make that probable for multi-kbit hashes.

Ah! I see what you mean.

What I was initially thinking:

Data: aaaaaabbbbbbbccccccccc
One-way hash: 47 (calculated from the data; collisions possible but unlikely)
Reversible function: 6a7b9c (original data can be restored from this information alone)

But it sounds like you're really talking about this:

Data: aaaaaabbbbbbbccccccccc
One-way hash: 47 (calculated from the data, collisions possible but unlikely)
Reversible function: id=47 (unique identifier for a dictionary entry that contains "aaaaaabbbbbbbccccccccc" or "6a7b9c"; collisions impossible)

That is interesting about pre-compressing data before, well, compressing it again (on a larger scale). I'd have had the same initial expectation that you did.

That's essentially correct, one produces an identifier whereas one produces an invertible result ( F-'(F(x)) = x ) . As one would expect having a guaranteed unique identifier for each datum places hefty (O(n!) ? ) worst case search limit on it that makes it impractical to implement on a cost effective basis for some designs

wow ok numbers guys :)
I don't know how these guys are doing except that the algorithim is applied as data streams vs other vendors who do it post data being backed up. And I know that the only way to restore if you had a disaster with the appliance is from a mirror image that is elsewhere. Lose that and you're screwed.

As far as simple restores go you have to have the same things you need from tape. Bkp app asks where data is and it is restored I am not sure if the data is uncompressed or how it is restored it's aquedtion I can ask.

Malse who was the vendor you talked to? Only other one for us was IBM and there's was very similar to data domain.

We reviewed Data Domain, HP (more economical option by far), Sun, IBM, and EMC. HP and Sun are primarily doing VTLs as opposed to integrated appliances that do all kinds of backup magic and are thus generally cheaper to integrate into an existing setup, DataDomain was IMO the most forward thinking and working sort of like a NetApp. I expect them to become the dominant concept (though not necessarily vendor).

Lots of the next generation of these things are going to be doing CIFS/NFS or NDMP presentation, which is nice if you are like me and hate most backup software like the syphillic plague of overpriced bassackwards dimwittery it generally is.

So we're currently in the pilot with data domain. This is sort of their way of selling the concepts to the customer and letting them try things out before deciding to buy.

We did not choose the VTL option and instead are trying backups across CIFS. I am not impressed. We use CA's ArcServe Backup as our backup solution here and for whatever reason I have yet to get a successful windows backup to complete to the data domain CIFS devices. Well, let me correct that. I can run small windows backups but anything that is over 20GB times out at exactly the same spot. Not only that but one of my very large RMAN backups 2.5TB takes ... 30hrs (yes 30hrs) to complete. This after following their best practice guide for my RMAN script. The first attempt took 24hrs and that was using my own tuning I use for tape backups. Which, oh by the way this 2.5TB database backs up to tape in 5hrs.

So, my question for you Malse or anyone else familiar with this stuff is this. Do you think this is a CIFS issues? All of my connections are gig and are on non-routed VLANs; there's no traffic on those networks except my backup manager and the datadomain appliance.

I have done some CIFS tuning according to both CA and Data Domain specs but this is just pathetic. We're using a windows backup manager (something I'm new with as I'm a Unix admin). I'm wondering if the backups would perform better with a Unix manager rather than Windows?

Please feel free to chime in with anything you can think of. Also what HP product did you go with?

I'm thinking CIFS or Arcserve problem. Why use arcserve at all when you can just do RMAN directly to the backup server's share?

B/C we use ArcServe's reporting and vaulting of our offsite tapes, etc. The idea is we'll backup to ddomain and then archive to tape for longer term storage (not necessarily for the RMAN backups those will just go to disk; so we may not even use ArcServe for those backups).

I think you're right about CIFS or ArcServe as I just added the CIFS share to one of my file servers and just copied data manually (click and drag) and didn't have any timeouts and my data actually copied albeit slooooowly.

My next step is to remove the ddomain appliance from the middle and do backups to CIFS with ArcServe. I just hate having to do all this bullshit work when the vendors should already know what to tune/tweak/change.

On that note why aren't you bitching at your sales guy to get someone with a clue on site for you? Even if you don't have Racing Stripe Platinum support if it never worked they should be jumping hoops to make sure you like it.

We've had two Data Domain's for about 4 years. They have been bullet proof and have given us around 16:1 compression. We use them only for 2nd line backups basically as virtual tape with Netbackup.

I'm willing to bet ArcServe is doing synchronous writes and its own buffering or something equally stupid along those lines.

Yep I think you're 100% right. We dropped lmdd on the windows manager and did basically "dd" type things from the media manager to the datadomain box and we screamed. And for the 10th time someone has confirmed our TCP tuning is correct and our NiC drivers are current arrrrgh I don't want to hear that shit again! :D

Now the ball is back in CA's court /bounce /bounce *loud thud*

Something that kind of hurt my head when I was looking at one vendor's implementation was their recommendation to do traditional stream compression before passing it off to the deduplicator instead of after. My initial thought was that you would want to preserve the original input to avoid small changes resulting in larger downstream changes due to diverging dictionaries or curves resulting in less overall efficiency, but their practical test had found the converse was true; makes sense if you think about it but isn't intuitive.

Yeah, this sort of thing is as old as the original Stacker and Stuffit software, where they found that using a compressed drive was faster than using an uncompressed drive, because the time savings of not having to access the information from the drive was greater than the time spent compressing/decompressing the data. This savings could only be increased when you are talking network speeds rather than hard drive datapath speeds (on top of the actual hard drive acquisition speeds).

Basically, figure that the processor is a few orders of magnitude faster than any other device in a computer, so given a choice between crunching numbers and accessing data, best to choose number crunching every time...

Heck that's the same reason ethernet beat the pants off token ring.



of more current concern:
Seriously, talk to their tech support. If the tech support sucks, the answer is no. Period. Try new vendor. You are now at a point where you have sufficient information to talk intelligently to their tech support rep about what your needs are, you have a case study of having tried their product and gotten unacceptable results. You are now not sure whether this is you or the product...so....perfect time to test their tech support. No brainer.

Basically, figure that the processor is a few orders of magnitude faster than any other device in a computer, so given a choice between crunching numbers and accessing data, best to choose number crunching every time.

While that's correct for the case you were referring to, it doesn't really have anything to do with the de-duplicator. The de-duplication algorithm is most limited by hash-space checking, which very rapidly becomes astronomically large, not the data scanning or compression rates, which are logarithmic. These are both computationally bound for non-trivial data sets.

On that subject, though, it will be interesting to see if we see a revival of drive-level compression (possibly paired with encryption) as we may be soon seeing the end of the easy ride from GMR platter density now that block compression ASICs are practically free and very fast. Some of you may remember when this happened with modems :>

Some of you may remember when this happened with modems
You mean we're going to start seeing drives advertised with how much compressed data they can store (assuming some fixed compression ratio that may or may not have any bearing on reality!) rather than raw uncompressed data?

So while we let CA figure out what the heck is wrong with their product I attempted a test of my 2.5TB Oracle database to an NFS mount on the data domain device with RMAN... 4 channels and using data domain's best practices for filesperset, etc. This backup takes a little over 5hrs to LTO3s and 2 channels.....

I started it last night at 6:30 and it's still going......

Here is the script we are running....

################################################## #########
#cert1_rman_custom_online_bkp.ksh
#
# Purpose: To backup the cert database to disk.
# The database is available at this time.
#
# Schedule: As scheduled
################################################## ##########
#!/usr/bin/ksh
export ORACLE_HOME=/u01/oracle/product/9.2.0.5
export ORACLE_SID=cert1
TimeStamp=`date +%Y%m%d_%H%M%S`
export LOG=/u02/oracle/admin/rman/scripts/CERT1/${ORACLE_SID}_HOTO-${TimeStamp}.log
echo "Starting Time: `date`" > ${LOG}
/u01/oracle/product/9.2.0.5/bin/rman target / catalog rman/rman@rcat1 <<! | tee -a ${LOG}
set echo on;
run {
sql 'alter system archive log current';
allocate channel t1 type disk format '/DEDUPE/oracle/CERT1_HOT0-${TimeStamp}-%U';
allocate channel t2 type disk format '/DEDUPE/oracle/CERT1_HOT0-${TimeStamp}-%U';
allocate channel t3 type disk format '/DEDUPE/oracle/CERT1_HOT0-${TimeStamp}-%U';
allocate channel t4 type disk format '/DEDUPE/oracle/CERT1_HOT0-${TimeStamp}-%U';
setlimit channel t1 maxopenfiles 1;
setlimit channel t2 maxopenfiles 1;
setlimit channel t3 maxopenfiles 1;
setlimit channel t4 maxopenfiles 1;
backup incremental level = 0 cumulative database filesperset=1 TAG='CERT1_HOT0.....${TimeStamp}';
release channel t1;
release channel t2;
release channel t3;
release channel t4;
}
exit


Here is what the data domain box is showing from iostat....

eth1 is the connection we're concened with... basically maybe...maybe 40-50Meg per sec... using dd I've proven I can cap this thing out at 120M per sec...so what gives here? This is just as lousey as using ARCserve in the middle?

---- ---- ------- ----- ----- ----- ----- ----- ------ ------ ------ ------ ------ ------- ------- ------- ------ ------ ------ ------
12% 16% 1342 16% 79% 0% 5% 4 0 0 47728 1165 483 2698 1% 0 41891 0 0
14% 18% 1634 21% 75% 0% 4% 0 0 0 50428 1245 575 0 1% 0 52592 0 0
12% 14% 1661 19% 75% 0% 6% 2 0 0 52079 1316 536 54 1% 0 44856 0 0
14% 17% 1615 20% 76% 0% 4% 5 1 1 46867 1182 584 0 0% 0 50848 0 0
11% 15% 1365 17% 79% 0% 4% 0 0 0 46417 1139 488 452 2% 0 43043 0 0
16% 18% 1782 22% 73% 0% 5% 3 0 0 48431 1194 635 398 2% 0 53429 0 0
11% 14% 1033 19% 75% 0% 6% 3 0 0 51402 1327 245 18889 7% 0 26919 0 0
15% 17% 1211 11% 87% 0% 2% 0 0 0 17463 428 315 17671 5% 0 47535 0 0
15% 21% 1670 20% 75% 0% 5% 8 0 0 50843 1283 756 2705 1% 0 48697 0 0
12% 13% 1427 18% 79% 0% 3% 0 1 1 46541 1139 355 0 0% 0 47125 0 0
13% 16% 1529 20% 77% 0% 3% 0 0 0 51059 1248 576 27 1% 0 50055 0 0
6% 7% 718 9% 89% 0% 2% 6 0 0 47068 1151 172 0 0% 0 23061 0 0
13% 15% 1347 17% 80% 0% 3% 0 0 0 49566 1213 403 8122 1% 0 44165 0 0
12% 13% 1443 18% 79% 0% 3% 8 0 0 44487 1084 388 0 0% 0 45885 0 0
12% 14% 1428 18% 79% 0% 3% 0 1 1 46714 1147 476 40 0% 0 45705 0 0
9% 11% 1107 14% 84% 0% 2% 0 0 0 44744 1093 379 33 0% 0 36247 0 0
01/29 08:45:29
CPU CPU State NFS NFS NFS NFS NFS CIFS eth0 KB/s eth1 KB/s Disk KiB/s Disk NVRAM KiB/s Repl KB/s
busy max 'CDVMS' ops/s proc recv send idle ops/s in out in out read write busy read write in out
---- ---- ------- ----- ----- ----- ----- ----- ------ ------ ------ ------ ------ ------- ------- ------- ------ ------ ------ ------
11% 13% 1280 16% 81% 0% 3% 6 0 0 36969 899 423 2745 1% 0 39974 0 0
14% 17% 1675 21% 75% 0% 4% 0 0 0 41772 1029 673 0 1% 0 52188 0 0
11% 15% 1345 17% 79% 0% 4% 0 0 0 52980 1316 576 40 1% 0 40850 0 0
11% 13% 1399 17% 79% 0% 4% 8 1 1 41748 1044 434 33 0% 0 40349 0 0
6% 8% 534 6% 92% 0% 2% 0 0 0 16917 415 380 94 1% 0 26952 0 0
6% 8% 534 6% 92% 0% 2% 0 0 0 16917 415 380 94 1% 0 26952 0 0
13% 16% 1024 16% 81% 0% 3% 0 0 0 39277 960 376 3625 1% 0 43584 0 0
13% 16% 2199 18% 78% 0% 4% 6 0 0 52479 1300 478 2813 1% 0 42622 0 0
14% 17% 1676 21% 75% 0% 4% 0 0 0 40230 1001 585 0 1% 0 52037 0 0
12% 16% 1516 18% 77% 0% 5% 8 0 0 54611 1390 592 0 1% 0 42248 0 0
9% 10% 1189 15% 83% 0% 2% 0 1 1 39746 975 393 0 0% 0 39120 0 0
12% 14% 1350 17% 80% 0% 3% 0 0 0 41088 1001 435 27 1% 0 44299 0 0
9% 14% 1143 14% 83% 0% 3% 6 0 0 44078 1074 438 2698 1% 0 36671 0 0
11% 13% 1130 14% 83% 0% 3% 0 0 0 37357 909 300 2698 0% 0 37107 0 0
12% 15% 1368 17% 80% 0% 3% 0 0 0 37883 924 348 27 0% 0 45339 0 0
14% 18% 1632 21% 75% 0% 4% 8 0 0 46445 1135 526 6 1% 0 52003 0 0
01/29 08:46:03
CPU CPU State NFS NFS NFS NFS NFS CIFS eth0 KB/s eth1 KB/s Disk KiB/s Disk NVRAM KiB/s Repl KB/s
busy max 'CDVMS' ops/s proc recv send idle ops/s in out in out read write busy read write in out
---- ---- ------- ----- ----- ----- ----- ----- ------ ------ ------ ------ ------ ------- ------- ------- ------ ------ ------ ------
12% 14% 1395 17% 79% 0% 4% 0 1 1 53596 1310 351 0 0% 0 44493 0 0
15% 17% 1546 20% 76% 0% 4% 0 0 0 52085 1275 336 2860 0% 0 51118 0 0
8% 10% 899 15% 82% 0% 3% 6 0 0 29571 721 253 13 0% 0 27349 0 0
7% 10% 730 5% 93% 0% 2% 0 0 0 23559 573 236 155 0% 0 24709 0 0
11% 13% 1235 16% 81% 0% 3% 0 0 0 40080 977 221 54 0% 0 40519 0 0
15% 17% 1732 21% 74% 0% 5% 8 1 1 54296 1356 538 2698 1% 0 51306 0 0
12% 14% 1435 18% 79% 0% 3% 0 0 0 45990 1126 430 6 0% 0 45729 0 0
13% 16% 1662 19% 76% 0% 5% 6 0 0 49660 1263 532 0 0% 0 46043 0 0
11% 15% 1243 16% 80% 0% 4% 0 0 0 38618 942 589 2745 1% 0 40307 0 0
10% 14% 1271 16% 81% 0% 3% 0 0 0 43953 1090 281 0 0% 0 40397 0 0
11% 16% 1405 16% 79% 0% 5% 8 1 1 39264 982 372 0 0% 0 39731 0 0
9% 13% 1141 15% 82% 0% 3% 0 0 0 38813 965 433 27 0% 0 36491 0 0
12% 15% 1436 18% 78% 0% 4% 0 0 0 45952 1120 462 0 0% 0 47746 0 0
11% 15% 1328 16% 80% 0% 4% 6 0 0 44599 1089 418 0 0% 0 42825 0 0
12% 15% 1276 16% 80% 0% 4% 0 0 0 44018 1074 621 5430 1% 0 41880 0 0
10% 13% 1184 15% 82% 0% 3% 0 0 0 39610 965 351 13 0% 0 38939 0 0

trying again but this time with these changes....

7> setlimit channel t1 maxopenfiles 16 readrate 200;
8> setlimit channel t2 maxopenfiles 16 readrate 200;
9> setlimit channel t3 maxopenfiles 16 readrate 200;
10> setlimit channel t4 maxopenfiles 16 readrate 200;
11> backup incremental level = 0 cumulative database filesperset=16 TAG='CERT1_HOT0.....20090129_093127';


many of our data files are only 64Meg so we theorize that only allowing 1 file open at a time was causing too much overhead and was really just stupid. we'll see how this does. Malse or anyone who is a DBA please feel free to pipe up; I am far from a DBA.

I'd set it to 2 or 4 instead of 16 for the test, also do you have RMAN doing compression? I'd turn that off as well for testing purposes. Didn't see that in the status report.

No RMAN compression that I know of.
How would I check that though to be sure.

Ahhh so the problem has absolutely nothing to do with RMAN or compression or really anything other than the way our network nazi's have our vlans configured....

02/02 15:02:34
CPU CPU State NFS NFS NFS NFS NFS CIFS eth0 KB/s eth1 KB/s Disk KiB/s Disk NVRAM KiB/s Repl KB/s
busy max 'CDVMS' ops/s proc recv send idle ops/s in out in out read write busy read write in out
---- ---- ------- ----- ----- ----- ----- ----- ------ ------ ------ ------ ------ ------- ------- ------- ------ ------ ------ ------
22% 25% 1483 31% 63% 0% 6% 2 114651 3285 2 0 764 60 1% 0 94862 0 0
22% 27% 1558 32% 61% 0% 7% 0 116002 3601 1 0 819 0 1% 0 100359 0 0
22% 25% 1561 32% 62% 0% 6% 0 115475 3329 1 0 876 6 1% 0 100938 0 0
19% 24% 1518 31% 61% 0% 8% 1 115352 3552 1 0 785 33 1% 0 96973 0 0
21% 24% 1616 33% 60% 0% 7% 0 116537 3209 1 0 881 0 1% 0 103276 0 0
25% 29% 1640 33% 59% 0% 8% 0 116531 3414 1 0 737 0 1% 0 107041 0 0
22% 24% 1614 33% 61% 0% 6% 2 115915 3057 1 0 864 54 1% 0 103484 0 0
22% 26% 1510 31% 63% 0% 6% 0 116451 3527 1 0 746 0 1% 0 97492 0 0
23% 26% 1627 34% 58% 0% 8% 0 115033 3329 1 0 872 0 1% 0 113134 0 0
24% 26% 1631 31% 62% 0% 7% 1 114530 3371 1 0 938 108 2% 0 110507 0 0
23% 25% 1401 31% 61% 0% 7% 0 115028 3396 1 0 878 54 1% 0 104997 0 0
22% 25% 2370 33% 60% 0% 7% 2 115047 3137 2 0 808 114 1% 0 103293 0 0
23% 25% 1596 32% 61% 0% 7% 0 116301 3165 1 0 785 0 1% 0 101837 0 0
24% 28% 1648 34% 58% 0% 8% 0 115727 3415 1 0 805 0 1% 0 107185 0 0
21% 25% 1554 32% 61% 0% 7% 2 115401 3322 1 0 806 27 1% 0 99398 0 0
20% 23% 1435 29% 64% 0% 7% 0 116754 3466 1 0 760 0 1% 0 93115 0 0



I direct connected my RMAN database to the DDR device and it's smoking. So I query my network group and discover that our 5 private vlans for backup networks that while private are all using the same 1 gig uplink..... /boggle say what.... and so this explains why 8am on a Monday my speed is 10M/sec but late Friday evening it's 80-90M/sec. But direct connect I push well into 113M/sec. DOH!

So anyway my 2.5TB data base should be done in a couple of hours :)

Oh and our latest compression numbers look a little something like this: (before the most recent backup running now mind you)

From: 2009-01-26 07:00 To: 2009-02-02 07:00

Pre-Comp Post-Comp Global-Comp Local-Comp Total-Comp
(GiB) (GiB) Factor Factor Factor
(Reduction %)
--------------- -------- --------- ----------- ---------- -------------
Currently Used: 9054.2 1190.1 - - 7.6x (86.9)
Written:*
Last 7 days 7031.1 396.8 8.0x 2.2x 17.7x (94.4)
Last 24 hrs
--------------- -------- --------- ----------- ---------- -------------
* Does not include the effects of pre-comp file deletes/truncates
since the last cleaning on 2009/01/27 06:18:54.
Key:
Pre-Comp = Data written before compression
Post-Comp = Storage used after compression
Global-Comp Factor = Pre-Comp / (Size after de-dupe)
Local-Comp Factor = (Size after de-dupe) / Post-Comp
Total-Comp Factor = Pre-Comp/Post-Comp
Reduction % = ((Pre-Comp - Post-Comp) / Pre-Comp) * 100


The CA Arcserve piece is still being worked, this is all without a media manager just RMAN to device.

this one is more current....

sysadmin@dd01# filesys show compression summary
From: 2009-01-26 15:00 To: 2009-02-02 15:00
Pre-Comp Post-Comp Global-Comp Local-Comp Total-Comp
(GiB) (GiB) Factor Factor Factor
(Reduction %)
--------------- -------- --------- ----------- ---------- -------------
Currently Used: 9348.3 1191.4 - - 7.8x (87.3)
Written:*
Last 7 days 7373.0 392.9 8.4x 2.2x 18.8x (94.7)
Last 24 hrs 357.9 0.6 341.7x 1.8x 601.4x (99.8)
--------------- -------- --------- ----------- ---------- -------------
* Does not include the effects of pre-comp file deletes/truncates
since the last cleaning on 2009/01/27 06:18:54.
Key:
Pre-Comp = Data written before compression
Post-Comp = Storage used after compression
Global-Comp Factor = Pre-Comp / (Size after de-dupe)
Local-Comp Factor = (Size after de-dupe) / Post-Comp
Total-Comp Factor = Pre-Comp/Post-Comp
Reduction % = ((Pre-Comp - Post-Comp) / Pre-Comp) * 100
sysadmin@dd01#

I would try to reverse it by moving the data limit to 64 and adding a #^--
This would make it available to the third drive thereby adding additional data without cumbersome coding normally needed so the duplicate shims are fast moving hyper crenum through the last fax or the seventh read of time.

... so the duplicate shims are fast moving hyper crenum through the last fax or the seventh read of time.

I'm trying to decide if this is gibberish or not. I feel stupid for not knowing for sure.

I'm trying to decide if this is gibberish or not. I feel stupid for not knowing for sure.


It is purely gibberish....

lol
so hey I encountered a new fun thing...

anyone have known good NFS mount options for AIX and Oracle 10.2.0.3? My NFS mount works fabulous with 9.2.0.5 but when I try a 10g database it barfs with this:

Starting backup at 03-FEB-09
channel t1: starting incremental level 0 datafile backupset
channel t1: specifying datafile(s) in backupset
input datafile fno=00067 name=/dev/rdbuild_0000064
channel t1: starting piece 1 at 03-FEB-09
released channel: t1
RMAN-00571: ================================================== =========
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ================================================== =========
RMAN-03009: failure of backup command on t1 channel at 02/03/2009 11:28:01
ORA-19504: failed to create file "/DEDUPE/oracle/BUILD1_HOT0-20090203_112755-s4k6ea60_1_1"
ORA-27054: NFS file system where the file is created or resides is not mounted with correct options
IBM AIX RISC System/6000 Error: 13: Permission denied
Additional information: 4


I can create and delete files with no issues it's only RMAN barfing. I've tried various and sundry options... this was my latest try...

mount -o rw,bg,hard,nointr,proto=tcp,vers=3,timeo=600,rsize =32768,wsize=32768 dd01:/backup/woc/rep/rman2 /DEDUPE


fails with same error. I've tweaked other options and it still fails. I'm going to call IBM support tomorrow but I figure this is more oracle/DBA related so who knows if they can help me. And oh in case I haven't said before I am no DBA.

Good and AIX in the same sentence?!

Is anything doing work as root? It may be denying uid 0 or SUID operations without an explicit declaration that's ok.

No it's performing the backup as oracle, like I said a 9.2.0.5 database can run a backup to it with no issue. This is something special to 10g.

Wonder if it's trying to MMAP a file or similar and getting rejected and that's getting caught by the generic permissions denied fallthrough.

My teammate found this:


There is nothing that can be done on the AIX side. The only solution is to follow the Oracle Metalink Note:387700.1

Log in as Oracle DBA user
SQL>alter system set event='10298 trace name context forever, level 32' scope= spfile ;
then shutdown immediate/startup the database, then try backup again.




and so the DBA is bouncing the database at 8:30 so we'll see :D

Yep that event is what fixed it. The NFS mount options do not matter one bit.

Figured I'd update this since a few folks are interested in this technology...

We had them bring their VTL option in (which is basically just HBAs) so we could see if there was a performance improvement and also b/c CA's shitty software was limiting on to 1 RMAN channel per disk group. I need to run multiple channels for our large databases so the VTL option made more sense not to mention we can now make copies of our data too.

Basically we're squishing our 2.2TB database into 10GB every night on a full online backup. If we did incrementals we could probably get it down to 1G but we're happy with this compression.

Our next plan is to see how our VCBs (vmware) compress and continue file and print testing. It's definitely a neat product and very easy to setup and use. All of my issues have been on the backup software side of things oh and then finding out that our network group has our backup VLANs all using a shared 1gig uplink with other live data....idiots.

I've been doing restores from the replicated DDR (data domain device) to prove and emulate a full disaster since. Again, it's all very easy and quite spiffy. I'm sure it has some issues, one thing that concerns me is when I start running out of space how easy it will be to add more, how I can throttle the replication, etc. In all I think it's worth the $$$ especially since we're spending $10k in tape cost a month due to our asanine lawsuit requirements of not being able to overwrite ANYTHING. We have something like 2,000 tapes at our offsite storage facility, they love us. This solution we're implementing costs less than 1 year of tape storage so the org is happy.

If anyone has specific questions or wants to talk to one of their sales people I can give you at least the North East dude's name who can surely point you in the right direction.

Our latest stats is an 85.2% reduction percentage. We have 11.2 TB stored on 1.6GB

Anyone know if something like this is how Dropbox instantly syncs huge files sometimes (assuming someone else has the same file synced)?

Apparently Dropbox is partly based on rsync (http://en.wikipedia.org/wiki/Rsync#Algorithm), if that helps.

rsync, etc, can do extent comparisions within a data block and will only transfer and insert the changes. Similar principle in operation, although the way rsync does it is much simpler than the DD-storage systems.

All I know is if I drop a popular ebook or podcast mp3 into my Dropbox, it syncs in <10s, even though they're 10-50+mb. Dandy.

Ah, they may have some sort of DD on the backend, which is a good idea given the amount of common data. Probably has at minimum a per-object checksum that is computed for whatever you're uploading before it transfers any data.

Chances are if you're uploading some where then they probably have something on their backend doing the compression for you.

I'm continually impressed at our rate of compression. Given that we're healthcare and that we cannot delete ANYTHING we have shit tons of common data so every time we backup we squish tighter. And restores have proven very easy even from the replicated node that will eventually be 30miles away at our second hospital. I feel very good about this solution.

In interesting news, Sun has implemented ZFS deduplication and will apparently have it in Solaris releases in the near future. ZFS is CDDL open source and available for a lot of systems, meaning we could be seeing fairly ubiquitous dedup data stores in the near future.