Filer Jedi - Adventures in Storage

So recently we've been grappling with some odd corruption issues on Isilon clusters. Our application writes a bunch of data from one node, and then there are separate read servers. We've noticed some garbage data getting returned immediately after writing the file when the read nodes attempt to open the newly created file. If we later read it again, the file was fine. Huh?

Chalk it up to Isilon? Perhaps not.

Continue reading "Odd corruption issues with Linux NFS clients" »

Posted by Filer Ninja at 03:34 PM | Permalink | Comments (0) | TrackBacks (0)

Here's one of the xml files I mentioned in the previous file. It's almost straight from the man page, except I added nfs latency counters.

Posted by FilerJedi at 06:19 PM | Permalink | Comments (1) | TrackBacks (0)

Continue reading "stats - who is your Daddy and what does he do?" »

Posted by FilerJedi at 04:44 PM | Permalink | Comments (0) | TrackBacks (0)

Here's a matrix of maximum drives you can have in a single aggregate. Most of these numbers are extrapolated based on testing using 500GB and 750GB drives.

Continue reading "Actual NetApp usable sizes - Updated 3/22/2008" »

Posted by Filer Ninja at 08:05 AM | Permalink | Comments (9) | TrackBacks (0)

In order to account for the size of the next snapmirror transfer:

1.) Calculate the delta in ‘df’ output from source and destination (source used minus destination used)
- AND -
2.) Calculate the sum of space used for all snapshots since (and including) the most recent snapmirror snapshot (the baseline).

In order to calculate the size of the snapshots, you can use one of 2 methods:
1.) Use filerview to see the size of the snapshot
2.) Use ‘snap reclaimable’ to see the size of the snapshot

Add the total of the snapshot space and the the total of the 'df' space together and you’ve got your next snapmirror transfer size.

Enjoy!

Posted by cannotsuggestaname at 07:26 AM | Permalink | Comments (1) | TrackBacks (0)

Want to start monitoring your filers using SNMP? We've got some useful OIDs to get you started. For this discussion, we'll use the NET-SNMP commandline utilities available at http://www.net-snmp.org/

To start, I've gathered a short list of SNMP OIDs you can use to start monitoring your netapp filers using the NET-SNMP utilities.

Continue reading "Fun with netapp.mib" »

Posted by Filer Ninja at 10:23 AM | Permalink | Comments (8) | TrackBacks (0)

We had an event on a older fcp installation in production this week, and I thought I'd breifly talk about the steps to fix it. The system is a F880 (first netapp platform with a flash card) and OnTap 6.4.4P6. The Solaris system , a E3500, running Solaris 8. There are four total solaris hosts, each getting 2 luns from a singel f880.

We had one hosts FCP card fail on us. The card was replaced quickly by our operations team, but the lun couldn't be seen on the host until we added the new cards WWN to the igroup for that lun. Here's how I fixed it.

This is on a production box that's scheduled to be replaced soon with a 4 node Oracle 10g RAC on Linux with NFS as it's storage. And the NFS server is going to be a 3050 with FC disks. But until then, this config has to limp along.

Continue reading "Quickie on using FCP luns on a filer." »

Posted by FilerJedi at 08:15 AM | Permalink | Comments (0) | TrackBacks (0)

So the extended break should be behind us. Just been busy with a lot of things. Expect new articles shortly on things like:
1) Visting EMC World
2) ibrix
3) NetApp Monitoring
4) Evaluating Storage
5) Checking out Cisco FC switchs and (maybe) comparing them to Brocade.

We also got a 6070 cluster, and other fun things planned.

I've also pointed at this site a new name "adventuresinstorage.com" goes to this very site as well. It's ment to reflect the more 'storage in general' focus of this blog.

Anway thanks for hanging in though the dead space there, and stay tuned.

Posted by FilerJedi at 12:50 AM | Permalink | Comments (2) | TrackBacks (0)

I just got done working with NetApp on a sever performance problem on one of our high performance database systems. It's a 4 node Oracle 10g RAC cluster using NFS. We moved moved from around 70 spindles at 15k RPM to 110 spindles at 10k RPM; we then syncmirrored it giving us an aggreagte of 220 spindles to read from. In doing that, we changed a lot with Oracle as well, like tablespace block size, and some sql tunings. Once it was all done, our read thoughput tanked. From somewhere around 95MB / sec read thoughput to 40-50MB / sec read thoughput.

Read on to know how we solved this degredation.

Continue reading "Linux nfs client parameter - sunrpc.tcp_slot_table_entries" »

Posted by FilerJedi at 12:09 PM | Permalink | Comments (5) | TrackBacks (0)

Well, it looks like my company is sponsering me going to EMC world later this month. I didn't get a chance to check out SNW in San Diego this year, but it seemed like the place to be. Lots of cool stuff I wish I got to see.

I'll update a report on what I see of EMC world. I have no bias, but I am the NetApp bigot where I work, so this should be interesting.

Posted by FilerJedi at 12:28 AM | Permalink | Comments (1) | TrackBacks (0)