Over at IdeaJam, someone posted an idea on
certifying NTFS defragmentation utilities for Domino. Below is my comment there - I would love to see what some of you think about this. Also, here is an example of what I see time and again when looking at fragmentation on Domino systems:

I don't care what file system you use, if you are using Domino you will have fragmentation. Even the WAFL fs has a defrag utility, even though it may be less susceptible.
The "problem" with Domino databases is that right after you defrag the volume, the file system starts to fragment because you're adding to the databases and that data has to go somewhere. It cannot go into a contiguous block because there may not be block space available due to the defrag tool keeping all files contiguous when defragging. And this isn't necessarily a problem with Domino - it's more of an issue with the file system and how it allocates storage.
To deal with this and achieve good uptime, I had hoped that a copy-style compact would be a poor man's defrag. Initially, it would seem that this would work because a new copy of the database would be created. However, this is not so because Domino doesn't (or can't?) always allocate a contiguous block that will store the entire database. Besides, Domino creates the database and has to start allocating space for the individual documents. Even if Domino could allocate a contiguous block, it would still suffer from fragmentation when new documents are added. (note: I didn't put this at my Idea Jam comment, but I would note that other transactions on the system will likely start to allocate blocks where the copy-style compact would be placing the data, so this is why more fragmentation may occur).
If Domino could get around this and use a contiguous block, I would propose something like "load compact -c +10" so that you get 10% white space in the new copy. Then you could have some white space in which new documents can be created and not cause further fragmentation.
Taking Domino out of the equation, something that may actually be faster than a defrag is to move all of the databases (or at least the largest or most heavily fragmented) off of the disk onto another volume and then run a defrag on the remaining files on that drive. Then just move the data back and the file system should allocate contiguous blocks for the new data while it's creating them one at a time.
Any other thoughts?
1 Tim Tripcony Permalink I'm curious what others' take is on fragmentation in Linux as well... from what I've seen so far, public opinion is that Linux file systems handle block allocation so ideally that defrag is essentially unnecessary, but I have a hard time believing that. Granted, since switching my server at home to Suse, the performance and reliability gains have been undeniable, but on a heavily utilized Domino server, is defragmenting truly unnecessary just because it's running on Linux?
2 Chris Whisonant Permalink Good question Tim. If you look at the 2nd comment at the Idea Jam I linked to above, you can see that someone was having fragmentation problems on Linux. Like I said, NO file system is impervious to fragmentation. Some may handle it better than others, but you're dealing with bits and bytes. And unless the file system leaves chunks of free space between each file and then adds new data into that free space, then it will have to add the data elsewhere on the drive (some would call these fragments of data...).
Even the impenetrable i5/OS has the STRDSKRGZ command to reorg the
disk. But most people may not notice the fragmentation on SAN/NAS
or the mid-range/mainframe platforms because of the number of disk
arms. But this doesn't mean that they are impervious to
fragmentation. Sure some of them may be somewhat proactive in
handling the fragmentation, but that doesn't mean that they don't
fragment (the admins/users just don't notice it).
3 Richard H. Schwartz Permalink Years ago, I realized that compact and defrag are natural enemies, unless you use them carefully.
As you have noted, aggressive defrag and aggressive compact
together guarantee that rapid re-fragmentation starts immediately
after de-fragmentation. And once the largest database size exceeds
the largest free block on a volume, a rapid downward spiral occurs
if you are using copy-style compact.
Careful use of defrag and compact basically means this: run compact
only immediately following a successful defrag; and to insure that
defrag is successful make sure that you have sufficient free space
on the volume so that your defrag will create a contiguous free
block as large as your N largest databases combined. (N is up to
you to decide on, but the higher the better.)
I like your suggestion for a +10 argument to go with -c. I like it
a lot, actually. Put it on IdeaJam and let me know, so I remember
to vote on it.
4 Chris Whisonant Permalink Richard, a few months ago I actually found an OLD post from you sometime back in the 6/7 or 4/5 forum - but I can't seem to find it now. It helped me out a lot on some thoughts on fragmentation - very good info. Also, have you read Daniel's post in the NEXT forum on fragmentation? Very good stuff there and in the replies.
I posted an Idea on this +10 thing:
http://ideajam.net/IdeaJam/P/ij.nsf/0/D007A96ABD0EC28F862573F00075D18D?OpenDocument
5 Fabien Menigault Permalink Very Interesting Post ! personally i defrag often as possible to increase performance playing Crysis .-p
More seriously, most of us when we format a partition, we use the
default allocation unit size ( 4096 )
What would be the better choice for Domino ?
512,1024,2048,4096,8192, 16ko, 32ko, 64ko
This, also has his role to play, i guess
++
6 Nathan T Freeman Permalink Made an idea linked to yours, Chris.
How about if I could just set a FIXED size on the NSF? Then
fragmentation of the file goes away.
Domino already has quota management, so there's a management
process in place to help admins do it.
http://ideajam.net/IdeaJam/P/ij.nsf/0/B3C64CE60BEC57AE862573F1001CBE0E?OpenDocument
7 Chris Whisonant Permalink Nathan, I saw that Idea - going over to promote it now. That's a really great idea as well!! :)