--- /dev/null
+<html><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"><title>Chapter 32. Handling Large Directories</title><link rel="stylesheet" href="samba.css" type="text/css"><meta name="generator" content="DocBook XSL Stylesheets V1.68.1"><link rel="start" href="index.html" title="The Official Samba-3 HOWTO and Reference Guide"><link rel="up" href="optional.html" title="Part III. Advanced Configuration"><link rel="prev" href="SambaHA.html" title="Chapter 31. High Availability"><link rel="next" href="cfgsmarts.html" title="Chapter 33. Advanced Configuration Techniques"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Chapter 32. Handling Large Directories</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="SambaHA.html">Prev</a> </td><th width="60%" align="center">Part III. Advanced Configuration</th><td width="20%" align="right"> <a accesskey="n" href="cfgsmarts.html">Next</a></td></tr></table><hr></div><div class="chapter" lang="en"><div class="titlepage"><div><div><h2 class="title"><a name="largefile"></a>Chapter 32. Handling Large Directories</h2></div><div><div class="author"><h3 class="author"><span class="firstname">Jeremy</span> <span class="surname">Allison</span></h3><div class="affiliation"><span class="orgname">Samba Team<br></span><div class="address"><p><code class="email"><<a href="mailto:jra@samba.org">jra@samba.org</a>></code></p></div></div></div></div><div><div class="author"><h3 class="author"><span class="firstname">John</span> <span class="othername">H.</span> <span class="surname">Terpstra</span></h3><div class="affiliation"><span class="orgname">Samba Team<br></span><div class="address"><p><code class="email"><<a href="mailto:jht@samba.org">jht@samba.org</a>></code></p></div></div></div></div><div><p class="pubdate">March 5, 2005</p></div></div></div><p>
+<a class="indexterm" name="id2639992"></a>
+<a class="indexterm" name="id2639999"></a>
+<a class="indexterm" name="id2640006"></a>
+Samba-3.0.12 and later implements a solution for sites that have experienced performance degradation due to the
+problem of using Samba-3 with applications that need large numbers of files (100,000 or more) per directory.
+</p><p>
+<a class="indexterm" name="id2640020"></a>
+<a class="indexterm" name="id2640027"></a>
+The key was fixing the directory handling to read only the current list requested instead of the old
+(up to samba-3.0.11) behavior of reading the entire directory into memory before doling out names.
+Normally this would have broken OS/2 applications, which have very strange delete semantics, but by
+stealing logic from Samba4 (thanks, Tridge), the current code in 3.0.12 handles this correctly.
+</p><p>
+<a class="indexterm" name="id2640044"></a>
+<a class="indexterm" name="id2640051"></a>
+To set up an application that needs large numbers of files per directory in a way that does not
+damage performance unduly, follow these steps:
+</p><p>
+<a class="indexterm" name="id2640063"></a>
+First, you need to canonicalize all the files in the directory to have one case, upper or lower take your
+pick (I chose upper because all my files were already uppercase names). Then set up a new custom share for the
+application as follows:
+</p><table class="simplelist" border="0" summary="Simple list"><tr><td> </td></tr><tr><td><em class="parameter"><code>[bigshare]</code></em></td></tr><tr><td><a class="indexterm" name="id2640092"></a><em class="parameter"><code>path = /data/manyfilesdir</code></em></td></tr><tr><td><a class="indexterm" name="id2640104"></a><em class="parameter"><code>read only = no</code></em></td></tr><tr><td><a class="indexterm" name="id2640117"></a><em class="parameter"><code>case sensitive = True</code></em></td></tr><tr><td><a class="indexterm" name="id2640130"></a><em class="parameter"><code>default case = upper</code></em></td></tr><tr><td><a class="indexterm" name="id2640143"></a><em class="parameter"><code>preserve case = no</code></em></td></tr><tr><td><a class="indexterm" name="id2640156"></a><em class="parameter"><code>short preserve case = no</code></em></td></tr></table><p>
+</p><p>
+<a class="indexterm" name="id2640172"></a>
+<a class="indexterm" name="id2640179"></a>
+<a class="indexterm" name="id2640186"></a>
+Of course, use your own path and settings, but set the case options to match the case of all the files in your
+directory. The path should point at the large directory needed for the application any new files created in
+there and in any paths under it will be forced by smbd into uppercase, but smbd will no longer have to scan
+the directory for names: it knows that if a file does not exist in uppercase, then it doesn't exist at all.
+</p><p>
+<a class="indexterm" name="id2640206"></a>
+<a class="indexterm" name="id2640212"></a>
+<a class="indexterm" name="id2640219"></a>
+The secret to this is really in the <a class="indexterm" name="id2640227"></a>case sensitive = True
+line. This tells smbd never to scan for case-insensitive versions of names. So if an application asks for a file
+called <code class="filename">FOO</code>, and it cannot be found by a simple stat call, then smbd will return file not
+found immediately without scanning the containing directory for a version of a different case. The other
+<code class="filename">xxx case xxx</code> lines make this work by forcing a consistent case on all files created by
+<span class="application">smbd</span>.
+</p><p>
+<a class="indexterm" name="id2640261"></a>
+<a class="indexterm" name="id2640268"></a>
+<a class="indexterm" name="id2640275"></a>
+Remember, all files and directories under the <em class="parameter"><code>path</code></em> directory must be in uppercase
+with this <code class="filename">smb.conf</code> stanza because <span class="application">smbd</span> will not be able to find lowercase filenames with these settings. Also
+note that this is done on a per-share basis, allowing this parameter to be set only for a share servicing an application with
+this problematic behavior (using large numbers of entries in a directory) the rest of your <span class="application">smbd</span> shares
+don't need to be affected.
+</p><p>
+This makes smbd much faster when dealing with large directories. My test case has over 100,000 files, and
+smbd now deals with this very efficiently.
+</p></div><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="SambaHA.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="optional.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="cfgsmarts.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">Chapter 31. High Availability </td><td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td><td width="40%" align="right" valign="top"> Chapter 33. Advanced Configuration Techniques</td></tr></table></div></body></html>
projects / samba / blobdiff