1 <html><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"><title>Chapter 32. Handling Large Directories</title><link rel="stylesheet" href="samba.css" type="text/css"><meta name="generator" content="DocBook XSL Stylesheets V1.68.1"><link rel="start" href="index.html" title="The Official Samba-3 HOWTO and Reference Guide"><link rel="up" href="optional.html" title="Part III. Advanced Configuration"><link rel="prev" href="SambaHA.html" title="Chapter 31. High Availability"><link rel="next" href="cfgsmarts.html" title="Chapter 33. Advanced Configuration Techniques"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Chapter 32. Handling Large Directories</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="SambaHA.html">Prev</a> </td><th width="60%" align="center">Part III. Advanced Configuration</th><td width="20%" align="right"> <a accesskey="n" href="cfgsmarts.html">Next</a></td></tr></table><hr></div><div class="chapter" lang="en"><div class="titlepage"><div><div><h2 class="title"><a name="largefile"></a>Chapter 32. Handling Large Directories</h2></div><div><div class="author"><h3 class="author"><span class="firstname">Jeremy</span> <span class="surname">Allison</span></h3><div class="affiliation"><span class="orgname">Samba Team<br></span><div class="address"><p><code class="email"><<a href="mailto:jra@samba.org">jra@samba.org</a>></code></p></div></div></div></div><div><div class="author"><h3 class="author"><span class="firstname">John</span> <span class="othername">H.</span> <span class="surname">Terpstra</span></h3><div class="affiliation"><span class="orgname">Samba Team<br></span><div class="address"><p><code class="email"><<a href="mailto:jht@samba.org">jht@samba.org</a>></code></p></div></div></div></div><div><p class="pubdate">March 5, 2005</p></div></div></div><p>
2 <a class="indexterm" name="id2639992"></a>
3 <a class="indexterm" name="id2639999"></a>
4 <a class="indexterm" name="id2640006"></a>
5 Samba-3.0.12 and later implements a solution for sites that have experienced performance degradation due to the
6 problem of using Samba-3 with applications that need large numbers of files (100,000 or more) per directory.
8 <a class="indexterm" name="id2640020"></a>
9 <a class="indexterm" name="id2640027"></a>
10 The key was fixing the directory handling to read only the current list requested instead of the old
11 (up to samba-3.0.11) behavior of reading the entire directory into memory before doling out names.
12 Normally this would have broken OS/2 applications, which have very strange delete semantics, but by
13 stealing logic from Samba4 (thanks, Tridge), the current code in 3.0.12 handles this correctly.
15 <a class="indexterm" name="id2640044"></a>
16 <a class="indexterm" name="id2640051"></a>
17 To set up an application that needs large numbers of files per directory in a way that does not
18 damage performance unduly, follow these steps:
20 <a class="indexterm" name="id2640063"></a>
21 First, you need to canonicalize all the files in the directory to have one case, upper or lower take your
22 pick (I chose upper because all my files were already uppercase names). Then set up a new custom share for the
23 application as follows:
24 </p><table class="simplelist" border="0" summary="Simple list"><tr><td> </td></tr><tr><td><em class="parameter"><code>[bigshare]</code></em></td></tr><tr><td><a class="indexterm" name="id2640092"></a><em class="parameter"><code>path = /data/manyfilesdir</code></em></td></tr><tr><td><a class="indexterm" name="id2640104"></a><em class="parameter"><code>read only = no</code></em></td></tr><tr><td><a class="indexterm" name="id2640117"></a><em class="parameter"><code>case sensitive = True</code></em></td></tr><tr><td><a class="indexterm" name="id2640130"></a><em class="parameter"><code>default case = upper</code></em></td></tr><tr><td><a class="indexterm" name="id2640143"></a><em class="parameter"><code>preserve case = no</code></em></td></tr><tr><td><a class="indexterm" name="id2640156"></a><em class="parameter"><code>short preserve case = no</code></em></td></tr></table><p>
26 <a class="indexterm" name="id2640172"></a>
27 <a class="indexterm" name="id2640179"></a>
28 <a class="indexterm" name="id2640186"></a>
29 Of course, use your own path and settings, but set the case options to match the case of all the files in your
30 directory. The path should point at the large directory needed for the application any new files created in
31 there and in any paths under it will be forced by smbd into uppercase, but smbd will no longer have to scan
32 the directory for names: it knows that if a file does not exist in uppercase, then it doesn't exist at all.
34 <a class="indexterm" name="id2640206"></a>
35 <a class="indexterm" name="id2640212"></a>
36 <a class="indexterm" name="id2640219"></a>
37 The secret to this is really in the <a class="indexterm" name="id2640227"></a>case sensitive = True
38 line. This tells smbd never to scan for case-insensitive versions of names. So if an application asks for a file
39 called <code class="filename">FOO</code>, and it cannot be found by a simple stat call, then smbd will return file not
40 found immediately without scanning the containing directory for a version of a different case. The other
41 <code class="filename">xxx case xxx</code> lines make this work by forcing a consistent case on all files created by
42 <span class="application">smbd</span>.
44 <a class="indexterm" name="id2640261"></a>
45 <a class="indexterm" name="id2640268"></a>
46 <a class="indexterm" name="id2640275"></a>
47 Remember, all files and directories under the <em class="parameter"><code>path</code></em> directory must be in uppercase
48 with this <code class="filename">smb.conf</code> stanza because <span class="application">smbd</span> will not be able to find lowercase filenames with these settings. Also
49 note that this is done on a per-share basis, allowing this parameter to be set only for a share servicing an application with
50 this problematic behavior (using large numbers of entries in a directory) the rest of your <span class="application">smbd</span> shares
51 don't need to be affected.
53 This makes smbd much faster when dealing with large directories. My test case has over 100,000 files, and
54 smbd now deals with this very efficiently.
55 </p></div><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="SambaHA.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="optional.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="cfgsmarts.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">Chapter 31. High Availability </td><td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td><td width="40%" align="right" valign="top"> Chapter 33. Advanced Configuration Techniques</td></tr></table></div></body></html>
projects / samba / blob