Spotlight tips

This FAQ, derived from information in the "Spotlight" chapters of the Tiger® and Leopard® Editions of our book Troubleshooting Mac OS X, addresses the following topics related to Spotlight™ in Mac OS X 10.4 Tiger and Mac OS X 10.5 Leopard:

• A brief tutorial on Spotlight.
• Rebuilding Spotlight indices.
• Stopping Spotlight indexing, which addresses:
  • Anomalies from stopping indexing using Privacy under Tiger.
  • Disabling Spotlight indexing while indexing is in progress.
• Preventing Spotlight from indexing backup volumes.

A comprehensive treatment of Spotlight, including extensive troubleshooting information, can be found in the "Spotlight" chapter of our book, Troubleshooting Mac OS X, Tiger Edition.

A brief tutorial on Spotlight

• Spotlight will begin indexing any volume as soon as it is mounted, with certain exceptions: CDs, DVDs, and network-attached or shared volumes are not indexed when mounted. By volume we mean either a hard drive with a single partition, or a specific partition on a hard drive with multiple partitions.
• By default, Spotlight indexes and searches a limited number of folders, specifically those listed in the AppleCare® Knowledge Base document "Mac OS X 10.4: Where does Spotlight search?"
• Furthermore, Spotlight will neither index nor search:
  • Hidden files: Files hose names begin with a period (.).
  • Invisible files: Files whose invisible file-system attribute has been enabled, specifically the kIsInvisible Finder flag.
  • Files within hidden or invisible folders.
  • Under Tiger, System-related files or folders. These are indexed and can be searched under Leopard.
• Spotlight saves all its index-related files in the .Spotlight-V100 directory at the root level of a volume it has indexed.
• If you open Activity Monitor, the process mds is active when Spotlight is either indexing or searching. The mdimport process is also active during indexing. Under Leopard, the mdworker process is also active during indexing.
• Under Tiger, Spotlight is disabled in Safe Mode. Under Leopard, Spotlight is available in Safe Mode.
• Spotlight employs SearchKit for content indexing and finding files by their content. SearchKit was first introduced in Mac OS X 10.3 Panther® for content indexing and Find By Content. However, Spotlight does not remove content indices created under previous versions of Mac OS X, which can consume considerable space on the hard drive.
• Removing these indices is especially tedious since Spotlight will not find hidden files whose names begin with a period (.) and index files created under Panther or Mac OS X 10.2 Jaguar have file names beginning with .FBC .
• As a result, you must use a utility like DEVONtechnologies EasyFind to locate and remove index files created under Panther or Jaguar.
• See our "Troubleshooting a failed indexing operation" FAQ for information about index files created by ContentIndexing under prior versions of Mac OS X.
• A number of Terminal commands are available for working with Spotlight:
  • mdfind : Search Spotlight metadata indices.
  • mdimport : Force Spotlight to import metadata for a specific directory.
  • mdls : List the metadata attributes of a specific file.
  • mdutil : A utility for managing Spotlight indices.
• You can find a variety of hints for manipulating the Spotlight index files, overcoming some of Spotlight's default limitations, and for using the Spotlight-specific Terminal commands by searching the Mac OS X Hints site for spotlight.
• Additional information about Spotlight can be found by reviewing the documents listed in the Related links section of this FAQ.

Rebuilding Spotlight indices

There are two approaches to rebuilding the Spotlight indices on a given volume:

Stopping Spotlight indexing

A common practice to stop Spotlight from indexing volumes is to add the volume to the Privacy while the volume is being indexed.

Under Tiger, using Privacy to stop indexing can cause a variety of anomalous behaviors which are the result of inconsistencies in how the Privacy tab works vs. certain functions of the mdutil command. These anomalies have been corrected in Leopard.

Anomalies from stopping indexing using Privacy under Tiger

The following anomalous interactions between Privacy and mdutil, whether bugs or undocumented effects, were found testing with Tiger:

1. If you add a volume to Privacy while Spotlight is indexing that volume:
   1. Spotlight immediately stops indexing the volume.
   2. You will be unable to search that volume with Find (Finder > Command-F).
      1. Find is implemented in Spotlight and respects the settings specified in the Privacy tab of Spotlight Preferences.
      2. Adding a volume to Privacy creates the file _exclusions.plist in the volume's .Spotlight-V100 directory, with a setting specifying that the volume cannot be searched.
   3. Incomplete index files may remain in the volume's .Spotlight-V100 directory.
   4. The Spotlight indexing process may crash, leaving the file mds-crash-state in the volume's .Spotlight-V100 directory. This file appears to prevent Spotlight from indexing the volume again — even if the volume is removed from Privacy — until the volume's .Spotlight-V100 directory has been removed via the command:
      1. sudo rm -ri /path_to_volume/.Spotlight-V100
   5. where /path_to_volume is the path of the volume in question. [1]
   6. The volume can then be reindexed by any of the following methods:
      1. Restarting your Mac.
      2. Using Disk Utility to first unmount, then mount the volume.
      3. Issuing the command: sudo mdutil -E /path_to_volume
2. If you add a volume to Privacy after Spotlight has indexed that volume:
   1. As noted previously, you will be unable to search the volume with Find.
   2. Spotlight will no longer index the volume when its contents change. However, indexing is still enabled as can be shown by the command:
      1. sudo mdutil -s /path_to_volume
   3. The _exclusions.plist file, created when the volume was added to Privacy, prevents both searching the volume and subsequent indexing of that volume while it is listed in Privacy.
   4. The ContentIndex.db file is removed from the volume's .Spotlight-V100 directory.
   5. The metadata index file store.db in the volume's .Spotlight-V100 directory is reduced in size. Apparently, all metadata is removed from this file for files whose contents were included in the now-removed ContentIndex.db file.
   6. Attempting to remove the remaining index files on the volume using the command:
      1. sudo mdutil -E /path_to_volume
   7. has no effect.
3. If you add a volume to Privacy and then disable indexing on that volume via the command:
   1. sudo mdutil -i off /path_to_volume
   2. The volume no longer appears in Privacy. If you attempt to add the volume to Privacy again, it will not appear in Privacy.
      1. The volume will again appear in Privacy if you enable indexing with the command:
         1. sudo mdutil -i on /path_to_volume
      2. Despite this command, indexing is still disabled on the volume: the setting in the _exclusions.plist file in the volume's .Spotlight-V100 directory, created when it was added to Privacy, continues to prohibit indexing.
      3. The general rule here is the following: you cannot add to Privacy any volume for which indexing has been disabled with the -i off flag of mdutil.
   3. The volume can be searched to some degree with both Spotlight and Find.
      1. Spotlight and Find ignore the setting in the _exclusions.plist file in the volume's .Spotlight-V100 directory, despite the fact that adding the volume to Privacy sets this file to indicate the volume cannot be searched.
      2. The results of a search using Spotlight or Find for a string, such as xyz, on the volume will be equivalent to the results from the UNIX® find command:
         1. find -x /path_to_volume -iname "*xyz*"
      3. with the exception that neither Spotlight nor Find will not search within application packages whereas UNIX find will, the latter depending on permissions.
      4. The general rule here is the following: disabling indexing on a volume with the -i off flag of mdutil after the volume has been added to Privacy enables the volume to be searched, to some degree, in both Spotlight and Find.
4. The only way to remove all the index-related files after stopping indexing via Privacy is to use mdutil to both disable indexing on the volume and erase the index files. This is accomplished by issuing the following two commands in Terminal in the order specified:
   1. sudo mdutil -i off /path_to_volume
   2. sudo mdutil -E /path_to_volume
5. While one may be tempted to combine both the -i off and -E flags in a single mdutil command, i.e.
   1. sudo mdutil -i off -E /path_to_volume
6. we have seen mixed results from such: sometimes it works, sometimes it does not. Using the commands individually, in the order specified, works consistently.
7. You will then also be able to search the volume with Find or Spotlight as noted item 3 above. Once again, any volume added to Privacy will no longer appear in Privacy after indexing has been disabled on that volume via the -i off flag of mdutil.

Disabling Spotlight indexing while indexing is in progress

Adding a volume to Privacy while it is being indexed is not the best method for stopping the indexing of that volume. Instead, we recommend the following:

Preventing Spotlight from indexing backup volumes

If you use a backup or synchronization utility to duplicate one volume on your Mac — the source volume — to another volume, such as a FireWire drive — the target volume, you may notice that Spotlight will index the target volume even though you may have previously disabled indexing on such. This is especially true if:

• Your backup utility performs either:
  • An incremental duplicate, copying only new or changed files from the source volume to the target volume.
  • A full duplicate, copying all files from the source volume to the target volume, replacing the contents of the target volume.
• Spotlight indexing is enabled on the source volume.

As part of the duplication, your backup utility may copy the .Spotlight-V100 directory from the source volume to the target volume. If Spotlight is enabled on the source volume, then it will be enabled on the target volume after the .Spotlight-V100 directory is copied to such from the source volume.

The information Spotlight uses to ascertain if indexing is enabled on a volume is saved in the .Spotlight-V100 directory at the root level of that volume. Therefore, if you backup a volume on which indexing is permitted, after the /.Spotlight-V100 directory is copied from the source volume to the target volume, indexing will be enabled on the target volume.

Additionally, Spotlight will begin to index the target volume while the backup is in progress. This can slow the backup process considerably: in our tests, this has caused backups to require up to 150% of their normal time, i.e. a one-hour backup could require 1.5 hours due to Spotlight indexing the target volume.

The latest versions of the backup and recovery applications recommended in our "Backup and Recovery" FAQ will respect the state of Spotlight indexing on the target volume. If you use one of the recommended applications, you can avoid the problem of Spotlight indexing backup volumes while the backup is in progress by simply disabling Spotlight indexing on the target volume before performing the first backup.

Related links

• AppleCare Knowledge Base documents related to Spotlight and Tiger or Leopard.
• Spotlight Technology Brief (PDF).
• "Working with Spotlight".
• Spotlight Overview.
• Spotlight Query Programming Topics.
• Spotlight Importer Programming Guide.
• Spotlight Metadata Attributes Reference.
• Search Kit Programming Guide.
• Search Kit Reference.
• Finding corrupted files.

Notes

[1] The value for /path_to_volume in the Terminal commands specified in this FAQ depends on the volume in question:

• If the volume is your current Mac OS X startup disk, substitute the single character / for the strings /path_to_volume or /path_to_volume/ in the commands.
• If the volume is other than your current Mac OS X startup disk, substitute:
• /Volumes/volume_name
• for /path_to_volume, where volume_name is the name of the volume in question.
• Note that if volume_name contains spaces or special characters, either the entire path must be enclosed in double quotes or the spaces and special characters must be escaped with a reverse solidus, aka a backslash (\).
• Example: to stop and prohibit indexing on a volume named Backup Drive you would issue one of the following commands:
• sudo mdutil -i off /Volumes/Backup\ Drive
• sudo mdutil -i off "/Volumes/Backup Drive"
• The first escapes the space in Backup Drive with a backslash, while the second encloses the entire path in double quotes.