Rebuilding Files

The -rebuild option is used to rebuild or recreate an indexed file. You should rebuild a file that has become corrupt, or one that contains a large number of deleted records that you want to remove from the file. The command is:

vutil -rebuild  [ --slow ]  [ -l ]  [ -t tmpfile ]  [ -b # ]
  [ -3456 ]  [ -ac ]  [ +ce ]  [ -k keynum ]  [ -d dir ]
  [ -f factor ]  [ -s spoolfile [ -r ] [-m size] ]
  [ -p pre_factor ]  [ -g ext_factor ]  [ -q ]  [ -j #threads] [ files ]

Each file listed on the command line will be rebuilt. If no files are listed, then the standard input is read for the list. If, under UNIX, the named file is a symbolic link, the link is removed and the restored file is put in its place.

This option by default applies a read lock to the file that is rebuilt. The -l option applies a write lock instead.

When a file is rebuilt, a temporary file is created and each record from the original file is written to it. The -t option allows you to specify the name of the temporary file used during the rebuild. (You may not specify a directory, just a file name.) When -t is not specified, the temporary file's name begins with VTMP, followed by a six-character system-generated sequence. On Windows systems, the file's name begins with V. The rebuilding process reports the number of records found and the number of deleted records that were skipped. After the rebuild is complete, you are given the option of replacing the original file with the new one. If you do not replace it, you can examine the temporary file for correctness and replace it manually later. This is recommended if you suspect any difficulties.

When doing a rebuild, vutil places records that are rejected due to illegal duplicate keys into a file. Should this happen, vutil will report the name of the file that contains the rejected records. The format of this file is the same as a COBOL binary sequential file with variable-size records.

-a     This option may be used to specify automatic replacement of the original file by the newly created one. This is useful when you are calling vutil from a program or a script.

When used once, this option causes automatic replacement only if no records are skipped. If any records are skipped, you are prompted before file replacement takes place. When used more than once, this option causes automatic replacement of the file even if records were lost in the process.

  The multiple specification of option -a may be given in the following syntax formats:

-aa

-a -a

-a (other options) -a

-b # This option sets a new blocking factor for the file. The blocking factor specifies the size of the blocks to be used by the file. Blocks are sized in 512 byte increments. Vision 5 and Vision 6 files support blocking factors from 1 to 16 (16 = 8192 bytes). Vision 3 and 4 files support blocking factors from 1 to 2.

When you rebuild a file, if the file is very large, or has a tree height of more than five, or key lengths in excess of 40 bytes, you may want to experiment with larger blocking factors. You will need to perform some benchmarking to determine if a larger block size improves performance. For more about how block size can affect performance, see Other Vision Features.

If you specify a blocking factor greater than 2 for a Vision 3 or 4 file, the factor is automatically and silently reduced to the maximum of 2.

-c This option removes record compression from the file.
+c This option adds record compression to the file.
-d dir This option specifies an alternate directory for placing the rebuilt file. Dir should be the name of a directory on the host machine other than the directory containing the files to be rebuilt. When this option is used, the original files are not modified or destroyed. The rebuilt files are placed in dir with the same base name as the original files. This option can be useful if you do not have enough disk space on the device holding the files to rebuild them, but you do have space on another disk. This option implies the -a option because you are not prompted before the rebuild completes.
+e This option adds record encryption. It is not possible to remove record encryption (this would make encryption pointless).

Record compression and encryption may be added to a file, and compression may be removed from a file, regardless of the presence or absence of the WITH COMPRESSION and WITH ENCRYPTION phrases in the file's SELECT.

-f factor This option allows you to specify a compression factor. The factor must be an integer that specifies how much of the space saved by compression is actually to be removed from the record. Zero means no compression. A value of 1 means use the default factor (70).

For factors from 2 through 100, the factor is considered to be a percentage. It specifies how much of the space saved by compression is actually to be removed from each record. For example, suppose an 80-byte record is compressed to 30 bytes. Then the compression factor is used to determine how much of the 50 bytes of saved space is actually to be removed from the record. A compression factor of 70 would mean that 70% of the 50 bytes (35 bytes total) will be removed. This leaves 15 bytes for future expansion, and results in a compressed record size of 45 bytes (30 compressed size plus 15 extra for growth). The larger the compression factor, the more of the saved space is removed. A compression factor of 100 removes all saved space and allows no room for expansion.

-g ext_factor This option sets a new extension factor for the file. This is the number of blocks that are added to a file's size when the file needs to be expanded. The default is one block. Specifying more than one enables you to take advantage of contiguous disk space, and thus may help to prevent fragmentation of the file as it grows.
-j #threads This option enables you to use multiple threads when rebuilding multi-keyed, Vision 6-formatted (or later) Vision files, which can improve the performance of the rebuild.

#threads must be a number between 2 and 120, and specifies the number of threads to be used in the rebuild process. Each thread is used to rebuild a key in the file; a new thread being spawned when a previous one completes rebuilding a key, until all keys are rebuilt.

The optimum number of threads that gives the best performance will vary depending on your system, and so trial and error in specifying different numbers of threads may be required. As a general guideline, specify a similar number of threads as your system has CPUs.

An important consideration when specifying the number of threads is that each thread is allocated memory as if it were its own version of vutil; therefore, the memory you have allocated for the V_BULK_MEMORY, V_BUFFERS, MAX_FILES, etc... variables is multiplied by the number of threads.

Another consideration is that rebuilding using multiple threads also uses additional temporary disk space, to hold the intermediate results of each key rebuild. This could be up to double the amount of space that a non-threaded rebuild would use.

-k keynum This option specifies that you want to rebuild the file in key order. The -k must be followed (as the next separate argument) by the number of the key that you want to use, with zero indicating the primary key, one indicating the first alternate key, and so forth. For example, to rebuild file1 in primary key order, you would specify:
vutil -rebuild -k 0 file1

There are two situations in which the -k option is particularly valuable. If you are repeatedly processing a file along a particular key, then you can improve performance by rebuilding the file in key order. This is particularly true if you do a great deal of sequential processing (common in reports). When you rebuild in key order, records that are logically adjacent (according to their key values) are placed next to each other on the disk. This maximizes the runtime's ability to improve performance with its read caching capabilities. It also minimizes the distance that the disk must seek when you are reading records sequentially by that key. Write performance also improves in applications that write large numbers of records in keyed sequence.

A second situation in which the -k option is valuable is when the default rebuild method fails to recover a file fully. This can occur if the chain of data records has been corrupted. When -k is specified, vutil will use the index you provide to try to locate the records, and will often find more records this way.

-p pre_factor This option allows you to specify the number of blocks that vutil is to preallocate to the file. pre_factor must be a numeric value between one and 2,097,152. The maximum pre-allocation factor varies with Vision version. Vision 5 and Vision 6 files accept the upper limit of 2,097,152 blocks. Vision 3, and 4 files are restricted to a maximum of 65,535 blocks. If a larger pre-allocation factor is specified than the Vision version allows, the factor is automatically and silently reduced to the allowable limit.
-q This option causes vutil to exit (with status 99) if user interaction is required.
-s spoolfile This option indicates that you want to use the spooling form of rebuild. This is especially helpful if you do not have adequate disk space to hold the new file. This option spools the records to removable media and then rebuilds the file over the existing file. This keeps only one copy of the file on disk and thus allows you to rebuild even when free disk space is limited. Note that the spooled file is not compressed.

The -s option must be followed by the name of the file to which you want to spool records. This can be any file but is usually the name of a tape or diskette device. For example, you might specify

vutil -rebuild -s /dev/rmt0 badfile

to rebuild the file badfile by spooling records to the tape device /dev/rmt0.

When -s is specified, vutil writes all the records it can recover from the corrupt file to the spool file, and then rebuilds the file using these records. You will be prompted to change media if the spool file gets full.

There are two additional options that can be used with the -s option:

  • -- allows you to recover an interrupted rebuild. When -r is specified, vutil skips the step of writing records to the spool device. Instead, it prompts you to mount the first volume of the spool file before it begins the rebuilding process.
  • -m size -- allows you to specify the size of the spool media. It is followed by the number of 1024-byte records that can fit on the media. This is useful when the spool device driver does not handle the end-of-media condition correctly. For example, if you were spooling to a 1.2 MB floppy disk, you could specify:
    -m 1200
--slow This option causes vutil to open the file for mass update instead of for bulk addition. This usually causes vutil to run slower. The only reason for using this option is as a possible work-around to some difficulty with using bulk addition.
-# This option causes vutil to rebuild the file in the Vision file format specified by the integer. Valid values include 3, 4, 5 and 6. If the -# option is not included, the file is rebuilt in the same format as the original file.

When you perform vutil -rebuild, one of the following status values is returned to the host operating system when vutil quits:

0 file successfully rebuilt
1 rebuild not performed because the file is locked
2 rebuild not fully performed because some records were not recovered
99 user interaction was required, and the -q switch was set
255 other errors