Comprestimator is a command line host-based utility that can be used to estimate an expected compression rate for block devices.
The Comprestimator utility uses advanced mathematical and statistical algorithms to perform the sampling and analysis process in a very short and efficient way. The utility also displays its accuracy level by showing the maximum error range of the results achieved based on the formulas it uses. The utility runs on a host that has access to the devices that will be analyzed, and performs only read operations so it has no effect whatsoever on the data stored on the device. The following section provides useful information on installing Comprestimator on a host and using it to analyze devices on that host. Depending on the environment configuration, in many cases Comprestimator will be used on more than one host, in order to analyze additional data types.
It is important to understand block device behavior when analyzing traditional (fully-allocated) volumes. Traditional volumes that were created without initially zeroing the device may contain traces of old data in the block device level. Such data will not be accessible or viewable in the file system level. When using Comprestimator to analyze such volumes, the expected compression results will reflect the compression rate that will be achieved for all the data in the block device level, including the traces of old data. This simulates the volume mirroring process of the analyzed device into a compressed volume. Later, when volume mirroring is actually used to compress the data on the storage system, it will process all data on the device (including both active data and traces of old data) and get it compressed. After that when storing more active data on the compressed volume, traces of old data will start getting deleted by new data that is written into the volume. As more active data accumulates in the device the compression rate achieved will be adjusted to reflect the accurate savings achieved for the active data. This block device behavior is limited to traditional volumes and will not occur when analyzing thinly provisioned volumes.
Regardless of the type of block device being scanned, it is also important to understand a few characteristics of common file systems space management. When files are deleted from a file system, the space they occupied before being deleted will be freed and available to the file system even though the data on disk was not actually deleted but rather the file system index and pointers were updated to reflect this change. When using Comprestimator to analyze a block device used by a file system all underlying data in the device will be analyzed, regardless of whether this data belongs to files that were already deleted from the file system. For example you can fill a 100GB file system and make it 100% used, then delete all the files in the file system making it 0% used. When scanning the block device used for storing the file system in this example, Comprestimator (or any other utility for that matter) will access the data that belongs to the files that were already deleted.
In order to reduce the impact of block device and file system behavior mentioned above it is highly recommended to use Comprestimator to analyze volumes that contain as much active data as possible rather than volumes that are mostly empty of data. This increases accuracy level and reduces the risk of analyzing old data that is already deleted but may still have traces on the device.
Comprestimator version 184.108.40.206 adds support for analyzing expected compression savings in accordance with XIV Gen3 storage systems running version 11.6, and Storwize V7000, SAN Volume Controller (SVC) and FlashSystem V9000 storage systems running software version 7.3. Among other enhancements in the software, version 7.3 adds support for the 2014 hardware models Storwize V7000 Gen2, SVC DH8, FlashSystem V9000 AC1 and XIV Gen3.
Comprestimator can be installed only on supported Windows operating systems (see list below). After installation completes, the binary files for other supported operating systems are available in the Windows installation folder.
By default, the files are copied to:
- In Windows 64-bit:
C:\Program Files (x86)\IBM\Comprestimator
- In Windows 32-bit:
Comprestimator is supported and can be used on the following client operating system versions:
- Windows 2003 Server, Windows 2008 R2 Server, Windows 2012, Windows 7, Windows 8
- Red Hat Enterprise Linux Version 5.x, 6.x, 7.1 (x86 64bit)
- ESXi 4.1, 5.5, 6.0
- Sun Solaris 10, 11
- AIX 6.1, 7.1
- HPUX 11.31
- SUSE SLES 11 (x86 64bit)
- Ubuntu 12 (x86 64bit)
To install Comprestimator on a Linux, AIX, Solaris or HP-UX host:
- Log into the host using the root account.
- Copy the Comprestimator binary file from the Windows installation folder to any folder on the host using an SCP utility such as WinSCP or PuTTY.
- Login to the UNIX host, and confirm that the file has execute permissions, to add execute permissions to the binary file, type chmod +x comprestimator.
To install Comprestimator on IBM i-series through VIOS host:
- Copy the AIX/comprestimator_aix tool and ftp it to VIOS in binary mode.
- Log into the VIOS host using the padmin account.
- Enter "oem_setup_env" to exit the padmin shell to a # prompt.
- Confirm that the file has executable permissions. To add execute permissions to the binary file, type chmod +x comprestimator.
To install Comprestimator on ESXi:
- Enable SSH on the ESXi server. Check the following URL for instructions or
skip this step if SSH is already enabled:
- Copy the ESX/comprestimator_esx binary file to any folder on the ESXi server. You can copy the file to the ESXi server by using the secure copy protocol (SCP), for example 'scp' on Linux or 'WinSCP' on Microsoft Windows.
To install Comprestimator on another Windows host:
- Log into the host.
- Copy the 32/64-bit binary file from the corresponding folder to any folder on the host.
To use Comprestimator on a Linux, AIX/VIOS, Solaris, HP-UX, ESXi server:
- Log into the server using the root account.
- Obtain the list of device names:
In Linux: Use the "fdisk –l" command.
In AIX/VIOS: Use the "lsdev –Cc disk" command.
In Solaris: Use the "format" command.
In HP-UX: Use the "ioscan –kfnC disk" command.
In ESXi 4.0: Use the "esxcli corestorage device list | grep Dev" command.
In ESXi 5.0: Use the "esxcli storage core device list | grep Dev" command.
- Run Comprestimator with the –d <device_name> flags to analyze a device or a partition.
Comprestimator is designed to scan any block device that is readable by the OS itself. This typically includes devices managed by logical volume managers (LVMs) or partitioned by the OS. However, for practical reasons, since compression is applied to physical volumes, it is recommended to estimate compression by running Comprestimator on the same block device/physical volume that will be compressed, and not on a logical volume, which may be spanning on those volumes. It is thereby highly recommended to always analyze the native block-device when using Comprestimator.
Some volume managers "reserve" some of the LUN capacity for internal use. Since Comprestimator reads directly from the block device, some of these IOs may fail. The tool will tolerate up to 1% failed IOs and a scan will be aborted if this threshold is reached.
To use Comprestimator on a Windows server:
- Log into the server using an account with administrator privileges.
- Open an elevated Command Prompt with administrator rights (Run as Administrator).
- Run Comprestimator with the –l flag to list the disk.
- Run Comprestimator with the –n flag to analyze the specific disk.
- Alternatively, run Comprestimator with –d and the device name. Note that when using –d option with the long device name, the device name must be wrapped in double quotation marks.
Using Linux, ESXi, AIX/VIOS, Solaris and HP-UX:
comprestimator -s SVC|XIV -d <device> [-p <number_of_processes>] [-P] [--storageVer=6.4|7.1|7.2|7.3|11.6] [-h] [-c <csv_file>] [-v]
comprestimator -s SVC|XIV [-l | -n <disk_number> | -d <device>] [-p <number_of_processes>] [-c <csv_file>] [-v] [-p <number_of_threads>] [-P] [--storageVer=6.4|7.1|7.2|7.3|11.6] [--config=<file>] [-h] [-c <csv_file>] [-v]
|-s, --storageSys||Specifies storage system. Supported values are: SVC, XIV|
For FlashSystem and Storwize systems, use 'SVC'
|-d||Specifies the device name. Note that the name must be wrapped in double quotation|
|device||Path of device to analyze (e.g. /dev/hdisk0 in AIX).|
|disk_number||Only in Windows - The disk number. Identify this number by running Comprestimator with the –l flag first.|
|-l||List the disk numbers and names available to use by Comprestimator. The drive numbers reported match the "Disk #" reported in Windows Disk Management.|
|-p||Specifies the number of processes (or threads in Windows) between 1 and 50. The default value is 10.|
|number_of_processes||The number of processes (or threads in Windows) used by Comprestimator.|
|-P||Displays the results using a paragraph format.|
|-c||Export the results to a CSV-formatted output file.|
|csv_file||File name of CSV-formatted output file.|
|-h||Display usage information.|
|-v||Display the results of every few samples. Note that the output is extended to a few thousands of lines as a result.|
|--storageVer||Target Storwize/SVC/V9000 storage system version. Storwize/SVC/V9000 systems options include 6.4, 7.1, 7.2, 7.3; default is 7.3. Use the default value when planning to use Real-time Compression with the 2014 hardware models Storwize V7000 Gen2, SVC DH8, and V9000 AC1. Since no compression algorithm changes were made in 7.4, relevant version for it is 7.3. XIV Gen3 systems options include only 11.6, which is the default value.|
|--config||Configuration file that contains list of devices to analyze. Used for automation.|
|file||Configuration file that contains list of devices to analyze. The file should contain a list of devices in the same format of the -d option, one device in each line.
# cat devlist_file
Additional Real-time Compression resources: