Evaluation Metrics for Image Compression

For my show and tell today, I wanted to compare multiple compression algorithms, which needs a solid set of evaluation metrics.

Compression isn't always about getting the smallest size. I'd learned about it when I was tinkering around with the BOA-constrictor from CERN-HEP. It is always a trade-off between size reduction and runtime efficiency, thus I began looking for what the usual benchmarks are.

1. Compression Ratio & Space Savings (The Size Metrics)¶

Compression Ratio ($CR$)¶

$$\text{CR} = \frac{\text{Uncompressed File Size}}{\text{Compressed File Size}}$$

• What it tells us: How many times smaller the data became (e.g. $10\times$ means the compressed file is one-tenth the original size).
• Why we use it: It is intuitive and scale-independent across different image resolutions and file types.

Space Savings ($\%$)¶

$$\text{Savings} = \left(1 - \frac{\text{Compressed Size}}{\text{Original Size}}\right) \times 100\%$$

• What it tells us: The percentage of disk space or network bandwidth eliminated (e.g., $10\times$ compression is a $90\%$ savings ratio).

Bits Per Pixel ($\text{bpp}$)¶

$$\text{bpp} = \frac{\text{Total Compressed Bits}}{\text{Width} \times \text{Height}}$$

• Why we use it: Raw 24-bit RGB images start at $24\text{ bpp}$ ($8\text{ bits} \times 3\text{ channels}$). When evaluating image codecs like JPEG, PNG, or Wavelets, $\text{bpp}$ normalizes file size across different dimensions, letting us measure the true information density per pixel. A good lossy photo typically lands between $0.5\text{--}1.5\text{ bpp}$.

2. Signal Fidelity: MSE vs. PSNR (Lossy Quality Metrics)¶

For lossless algorithms (LZ77, LZ78, Huffman, BPE, PNG), reconstructed data must match the original bit-for-bit ($\text{Error} = 0$). But for lossy transforms (JPEG DCT quantization, DWT thresholding, NCA generative growth), we discard imperceptible frequencies, which requires measuring reconstruction error.

Mean Squared Error ($\text{MSE}$)¶

$$\text{MSE} = \frac{1}{H \cdot W \cdot C} \sum_{x,y,c} (I(x,y,c) - \hat{I}(x,y,c))^2$$

• What it is: The raw average squared difference between original pixels $I$ and reconstructed pixels $\hat{I}$.
• Why it's insufficient on its own: $\text{MSE}$ is linear and depends on dynamic range. An $\text{MSE}$ of $25$ means an average pixel shift of $5$ intensity levels, but human eyes perceive brightness logarithmically (Weber-Fechner Law), not linearly.

Peak Signal-to-Noise Ratio ($\text{PSNR}$)¶

$$\text{PSNR} = 10 \cdot \log_{10}\left(\frac{\text{MAX}_I^2}{\text{MSE}}\right) \quad \text{(measured in dB)}$$ (where $\text{MAX}_I = 255$ for 8-bit image channels) - Why we use it: By taking the ratio of peak signal power ($255^2$) to noise power ($\text{MSE}$) on a decibel scale, $\text{PSNR}$ matches human perception of degradation: - $> 40\text{ dB}$: Excellent; virtually indistinguishable from the original. - $30\text{--}40\text{ dB}$: Good; minor compression artifacts upon close zoom. - $< 25\text{ dB}$: Noticeable distortion, blockiness, or blurring.

Structural Similarity Index ($\text{SSIM}$)¶

$$\text{SSIM}(x,y) = \frac{(2\mu_x\mu_y + c_1)(2\sigma_{xy} + c_2)}{(\mu_x^2 + \mu_y^2 + c_1)(\sigma_x^2 + \sigma_y^2 + c_2)}$$

• Why PSNR isn't enough: If you shift an image 1 pixel to the right, $\text{MSE}$ skyrockets and $\text{PSNR}$ drops drastically, even though the image looks identical to human eyes! $\text{SSIM}$ measures structural degradation across luminance ($\mu$), contrast ($\sigma$), and correlation ($\sigma_{xy}$), scoring between $0.0$ and $1.0$.

3. Rate-Distortion Curves ($R\text{-}D$)¶

Instead of testing a single quality setting, we sweep across quantization levels or threshold percentiles and plot Rate (Compressed Size in KB or bpp) on the X-axis against Distortion (PSNR in dB) on the Y-axis.

• The Pareto frontier (the curve sitting furthest towards the top-left) represents the optimal codec: delivering the highest visual fidelity at the smallest file size.

4. Operational Efficiency: Latency & Memory Footprint¶

A codec with a $15\times$ compression ratio is useless in real-time streaming if encoding takes 10 seconds per frame.

• Encoding / Decoding Throughput ($\text{MB/s}$ and $\text{ms}$): Codecs like LZ77 and JPEG are engineered such that decompression is nearly instantaneous for clients, even if compression takes longer.
• Memory Complexity:
• LZ77 (Sliding Window): Strictly bounded $O(W)$ RAM making it predictable.
• LZ78 / LZW (Dynamic Trie): Memory grows dynamically with the number of unique phrase nodes $O(D)$.

Summary Table¶