Pokemon Color Transfer

Background

Palette Extraction

Palette extraction is the process of analyzing an image and summarizing its millions of pixel colors into a small, representative set of key colors — known as a color palette. Instead of working directly with every pixel’s RGB value, palette extraction identifies the dominant or most perceptually important colors that define the visual appearance of the image. A palette typically contains only 4–8 colors, yet these colors capture the essential chromatic structure of the image. This compact representation removes noise, eliminates redundant colors, and preserves the underlying style of the original artwork. We implement two palette extraction methods.

K-Means

We use K-Means to cluster pixel values and use its center as palette. To avoid clustering on millions of pixels, we quantize RGB space into 16 bins per channel. For each pixel with RGB value $(r,g,b)$:

$$(\lfloor \frac{r}{16}\rfloor ,\lfloor \frac{g}{16}\rfloor ,\lfloor \frac{b}{16}\rfloor )$$

For each non-empty bin, we count pixels and compute average LAB value of all pixels in that bin. To avoid randomness in K-Means, the method uses a weighted farthest-point initialization. We select the bin with the largest weight as the first center. For each new center, compute squared distance from existing centers and apply attenuation:

$${w_i}^{\prime }=w_i\cdot (1-e^{-d_i^2/{\sigma }_a^2})$$

We then pick the bin with the largest attenuated weight. Each histogram bin $i$ with LAB color $x_i$ and weight $w_i$ is assigned to the nearest center. The center update rule is:

$$C_j=\frac{\sum _{i\in j}{w}_i\cdot x_i}{\sum _{i\in j}{w}_i}$$

Black and white anchors remain fixed. The convergence criterion is:

$$\Vert C_{\text{new}}-C_{\text{old}}{\Vert }^2<10^{-4}$$

Blind Separation Palette Extraction (BSS-LLE Method)

This second method treats palette extraction as a blind unmixing problem with spatial smoothness constraints. It is computationally more expensive but yields globally coherent palettes.

Each pixel forms a 5-D feature vector:

$$X=[R,G,B,\alpha x,\alpha y]$$

where: $(R,G,B)$ ∈ $[0,1]$, $(x,y)$ are normalized coordinates, $\alpha$ controls spatial smoothness. For each pixel, we find $K=30$ nearest neighbors and compute LLE weights by solving:

$$Gw=\mathrm{𝟏},G=(X_j-X_i)(X_j-X_i{)}^{\mathrm{\top }}+ϵI$$

Construct Laplacian:

$$L=I-Z,L^{T}L$$

We assume each pixel’s color can be expressed as a mixture of $m$ palette colors:

$$I=WC$$

Where $W\in {ℝ}^{N\times m}$ are mixture weights, and $C\in {ℝ}^{m\times 3}$ are palette colors. We minimize:

$$\Vert I-WC{\Vert }^2+{\lambda }_s\Vert W-ZW{\Vert }^2+{\lambda }_u\Vert W\mathrm{𝟏}-\mathrm{𝟏}{\Vert }^2+{\lambda }_{sp}\Vert W{\Vert }_0$$

The optimization uses alternating minimization:

1. Update W (closed-form linear system) 2. Hard-threshold W to enforce sparsity 3. Update C by solving

$$(W^{T}W)C=W^{T}I$$

4. Increase β to gradually enforce sparsity (continuation method)

The learned palette may lie off-manifold. Thus each palette color is replaced by the mean of its nearest real RGB pixels. This ensures interpretability and consistent color reproduction.

Final Output

Both methods return a palette:

$$\{(R_1,G_1,B_1),\dots ,(R_m,G_m,B_m)\}.$$

Methods

In this section, we describe the methods we use to transfer color between pokemons.

Baseline: Palette Based Random Transfer

Palette-based random transfer works by first extracting a compact color palette from each image, then randomly matching colors between the two palettes to generate a playful and diverse recoloring. Instead of enforcing a strict one-to-one correspondence or optimizing for perceptual similarity, the method randomly permutes or samples palette colors and maps all pixels associated with a source palette color to a randomly chosen target palette color. This allows the transferred image to preserve structural details while producing vivid, surprising, and stylistically varied recolorings. Because it operates only on palette colors rather than individual pixels, palette-based random transfer is fast, interpretable, and ideal for generating creative variations in tasks like Pokemon color stylization. We use Palette-based random transfer as the baseline.

Neighbor Segments

Neighbor Segments (NS) Method groups pixels into local, perceptually coherent regions and uses neighborhood relationships to guide smooth and consistent recoloring. Instead of treating each pixel independently, the image is first segmented into small regions (superpixels or color-coherent clusters), where each segment represents a set of spatially adjacent pixels with similar color statistics. Let the image be segmented into $S=s_1,s_2,\dots ,s_K$, where each segment $s_k$ contains pixels with similar color features. These segments form a neighborhood graph $G=(S,E)$, where an undirected edge $(s_i,s_j)\in E$ indicates that the two segments touch in the spatial domain. The adjacency matrix $A$ of the graph satisfies: $A_{ij}=1,if(s_i,s_j)\in E$

During color transfer, each segment is assigned one palette color. Let $P=p_1,p_2,\dots ,p_m$ be the target palette, and let $c_k$ denote the transferred color assigned to segment $s_k$. The NS method encourages smoothness by minimizing a neighborhood-consistency energy:

$$E_{\text{smooth}}=\sum _{(i,j)\in E}{w}_{ij}\Vert c_i-c_j{\Vert }^2,$$

where $w_{ij}$ is a weight encoding the similarity of the segments (e.g., based on LAB difference or boundary strength). This term penalizes large color differences between adjacent segments, preventing abrupt color transitions or blocky artifacts. At the same time, the transferred color for each segment should remain close to its mapped palette color $p_{\pi (k)}$ determined by the palette mapping rule:

$$E_{\text{data}}=\sum _{k=1}^K\Vert c_k-p_{\pi (k)}{\Vert }^2.$$

The final transferred colors are obtained by minimizing the combined objective:

$$E=E_{\text{data}}+\lambda E_{\text{smooth}},$$

where $\lambda$ controls the strength of neighborhood smoothing. This neighborhood-aware propagation allows the algorithm to maintain structural consistency, preserve texture boundaries, and generate recolorings that are both stable and visually coherent. Overall, Neighbor Segments provides a lightweight way to incorporate spatial smoothness into palette-based transfer, producing natural transitions while keeping computation efficient.

Neighbor Segments with Superpixel (NS-S)

Neighbor Segments with Superpixels (NS-S) method extends palette-based color transfer by incorporating spatially coherent superpixel regions. Instead of operating on individual pixels, the image is first partitioned into a set of superpixels, each representing a compact region of adjacent pixels with similar color and texture characteristics. Let the superpixel segmentation produce $S=\{s_1,s_2,\dots ,s_K\}$, where each superpixel $s_k$ is treated as a unit for color assignment. A neighborhood graph $G=(S,E)$ is constructed, where an edge $(s_i,s_j)\in E$ indicates that two superpixels touch or share a boundary in the image plane. The adjacency matrix is defined as:

$$A_{ij}=\{\begin{array}{cc}1,& (s_i,s_j)\in E,\\ 0,& \text{otherwise}.\end{array}$$

Given a target palette $P=\{p_1,\dots ,p_m\}$, each superpixel $s_k$ receives a transferred color $c_k$, determined by a palette-matching function $\pi (k)$ (e.g., hard assignment, soft matching, or nearest palette color).

To ensure smooth and visually coherent recoloring across the image, the NS-SP method minimizes the following energy:

$$E=\underset{E_{\text{data}}}{\underbrace{\sum _{k=1}^K\Vert c_k-p_{\pi (k)}{\Vert }^2}}+\lambda \underset{E_{\text{smooth}}}{\underbrace{\sum _{(i,j)\in E}{w}_{ij}\Vert c_i-c_j{\Vert }^2}},$$

where:

• $E_{\text{data}}$ encourages each superpixel to adopt its intended palette color,
• $E_{\text{smooth}}$ enforces spatial smoothness between adjacent superpixels,
• $w_{ij}$ weights the similarity between boundary segments (e.g., based on color difference, gradient magnitude, or LAB distance),
• $\lambda$ controls the strength of neighborhood smoothing.

By working at the superpixel level, NS-SP effectively reduces noise, prevents pixel-level flicker, and maintains clean boundaries between meaningful regions. The neighborhood smoothing further prevents abrupt color jumps, producing recolored images that maintain Pokémon shapes, shading, and visual consistency while still allowing strong palette-driven stylistic changes.

Overall, the NS-SP formulation provides a computationally efficient and perceptually stable approach for palette-based color transfer, balancing stylization with structural fidelity.

Results

Table 1: Quantitative Evaluation Results on Bulbasaur and Squirtle

Metric	Baseline	Clustering	Clustering-NP	Convex Hall	NS	NS-S
FID	1	2	3	4	5	6
Histogram Similarity	1	2	3	4	5	6
CIELAB	1	2	3	4	5	6
CIE94	1	2	3	4	5	6
CIEDE2000	1	2	3	4	5	6
SSIM	1	2	3	4	5	6
VGG Latent Space Distance	1	2	3	4	5	6

Appendix II

Wenxiao Cai:

Yifei Deng: