User-aware Multi-modal Animation Illustration Recommendation Fusion with Painting StyleThis work was supported in part by the National Natural Science Foundation of China, Grant No. 62366060, 61762092, Open Foundation of Yunnan Key Laboratory of Software (2024)

2.1 Problem Definition

2.2 Dual-output Image Encoder

Different painting styles give the illustration personality and artistry, making it more attractive, expressive, and recognizable. In the field of illustrations, not only the semantic features of its images but also the stylistic features of its paintings influence user preferences. Therefore, we have built an image encoder $\mathbf{E}^{\text{img}}$ with the semantic feature $e^{\text{SEM}}$ and style feature $e^{\text{STY}}$ outputs according to the following steps: 1) To extract image features that align with animation domain knowledge, we use $i^{\text{IMG}}$ as features and $i^{\text{L}}$ as labels, building a large-scale anime image multi-label multi-classification dataset. 2) We chose the structure of ResNet101 to construct the encoder and pre-train on the dataset. 3) The classification head is removed from the pre-training model after fitting the pretext task. Then, as shown in Fig. LABEL:image_encoder, the feature map (FM) generated from various layers of $\mathbf{E}^{\text{img}}$ is modified to obtain dual outputs:

Semantic Feature Output. Higher-level FM is obtained from convolutional neural networks (CNN) to describe more complex features. In the pretext task, we chose the text annotations $i^{\text{L}}$ made by artists as the labels for the images. As a result, $\mathbf{E}^{\text{img}}$ can effectively learn these semantic features represented by pre-trained task. For reducing the spatial resolution of features to capture high-level semantic features of images better, we add a mean pooling layer after the output of the last convolutional layer of $\mathbf{E}^{\text{img}}$ to obtain feature $e^{\text{SEM}}$ as:

Since direct combination will result in excessive output dimensions, max pooling is utilized to remove local details of the image and allows the FM to focus more on the global texture and structure of the image. This abstract representation is more suitable for representing style feature of an image. Therefore, the layer-wise Gram matrix representation and $e^{\text{STY}}$ are given by merging the $G^{l}$ of the first three layers by max pooling.

2.3 Multi-perspective Text Encoder

Although general pre-trained models (GPMs) perform well in various natural language processing tasks, they often face challenges when adapting to specific domains. For example, misunderstandings or errors may occur in domain-specific terminology and knowledge. It is difficult for GPMs to understand vocabulary such as character names, specific anime titles, concepts, and settings within animation works. The combination of multilingualism such as Chinese, Japanese, and English and domain-specific vocabulary leads to poor performance of GPMs in our datasets. Therefore, we obtain multi-perspective text pair to fine-tune the GPM and construct a text encoder $\mathbf{E}^{\text{text}}$ integrated with domain knowledge from diverse views as two main steps:

1) As shown in Fig. LABEL:text_encoder, we collect publicly available information from various websites such as Bangumi, Moegirlpedia, MyAnimeList, Wikipedia etc., to create a dataset consisting of text pairs from multiingual relation, domain relationship, term explanation perspectives as shown in Table 1.

Specifically, texts from different sources can provide multi-hierarchy information about a specific topic. The comparison of multiple languages enables the model to learn the correspondence between multiple languages.

2.4 User-aware Multi-modal Contribution Measurement

Given the effectiveness of individual UMCM, inspired by the MoE [11] framework in multi-task RS, we employ parallel UMCMs to simulate experts for different decisions. Furthermore, we utilize the dot-product attention mechanism to implement an aggregation gate, consolidating the decisions from multiple experts, thereby further enhancing the modeling of MM contributions.

2.5 Multi-modal Crosses

Utilizing MLP to simulate implicit feature intersections of any order is time-consuming. To avoid unnecessary computations, we utilize a lightweight DCN-V2 [9] module to simulate explicit and bounded-degree MM crosses. The core idea of DCN-V2 is to create explicit feature cross layers.

3.1 Experimental Settings

Dataset. To the best of our knowledge, there currently exists yet to be a high-quality public dataset for anime illustration RS. We collected user interaction logs from a commercial illustration website to construct our dataset. The detailed information of the dataset is shown in Table 2. Fig. LABEL:fig:userandillustraion (a) and (b) depict the distribution of interaction counts from the perspectives of illustrations and users, respectively, showcasing the long-tail distribution challenge present in the dataset, underscoring the complexities involved in anime illustrations RS.

Evaluation Metrics. As in [12, 13, 14, 9], we use the Area Under the Curve and Binary Cross-Entropy Loss to evaluate the performance of all methods.

Baselines.To demonstrate the effectiveness of our proposed method, we compare it with six SOTA general RS approaches.

Implementation Details. We implement UMAIR-FPS on the TensorFlow framework, and use the Adam optimizer with a learning rate of 1e-3 and train on the dataset for just one epoch to avoid the overfitting issue [15]. We adhere to default settings of most hyperparameters in the baseline methods except outputting an embedding layer with a dimension of 8 and an L2 regularization parameter of 1e-3.

3.2 Performance Comparison

Table 3 presents the performance comparison of UMAIR-FPS with all other methods on the dataset. It also reports the parameters for each model to assess the space complexity. As shown in Table 3, our model exhibits significant improvement compared to six baseline models. Compared to the best-performing baseline, UMAIR-FPS still improves AUC by 5.4% and, notably, boosts the BCE loss by an impressive 25.35%. We believe the superior performance of our model can be attributed to the following factors: 1) Combination of textual and visual information. This allows us to model the intrinsic correlations, generating a more comprehensive item representation, and to some extent, alleviates data sparsity caused by the long-tail distribution. 2) UMCM mechanism. By re-weighting the MM features, our model can simulate the real selection process of users more closely. 3) MM crosses. This increases the non-linearity capabilities of the model, capturing user preferences in a more granular manner.

Even though UMAIR-FPS introduces features from multiple modalities, its parameter count remains on par with other models. This ensures that our method does not compromise efficiency while guaranteeing predictive accuracy.

3.3 Ablation Study

To further investigate the effectiveness of our key modules, we conduct ablation studies on four groups of variants, as presented in Table 4.

Impact of Multi-modal Features.To explore the effectiveness of the MM features, we establish four variants. In Variant ${w_{/}o\ \text{ALL}}$, we eliminate all MM features, including those representing the style and semantic features of images, as well as semantic features of text labels. As a result, the AUC decreases by 9.00%, the BLE loss increases by 70.37%, and the parameter count drops by 2.96%. This trend underscores the significance of combined text and visual multimodal features in accurately capturing user preferences, and the model’s performance is hardly impacted by this integration.

In Variants ${w_{/}o\ \text{STY}}$, ${w_{/}o\ \text{SEM}}$, and ${w_{/}o\ \text{TSEM}}$, we respectively remove the style features of images $e^{\text{STY}}$, the semantic features of images $e^{\text{SEM}}$, and the semantic features of text labels $e^{\text{TSEM}}$. The most notable impact comes from removing $e^{\text{STY}}$, indicating that it bring substantial improvements in anime recommendations. Among all features, the enhancement brought by $e^{\text{SEM}}$ is the smallest, as it is guided by the actual image text labels. Moreover, the information obtained by selecting lower-level features might still overlap with $e^{\text{TSEM}}$.

The effectiveness of the image encoder $\mathbf{E}^{\text{img}}$ is shown in Fig. LABEL:heatmap. The cosine distances between the feature vectors of similar artworks at the semantic feature and the style feature level are close, while they are far for dissimilar ones.

Impact of UMCM. We design two new variants: one without UMCM, denoted as ${w_{/}o\ \text{UMCM}}$, one replaced the ATT with the SENet module, denoted as ${w/\ \text{UMCM\_SEN}}$, and one implemented with the dot product attention mechanism, ${w/\ \text{UMCM\_ATT}}$. Compared to ${w_{/}o\ \text{UMCM}}$, ${w/\ \text{UMCM\_ATT}}$ reduces the loss by 13.52%, increases the AUC by 2.085%, and augments the model parameters by 0.382%. Compared to ${w/\ \text{UMCM\_SEN}}$, the loss decreases, while the AUC remains roughly consistent. This suggests that dynamically reweighting illustrations based on user feature vectors can effectively model user preferences.

When utilizing UMCM by MoE, We enhance UMCM with the MoE framework. Our observations indicate that, for multimodal contribution reweighting, a three-module stack is optimal for both ATT and SNET. Adding additional modules tends to degrade performance. Thus, after introducing MM cross-interactions into the multi-task RS framework, we can enhance recommendation performance by leveraging multiple weight modeling modules formed by expert outputs, facilitating representation learning for specific tasks.

Impact of MM Crosses. We compare UMAIR-FPS with the variant without FX, denoted as ${w_{/}o\ \text{DCN-V2}}$. denoted as ${w_{/}o\ \text{DCN-V2}}$. The AUC decreases by 1.80% and the BCE loss increases by 52.14%, with the model parameters reduced by 0.70%. These results underscore the efficacy of FX. Moreover, undergoing user-aware weight fusion, can still benefit from FX with user features.