Unsupervised Visible-Infrared Person Re-identification (USL-VI-ReID) is critical for cross-modal intelligent surveillance. While vision-language models (e.g., CLIP) present powerful representational capabilities, directly fine-tuning them for USL-VI-ReID often causes catastrophic feature collapse and prompt degradation due to massive domain gaps and noisy pseudo-labels. Furthermore, traditional discrete matching and heuristic denoising strategies suffer from severe cross-modal information starvation and numerical bias against hard positives. To address these challenges, we propose a robust, CLIP-based unsupervised cross-modal fine-tuning framework. First, we design an implicit adapter fine-tuning strategy coupled with decoupled multi-dimensional semantic prompting to isolate domain biases without destroying pre-trained priors. Second, a Cluster-Aware Cross-Modal Semantic Alignment (CCSA) mechanism maps dynamic visual centers to modality-shared textual proxies via visual-conditioned prompting, facilitating an implicit soft alignment decoupled from hard clustering noise. Third, we frame cross-modal association as a Topology-Aware Optimal Transport (TOTO) problem. Utilizing Fused Gromov-Wasserstein (FGW) constraints and Argmax assignments, TOTO injects potent hard regularization to overcome optimization inertia on difficult samples. Finally, our Pure Relative Confidence Ratio and Dual Adaptive Denoising (RCR-DAD) module eliminates numerical bias, formulating a robust self-paced learning trajectory. Extensive experiments on SYSU-MM01 and RegDB demonstrate our framework achieves state-of-the-art performance. The code will be released.