With the widespread adoption of genome sequencing technologies, predicting complex traits using genomic markers has become a key component in breeding programs. However, the high dimensionality and sparsity of genomic data, along with the complex nonlinear interactions among genetic markers, significantly increase the difficulty of accurate data analysis and the cost of hardware deployment. Therefore, this study proposes a chromosome-encoded multi-head self-attention model, named ChrFormer, for genomic prediction. The model employs a chromosome encoder to compress whole-genome SNP data into 20 chromosome-specific feature vectors and one global feature vector. It leverages the multi-head self-attention mechanism to dynamically capture long-range interactive effects across chromosomes, and a multilayer perceptron (MLP) precisely predicts phenotype from the refined genomic features. The study selected genotyping data from 50,000 SNPs of 4,875 Large White pigs, along with four key production traits, including backfat thickness at 100 kg and 115 kg, and age at 100 kg and 115 kg. A ten-fold cross-validation approach and the Pearson correlation coefficient were used to evaluate prediction accuracy. The predictive performance of ChrFormer was systematically compared with genomic best linear unbiased prediction (GBLUP), Bayesian method A (BayesA), and representative deep learning methods, including the visual geometry group (VGG) network and the feedforward neural network (FNN). Furthermore, the study analyzed the strengths and weaknesses of each deep learning model from multiple aspects, including the number of model parameters, training time, and the extent of overfitting. The results show that ChrFormer significantly outperforms the VGG and FNN deep learning models in predictive accuracy across all tested traits. For three of the traits (backfat thickness at 100 kg and 115 kg, and days to 115 kg), its prediction accuracy surpasses that of the traditional GBLUP and BayesA methods. Although ChrFormer requires a longer training time per iteration (54.88 s), its number of parameters is only about one-tenth of that of VGG and FNN, and it demonstrates more stable resistance to overfitting. These results demonstrate that the self-attention-based ChrFormer is a practical tool for genomic phenotype prediction in animal breeding, and its lightweight architecture and stable performance offer a readily deployable solution for breeding stations with limited computational resources.