Objective: To achieve the segmentation of different layers and fluid areas on the optical coherence tomography (OCT) image of the retina. Methods: A lightweight neural network based on deep learning was proposed. The network structure adopted in this study was designed based on the architecture of dilated-residual U-Net. By connecting the upsampling output obtained at different depth networks, multi-scale feature fusion was performed to enable the system to accurately identify the boundaries on the OCT image. Results: Compared with U-Net, this algorithm could achieve the same accuracy with 1–2 epochs less, and the accuracy was also improved by 1.25%. Conclusion: The proposed network improves the segmentation performance of retinal OCT images, and reduces the number of parameters, which demonstrates the network has great application potential.