We introduce a two-stage approach for 3D segmentation of 5 abdominal organs in volumetric CT images. First, the relevant volume of interest for each organ is extracted as a bounding box. The obtained volume serves as input for a second step, in which two U-Net architectures of different dimensionalities are applied to reconstruct an organ segmentation as a label mask. The focus of this work is on the comparison of 2D U-Nets vs. 3D U-Nets counterparts. Overall, the results of our study show Dice improvements of at most 6%. Regarding computation time, to our surprise, the segmentation of liver and kidney was performed significantly faster by the GPU memory-saving 2D-U Nets. There were no significant differences for other major abdominal organs, but for all organs tested, we observe highly significant advantages for the 2D U-Net in terms of GPU computational effort.

Organ bounds; U-Net; Architecture; Abdomen; Segmentation; 3D CT images.

We introduce a two-stage approach for 3D segmentation of 5 abdominal organs in volumetric CT images. First, the relevant volume of interest for each organ is extracted as a bounding box. The obtained volume serves as input for a second step, in which two U-Net architectures of different dimensionalities are applied to reconstruct an organ segmentation as a label mask. The focus of this work is on the comparison of 2D U-Nets vs. 3D U-Nets counterparts. Overall, the results of our study show Dice improvements of at most 6%. Regarding computation time, to our surprise, the segmentation of liver and kidney was performed significantly faster by the GPU memory-saving 2D-U Nets. There were no significant differences for other major abdominal organs, but for all organs tested, we observe highly significant advantages for the 2D U-Net in terms of GPU computational effort.

Organ bounds; U-Net; Architecture; Abdomen; Segmentation; 3D CT images.

Steady progress in the field of deep learning and artificial intelligence is opening up previously unimagined possibilities for medical research. However, even with the increasing availability of publicly accessible databases, the automated just-in-time (JIT) segmentation of 3D patient organ models remains an unsolved challenge. As manual segmentation of abdominal organ structures in axial CT slices is a very time consuming task, automatic and JIT reconstruction procedures of 3D models are highly relevant for surgery planning and navigation.

Over the years, various Convolutional Neural Network (CNN) models for image segmentation have emerged, some of which differ greatly in memory requirements and the saving of computational power. In 2015, the 2D U-Net architecture showed promising results that outperformed conventional models on 2D biomedical segmentation problems and was also found to be effective on lower resolution images [1]. In contrast to the previously known CNN models, the U-Net concept uses downand up-sampling steps to resample the condensed feature map to the original size of the input image. With the inclusion of higher resolution feature information in each up-sampling step, semantic segmentation of the input images can be accomplished efficiently by skip connections. In 2016, the resulting U-shaped architecture was extended to a 3D U-Net edition by Cicek et al. by replacing 2D operations with their 3D equivalents [2]. The primary focus in this paper is to address the question of whether the 3D U-Net is truly better suited for 3D data.

We expect that in Big Data studies, parallel segmentation of thousands of 3D volumetric images requires high computational time due to the limited amount of processing nodes and sub-processes for parallel processing. We conjecture that 3D U-Nets will require even higher GPU computational and memory overhead for 3D image processing. In this work, we focus on proposing solutions with the lowest possible cost while maintaining or even improving segmentation quality. The goals of our approach can be summarized in three simple steps:

We detect and apply U-Nets to local sub-images for each task (liver, kidneys, spleen, pancreas).

We compare two U-Net architectures in terms of quality and GPU-performance when executing semantic segmentations.

We recommend the more suitable architectural model to the interested reader based on a detailed statistical analysis.

Finally, using an image database of 80 CT scans and their ground truth segmentations, the question whether the simpler 2D U-Net might perform better than the complex and theoretically more powerful 3D U-Net will be answered.

Particularly in the emerging field of visuo-haptic training and planning interventions using virtual reality (VR) techniques, fast and accurate 3D segmentation results are of utmost importance [3-6]. Time-variant 4D VR simulations with breathing simulation are readily available for training and planning of liver needle interventions [7-11]. A major obstacle remains the JIT and high-quality reconstruction of all required 3D models, i.e., abdominal organs in particular are rarely easy to segment. This is due to varying imaging conditions such as contrast agent administration, structure variations and noise.

In the literature, 3D U-Nets are often suggested to be more powerful [12-16]. However, there are some studies that highlight the advantages of 2D U-Nets in 3D segmentation tasks, such as [17-19]. Nemoto et al. find that 2D U-Nets with low computational complexity are effective and equivalent to 3D U-Nets for semantic lung segmentation, except for the trachea and bronchi [17].

A hybrid approach of 3D and 2D inputs for evaluating hemorrhage on CT scans of the head was proposed by Chang et al. [20]. This method was later adapted by Ushinsky et al., applying it to segmentation of the prostate in MRI images, demonstrating that 2D U-Nets are very effective for 3D data [19]. Christ et al. provided a slice-wise application of 2D U-Nets for liver segmentation in combination with 3D random fields [21]. Similarly, Meine et al. proposed liver segmentation methods with the assistance of 3D, 2D and three fused 2D U-Net sectional (axial, coronal, sagittal) results in a 2.5D ensemble approach [18]. They find that the 2.5D U-Net ensemble results are statistically superior for liver segmentation, especially for images with pathologies.

Other recent approaches aim to combine multiple stacked 2D U-Nets and further improve information flow through semantic connections between different components [22,23]. The same concept has also been used in application areas such as colon polyp segmentation and face recognition [24,25]. Zwettler et al. have recently shown that extending datasets with synthesized slices can notably improve the results of 2D U-Nets applied on a small number of training datasets, indicating further potential for the 2D approach [26].

Regarding our 2D U-Net setup for the abdominal organs liver, spleen, kidneys and pancreas, we prefer axial slice training within the previously determined organ-specific VOIs. This results in significant savings in radiation dosage savings due to high scan pitch. Yet image resolution in axial CT slices is very high. New aspects in this work include the organ-specific VOI approach for the investigated organs such as liver, spleen, kidneys and pancreas. Finally, an organ-specific architecture dimensionality recommendation is given for each of the organs.

For the training and testing of our U-Nets, eighty CT scans and the corresponding labeled images were used, which can be found in various public sources1. Besides differences in image information in terms of quality, noise, and field of view, patient-specific volume representation also varied in the number of slices (64 to 861), pathologic lesions, slice width (1 to 5 mm), and contrast agent used.

The main problem concerning data annotation was the weakly contrasted pancreas organ in the CTs, which was not present in some image sets consisting of congruent intensity and label images from the public sources. We coped with that by four eye reviewed manual segmentation of this occasionally missing structure in the label maps. However, most organ reference segmentation were readily available in the public data sources.

To ensure a fair processing of the data, the orientation of images was changed to Right-Anterior-Inferior (RAI) and zero origin (0.0, 0.0, 0.0). Because CNNs are not able to natively interpret voxel spacings, an isotropic image resampling with 2.03 mm was performed as a compromise for varying xyz-distances (xy ≤ 2 mm, z ≤ 5 mm).

Our approach consists of two different machine learning techniques, bounding box detection using random regression forests (RRF) and U-Nets for semantic segmentation. After using the RRFs to detect the organ VOIs in the CT data, the U-Nets are applied to segment the organs contained in the VOIs.

For the evaluation of this work, the ground truth of the corresponding organ VOI bounding box (BB) is required. It can be created by scanning the reference segmentation maps for labeled voxel coordinate extremes. To create a three-dimensional BB vector for each organ, we iterate in orthogonal slices through the organ’s label map for each coordinate direction (x, y, z) and store the extreme limits in a 6D BB vector that functions as ground truth BBs.

http://visceral.eu,

http://sliver07.org,

http://competitions.codalab.org/competitions/17094

Alternatively, VOI-based organ extraction is readily available for organ-specific VOI detection of the organ ensemble (liver, still left kidney, right kidney, spleen, pancreas) [27-29]. This step also solves the FoV problem in the z-direction of the scanner, since CT scans cover variable body portions. Our currently studied scheme for learning bounding boxes of VOIs using RRFs is summarized below.

We use Random Regression Forests (RRF) to determine the location and extent of abdominal organs [29]. As shown in figure 1, the RRF training step expects scans and ground truth VOIs as input.

A three-dimensional VOI bc of an organ c can be described by using a 6D vector $b_c= (b_c^{Left}, b_c^{Right}, b_c^{Anterior}, b_c^{Posterior}, b_c^{Head}, b_c^{Foot})$ with coordinates in mm [29]. To reduce the amount of data and speed up the algorithm runtime, only a fraction of the voxels in the scan is used for bounding box calculation. Starting from the scan center, we calculate the position of the voxels in each axis direction that are 15 mm away from the center, forming a cuboid box around the center. All voxels inside this area are part of the subset used for training and application of the algorithm.

Each voxel in the subset is assigned a feature and an offset vector. As described in eq. 1, the offset vector of a voxel $p= (p_{x_{min}}, p_{x_{max}}, p_{y_{min}}, p_{y_{max}}, p_{z_{min}}, p_{z_{max}})$ is the difference d(p,c)=p-b_c

between its position and the position b_c of eachof the six bounding box walls of the organ. Figure 2 shows this principle on the example of a kidney.

The feature vector of a voxel is derived from several feature boxes generated by traversing the voxels that are within a specified radial distance (r = 5 cm) from the scan medial axis. By computing the average intensities of all feature boxes obtained, the aim is to capture the spatial and intensity context around the voxel. Eq. 2 describes the calculation of a feature vi;

$v_i= \frac{1}{\left|F_{p;i}\right|}{\displaystyle \sum }_{q\in F_{p;i}}J\left(q\right).$

F_p;i represents the feature box of the ith feature, q indicates the points inside F_p;i and J (q) determines the intensity at position q. The feature box is defined related to the current voxel. In contrast to Criminisi et al. [29], we use only 50 feature boxes, that are evenly distributed on three spheres (r = 5 cm, 2.5 cm, 1.25 cm). The input feature vector is finally composed of the mean intensities of the feature boxes. Figure 3 shows an example feature box, that was created in correlation to the selected voxel.

During the first step of our application scheme (figure 4, left), the RRF localizes the VOI for each individual organ as BB. Finally, the information of a detected VOI (intensities, labels) is resampled to 96x96(x96) for CNN input (figure 5).

In the second stage, as shown in the bird’s eye view in figure 4, right, the resulting VOI is passed to either a 2D [1] or a 3D U-Net for semantic labeling [2].

The training data for our U-Net is composed of the expert segmentations and ground truth bounding box interiors, as shown schematically in figures 6,4. The VOIs are then used to locally extract the intensity and label data from the CT scans. As input, a U-Net obtains a VOI from the intensity data, whereas the corresponding label data is linked to the output. We use slightly modified 2D and 3D U-Net architectures compared to Ronneberger et al. [1].

As can be seen in figure 5, our architecture consists of four down- and up-scaling steps that act as analysis and synthesis paths. Skip connections between the corresponding levels allow for the incorporation of additional information in the form of high-resolution features at each up-scaling step. Starting with the VOI input of size 96x96(x96), each downscaling step consists of two 3x3(x3) convolutions with a small dropout layer in between to prevent overfitting. Each step is then followed by a 2x2(x2) max-pooling layer until the final down-scaling step. In the up-scaling path, each step is introduced by a 2x2(x2) transposed convolution, which is then concatenated with the skip connection output from the corresponding down-scaling step. The last step is followed by a 1x1(x1) sigmoid activation function that provides the final segmentation result. For the 3D U-Net, an additional layer depth dimension is added while all other design elements are kept constant.

By using the data contained in a given VOI bounding box to segment the corresponding organ, the resulting output is a local segmentation map of the entire target organ in a probability range of 0 to 100%. The threshold for all organs except pancreas was chosen as 50%. For pancreas 30% was empirically found best.

The U-Nets were trained with batches of size 8 over 100 epochs. In addition, Adam optimization and a cross entropy loss function were used. A separate U-Net is trained for each organ, using a ReLU activation function.

To allow for a fair comparison, we use the same 64 training images for both U-Nets in each iteration of the crossvalidation and perform testing with the same 16. In this way, we avoid data contamination due to separate training and test sets. The only difference between the U-Net architectures is the dimension of the input, output, and filter kernel layers. For 2D U-Nets, the input and output dimensions are 96x96 and 2D filter kernels are applied in the layers. Respectively, for 3D U-Nets we have 96x96x96 inputs and outputs and apply 3D convolutions in the architecture layers.

A 5-fold randomized cross-validation using 4:1 splits was used. Five iterations yield 80 quality measures for each organ used in our statistics (table 1). This means that five new models for each organ are trained with randomly selected training data and used in the analysis. Figure 7 illustrates the partition of training and testing data. To focus solely on the influence of the U-Net dimensionality, the reference VOIs were used for the evaluation of this study. The Dice similarity coefficient was used as metric for the analysis:

$DSC= \frac{2.\left|U{{\displaystyle \cap }}^{\text{}}G\right|}{\left|U\right|+\left|G\right|}$

Where, U is the set of voxels from U-Net object segmentation and G is the set of ground truth voxels. A DSC value of 1 indicates perfect segmentation, a value close to 0 implies poor segmentation. From the DSC results, we calculate means and standard deviations, medians and Inter-Quartile-Ranges (IQR) as measures of accuracy and precision. Statistical analyses with paired T-tests and Wilcoxon-Signed-Rank(WSR)-tests were performed using GNU-R 4.0.3. Finally, we recommend the dimensionality of the U-Net architecture based on accuracy, i.e. greater mean or median, and precision, i.e. smaller standard deviation or IQR and smaller number of outliers. We also keep in mind the relationship between quality and GPU overhead in time and memory.

The qualitative liver results (figure 8) show that the 2D U-Net (top) is superior with more uniform coverage of the segmentation area. The 3D U-Net (bottom) obviously suffers from under-segmentation as more brown ground truth surface is visible in figure 8 (bottom). With reference to table 1, the precision of the 2D U-Net is also higher due to the lower std. deviations and IQRs.

Regarding the kidney results, the 2D U-Net is highly significantly better in both T- and WSR-tests. The boxplots with a small x for the mean in figures 9, 10 visually confirm: the 2D U-Net is significantly better for liver and highly significantly better for kidneys (figures 9a, 10a, 9b, 10b).

For spleen segmentation, the 2D U-Net tends to have an advantage, as indicated in figures 9c, 10c and table 1.

The 3D U-Net has higher accuracy and precision for the low-contrast pancreas. The higher precision is reflected in smaller standard deviations and fewer outliers in table 1 and figures 9d and 10d. In terms of accuracy measured by medians, the 3D U-Net has trend advantage of 0.02 over the 2D U-Net, while precision measured by IQR is equal.

The GPU-Performance evaluation in table 2 shows the superiority of the 2D U-Net at the scale of this study with an 80-image data base. The ratio of quality to GPU resources is always better for the 2D U-Net. In table 2 GPU memory saving is >6-fold in the training phase and >5-fold in the model application, trivially a highly significant result since there is no variation, i.e. no standard deviation and IQR. In terms of GPU calculation times measured during training, the 2D U-Net is exactly 40 seconds or 7% faster on average, again a highly significant average result. In the U-Net model application, we can observe a weakly significant ( p < 0.1) advantage for the 2D U-Net from 37% to 75%.

Surprisingly in this study, 2D U-Nets are favorable regarding the ratio of quality vs. computing costs.

The liver and spleen hold the greatest volume in our abdominal organ group. The liver and especially kidney competition is significantly won by the 2D U-Net ( p < 0.05). The liver is a difficult organ often with a variably filled stomach as a neighbor.

Kidneys can be regarded as easy organs lighted by contrast agent and inside fatty tissue with low CT intensity.

A better posed training for the 2D U-Net could be the reason for the 2D U-Nets’ better results for liver and kidney tasks. A higher relative number training elements is used, i.e. axial slice pixels, vs. the number of net weights. The spleen results are in favor of the 2D U-Net regarding the mean and medians by trend. The 2D U-Net is also favorable for a less number of outliers (figure 9c).

The difficult pancreas does not provide many axial training slices useful for the 2D U-Net, as its elongation is not prominent on the z-axis. We suppose this is the reason, why and precision, i.e. higher mean and lower standard deviation and lower number of outliers (figure 9d). This win is supported by higher accuracy for the 3D U-Net in terms of medians. However, the race is not decided by significant differences making the 2D U-Net still very attractive for some users with GPU performance and memory concerns. Abdominal volumetric CT images and key organ segmentation were analyzed.

This new study shows interesting results from competing U-Net architectures, especially focusing different dimensionalities of net filter bank kernels and quality vs. GPU performance. The interested reader can now select a particular U-Net architecture, primarily whether to use a computational inexpensive design. Finally, in this study’s scope, a humble recommendation for the 3D U-Net could be given for the pancreatic organ in terms of better accuracy by trend and smaller standard deviation only. The IQRs as an alternative measure of precision are on par, and regarding the median the 3D U-Net wins just by a small trend. We coconclude, because of the deeper layering structure and thus more trainable weights, a 3D U-Net needs significantly more training data vs. possible overfitting to outpace 2D U-Nets.

As training volumes are normalized to a square or cube of 96 voxels, the GPU memory consumption in table 2 is always constant. Therefore, differences are trivially highly significant, as no varying results occur. We observe consistently lower memory consumption for 2D U-Nets. The memory effort in training is higher for the 3D U-Net including more space for administrative overhead data. Thus, 2D U-Nets can run on affordable 2-4 GB GPUs for 3D CT volume segmentation. 3D U-Nets definitely need currently totally overpriced 12 GB GPUs.

The timing measures in table 2 clearly speak out for the 2D U-Net. Training is highly significantly 40 seconds or 7% faster on average using the 2D U-Net. However, regarding application, in the trained model prediction, the differences are not so striking with a weak significance by median. 37% to 75% improvement can be achieved, however in the range of one second, which is practically unimportant.

As final and bold conclusion regarding our study design and results, we can recommend using the 3D U-Net subject to the amount of data we used here - for pancreas only. The conclusions are justified by statistically significant or by trend quality and GPU-computation performance results for all organs under study. We suppose, 3D U-Nets may overcome in quality when using several hundreds of training images. However, this comes approximately with an order of magnitude higher additional computational burden.

For the first time, an original and significant comparison of U-Net architecture dimension is provided to the reader focusing the key abdominal organs of liver, spleen, kidneys and pancreas. The reader can decide, which approach is appropriate for his concrete target organ, amount of training data and used GPU or cluster nodes, parallel process design, e.g. for atlas-based usage of U-Nets as encountered in multiclassifier fusion [18].

Regarding the difficult pancreas with mixed DSC results in this study, we plan to train 2D U-Nets using a elongation optimized algorithm to provide slices oriented axially along its main central curve, to better reflect its orientation to generate more training slices for 2D U-Nets [30,31,11]. On the other hand, more training images shall be used to explore the 3D U-Nets’ theoretical advantage under better conditions, as we have discussed, for its training and application.

The scientific explanation of the U-Net methods is given al. and we lift these methods here to compare them [1,2].

The current limitations of this study will manifest as improvements in the future. Development will cover improved bounding box detection. At this state of our research, our current RRF bounding box detection would confound the core message of this paper, the aim of which was to focus purely on U-Net performances.

German Research Foundation: DFG MA 6791/1-1.