泰格的技术分享

菜单

tanger
发布于 2024-04-30 / 13 阅读 / 0 评论 / 0 点赞

Springboot多模块，多环境，docker分层镜像打包的最佳实践

Springboot多模块，多环境，docker分层镜像打包的最佳实践

docker镜像为什么要分层打包？以我自己的实际经验，springboot打fat包，然后以 openjdk:21 构建成镜像，镜像大小差不多得5-600mb起步。即使除去openjdk:21的基础层，光应用的jar包就有100mb+。

镜像上传到服务器的时候，网速一般的情况下，非常慢。还有另外一个原因就是每个镜像重复占用硬盘，对于devOps不停打包，硬盘空间很快就消耗完了。

从springboot2.3之后，springboot打包插件（spring-boot-maven-plugin）支持了docker分层打包，可以把springboot依赖，项目依赖单独领出来作为docker的层，可以为后续的镜像提供重用镜像层。这就意味着只有自己项目的代码的jar包（这部分内容非常小，10几到几百kb），才会触发上传和镜像空间占用。

docker的一大特色就是镜像的存储是分层的，参考下面这张官图

我们在Dockerfile中的每一个指令会对应到镜像的每一层，docker在更新镜像时，只会推送变更过的层，当它计算出来这一层的摘要和之前的版本一致时，会复用上一次打包镜像时的缓存，会极大的提高打包镜像以及镜像push/pull操作的速度。

分层jar
Spring Boot始终支持它的fat格式，jar包解压后的目录结构如下：

为三个主要部分：

  • classes是用于引导jar加载的类

  • Spring Boot应用程序类是在 BOOT-INF/classes

  • 依赖是在 BOOT-INF/lib

2.3.0.M1之后springboot（spring-boot-maven-plugin）提供了layout一种称为的新类型LAYERED_JAR，默认会打开，此时会多出layers.idx和classpath.idx两个分层索引文件。

执行一下命令来查看当前jar包支持的分层：

java -Djarmode=layertools -jar target/demo-0.0.1-SNAPSHOT.jar list

输出：

dependencies
spring-boot-loader
snapshot-dependencies
application

  • dependencies 项目的jar依赖（变化小）

  • spring-boot-loader springboot的启动引导class（基本不变）

  • snapshot-dependencies 项目的快照jar依赖 （变化小）

  • application 项目自身的jar （变化大）

编写dockerfile的时候，从上到下copy分层，从上到下的变更可能性逐渐变大，因为一旦上层的数据变动，将会引起下层全部分层缓存失效，不能复用。

接下来使用命令看看分层后的目录结构（list命令只是查看支持什么分层，并没有进行分层）：

java -Djarmode=layertools -jar demo-0.0.1-SNAPSHOT.jar extract

结果如下：（由于没有快照包，所以没有snapshot-dependencies目录）

接下来制作一个dockerfile用于提取并复制每一层的文件。这是一个例子：


ROM openjdk:11 as builder
WORKDIR application
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} application.jar
RUN java -Djarmode=layertools -jar application.jar extract

FROM openjdk:11
WORKDIR application
COPY --from=builder application/dependencies/ ./
COPY --from=builder application/snapshot-dependencies/ ./
COPY --from=builder application/resources/ ./
COPY --from=builder application/application/ ./
ENTRYPOINT ["java", "org.springframework.boot.loader.JarLauncher"]

最佳实践

以上说明了分层的基本原理和操作，接下来按照实际项目进行改造。

假设当前有如下多模块项目：

编写项目根pom

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.1.0</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>org.example</groupId>
    <artifactId>docker-spring-build</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>pom</packaging>
    <modules>
        <module>dsb-api</module>
        <module>dsb-boot</module>
    </modules>

    <properties>
        <maven.compiler.source>21</maven.compiler.source>
        <maven.compiler.target>21</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

</project>

编写启动模块的pom

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.example</groupId>
        <artifactId>docker-spring-build</artifactId>
        <version>1.0-SNAPSHOT</version>
    </parent>

    <artifactId>dsb-boot</artifactId>

    <properties>
        <maven.compiler.source>21</maven.compiler.source>
        <maven.compiler.target>21</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.example</groupId>
            <artifactId>dsb-api</artifactId>
            <version>${project.version}</version>
        </dependency>
    </dependencies>

    <build>
        <resources>
            <!--设置包含哪些文件到“target/classes”目录下-->
            <resource>
                <directory>src/main/resources</directory>
                <filtering>true</filtering>
                <includes>
                    <include>**/**</include>
                </includes>
            </resource>
        </resources>

        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-resources-plugin</artifactId>
                <version>3.3.1</version>
                <configuration>
                    <!--使用默认的替换展位符 ${}-->
                    <useDefaultDelimiters>true</useDefaultDelimiters>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <layers>
                        <enabled>true</enabled>
                        <!--支持自定义分层-->
                        <configuration>${project.basedir}/layers.xml</configuration>
                    </layers>
            </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>3.2.0</version>
                <!--排除 META-INFO/maven 文件夹-->
                <configuration>
                    <archive>
                        <addMavenDescriptor>false</addMavenDescriptor>
                    </archive>
                </configuration>
            </plugin>
            <plugin>
                <!--打包完成之后如归档插件，就是把jar，配置文件等打成tar包-->
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <executions>
                    <execution>
                        <id>assembly</id>
                        <goals>
                            <goal>single</goal>
                        </goals>
                        <phase>package</phase>
                    </execution>
                </executions>
                <configuration>
                    <descriptors>
                        <!--设置详细的打包规则文件-->
                        <descriptor>${project.basedir}/src/main/assembly/release.xml</descriptor>
                    </descriptors>
                    <appendAssemblyId>false</appendAssemblyId>
                    <outputDirectory>${project.basedir}/target</outputDirectory>
                    <finalName>${project.name}_${project.version}_${env.active}</finalName>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <!--支持多环境配置-->
    <profiles>
        <profile>
            <id>dev</id>
            <activation>
                <activeByDefault>true</activeByDefault>
            </activation>
            <properties>
                <env.active>dev</env.active>
            </properties>
        </profile>
        <profile>
            <id>test</id>
            <properties>
                <env.active>dev</env.active>
            </properties>
        </profile>
        <profile>
            <id>prod</id>
            <properties>
                <env.active>prod</env.active>
            </properties>
        </profile>
    </profiles>

</project>

详细说明大家请看注释内容

release.xml

<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
    <id>dist</id>
    <formats>
        <format>tar.gz</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <fileSets>
        <fileSet>
            <directory>.</directory>
            <outputDirectory>/</outputDirectory>
            <includes>
                <include>README*</include>
            </includes>
        </fileSet>
        <fileSet>
            <directory>target</directory>
            <outputDirectory>/</outputDirectory>
            <includes>
                <include>*.jar</include>
            </includes>
        </fileSet>
        <fileSet>
            <directory>./src/main/bin</directory>
            <outputDirectory>bin</outputDirectory>
            <includes>
                <include>**/*</include>
            </includes>
            <fileMode>0755</fileMode>
        </fileSet>
        <fileSet>
            <directory>target/classes</directory>
            <outputDirectory>/config</outputDirectory>
            <includes>
                <include>application.yml</include>
                <include>application-*.yml</include>
                <include>spy.properties</include>
                <include>*.xml</include>
                <include>templates/**/**.ftl</include>
                <include>i18n/**/**.properties</include>
            </includes>
        </fileSet>
        <fileSet>
            <directory>target/classes</directory>
            <outputDirectory>/</outputDirectory>
            <includes>
                <include>version.txt</include>
                <include>applicationName.txt</include>
            </includes>
        </fileSet>
    </fileSets>
</assembly>

自定义docker分层

dsb-boot目录下添加layers.xml分层文件

<layers xmlns="http://www.springframework.org/schema/boot/layers"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.springframework.org/schema/boot/layers
                          https://www.springframework.org/schema/boot/layers/layers-3.2.xsd">

    <application>
        <into layer="spring-boot-loader">
            <include>org/springframework/boot/loader/**</include>
        </into>
        <into layer="application" />
    </application>
    <dependencies>
        <!--这里的顺序决定依赖处理的分层，从上到下处理-->
        <!--先处理二方依赖-->
        <into layer="second-snapshot-dependencies">
            <!--规则是  groupId:artifactId:version -->
            <include>org.example:*:*SNAPSHOT</include>
            <exclude>org.example:dsb-api:*SNAPSHOT</exclude>
        </into>
        <into layer="second-dependencies">
            <include>org.example:*:*</include>
            <!--排除一方依赖，保留到下面的application-->
            <exclude>org.example:dsb-api:*</exclude>
        </into>
        <into layer="application">
            <includeModuleDependencies />
        </into>
        <!--三方快照包单独一层-->
        <into layer="snapshot-dependencies">
            <include>*:*:*SNAPSHOT</include>
        </into>
        <into layer="dependencies" />
    </dependencies>
    <layerOrder>
        <layer>spring-boot-loader</layer>
        <layer>dependencies</layer>
        <!--三方快照修改频率较小-->
        <layer>snapshot-dependencies</layer>
        <layer>second-dependencies</layer>
        <layer>second-snapshot-dependencies</layer>
        <layer>application</layer>
    </layerOrder>
</layers>

编写Dockerfile

FROM openjdk:21 as builder
WORKDIR application
ARG TAR_FILE=target/*.tar.gz
ADD ${TAR_FILE} /application
RUN mv *.jar app.jar
RUN java -Djarmode=layertools -jar app.jar extract


FROM openjdk:21
WORKDIR application
COPY --from=builder application/spring-boot-loader/ ./
COPY --from=builder application/dependencies/ ./
RUN mkdir ./bin
COPY --from=builder application/bin/** ./bin
COPY --from=builder application/snapshot-dependencies/ ./
COPY --from=builder application/second-dependencies/ ./
COPY --from=builder application/second-snapshot-dependencies/ ./
RUN mkdir ./config
COPY --from=builder application/config/ ./config
COPY --from=builder application/application/ ./
COPY --from=builder application/version.txt ./
ENTRYPOINT ["/application/bin/dockerStartup.sh"]

dsb-boot添加启动脚本

src/main/bin/dockerStartup.sh

#!/bin/bash

current_path=`pwd`

# $0 代表当前执行脚本的位置（名称） 例如执行 /application/bin/dockerStartup.sh  $0=/application/bin/dockerStartup.sh
# 执行 ./bin/dockerStartup.sh $0=./bin/dockerStartup.sh

# 获取脚本的绝对路径
case "`uname`" in
    Linux)
		bin_abs_path=$(readlink -f $(dirname $0))
		;;
	*)
		bin_abs_path=`cd $(dirname $0); pwd`
		;;
esac
base=${bin_abs_path}/..

export LANG=en_US.UTF-8
export BASE=$base

#环境变量设置
if [ -z "$JAVA_MEM_OPTS" ] ; then
  JAVA_MEM_OPTS="-Xms512m -Xmx512m "
fi
if [ -z "$JAVA_GC_OPTS" ] ; then
  JAVA_GC_OPTS="-XX:+UseG1GC -XX:MaxGCPauseMillis=250 -XX:+UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent "
fi
if [ -z "$JAVA_OTHER_OPTS" ] ; then
  JAVA_OTHER_OPTS="-Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 "
fi
if [ -z "$APP_NAME" ] ; then
  APP_NAME="default-app"
fi
JAVA="java -server"
SERVER_NAME="-DappName=$APP_NAME"


echo "cd to $bin_abs_path for workaround relative path"
cd $base
#echo CLASSPATH :$CLASSPATH
exec $JAVA $SERVER_NAME $JAVA_MEM_OPTS $JAVA_GC_OPTS $JAVA_OTHER_OPTS $JAVA_DEBUG_OPT org.springframework.boot.loader.JarLauncher

Maven打包

mvn clean package -Dmaven.test.skip=true

打包镜像

cd  dsb-boot
docker build --tag dsb:1.55 .

分层效果

首先打包一次镜像，观察日志

cd  dsb-boot
docker build --tag dsb:1.55 .

dsb:1.55日志：

docker build --tag dsb:1.55 .
[+] Building 17.3s (21/21) FINISHED                                                                                                 docker:orbstack
 => [internal] load build definition from Dockerfile                                                                                           0.0s
 => => transferring dockerfile: 809B                                                                                                           0.0s
 => [internal] load metadata for docker.io/library/openjdk:21                                                                                 15.7s
 => [internal] load .dockerignore                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                0.0s
 => [internal] load build context                                                                                                              0.2s
 => => transferring context: 17.03MB                                                                                                           0.2s
 => [stage-1  1/13] FROM docker.io/library/openjdk:21@sha256:af9de795d1f8d3b6172f6c55ca9ba1c5768baa11bb2dc8af7045c7db9d4c33ac                  0.0s
 => CACHED [stage-1  2/13] WORKDIR application                                                                                                 0.0s
 => [builder 3/5] ADD target/*.tar.gz /application                                                                                             0.1s
 => [builder 4/5] RUN mv *.jar app.jar                                                                                                         0.3s
 => [builder 5/5] RUN java -Djarmode=layertools -jar app.jar extract                                                                           0.6s
 => [stage-1  3/13] COPY --from=builder application/spring-boot-loader/ ./                                                                     0.0s
 => [stage-1  4/13] COPY --from=builder application/dependencies/ ./                                                                           0.0s
 => [stage-1  5/13] RUN mkdir ./bin                                                                                                            0.1s
 => [stage-1  6/13] COPY --from=builder application/bin/** ./bin                                                                               0.0s
 => [stage-1  7/13] COPY --from=builder application/snapshot-dependencies/ ./                                                                  0.0s
 => [stage-1  8/13] COPY --from=builder application/second-dependencies/ ./                                                                    0.0s
 => [stage-1  9/13] COPY --from=builder application/second-snapshot-dependencies/ ./                                                           0.0s
 => [stage-1 10/13] RUN mkdir ./config                                                                                                         0.1s
 => [stage-1 11/13] COPY --from=builder application/config/ ./config                                                                           0.0s
 => [stage-1 12/13] COPY --from=builder application/application/ ./                                                                            0.0s
 => [stage-1 13/13] COPY --from=builder application/version.txt ./                                                                             0.0s
 => exporting to image                                                                                                                         0.0s
 => => exporting layers                                                                                                                        0.0s
 => => writing image sha256:94fc88ee72602d085129974f2c98e2dcc8cc5bd041be787b8f1e66e9747710de                                                   0.0s
 => => naming to docker.io/library/dsb:1.55

修改一些代码，重新打包

mvn clean package -Dmaven.test.skip=true
cd  dsb-boot
docker build --tag dsb:1.56 .

观察日志：

 docker build --tag dsb:1.56 .
[+] Building 19.4s (21/21) FINISHED                                                                                                 docker:orbstack
 => [internal] load build definition from Dockerfile                                                                                           0.0s
 => => transferring dockerfile: 809B                                                                                                           0.0s
 => [internal] load metadata for docker.io/library/openjdk:21                                                                                 18.2s
 => [internal] load .dockerignore                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                0.0s
 => [stage-1  1/13] FROM docker.io/library/openjdk:21@sha256:af9de795d1f8d3b6172f6c55ca9ba1c5768baa11bb2dc8af7045c7db9d4c33ac                  0.0s
 => [internal] load build context                                                                                                              0.1s
 => => transferring context: 17.03MB                                                                                                           0.1s
 => CACHED [stage-1  2/13] WORKDIR application                                                                                                 0.0s
 => [builder 3/5] ADD target/*.tar.gz /application                                                                                             0.1s
 => [builder 4/5] RUN mv *.jar app.jar                                                                                                         0.2s
 => [builder 5/5] RUN java -Djarmode=layertools -jar app.jar extract                                                                           0.5s
 => CACHED [stage-1  3/13] COPY --from=builder application/spring-boot-loader/ ./                                                              0.0s
 => CACHED [stage-1  4/13] COPY --from=builder application/dependencies/ ./                                                                    0.0s
 => CACHED [stage-1  5/13] RUN mkdir ./bin                                                                                                     0.0s
 => CACHED [stage-1  6/13] COPY --from=builder application/bin/** ./bin                                                                        0.0s
 => CACHED [stage-1  7/13] COPY --from=builder application/snapshot-dependencies/ ./                                                           0.0s
 => CACHED [stage-1  8/13] COPY --from=builder application/second-dependencies/ ./                                                             0.0s
 => CACHED [stage-1  9/13] COPY --from=builder application/second-snapshot-dependencies/ ./                                                    0.0s
 => CACHED [stage-1 10/13] RUN mkdir ./config                                                                                                  0.0s
 => CACHED [stage-1 11/13] COPY --from=builder application/config/ ./config                                                                    0.0s
 => [stage-1 12/13] COPY --from=builder application/application/ ./                                                                            0.0s
 => [stage-1 13/13] COPY --from=builder application/version.txt ./                                                                             0.0s
 => exporting to image                                                                                                                         0.0s
 => => exporting layers                                                                                                                        0.0s
 => => writing image sha256:997ce0e663d31f104d719b65d5bf6ea8f56ff5552dd7a45f195de93b84967f0b                                                   0.0s
 => => naming to docker.io/library/dsb:1.56

步骤3-11均使用到了CACHED（缓存），使用缓存后可以减小硬盘占用空间合镜像的推送和拉取速度。

源码

docker-spring-build.zip

docker swarm 错误日志...
Netty报错Too many o...

评论