使用DockerFile构建Bare Metal镜像

Mutable和Immutable介绍

云原生的代表技术包括容器、服务网格、微服务、不可变基础设施和声明式API。容器技术的最大创造就是通过Dockerfile将应用打包为容器镜像，实现了不可变基础设施，标准化了应用模板。
在容器之前叫Mutable（可变的基础设施）在OS上部署应用，重启生效，可以随时进行修改。

容器技术就是Immutable的代表，引入容器镜像，通过Dockerfile将应用标准化打包为容器镜像，通过容器镜像启动容器，无法在容器中进行永久性修改，需要修改只能通过更新Dockerfile方式进行。
现如今Immutable理念也开始逐步从容器下沉到Bare Metal OS，通过Dockerfile构建Bare Metal镜像，实现Bare Metal OS Immutable。典型的开源项目技术Elemental项目

Elemental概述

Elemental 是一系列工具集合，主要是想通过 Kubernetes 实现集中式、完整的云原生操作系统构建和管理。集群节点操作系统是通过Elemental CLI通过容器映像构建和维护的，并使用Elemental CLI安装在新主机上。
Elemental Operator和Rancher System Agent使Rancher Manager 能够完全控制 Elemental 集群，从在节点上安装和管理操作系统到以集中方式配置新的 K3s 或 RKE2 集群。 Elemental项目组成

• elemental-toolkit – 包括一组操作系统实用程序，可通过容器启用操作系统管理。包括 dracut 模块、引导加载程序配置、cloud-init 自定义配置服务等。
• elemental-operator – 这连接到 Rancher Manager 并处理 machineRegistration 和 machineInventory CRD
• elemental-register – 这通过 machineRegistrations 注册机器并通过 elemental-cli 安装
• elemental-cli – 这会安装任何基于 elemental-toolkit 的衍生工具。实现OCI容器镜像构建为可在虚拟机、物理机、嵌入式设备运行的ISO镜像。
• rancher-system-agent – 在已安装的系统上运行并从 Rancher Manager 获取命令在系统上安装和运行rancher-agent，注册到Rancher中。

项目地址：https://github.com/rancher/elemental-toolkit

配置使用

在一台装有Docker的主机上进行提前准备项：

• 一台安装了Docker的主机
• Harbor镜像仓库
• EXSI或物理pc、服务器用于build后的ISO测试

使用Elemental-toolkit构建ISO流程

• 基础base镜像发行版：
  teal: SLE Micro for Rancher based one, shipping packages from Sle Micro 5.3.
  green: openSUSE based one, shipping packages from OpenSUSE Leap 15.4 repositories.
  blue: Fedora based one, shipping packages from Fedora 33 repositories
  orange: Ubuntu based one, shipping packages form Ubuntu 20.10 repositories
• 自定义镜像并制作OCI Image
• 在装有Docker的机器启动Elemental Build
  UEFI Boot，选择合适的实例类型
  Clout-init userdata 初始化
  Default user/pass: root/cos
• 升级自定义镜像
  elemental upgrade –no-verify –reboot -d niusmallnan/containeros\:dev

在安装了Docker的主机上创建/root/derivative目录。整体目录结构

/root/derivative/├── Dockerfile├── cloud-init.yaml├── install.sh├── installer.sh├── k3s├── k3s-airgap-images-amd64.tar.gz├── manifest.yaml├── nginx.yaml├── overlay│   └── iso│       └── boot│           └── grub2│               └── grub.cfg└── repositories.yaml

Demo架构

• 通过Elemental构建的OS中包含K3S
• 将需要部署的应用yaml放置到 /var/lib/rancher/k3s/server/manifests目录，K3S启动成功后会自动部署yaml启动应用。

下载K3S离线镜像包和CLI文件
https://github.com/k3s-io/k3s/releasesnginx.yaml文件用于k3s启动后加载此yaml文件，模拟演示是个应用

Dockerfile文件创建

ARG LUET_VERSION=0.32.0FROM quay.io/luet/base:$LUET_VERSION AS luetFROM registry.suse.com/suse/sle-micro-rancher/5.2ARG ARCH=amd64ENV ARCH=${ARCH}# Copy the luet config file pointing to the upgrade repositoryCOPY repositories.yaml /etc/luet/luet.yaml# Copy luet from the official imagesCOPY --from=luet /usr/bin/luet /usr/bin/luetENV LUET_NOLOCK=trueRUN luet install -y \toolchain/yip \toolchain/luet \utils/installer \system/cos-setup \system/immutable-rootfs \system/grub2-config \system/base-dracut-modulesRUN  mkdir /var/lib/rancher/k3s/agent/images/ -p &&  mkdir /var/lib/rancher/k3s/server/manifests -pCOPY install.sh /system/oem/COPY k3s /usr/local/binCOPY nginx.yaml /system/oem/COPY k3s-airgap-images-amd64.tar.gz /system/oem/RUN  chmod a+x /usr/local/bin/k3s && chmod a+x /system/oem/install.shWORKDIR /system/oemRUN  INSTALL_K3S_SKIP_START="true" INSTALL_K3S_SKIP_ENABLE="true" INSTALL_K3S_SKIP_DOWNLOAD="true" sh install.sh## System layout# Required by k3s etc.RUN mkdir /usr/libexec && mkdir /usr/local/bin -p && touch /usr/libexec/.keep# Copy custom files# COPY files/ /# Copy cloud-init default configurationCOPY cloud-init.yaml /system/oem/# Generate initrdRUN mkinitrd# OS level configurationRUN echo "VERSION=999" > /etc/os-releaseRUN echo "GRUB_ENTRY_NAME=derivative" >> /etc/os-releaseRUN echo "welcome to our derivative" >> /etc/issue.d/01-derivative

cloud-init文件创建，主要用于磁盘分区配置和登录用户名和密码配置
cloud-init.yaml

name: "Default settings"stages:initramfs:# Setup default hostname- name: "Branding"hostname: "derivative"# Setup an admin group with sudo access- name: "Setup groups"ensure_entities:- entity: |kind: "group"group_name: "admin"password: "x"gid: 900            # Setup network - openSUSE specific- name: "Network setup"files:- path: /etc/sysconfig/network/ifcfg-eth0content: |BOOTPROTO='dhcp'STARTMODE='onboot'                  permissions: 0600owner: 0group: 0# Setup a custom user- name: "Setup users"users:# Replace the default user name here and settingsjoe:# Comment passwd for no passwordpasswd: "joe"shell: /bin/bashhomedir: "/home/joe"groups:- "admin"#authorized_keys:# Replace here with your ssh keys# joe: # - ssh-rsa ....# Setup sudo- name: "Setup sudo"files:- path: "/etc/sudoers"owner: 0group: 0permsisions: 0600content: |Defaults always_set_homeDefaults secure_path="/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin:/usr/local/sbin"Defaults env_resetDefaults env_keep = "LANG LC_ADDRESS LC_CTYPE LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE LC_ATIME LC_ALL LANGUAGE LINGUAS XDG_SESSION_COOKIE"Defaults !insultsroot ALL=(ALL) ALL%admin ALL=(ALL) NOPASSWD: ALL@includedir /etc/sudoers.d            commands:- passwd -l root# Setup persistency so k3s works properly# See also: https://rancher.github.io/elemental-toolkit/docs/reference/immutable_rootfs/#configuration-with-an-environment-filerootfs.after:- name: "Immutable Layout configuration"environment_file: /run/cos/cos-layout.envenvironment:VOLUMES: "LABEL=COS_OEM:/oem LABEL=COS_PERSISTENT:/var"OVERLAY: "tmpfs:25%"RW_PATHS: "/usr/local /etc /srv"PERSISTENT_STATE_PATHS: >-/etc/systemd/etc/rancher/etc/ssh/etc/iscsi /etc/cni/home/opt/root/usr/libexec/var/log/var/lib/wicked/var/lib/longhorn/var/lib/cni/usr/local/binPERSISTENT_STATE_TARGET: >-/etc/systemd/etc/rancher/etc/ssh/etc/iscsi/etc/cni/home/opt/root/usr/libexec/var/log/var/lib/kubelet/var/lib/wicked/var/lib/longhorn/var/lib/cni/usr/local/binPERSISTENT_STATE_BIND: "true"# Finally, let's start k3s when network is available, and download the SSH key from github for the joe usernetwork:- name: "Deploy cos-system"commands:- elemental install /dev/sda - systemctl enable k3s && systemctl start  k3safter-install:- name: "install k3s"commands:- mount /dev/sda5 /var- mkdir -p  /var/lib/rancher/k3s/agent/images/  && mkdir /var/lib/rancher/k3s/server/manifests -p- cp /system/oem/k3s-airgap-images-amd64.tar.gz  /var/lib/rancher/k3s/agent/images/- cp /system/oem/nginx.yaml /var/lib/rancher/k3s/server/manifests- reboot

创建manifest.yaml文件定义OS启动引导所需要文件

iso:rootfs:- channel:system/cosuefi:- channel:live/grub2-efi-imageimage:- channel:live/grub2- channel:live/grub2-efi-imagelabel: "COS_LIVE"name: "cOS-0"# Raw disk creation values startraw_disk:x86_64:# which packages to install and the target to install them atpackages:- name: channel:system/grub2-efi-imagetarget: efi- name: channel:system/grub2-configtarget: root- name: channel:system/grub2-artifactstarget: root/grub2- name: channel:recovery/cos-imgtarget: root/cOSrepositories:- uri: quay.io/costoolkit/releases-tealarch: "x86_64"

创建repositories.yaml文件

logging:color: falseenable_emoji: falsegeneral:debug: falsespinner_charset: 9repositories:- name: "cos"description: "cOS official"type: "docker"enable: truecached: truepriority: 1verify: falseurls:- "quay.io/costoolkit/releases-green"

创建grub文件配置内核引导
在/root/derivative/overlay/iso/boot/目录创建grub2grub.cfg文件

search --no-floppy --file --set=root /boot/kernelset default=0set timeout=10set timeout_style=menuset linux=linuxset initrd=initrdif [ "${grub_cpu}" = "x86_64" -o "${grub_cpu}" = "i386" -o "${grub_cpu}" = "arm64" ];thenif [ "${grub_platform}" = "efi" ]; thenif [ "${grub_cpu}" != "arm64" ]; thenset linux=linuxefiset initrd=initrdefifififiif [ "${grub_platform}" = "efi" ]; thenecho "Please press 't' to show the boot menu on this console"fiset font=($root)/boot/${grub_cpu}/loader/grub2/fonts/unicode.pf2if [ -f ${font} ];thenloadfont ${font}fimenuentry "cOS" --class os --unrestricted {echo Loading kernel...$linux ($root)/boot/kernel.xz cdroot root=live:CDLABEL=COS_LIVE rd.live.dir=/ rd.live.squashimg=rootfs.squashfs console=tty1 console=ttyS0 rd.cos.disableecho Loading initrd...$initrd ($root)/boot/rootfs.xz}if [ "${grub_platform}" = "efi" ]; thenhiddenentry "Text mode" --hotkey "t" {set textmode=trueterminal_output console}fi

先构建镜像

docker build -t 172.16.1.208/library/example:v4.0 .

镜像要上传到镜像仓库才能build iso

docker push 172.16.1.208/library/example:v4.0

构建ISO

注：目前只支持公开的镜像仓库，不支持私有的镜像仓库
https://github.com/rancher/elemental-cli/issues/389构建完成，生成此cOS-0.iso镜像文件

将cOS-0.iso下载到ESXI或其他虚拟化平台也可以刻录U盘直接安装物理机。配置选4c4G 60G磁盘
加载ISO后自动分区，自动进行初始化，安装系统，完成后自动重启进入系统。密码ssh账号密码joe/joe
在安装后的系统查看已经部署好的K3S。 查看自动部署的应用
访问应用因为整个系统都限制了修改，所以在操作系统任何目录执行修改命令都无法修改。如

总结

通过Elemental实现了操作系统为不变基础设施，同时也可以将我们传统的OS带入云原生，通过Dockerfile去构建，通过CICD去统一发版维护，目前能想到的一个比较大的应用场景在于，一个是边缘场景，边缘设备操作系统批量部署安装。另外就是一些to b的客户将自己业务+容器编排和OS通过Elemental构建打包，直接到客户现场加载ISO就部署完了，开箱即用。另外OS也可以标准化，统一化管理。