Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
| ... | ... |
@@ -26,8 +26,8 @@ github.com/imdario/mergo v0.3.6 |
| 26 | 26 |
golang.org/x/sync 1d60e4601c6fd243af51cc01ddf169918a5407ca |
| 27 | 27 |
|
| 28 | 28 |
# buildkit |
| 29 |
-github.com/moby/buildkit c7bb575343df0cbfeab8b5b28149630b8153fcc6 |
|
| 30 |
-github.com/tonistiigi/fsutil f567071bed2416e4d87d260d3162722651182317 |
|
| 29 |
+github.com/moby/buildkit 8cf9bec86a7f11fe6591804aee152c8e8a7a8a0d # v0.3.3 |
|
| 30 |
+github.com/tonistiigi/fsutil 2862f6bc5ac9b97124e552a5c108230b38a1b0ca |
|
| 31 | 31 |
github.com/grpc-ecosystem/grpc-opentracing 8e809c8a86450a29b90dcc9efbf062d0fe6d9746 |
| 32 | 32 |
github.com/opentracing/opentracing-go 1361b9cd60be79c4c3a7fa9841b3c132e40066a7 |
| 33 | 33 |
github.com/google/shlex 6f45313302b9c56850fc17f99e40caebce98c716 |
| ... | ... |
@@ -27,9 +27,11 @@ Read the proposal from https://github.com/moby/moby/issues/32925 |
| 27 | 27 |
|
| 28 | 28 |
Introductory blog post https://blog.mobyproject.org/introducing-buildkit-17e056cc5317 |
| 29 | 29 |
|
| 30 |
+:information_source: If you are visiting this repo for the usage of experimental Dockerfile features like `RUN --mount=type=(bind|cache|tmpfs|secret|ssh)`, please refer to [`frontend/dockerfile/docs/experimental.md`](frontend/dockerfile/docs/experimental.md). |
|
| 31 |
+ |
|
| 30 | 32 |
### Used by |
| 31 | 33 |
|
| 32 |
-[Moby](https://github.com/moby/moby/pull/37151) |
|
| 34 |
+[Moby & Docker](https://github.com/moby/moby/pull/37151) |
|
| 33 | 35 |
|
| 34 | 36 |
[img](https://github.com/genuinetools/img) |
| 35 | 37 |
|
| ... | ... |
@@ -37,6 +39,12 @@ Introductory blog post https://blog.mobyproject.org/introducing-buildkit-17e056c |
| 37 | 37 |
|
| 38 | 38 |
[container build interface](https://github.com/containerbuilding/cbi) |
| 39 | 39 |
|
| 40 |
+[Knative Build Templates](https://github.com/knative/build-templates) |
|
| 41 |
+ |
|
| 42 |
+[boss](https://github.com/crosbymichael/boss) |
|
| 43 |
+ |
|
| 44 |
+[Rio](https://github.com/rancher/rio) (on roadmap) |
|
| 45 |
+ |
|
| 40 | 46 |
### Quick start |
| 41 | 47 |
|
| 42 | 48 |
Dependencies: |
| ... | ... |
@@ -79,6 +87,7 @@ See [`solver/pb/ops.proto`](./solver/pb/ops.proto) for the format definition. |
| 79 | 79 |
Currently, following high-level languages has been implemented for LLB: |
| 80 | 80 |
|
| 81 | 81 |
- Dockerfile (See [Exploring Dockerfiles](#exploring-dockerfiles)) |
| 82 |
+- [Buildpacks](https://github.com/tonistiigi/buildkit-pack) |
|
| 82 | 83 |
- (open a PR to add your own language) |
| 83 | 84 |
|
| 84 | 85 |
For understanding the basics of LLB, `examples/buildkit*` directory contains scripts that define how to build different configurations of BuildKit itself and its dependencies using the `client` package. Running one of these scripts generates a protobuf definition of a build graph. Note that the script itself does not execute any steps of the build. |
| ... | ... |
@@ -145,6 +154,10 @@ buildctl build --frontend=gateway.v0 --frontend-opt=source=tonistiigi/dockerfile |
| 145 | 145 |
buildctl build --frontend gateway.v0 --frontend-opt=source=tonistiigi/dockerfile --frontend-opt=context=git://github.com/moby/moby --frontend-opt build-arg:APT_MIRROR=cdn-fastly.deb.debian.org |
| 146 | 146 |
```` |
| 147 | 147 |
|
| 148 |
+##### Building a Dockerfile with experimental features like `RUN --mount=type=(bind|cache|tmpfs|secret|ssh)` |
|
| 149 |
+ |
|
| 150 |
+See [`frontend/dockerfile/docs/experimental.md`](frontend/dockerfile/docs/experimental.md). |
|
| 151 |
+ |
|
| 148 | 152 |
### Exporters |
| 149 | 153 |
|
| 150 | 154 |
By default, the build result and intermediate cache will only remain internally in BuildKit. Exporter needs to be specified to retrieve the result. |
| ... | ... |
@@ -207,15 +220,22 @@ buildctl debug workers -v |
| 207 | 207 |
|
| 208 | 208 |
BuildKit can also be used by running the `buildkitd` daemon inside a Docker container and accessing it remotely. The client tool `buildctl` is also available for Mac and Windows. |
| 209 | 209 |
|
| 210 |
+We provide `buildkitd` container images as [`moby/buildkit`](https://hub.docker.com/r/moby/buildkit/tags/): |
|
| 211 |
+ |
|
| 212 |
+* `moby/buildkit:latest`: built from the latest regular [release](https://github.com/moby/buildkit/releases) |
|
| 213 |
+* `moby/buildkit:rootless`: same as `latest` but runs as an unprivileged user, see [`docs/rootless.md`](docs/rootless.md) |
|
| 214 |
+* `moby/buildkit:master`: built from the master branch |
|
| 215 |
+* `moby/buildkit:master-rootless`: same as master but runs as an unprivileged user, see [`docs/rootless.md`](docs/rootless.md) |
|
| 216 |
+ |
|
| 210 | 217 |
To run daemon in a container: |
| 211 | 218 |
|
| 212 | 219 |
``` |
| 213 |
-docker run -d --privileged -p 1234:1234 tonistiigi/buildkit --addr tcp://0.0.0.0:1234 |
|
| 220 |
+docker run -d --privileged -p 1234:1234 moby/buildkit:latest --addr tcp://0.0.0.0:1234 |
|
| 214 | 221 |
export BUILDKIT_HOST=tcp://0.0.0.0:1234 |
| 215 | 222 |
buildctl build --help |
| 216 | 223 |
``` |
| 217 | 224 |
|
| 218 |
-The `tonistiigi/buildkit` image can be built locally using the Dockerfile in `./hack/dockerfiles/test.Dockerfile`. |
|
| 225 |
+The images can be also built locally using `./hack/dockerfiles/test.Dockerfile` (or `./hack/dockerfiles/test.buildkit.Dockerfile` if you already have BuildKit). |
|
| 219 | 226 |
|
| 220 | 227 |
### Opentracing support |
| 221 | 228 |
|
| ... | ... |
@@ -232,7 +252,7 @@ export JAEGER_TRACE=0.0.0.0:6831 |
| 232 | 232 |
|
| 233 | 233 |
### Supported runc version |
| 234 | 234 |
|
| 235 |
-During development, BuildKit is tested with the version of runc that is being used by the containerd repository. Please refer to [runc.md](https://github.com/containerd/containerd/blob/v1.1.3/RUNC.md) for more information. |
|
| 235 |
+During development, BuildKit is tested with the version of runc that is being used by the containerd repository. Please refer to [runc.md](https://github.com/containerd/containerd/blob/v1.2.0-rc.1/RUNC.md) for more information. |
|
| 236 | 236 |
|
| 237 | 237 |
### Running BuildKit without root privileges |
| 238 | 238 |
|
| ... | ... |
@@ -628,7 +628,7 @@ func getFollowLinksWalk(root *iradix.Node, k []byte, linksWalked *int) ([]byte, |
| 628 | 628 |
if dirPath == "." || dirPath == "/" {
|
| 629 | 629 |
dirPath = "" |
| 630 | 630 |
} |
| 631 |
- link := parent.Linkname |
|
| 631 |
+ link := path.Clean(parent.Linkname) |
|
| 632 | 632 |
if !path.IsAbs(link) {
|
| 633 | 633 |
link = path.Join("/", path.Join(path.Dir(dirPath), link))
|
| 634 | 634 |
} |
| ... | ... |
@@ -126,30 +126,11 @@ func Image(ref string, opts ...ImageOption) State {
|
| 126 | 126 |
if err != nil {
|
| 127 | 127 |
src.err = err |
| 128 | 128 |
} else {
|
| 129 |
- var img struct {
|
|
| 130 |
- Config struct {
|
|
| 131 |
- Env []string `json:"Env,omitempty"` |
|
| 132 |
- WorkingDir string `json:"WorkingDir,omitempty"` |
|
| 133 |
- User string `json:"User,omitempty"` |
|
| 134 |
- } `json:"config,omitempty"` |
|
| 135 |
- } |
|
| 136 |
- if err := json.Unmarshal(dt, &img); err != nil {
|
|
| 137 |
- src.err = err |
|
| 138 |
- } else {
|
|
| 139 |
- st := NewState(src.Output()) |
|
| 140 |
- for _, env := range img.Config.Env {
|
|
| 141 |
- parts := strings.SplitN(env, "=", 2) |
|
| 142 |
- if len(parts[0]) > 0 {
|
|
| 143 |
- var v string |
|
| 144 |
- if len(parts) > 1 {
|
|
| 145 |
- v = parts[1] |
|
| 146 |
- } |
|
| 147 |
- st = st.AddEnv(parts[0], v) |
|
| 148 |
- } |
|
| 149 |
- } |
|
| 150 |
- st = st.Dir(img.Config.WorkingDir) |
|
| 129 |
+ st, err := NewState(src.Output()).WithImageConfig(dt) |
|
| 130 |
+ if err == nil {
|
|
| 151 | 131 |
return st |
| 152 | 132 |
} |
| 133 |
+ src.err = err |
|
| 153 | 134 |
} |
| 154 | 135 |
} |
| 155 | 136 |
return NewState(src.Output()) |
| ... | ... |
@@ -2,8 +2,10 @@ package llb |
| 2 | 2 |
|
| 3 | 3 |
import ( |
| 4 | 4 |
"context" |
| 5 |
+ "encoding/json" |
|
| 5 | 6 |
"fmt" |
| 6 | 7 |
"net" |
| 8 |
+ "strings" |
|
| 7 | 9 |
|
| 8 | 10 |
"github.com/containerd/containerd/platforms" |
| 9 | 11 |
"github.com/moby/buildkit/identity" |
| ... | ... |
@@ -171,6 +173,31 @@ func (s State) WithOutput(o Output) State {
|
| 171 | 171 |
return s |
| 172 | 172 |
} |
| 173 | 173 |
|
| 174 |
+func (s State) WithImageConfig(c []byte) (State, error) {
|
|
| 175 |
+ var img struct {
|
|
| 176 |
+ Config struct {
|
|
| 177 |
+ Env []string `json:"Env,omitempty"` |
|
| 178 |
+ WorkingDir string `json:"WorkingDir,omitempty"` |
|
| 179 |
+ User string `json:"User,omitempty"` |
|
| 180 |
+ } `json:"config,omitempty"` |
|
| 181 |
+ } |
|
| 182 |
+ if err := json.Unmarshal(c, &img); err != nil {
|
|
| 183 |
+ return State{}, err
|
|
| 184 |
+ } |
|
| 185 |
+ for _, env := range img.Config.Env {
|
|
| 186 |
+ parts := strings.SplitN(env, "=", 2) |
|
| 187 |
+ if len(parts[0]) > 0 {
|
|
| 188 |
+ var v string |
|
| 189 |
+ if len(parts) > 1 {
|
|
| 190 |
+ v = parts[1] |
|
| 191 |
+ } |
|
| 192 |
+ s = s.AddEnv(parts[0], v) |
|
| 193 |
+ } |
|
| 194 |
+ } |
|
| 195 |
+ s = s.Dir(img.Config.WorkingDir) |
|
| 196 |
+ return s, nil |
|
| 197 |
+} |
|
| 198 |
+ |
|
| 174 | 199 |
func (s State) Run(ro ...RunOption) ExecState {
|
| 175 | 200 |
ei := &ExecInfo{State: s}
|
| 176 | 201 |
if p := s.GetPlatform(); p != nil {
|
| ... | ... |
@@ -99,7 +99,7 @@ func Build(ctx context.Context, c client.Client) (*client.Result, error) {
|
| 99 | 99 |
name := "load build definition from " + filename |
| 100 | 100 |
|
| 101 | 101 |
src := llb.Local(LocalNameDockerfile, |
| 102 |
- llb.IncludePatterns([]string{filename}),
|
|
| 102 |
+ llb.FollowPaths([]string{filename}),
|
|
| 103 | 103 |
llb.SessionID(c.BuildOpts().SessionID), |
| 104 | 104 |
llb.SharedKeyHint(defaultDockerfileName), |
| 105 | 105 |
dockerfile2llb.WithInternalName(name), |
| ... | ... |
@@ -189,7 +189,7 @@ func Build(ctx context.Context, c client.Client) (*client.Result, error) {
|
| 189 | 189 |
if dockerignoreState == nil {
|
| 190 | 190 |
st := llb.Local(LocalNameContext, |
| 191 | 191 |
llb.SessionID(c.BuildOpts().SessionID), |
| 192 |
- llb.IncludePatterns([]string{dockerignoreFilename}),
|
|
| 192 |
+ llb.FollowPaths([]string{dockerignoreFilename}),
|
|
| 193 | 193 |
llb.SharedKeyHint(dockerignoreFilename), |
| 194 | 194 |
dockerfile2llb.WithInternalName("load "+dockerignoreFilename),
|
| 195 | 195 |
) |
| ... | ... |
@@ -35,7 +35,7 @@ const ( |
| 35 | 35 |
localNameContext = "context" |
| 36 | 36 |
historyComment = "buildkit.dockerfile.v0" |
| 37 | 37 |
|
| 38 |
- DefaultCopyImage = "tonistiigi/copy:v0.1.7@sha256:9aab7d9ab369c6daf4831bf0653f7592110ab4b7e8a33fee2b9dca546e9d3089" |
|
| 38 |
+ DefaultCopyImage = "docker/dockerfile-copy:v0.1.9@sha256:e8f159d3f00786604b93c675ee2783f8dc194bb565e61ca5788f6a6e9d304061" |
|
| 39 | 39 |
) |
| 40 | 40 |
|
| 41 | 41 |
type ConvertOpt struct {
|
| ... | ... |
@@ -327,6 +327,7 @@ func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State, |
| 327 | 327 |
targetPlatform: platformOpt.targetPlatform, |
| 328 | 328 |
extraHosts: opt.ExtraHosts, |
| 329 | 329 |
copyImage: opt.OverrideCopyImage, |
| 330 |
+ llbCaps: opt.LLBCaps, |
|
| 330 | 331 |
} |
| 331 | 332 |
if opt.copyImage == "" {
|
| 332 | 333 |
opt.copyImage = DefaultCopyImage |
| ... | ... |
@@ -441,6 +442,7 @@ type dispatchOpt struct {
|
| 441 | 441 |
buildPlatforms []specs.Platform |
| 442 | 442 |
extraHosts []llb.HostIP |
| 443 | 443 |
copyImage string |
| 444 |
+ llbCaps *apicaps.CapSet |
|
| 444 | 445 |
} |
| 445 | 446 |
|
| 446 | 447 |
func dispatch(d *dispatchState, cmd command, opt dispatchOpt) error {
|
| ... | ... |
@@ -467,7 +469,9 @@ func dispatch(d *dispatchState, cmd command, opt dispatchOpt) error {
|
| 467 | 467 |
err = dispatchCopy(d, c.SourcesAndDest, opt.buildContext, true, c, "", opt) |
| 468 | 468 |
if err == nil {
|
| 469 | 469 |
for _, src := range c.Sources() {
|
| 470 |
- d.ctxPaths[path.Join("/", filepath.ToSlash(src))] = struct{}{}
|
|
| 470 |
+ if !strings.HasPrefix(src, "http://") && !strings.HasPrefix(src, "https://") {
|
|
| 471 |
+ d.ctxPaths[path.Join("/", filepath.ToSlash(src))] = struct{}{}
|
|
| 472 |
+ } |
|
| 471 | 473 |
} |
| 472 | 474 |
} |
| 473 | 475 |
case *instructions.LabelCommand: |
| ... | ... |
@@ -627,7 +631,12 @@ func dispatchRun(d *dispatchState, c *instructions.RunCommand, proxy *llb.ProxyE |
| 627 | 627 |
return err |
| 628 | 628 |
} |
| 629 | 629 |
opt = append(opt, runMounts...) |
| 630 |
- opt = append(opt, llb.WithCustomName(prefixCommand(d, uppercaseCmd(processCmdEnv(dopt.shlex, c.String(), env)), d.prefixPlatform, d.state.GetPlatform()))) |
|
| 630 |
+ |
|
| 631 |
+ shlex := *dopt.shlex |
|
| 632 |
+ shlex.RawQuotes = true |
|
| 633 |
+ shlex.SkipUnsetEnv = true |
|
| 634 |
+ |
|
| 635 |
+ opt = append(opt, llb.WithCustomName(prefixCommand(d, uppercaseCmd(processCmdEnv(&shlex, c.String(), env)), d.prefixPlatform, d.state.GetPlatform()))) |
|
| 631 | 636 |
for _, h := range dopt.extraHosts {
|
| 632 | 637 |
opt = append(opt, llb.AddExtraHost(h.Host, h.IP)) |
| 633 | 638 |
} |
| ... | ... |
@@ -729,6 +738,13 @@ func dispatchCopy(d *dispatchState, c instructions.SourcesAndDest, sourceState l |
| 729 | 729 |
if d.ignoreCache {
|
| 730 | 730 |
runOpt = append(runOpt, llb.IgnoreCache) |
| 731 | 731 |
} |
| 732 |
+ |
|
| 733 |
+ if opt.llbCaps != nil {
|
|
| 734 |
+ if err := opt.llbCaps.Supports(pb.CapExecMetaNetwork); err == nil {
|
|
| 735 |
+ runOpt = append(runOpt, llb.Network(llb.NetModeNone)) |
|
| 736 |
+ } |
|
| 737 |
+ } |
|
| 738 |
+ |
|
| 732 | 739 |
run := img.Run(append(runOpt, mounts...)...) |
| 733 | 740 |
d.state = run.AddMount("/dest", d.state).Platform(platform)
|
| 734 | 741 |
|
| ... | ... |
@@ -2,6 +2,7 @@ package shell |
| 2 | 2 |
|
| 3 | 3 |
import ( |
| 4 | 4 |
"bytes" |
| 5 |
+ "fmt" |
|
| 5 | 6 |
"strings" |
| 6 | 7 |
"text/scanner" |
| 7 | 8 |
"unicode" |
| ... | ... |
@@ -17,7 +18,9 @@ import ( |
| 17 | 17 |
// It doesn't support all flavors of ${xx:...} formats but new ones can
|
| 18 | 18 |
// be added by adding code to the "special ${} format processing" section
|
| 19 | 19 |
type Lex struct {
|
| 20 |
- escapeToken rune |
|
| 20 |
+ escapeToken rune |
|
| 21 |
+ RawQuotes bool |
|
| 22 |
+ SkipUnsetEnv bool |
|
| 21 | 23 |
} |
| 22 | 24 |
|
| 23 | 25 |
// NewLex creates a new Lex which uses escapeToken to escape quotes. |
| ... | ... |
@@ -58,17 +61,21 @@ func (s *Lex) ProcessWordsWithMap(word string, env map[string]string) ([]string, |
| 58 | 58 |
|
| 59 | 59 |
func (s *Lex) process(word string, env map[string]string) (string, []string, error) {
|
| 60 | 60 |
sw := &shellWord{
|
| 61 |
- envs: env, |
|
| 62 |
- escapeToken: s.escapeToken, |
|
| 61 |
+ envs: env, |
|
| 62 |
+ escapeToken: s.escapeToken, |
|
| 63 |
+ skipUnsetEnv: s.SkipUnsetEnv, |
|
| 64 |
+ rawQuotes: s.RawQuotes, |
|
| 63 | 65 |
} |
| 64 | 66 |
sw.scanner.Init(strings.NewReader(word)) |
| 65 | 67 |
return sw.process(word) |
| 66 | 68 |
} |
| 67 | 69 |
|
| 68 | 70 |
type shellWord struct {
|
| 69 |
- scanner scanner.Scanner |
|
| 70 |
- envs map[string]string |
|
| 71 |
- escapeToken rune |
|
| 71 |
+ scanner scanner.Scanner |
|
| 72 |
+ envs map[string]string |
|
| 73 |
+ escapeToken rune |
|
| 74 |
+ rawQuotes bool |
|
| 75 |
+ skipUnsetEnv bool |
|
| 72 | 76 |
} |
| 73 | 77 |
|
| 74 | 78 |
func (sw *shellWord) process(source string) (string, []string, error) {
|
| ... | ... |
@@ -103,10 +110,8 @@ func (w *wordsStruct) addRawChar(ch rune) {
|
| 103 | 103 |
} |
| 104 | 104 |
|
| 105 | 105 |
func (w *wordsStruct) addString(str string) {
|
| 106 |
- var scan scanner.Scanner |
|
| 107 |
- scan.Init(strings.NewReader(str)) |
|
| 108 |
- for scan.Peek() != scanner.EOF {
|
|
| 109 |
- w.addChar(scan.Next()) |
|
| 106 |
+ for _, ch := range str {
|
|
| 107 |
+ w.addChar(ch) |
|
| 110 | 108 |
} |
| 111 | 109 |
} |
| 112 | 110 |
|
| ... | ... |
@@ -196,14 +201,20 @@ func (sw *shellWord) processSingleQuote() (string, error) {
|
| 196 | 196 |
|
| 197 | 197 |
var result bytes.Buffer |
| 198 | 198 |
|
| 199 |
- sw.scanner.Next() |
|
| 199 |
+ ch := sw.scanner.Next() |
|
| 200 |
+ if sw.rawQuotes {
|
|
| 201 |
+ result.WriteRune(ch) |
|
| 202 |
+ } |
|
| 200 | 203 |
|
| 201 | 204 |
for {
|
| 202 |
- ch := sw.scanner.Next() |
|
| 205 |
+ ch = sw.scanner.Next() |
|
| 203 | 206 |
switch ch {
|
| 204 | 207 |
case scanner.EOF: |
| 205 | 208 |
return "", errors.New("unexpected end of statement while looking for matching single-quote")
|
| 206 | 209 |
case '\'': |
| 210 |
+ if sw.rawQuotes {
|
|
| 211 |
+ result.WriteRune(ch) |
|
| 212 |
+ } |
|
| 207 | 213 |
return result.String(), nil |
| 208 | 214 |
} |
| 209 | 215 |
result.WriteRune(ch) |
| ... | ... |
@@ -225,14 +236,20 @@ func (sw *shellWord) processDoubleQuote() (string, error) {
|
| 225 | 225 |
|
| 226 | 226 |
var result bytes.Buffer |
| 227 | 227 |
|
| 228 |
- sw.scanner.Next() |
|
| 228 |
+ ch := sw.scanner.Next() |
|
| 229 |
+ if sw.rawQuotes {
|
|
| 230 |
+ result.WriteRune(ch) |
|
| 231 |
+ } |
|
| 229 | 232 |
|
| 230 | 233 |
for {
|
| 231 | 234 |
switch sw.scanner.Peek() {
|
| 232 | 235 |
case scanner.EOF: |
| 233 | 236 |
return "", errors.New("unexpected end of statement while looking for matching double-quote")
|
| 234 | 237 |
case '"': |
| 235 |
- sw.scanner.Next() |
|
| 238 |
+ ch := sw.scanner.Next() |
|
| 239 |
+ if sw.rawQuotes {
|
|
| 240 |
+ result.WriteRune(ch) |
|
| 241 |
+ } |
|
| 236 | 242 |
return result.String(), nil |
| 237 | 243 |
case '$': |
| 238 | 244 |
value, err := sw.processDollar() |
| ... | ... |
@@ -269,7 +286,11 @@ func (sw *shellWord) processDollar() (string, error) {
|
| 269 | 269 |
if name == "" {
|
| 270 | 270 |
return "$", nil |
| 271 | 271 |
} |
| 272 |
- return sw.getEnv(name), nil |
|
| 272 |
+ value, found := sw.getEnv(name) |
|
| 273 |
+ if !found && sw.skipUnsetEnv {
|
|
| 274 |
+ return "$" + name, nil |
|
| 275 |
+ } |
|
| 276 |
+ return value, nil |
|
| 273 | 277 |
} |
| 274 | 278 |
|
| 275 | 279 |
sw.scanner.Next() |
| ... | ... |
@@ -285,7 +306,11 @@ func (sw *shellWord) processDollar() (string, error) {
|
| 285 | 285 |
switch ch {
|
| 286 | 286 |
case '}': |
| 287 | 287 |
// Normal ${xx} case
|
| 288 |
- return sw.getEnv(name), nil |
|
| 288 |
+ value, found := sw.getEnv(name) |
|
| 289 |
+ if !found && sw.skipUnsetEnv {
|
|
| 290 |
+ return fmt.Sprintf("${%s}", name), nil
|
|
| 291 |
+ } |
|
| 292 |
+ return value, nil |
|
| 289 | 293 |
case ':': |
| 290 | 294 |
// Special ${xx:...} format processing
|
| 291 | 295 |
// Yes it allows for recursive $'s in the ... spot |
| ... | ... |
@@ -301,19 +326,26 @@ func (sw *shellWord) processDollar() (string, error) {
|
| 301 | 301 |
|
| 302 | 302 |
// Grab the current value of the variable in question so we |
| 303 | 303 |
// can use to to determine what to do based on the modifier |
| 304 |
- newValue := sw.getEnv(name) |
|
| 304 |
+ newValue, found := sw.getEnv(name) |
|
| 305 | 305 |
|
| 306 | 306 |
switch modifier {
|
| 307 | 307 |
case '+': |
| 308 | 308 |
if newValue != "" {
|
| 309 | 309 |
newValue = word |
| 310 | 310 |
} |
| 311 |
+ if !found && sw.skipUnsetEnv {
|
|
| 312 |
+ return fmt.Sprintf("${%s:%s%s}", name, string(modifier), word), nil
|
|
| 313 |
+ } |
|
| 311 | 314 |
return newValue, nil |
| 312 | 315 |
|
| 313 | 316 |
case '-': |
| 314 | 317 |
if newValue == "" {
|
| 315 | 318 |
newValue = word |
| 316 | 319 |
} |
| 320 |
+ if !found && sw.skipUnsetEnv {
|
|
| 321 |
+ return fmt.Sprintf("${%s:%s%s}", name, string(modifier), word), nil
|
|
| 322 |
+ } |
|
| 323 |
+ |
|
| 317 | 324 |
return newValue, nil |
| 318 | 325 |
|
| 319 | 326 |
default: |
| ... | ... |
@@ -364,13 +396,13 @@ func isSpecialParam(char rune) bool {
|
| 364 | 364 |
return false |
| 365 | 365 |
} |
| 366 | 366 |
|
| 367 |
-func (sw *shellWord) getEnv(name string) string {
|
|
| 367 |
+func (sw *shellWord) getEnv(name string) (string, bool) {
|
|
| 368 | 368 |
for key, value := range sw.envs {
|
| 369 | 369 |
if EqualEnvKeys(name, key) {
|
| 370 |
- return value |
|
| 370 |
+ return value, true |
|
| 371 | 371 |
} |
| 372 | 372 |
} |
| 373 |
- return "" |
|
| 373 |
+ return "", false |
|
| 374 | 374 |
} |
| 375 | 375 |
|
| 376 | 376 |
func BuildEnvs(env []string) map[string]string {
|
| ... | ... |
@@ -24,7 +24,7 @@ const ( |
| 24 | 24 |
// Dialer returns a connection that can be used by the session |
| 25 | 25 |
type Dialer func(ctx context.Context, proto string, meta map[string][]string) (net.Conn, error) |
| 26 | 26 |
|
| 27 |
-// Attachable defines a feature that can be expsed on a session |
|
| 27 |
+// Attachable defines a feature that can be exposed on a session |
|
| 28 | 28 |
type Attachable interface {
|
| 29 | 29 |
Register(*grpc.Server) |
| 30 | 30 |
} |
| ... | ... |
@@ -66,7 +66,7 @@ func NewSession(ctx context.Context, name, sharedKey string) (*Session, error) {
|
| 66 | 66 |
return s, nil |
| 67 | 67 |
} |
| 68 | 68 |
|
| 69 |
-// Allow enable a given service to be reachable through the grpc session |
|
| 69 |
+// Allow enables a given service to be reachable through the grpc session |
|
| 70 | 70 |
func (s *Session) Allow(a Attachable) {
|
| 71 | 71 |
a.Register(s.grpcServer) |
| 72 | 72 |
} |
| ... | ... |
@@ -171,6 +171,7 @@ func (sb *subBuilder) Build(ctx context.Context, e Edge) (CachedResult, error) {
|
| 171 | 171 |
} |
| 172 | 172 |
|
| 173 | 173 |
func (sb *subBuilder) Context(ctx context.Context) context.Context {
|
| 174 |
+ ctx = session.NewContext(ctx, sb.state.getSessionID()) |
|
| 174 | 175 |
return opentracing.ContextWithSpan(progress.WithProgress(ctx, sb.mpw), sb.mspan) |
| 175 | 176 |
} |
| 176 | 177 |
|
| ... | ... |
@@ -475,6 +476,7 @@ func (j *Job) Discard() error {
|
| 475 | 475 |
} |
| 476 | 476 |
|
| 477 | 477 |
func (j *Job) Context(ctx context.Context) context.Context {
|
| 478 |
+ ctx = session.NewContext(ctx, j.SessionID) |
|
| 478 | 479 |
return progress.WithProgress(ctx, j.pw) |
| 479 | 480 |
} |
| 480 | 481 |
|
| ... | ... |
@@ -1,113 +1,40 @@ |
| 1 | 1 |
package specconv |
| 2 | 2 |
|
| 3 | 3 |
import ( |
| 4 |
- "os" |
|
| 5 |
- "sort" |
|
| 6 | 4 |
"strings" |
| 7 | 5 |
|
| 8 |
- "github.com/opencontainers/runc/libcontainer/system" |
|
| 9 |
- "github.com/opencontainers/runc/libcontainer/user" |
|
| 10 | 6 |
"github.com/opencontainers/runtime-spec/specs-go" |
| 11 |
- "github.com/pkg/errors" |
|
| 12 | 7 |
) |
| 13 | 8 |
|
| 14 | 9 |
// ToRootless converts spec to be compatible with "rootless" runc. |
| 15 |
-// * Adds userns (Note: since we are already in userns, ideally we should not need to do this. runc-side issue is tracked at https://github.com/opencontainers/runc/issues/1837) |
|
| 16 |
-// * Fix up mount flags (same as above) |
|
| 17 |
-// * Replace /sys with bind-mount (FIXME: we don't need to do this if netns is unshared) |
|
| 10 |
+// * Remove /sys mount |
|
| 11 |
+// * Remove cgroups |
|
| 12 |
+// |
|
| 13 |
+// See docs/rootless.md for the supported runc revision. |
|
| 18 | 14 |
func ToRootless(spec *specs.Spec) error {
|
| 19 |
- if !system.RunningInUserNS() {
|
|
| 20 |
- return errors.New("needs to be in user namespace")
|
|
| 21 |
- } |
|
| 22 |
- uidMap, err := user.CurrentProcessUIDMap() |
|
| 23 |
- if err != nil && !os.IsNotExist(err) {
|
|
| 24 |
- return err |
|
| 25 |
- } |
|
| 26 |
- gidMap, err := user.CurrentProcessUIDMap() |
|
| 27 |
- if err != nil && !os.IsNotExist(err) {
|
|
| 28 |
- return err |
|
| 29 |
- } |
|
| 30 |
- return toRootless(spec, uidMap, gidMap) |
|
| 31 |
-} |
|
| 32 |
- |
|
| 33 |
-// toRootless was forked from github.com/opencontainers/runc/libcontainer/specconv |
|
| 34 |
-func toRootless(spec *specs.Spec, uidMap, gidMap []user.IDMap) error {
|
|
| 35 |
- if err := configureUserNS(spec, uidMap, gidMap); err != nil {
|
|
| 36 |
- return err |
|
| 37 |
- } |
|
| 38 |
- if err := configureMounts(spec); err != nil {
|
|
| 39 |
- return err |
|
| 40 |
- } |
|
| 41 |
- |
|
| 42 |
- // Remove cgroup settings. |
|
| 43 |
- spec.Linux.Resources = nil |
|
| 44 |
- spec.Linux.CgroupsPath = "" |
|
| 45 |
- return nil |
|
| 46 |
-} |
|
| 47 |
- |
|
| 48 |
-// configureUserNS add suserns and the current ID map to the spec. |
|
| 49 |
-// Since we are already in userns, ideally we should not need to add userns. |
|
| 50 |
-// However, currently rootless runc always requires userns to be added. |
|
| 51 |
-// https://github.com/opencontainers/runc/issues/1837 |
|
| 52 |
-func configureUserNS(spec *specs.Spec, uidMap, gidMap []user.IDMap) error {
|
|
| 53 |
- spec.Linux.Namespaces = append(spec.Linux.Namespaces, specs.LinuxNamespace{
|
|
| 54 |
- Type: specs.UserNamespace, |
|
| 55 |
- }) |
|
| 56 |
- |
|
| 57 |
- sort.Slice(uidMap, func(i, j int) bool { return uidMap[i].ID < uidMap[j].ID })
|
|
| 58 |
- uNextContainerID := int64(0) |
|
| 59 |
- for _, u := range uidMap {
|
|
| 60 |
- spec.Linux.UIDMappings = append(spec.Linux.UIDMappings, |
|
| 61 |
- specs.LinuxIDMapping{
|
|
| 62 |
- HostID: uint32(u.ID), |
|
| 63 |
- ContainerID: uint32(uNextContainerID), |
|
| 64 |
- Size: uint32(u.Count), |
|
| 65 |
- }) |
|
| 66 |
- uNextContainerID += int64(u.Count) |
|
| 67 |
- } |
|
| 68 |
- sort.Slice(gidMap, func(i, j int) bool { return gidMap[i].ID < gidMap[j].ID })
|
|
| 69 |
- gNextContainerID := int64(0) |
|
| 70 |
- for _, g := range gidMap {
|
|
| 71 |
- spec.Linux.GIDMappings = append(spec.Linux.GIDMappings, |
|
| 72 |
- specs.LinuxIDMapping{
|
|
| 73 |
- HostID: uint32(g.ID), |
|
| 74 |
- ContainerID: uint32(gNextContainerID), |
|
| 75 |
- Size: uint32(g.Count), |
|
| 76 |
- }) |
|
| 77 |
- gNextContainerID += int64(g.Count) |
|
| 78 |
- } |
|
| 79 |
- return nil |
|
| 80 |
-} |
|
| 81 |
- |
|
| 82 |
-func configureMounts(spec *specs.Spec) error {
|
|
| 15 |
+ // Remove /sys mount because we can't mount /sys when the daemon netns |
|
| 16 |
+ // is not unshared from the host. |
|
| 17 |
+ // |
|
| 18 |
+ // Instead, we could bind-mount /sys from the host, however, `rbind, ro` |
|
| 19 |
+ // does not make /sys/fs/cgroup read-only (and we can't bind-mount /sys |
|
| 20 |
+ // without rbind) |
|
| 21 |
+ // |
|
| 22 |
+ // PR for making /sys/fs/cgroup read-only is proposed, but it is very |
|
| 23 |
+ // complicated: https://github.com/opencontainers/runc/pull/1869 |
|
| 24 |
+ // |
|
| 25 |
+ // For buildkit usecase, we suppose we don't need to provide /sys to |
|
| 26 |
+ // containers and remove /sys mount as a workaround. |
|
| 83 | 27 |
var mounts []specs.Mount |
| 84 | 28 |
for _, mount := range spec.Mounts {
|
| 85 |
- // Ignore all mounts that are under /sys, because we add /sys later. |
|
| 86 | 29 |
if strings.HasPrefix(mount.Destination, "/sys") {
|
| 87 | 30 |
continue |
| 88 | 31 |
} |
| 89 |
- |
|
| 90 |
- // Remove all gid= and uid= mappings. |
|
| 91 |
- // Since we are already in userns, ideally we should not need to do this. |
|
| 92 |
- // https://github.com/opencontainers/runc/issues/1837 |
|
| 93 |
- var options []string |
|
| 94 |
- for _, option := range mount.Options {
|
|
| 95 |
- if !strings.HasPrefix(option, "gid=") && !strings.HasPrefix(option, "uid=") {
|
|
| 96 |
- options = append(options, option) |
|
| 97 |
- } |
|
| 98 |
- } |
|
| 99 |
- mount.Options = options |
|
| 100 | 32 |
mounts = append(mounts, mount) |
| 101 | 33 |
} |
| 102 |
- |
|
| 103 |
- // Add the sysfs mount as an rbind, because we can't mount /sys unless we have netns. |
|
| 104 |
- // TODO: keep original /sys mount when we have netns. |
|
| 105 |
- mounts = append(mounts, specs.Mount{
|
|
| 106 |
- Source: "/sys", |
|
| 107 |
- Destination: "/sys", |
|
| 108 |
- Type: "none", |
|
| 109 |
- Options: []string{"rbind", "nosuid", "noexec", "nodev", "ro"},
|
|
| 110 |
- }) |
|
| 111 | 34 |
spec.Mounts = mounts |
| 35 |
+ |
|
| 36 |
+ // Remove cgroups so as to avoid `container_linux.go:337: starting container process caused "process_linux.go:280: applying cgroup configuration for process caused \"mkdir /sys/fs/cgroup/cpuset/buildkit: permission denied\""` |
|
| 37 |
+ spec.Linux.Resources = nil |
|
| 38 |
+ spec.Linux.CgroupsPath = "" |
|
| 112 | 39 |
return nil |
| 113 | 40 |
} |
| ... | ... |
@@ -6,7 +6,7 @@ github.com/davecgh/go-spew v1.1.0 |
| 6 | 6 |
github.com/pmezard/go-difflib v1.0.0 |
| 7 | 7 |
golang.org/x/sys 1b2967e3c290b7c545b3db0deeda16e9be4f98a2 |
| 8 | 8 |
|
| 9 |
-github.com/containerd/containerd d97a907f7f781c0ab8340877d8e6b53cc7f1c2f6 |
|
| 9 |
+github.com/containerd/containerd 1a5f9a3434ac53c0e9d27093ecc588e0c281c333 |
|
| 10 | 10 |
github.com/containerd/typeurl a93fcdb778cd272c6e9b3028b2f42d813e785d40 |
| 11 | 11 |
golang.org/x/sync 450f422ab23cf9881c94e2db30cac0eb1b7cf80c |
| 12 | 12 |
github.com/sirupsen/logrus v1.0.0 |
| ... | ... |
@@ -16,9 +16,9 @@ golang.org/x/net 0ed95abb35c445290478a5348a7b38bb154135fd |
| 16 | 16 |
github.com/gogo/protobuf v1.0.0 |
| 17 | 17 |
github.com/gogo/googleapis b23578765ee54ff6bceff57f397d833bf4ca6869 |
| 18 | 18 |
github.com/golang/protobuf v1.1.0 |
| 19 |
-github.com/containerd/continuity f44b615e492bdfb371aae2f76ec694d9da1db537 |
|
| 19 |
+github.com/containerd/continuity bd77b46c8352f74eb12c85bdc01f4b90f69d66b4 |
|
| 20 | 20 |
github.com/opencontainers/image-spec v1.0.1 |
| 21 |
-github.com/opencontainers/runc 20aff4f0488c6d4b8df4d85b4f63f1f704c11abd |
|
| 21 |
+github.com/opencontainers/runc a00bf0190895aa465a5fbed0268888e2c8ddfe85 |
|
| 22 | 22 |
github.com/Microsoft/go-winio v0.4.11 |
| 23 | 23 |
github.com/containerd/fifo 3d5202aec260678c48179c56f40e6f38a095738c |
| 24 | 24 |
github.com/opencontainers/runtime-spec eba862dc2470385a233c7507392675cbeadf7353 # v1.0.1-45-geba862d |
| ... | ... |
@@ -28,8 +28,9 @@ google.golang.org/genproto d80a6e20e776b0b17a324d0ba1ab50a39c8e8944 |
| 28 | 28 |
golang.org/x/text 19e51611da83d6be54ddafce4a4af510cb3e9ea4 |
| 29 | 29 |
github.com/docker/go-events 9461782956ad83b30282bf90e31fa6a70c255ba9 |
| 30 | 30 |
github.com/syndtr/gocapability db04d3cc01c8b54962a58ec7e491717d06cfcc16 |
| 31 |
-github.com/Microsoft/hcsshim v0.7.3 |
|
| 31 |
+github.com/Microsoft/hcsshim v0.7.9 |
|
| 32 | 32 |
golang.org/x/crypto 0709b304e793a5edb4a2c0145f281ecdc20838a4 |
| 33 |
+github.com/containerd/cri 8506fe836677cc3bb23a16b68145128243d843b5 # release/1.2 branch |
|
| 33 | 34 |
|
| 34 | 35 |
github.com/urfave/cli 7bc6a0acffa589f415f88aca16cc1de5ffd66f9c |
| 35 | 36 |
github.com/morikuni/aec 39771216ff4c63d11f5e604076f9c45e8be1067b |
| ... | ... |
@@ -40,8 +41,8 @@ golang.org/x/time f51c12702a4d776e4c1fa9b0fabab841babae631 |
| 40 | 40 |
github.com/docker/docker 71cd53e4a197b303c6ba086bd584ffd67a884281 |
| 41 | 41 |
github.com/pkg/profile 5b67d428864e92711fcbd2f8629456121a56d91f |
| 42 | 42 |
|
| 43 |
-github.com/tonistiigi/fsutil f567071bed2416e4d87d260d3162722651182317 |
|
| 44 |
-github.com/hashicorp/go-immutable-radix 826af9ccf0feeee615d546d69b11f8e98da8c8f1 git://github.com/tonistiigi/go-immutable-radix.git |
|
| 43 |
+github.com/tonistiigi/fsutil 2862f6bc5ac9b97124e552a5c108230b38a1b0ca |
|
| 44 |
+github.com/hashicorp/go-immutable-radix 826af9ccf0feeee615d546d69b11f8e98da8c8f1 https://github.com/tonistiigi/go-immutable-radix |
|
| 45 | 45 |
github.com/hashicorp/golang-lru a0d98a5f288019575c6d1f4bb1573fef2d1fcdc4 |
| 46 | 46 |
github.com/mitchellh/hashstructure 2bca23e0e452137f789efbc8610126fd8b94f73b |
| 47 | 47 |
github.com/docker/go-connections 3ede32e2033de7505e6500d6c868c2b9ed9f169d |
| ... | ... |
@@ -66,6 +67,3 @@ github.com/opentracing-contrib/go-stdlib b1a47cfbdd7543e70e9ef3e73d0802ad306cc1c |
| 66 | 66 |
# used by dockerfile tests |
| 67 | 67 |
gotest.tools v2.1.0 |
| 68 | 68 |
github.com/google/go-cmp v0.2.0 |
| 69 |
- |
|
| 70 |
-# used by rootless spec conv test |
|
| 71 |
-github.com/seccomp/libseccomp-golang 32f571b70023028bd57d9288c20efbcb237f3ce0 |
| ... | ... |
@@ -5,6 +5,7 @@ import ( |
| 5 | 5 |
"os" |
| 6 | 6 |
"path/filepath" |
| 7 | 7 |
"strings" |
| 8 |
+ "syscall" |
|
| 8 | 9 |
"time" |
| 9 | 10 |
|
| 10 | 11 |
"github.com/docker/docker/pkg/fileutils" |
| ... | ... |
@@ -71,7 +72,7 @@ func Walk(ctx context.Context, p string, opt *WalkOpt, fn filepath.WalkFunc) err |
| 71 | 71 |
return err |
| 72 | 72 |
} |
| 73 | 73 |
defer func() {
|
| 74 |
- if retErr != nil && os.IsNotExist(errors.Cause(retErr)) {
|
|
| 74 |
+ if retErr != nil && isNotExist(retErr) {
|
|
| 75 | 75 |
retErr = filepath.SkipDir |
| 76 | 76 |
} |
| 77 | 77 |
}() |
| ... | ... |
@@ -216,3 +217,14 @@ func trimUntilIndex(str, sep string, count int) string {
|
| 216 | 216 |
} |
| 217 | 217 |
} |
| 218 | 218 |
} |
| 219 |
+ |
|
| 220 |
+func isNotExist(err error) bool {
|
|
| 221 |
+ err = errors.Cause(err) |
|
| 222 |
+ if os.IsNotExist(err) {
|
|
| 223 |
+ return true |
|
| 224 |
+ } |
|
| 225 |
+ if pe, ok := err.(*os.PathError); ok {
|
|
| 226 |
+ err = pe.Err |
|
| 227 |
+ } |
|
| 228 |
+ return err == syscall.ENOTDIR |
|
| 229 |
+} |