Browse code

vendor: update buildkit to v0.3.3

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

Tonis Tiigi authored on 2018/11/22 04:02:56
Showing 23 changed files
... ...
@@ -26,8 +26,8 @@ github.com/imdario/mergo v0.3.6
26 26
 golang.org/x/sync 1d60e4601c6fd243af51cc01ddf169918a5407ca
27 27
 
28 28
 # buildkit
29
-github.com/moby/buildkit c7bb575343df0cbfeab8b5b28149630b8153fcc6
30
-github.com/tonistiigi/fsutil f567071bed2416e4d87d260d3162722651182317
29
+github.com/moby/buildkit 8cf9bec86a7f11fe6591804aee152c8e8a7a8a0d # v0.3.3
30
+github.com/tonistiigi/fsutil 2862f6bc5ac9b97124e552a5c108230b38a1b0ca
31 31
 github.com/grpc-ecosystem/grpc-opentracing 8e809c8a86450a29b90dcc9efbf062d0fe6d9746
32 32
 github.com/opentracing/opentracing-go 1361b9cd60be79c4c3a7fa9841b3c132e40066a7
33 33
 github.com/google/shlex 6f45313302b9c56850fc17f99e40caebce98c716
... ...
@@ -27,9 +27,11 @@ Read the proposal from https://github.com/moby/moby/issues/32925
27 27
 
28 28
 Introductory blog post https://blog.mobyproject.org/introducing-buildkit-17e056cc5317
29 29
 
30
+:information_source: If you are visiting this repo for the usage of experimental Dockerfile features like `RUN --mount=type=(bind|cache|tmpfs|secret|ssh)`, please refer to [`frontend/dockerfile/docs/experimental.md`](frontend/dockerfile/docs/experimental.md).
31
+
30 32
 ### Used by
31 33
 
32
-[Moby](https://github.com/moby/moby/pull/37151)
34
+[Moby & Docker](https://github.com/moby/moby/pull/37151)
33 35
 
34 36
 [img](https://github.com/genuinetools/img)
35 37
 
... ...
@@ -37,6 +39,12 @@ Introductory blog post https://blog.mobyproject.org/introducing-buildkit-17e056c
37 37
 
38 38
 [container build interface](https://github.com/containerbuilding/cbi)
39 39
 
40
+[Knative Build Templates](https://github.com/knative/build-templates)
41
+
42
+[boss](https://github.com/crosbymichael/boss)
43
+
44
+[Rio](https://github.com/rancher/rio) (on roadmap)
45
+
40 46
 ### Quick start
41 47
 
42 48
 Dependencies:
... ...
@@ -79,6 +87,7 @@ See [`solver/pb/ops.proto`](./solver/pb/ops.proto) for the format definition.
79 79
 Currently, following high-level languages has been implemented for LLB:
80 80
 
81 81
 - Dockerfile (See [Exploring Dockerfiles](#exploring-dockerfiles))
82
+- [Buildpacks](https://github.com/tonistiigi/buildkit-pack)
82 83
 - (open a PR to add your own language)
83 84
 
84 85
 For understanding the basics of LLB, `examples/buildkit*` directory contains scripts that define how to build different configurations of BuildKit itself and its dependencies using the `client` package. Running one of these scripts generates a protobuf definition of a build graph. Note that the script itself does not execute any steps of the build.
... ...
@@ -145,6 +154,10 @@ buildctl build --frontend=gateway.v0 --frontend-opt=source=tonistiigi/dockerfile
145 145
 buildctl build --frontend gateway.v0 --frontend-opt=source=tonistiigi/dockerfile --frontend-opt=context=git://github.com/moby/moby --frontend-opt build-arg:APT_MIRROR=cdn-fastly.deb.debian.org
146 146
 ````
147 147
 
148
+##### Building a Dockerfile with experimental features like `RUN --mount=type=(bind|cache|tmpfs|secret|ssh)`
149
+
150
+See [`frontend/dockerfile/docs/experimental.md`](frontend/dockerfile/docs/experimental.md).
151
+
148 152
 ### Exporters
149 153
 
150 154
 By default, the build result and intermediate cache will only remain internally in BuildKit. Exporter needs to be specified to retrieve the result.
... ...
@@ -207,15 +220,22 @@ buildctl debug workers -v
207 207
 
208 208
 BuildKit can also be used by running the `buildkitd` daemon inside a Docker container and accessing it remotely. The client tool `buildctl` is also available for Mac and Windows.
209 209
 
210
+We provide `buildkitd` container images as [`moby/buildkit`](https://hub.docker.com/r/moby/buildkit/tags/):
211
+
212
+* `moby/buildkit:latest`: built from the latest regular [release](https://github.com/moby/buildkit/releases)
213
+* `moby/buildkit:rootless`: same as `latest` but runs as an unprivileged user, see [`docs/rootless.md`](docs/rootless.md)
214
+* `moby/buildkit:master`: built from the master branch
215
+* `moby/buildkit:master-rootless`: same as master but runs as an unprivileged user, see [`docs/rootless.md`](docs/rootless.md)
216
+
210 217
 To run daemon in a container:
211 218
 
212 219
 ```
213
-docker run -d --privileged -p 1234:1234 tonistiigi/buildkit --addr tcp://0.0.0.0:1234
220
+docker run -d --privileged -p 1234:1234 moby/buildkit:latest --addr tcp://0.0.0.0:1234
214 221
 export BUILDKIT_HOST=tcp://0.0.0.0:1234
215 222
 buildctl build --help
216 223
 ```
217 224
 
218
-The `tonistiigi/buildkit` image can be built locally using the Dockerfile in `./hack/dockerfiles/test.Dockerfile`.
225
+The images can be also built locally using `./hack/dockerfiles/test.Dockerfile` (or `./hack/dockerfiles/test.buildkit.Dockerfile` if you already have BuildKit).
219 226
 
220 227
 ### Opentracing support
221 228
 
... ...
@@ -232,7 +252,7 @@ export JAEGER_TRACE=0.0.0.0:6831
232 232
 
233 233
 ### Supported runc version
234 234
 
235
-During development, BuildKit is tested with the version of runc that is being used by the containerd repository. Please refer to [runc.md](https://github.com/containerd/containerd/blob/v1.1.3/RUNC.md) for more information.
235
+During development, BuildKit is tested with the version of runc that is being used by the containerd repository. Please refer to [runc.md](https://github.com/containerd/containerd/blob/v1.2.0-rc.1/RUNC.md) for more information.
236 236
 
237 237
 ### Running BuildKit without root privileges
238 238
 
... ...
@@ -628,7 +628,7 @@ func getFollowLinksWalk(root *iradix.Node, k []byte, linksWalked *int) ([]byte,
628 628
 		if dirPath == "." || dirPath == "/" {
629 629
 			dirPath = ""
630 630
 		}
631
-		link := parent.Linkname
631
+		link := path.Clean(parent.Linkname)
632 632
 		if !path.IsAbs(link) {
633 633
 			link = path.Join("/", path.Join(path.Dir(dirPath), link))
634 634
 		}
... ...
@@ -126,30 +126,11 @@ func Image(ref string, opts ...ImageOption) State {
126 126
 		if err != nil {
127 127
 			src.err = err
128 128
 		} else {
129
-			var img struct {
130
-				Config struct {
131
-					Env        []string `json:"Env,omitempty"`
132
-					WorkingDir string   `json:"WorkingDir,omitempty"`
133
-					User       string   `json:"User,omitempty"`
134
-				} `json:"config,omitempty"`
135
-			}
136
-			if err := json.Unmarshal(dt, &img); err != nil {
137
-				src.err = err
138
-			} else {
139
-				st := NewState(src.Output())
140
-				for _, env := range img.Config.Env {
141
-					parts := strings.SplitN(env, "=", 2)
142
-					if len(parts[0]) > 0 {
143
-						var v string
144
-						if len(parts) > 1 {
145
-							v = parts[1]
146
-						}
147
-						st = st.AddEnv(parts[0], v)
148
-					}
149
-				}
150
-				st = st.Dir(img.Config.WorkingDir)
129
+			st, err := NewState(src.Output()).WithImageConfig(dt)
130
+			if err == nil {
151 131
 				return st
152 132
 			}
133
+			src.err = err
153 134
 		}
154 135
 	}
155 136
 	return NewState(src.Output())
... ...
@@ -2,8 +2,10 @@ package llb
2 2
 
3 3
 import (
4 4
 	"context"
5
+	"encoding/json"
5 6
 	"fmt"
6 7
 	"net"
8
+	"strings"
7 9
 
8 10
 	"github.com/containerd/containerd/platforms"
9 11
 	"github.com/moby/buildkit/identity"
... ...
@@ -171,6 +173,31 @@ func (s State) WithOutput(o Output) State {
171 171
 	return s
172 172
 }
173 173
 
174
+func (s State) WithImageConfig(c []byte) (State, error) {
175
+	var img struct {
176
+		Config struct {
177
+			Env        []string `json:"Env,omitempty"`
178
+			WorkingDir string   `json:"WorkingDir,omitempty"`
179
+			User       string   `json:"User,omitempty"`
180
+		} `json:"config,omitempty"`
181
+	}
182
+	if err := json.Unmarshal(c, &img); err != nil {
183
+		return State{}, err
184
+	}
185
+	for _, env := range img.Config.Env {
186
+		parts := strings.SplitN(env, "=", 2)
187
+		if len(parts[0]) > 0 {
188
+			var v string
189
+			if len(parts) > 1 {
190
+				v = parts[1]
191
+			}
192
+			s = s.AddEnv(parts[0], v)
193
+		}
194
+	}
195
+	s = s.Dir(img.Config.WorkingDir)
196
+	return s, nil
197
+}
198
+
174 199
 func (s State) Run(ro ...RunOption) ExecState {
175 200
 	ei := &ExecInfo{State: s}
176 201
 	if p := s.GetPlatform(); p != nil {
... ...
@@ -99,7 +99,7 @@ func Build(ctx context.Context, c client.Client) (*client.Result, error) {
99 99
 	name := "load build definition from " + filename
100 100
 
101 101
 	src := llb.Local(LocalNameDockerfile,
102
-		llb.IncludePatterns([]string{filename}),
102
+		llb.FollowPaths([]string{filename}),
103 103
 		llb.SessionID(c.BuildOpts().SessionID),
104 104
 		llb.SharedKeyHint(defaultDockerfileName),
105 105
 		dockerfile2llb.WithInternalName(name),
... ...
@@ -189,7 +189,7 @@ func Build(ctx context.Context, c client.Client) (*client.Result, error) {
189 189
 			if dockerignoreState == nil {
190 190
 				st := llb.Local(LocalNameContext,
191 191
 					llb.SessionID(c.BuildOpts().SessionID),
192
-					llb.IncludePatterns([]string{dockerignoreFilename}),
192
+					llb.FollowPaths([]string{dockerignoreFilename}),
193 193
 					llb.SharedKeyHint(dockerignoreFilename),
194 194
 					dockerfile2llb.WithInternalName("load "+dockerignoreFilename),
195 195
 				)
... ...
@@ -35,7 +35,7 @@ const (
35 35
 	localNameContext = "context"
36 36
 	historyComment   = "buildkit.dockerfile.v0"
37 37
 
38
-	DefaultCopyImage = "tonistiigi/copy:v0.1.7@sha256:9aab7d9ab369c6daf4831bf0653f7592110ab4b7e8a33fee2b9dca546e9d3089"
38
+	DefaultCopyImage = "docker/dockerfile-copy:v0.1.9@sha256:e8f159d3f00786604b93c675ee2783f8dc194bb565e61ca5788f6a6e9d304061"
39 39
 )
40 40
 
41 41
 type ConvertOpt struct {
... ...
@@ -327,6 +327,7 @@ func Dockerfile2LLB(ctx context.Context, dt []byte, opt ConvertOpt) (*llb.State,
327 327
 			targetPlatform:    platformOpt.targetPlatform,
328 328
 			extraHosts:        opt.ExtraHosts,
329 329
 			copyImage:         opt.OverrideCopyImage,
330
+			llbCaps:           opt.LLBCaps,
330 331
 		}
331 332
 		if opt.copyImage == "" {
332 333
 			opt.copyImage = DefaultCopyImage
... ...
@@ -441,6 +442,7 @@ type dispatchOpt struct {
441 441
 	buildPlatforms    []specs.Platform
442 442
 	extraHosts        []llb.HostIP
443 443
 	copyImage         string
444
+	llbCaps           *apicaps.CapSet
444 445
 }
445 446
 
446 447
 func dispatch(d *dispatchState, cmd command, opt dispatchOpt) error {
... ...
@@ -467,7 +469,9 @@ func dispatch(d *dispatchState, cmd command, opt dispatchOpt) error {
467 467
 		err = dispatchCopy(d, c.SourcesAndDest, opt.buildContext, true, c, "", opt)
468 468
 		if err == nil {
469 469
 			for _, src := range c.Sources() {
470
-				d.ctxPaths[path.Join("/", filepath.ToSlash(src))] = struct{}{}
470
+				if !strings.HasPrefix(src, "http://") && !strings.HasPrefix(src, "https://") {
471
+					d.ctxPaths[path.Join("/", filepath.ToSlash(src))] = struct{}{}
472
+				}
471 473
 			}
472 474
 		}
473 475
 	case *instructions.LabelCommand:
... ...
@@ -627,7 +631,12 @@ func dispatchRun(d *dispatchState, c *instructions.RunCommand, proxy *llb.ProxyE
627 627
 		return err
628 628
 	}
629 629
 	opt = append(opt, runMounts...)
630
-	opt = append(opt, llb.WithCustomName(prefixCommand(d, uppercaseCmd(processCmdEnv(dopt.shlex, c.String(), env)), d.prefixPlatform, d.state.GetPlatform())))
630
+
631
+	shlex := *dopt.shlex
632
+	shlex.RawQuotes = true
633
+	shlex.SkipUnsetEnv = true
634
+
635
+	opt = append(opt, llb.WithCustomName(prefixCommand(d, uppercaseCmd(processCmdEnv(&shlex, c.String(), env)), d.prefixPlatform, d.state.GetPlatform())))
631 636
 	for _, h := range dopt.extraHosts {
632 637
 		opt = append(opt, llb.AddExtraHost(h.Host, h.IP))
633 638
 	}
... ...
@@ -729,6 +738,13 @@ func dispatchCopy(d *dispatchState, c instructions.SourcesAndDest, sourceState l
729 729
 	if d.ignoreCache {
730 730
 		runOpt = append(runOpt, llb.IgnoreCache)
731 731
 	}
732
+
733
+	if opt.llbCaps != nil {
734
+		if err := opt.llbCaps.Supports(pb.CapExecMetaNetwork); err == nil {
735
+			runOpt = append(runOpt, llb.Network(llb.NetModeNone))
736
+		}
737
+	}
738
+
732 739
 	run := img.Run(append(runOpt, mounts...)...)
733 740
 	d.state = run.AddMount("/dest", d.state).Platform(platform)
734 741
 
... ...
@@ -1,4 +1,4 @@
1
-// +build !dfrunmount,!dfextall
1
+// +build !dfrunmount
2 2
 
3 3
 package dockerfile2llb
4 4
 
... ...
@@ -1,4 +1,4 @@
1
-// +build dfrunmount dfextall
1
+// +build dfrunmount
2 2
 
3 3
 package dockerfile2llb
4 4
 
... ...
@@ -1,4 +1,4 @@
1
-// +build dfsecrets dfextall
1
+// +build dfsecrets
2 2
 
3 3
 package dockerfile2llb
4 4
 
... ...
@@ -1,4 +1,4 @@
1
-// +build dfssh dfextall
1
+// +build dfssh
2 2
 
3 3
 package dockerfile2llb
4 4
 
... ...
@@ -1,4 +1,4 @@
1
-// +build !dfsecrets,!dfextall
1
+// +build !dfsecrets
2 2
 
3 3
 package instructions
4 4
 
... ...
@@ -1,4 +1,4 @@
1
-// +build !dfssh,!dfextall
1
+// +build !dfssh
2 2
 
3 3
 package instructions
4 4
 
... ...
@@ -1,4 +1,4 @@
1
-// +build dfrunmount dfextall
1
+// +build dfrunmount
2 2
 
3 3
 package instructions
4 4
 
... ...
@@ -1,4 +1,4 @@
1
-// +build dfsecrets dfextall
1
+// +build dfsecrets
2 2
 
3 3
 package instructions
4 4
 
... ...
@@ -1,4 +1,4 @@
1
-// +build dfssh dfextall
1
+// +build dfssh
2 2
 
3 3
 package instructions
4 4
 
... ...
@@ -2,6 +2,7 @@ package shell
2 2
 
3 3
 import (
4 4
 	"bytes"
5
+	"fmt"
5 6
 	"strings"
6 7
 	"text/scanner"
7 8
 	"unicode"
... ...
@@ -17,7 +18,9 @@ import (
17 17
 // It doesn't support all flavors of ${xx:...} formats but new ones can
18 18
 // be added by adding code to the "special ${} format processing" section
19 19
 type Lex struct {
20
-	escapeToken rune
20
+	escapeToken  rune
21
+	RawQuotes    bool
22
+	SkipUnsetEnv bool
21 23
 }
22 24
 
23 25
 // NewLex creates a new Lex which uses escapeToken to escape quotes.
... ...
@@ -58,17 +61,21 @@ func (s *Lex) ProcessWordsWithMap(word string, env map[string]string) ([]string,
58 58
 
59 59
 func (s *Lex) process(word string, env map[string]string) (string, []string, error) {
60 60
 	sw := &shellWord{
61
-		envs:        env,
62
-		escapeToken: s.escapeToken,
61
+		envs:         env,
62
+		escapeToken:  s.escapeToken,
63
+		skipUnsetEnv: s.SkipUnsetEnv,
64
+		rawQuotes:    s.RawQuotes,
63 65
 	}
64 66
 	sw.scanner.Init(strings.NewReader(word))
65 67
 	return sw.process(word)
66 68
 }
67 69
 
68 70
 type shellWord struct {
69
-	scanner     scanner.Scanner
70
-	envs        map[string]string
71
-	escapeToken rune
71
+	scanner      scanner.Scanner
72
+	envs         map[string]string
73
+	escapeToken  rune
74
+	rawQuotes    bool
75
+	skipUnsetEnv bool
72 76
 }
73 77
 
74 78
 func (sw *shellWord) process(source string) (string, []string, error) {
... ...
@@ -103,10 +110,8 @@ func (w *wordsStruct) addRawChar(ch rune) {
103 103
 }
104 104
 
105 105
 func (w *wordsStruct) addString(str string) {
106
-	var scan scanner.Scanner
107
-	scan.Init(strings.NewReader(str))
108
-	for scan.Peek() != scanner.EOF {
109
-		w.addChar(scan.Next())
106
+	for _, ch := range str {
107
+		w.addChar(ch)
110 108
 	}
111 109
 }
112 110
 
... ...
@@ -196,14 +201,20 @@ func (sw *shellWord) processSingleQuote() (string, error) {
196 196
 
197 197
 	var result bytes.Buffer
198 198
 
199
-	sw.scanner.Next()
199
+	ch := sw.scanner.Next()
200
+	if sw.rawQuotes {
201
+		result.WriteRune(ch)
202
+	}
200 203
 
201 204
 	for {
202
-		ch := sw.scanner.Next()
205
+		ch = sw.scanner.Next()
203 206
 		switch ch {
204 207
 		case scanner.EOF:
205 208
 			return "", errors.New("unexpected end of statement while looking for matching single-quote")
206 209
 		case '\'':
210
+			if sw.rawQuotes {
211
+				result.WriteRune(ch)
212
+			}
207 213
 			return result.String(), nil
208 214
 		}
209 215
 		result.WriteRune(ch)
... ...
@@ -225,14 +236,20 @@ func (sw *shellWord) processDoubleQuote() (string, error) {
225 225
 
226 226
 	var result bytes.Buffer
227 227
 
228
-	sw.scanner.Next()
228
+	ch := sw.scanner.Next()
229
+	if sw.rawQuotes {
230
+		result.WriteRune(ch)
231
+	}
229 232
 
230 233
 	for {
231 234
 		switch sw.scanner.Peek() {
232 235
 		case scanner.EOF:
233 236
 			return "", errors.New("unexpected end of statement while looking for matching double-quote")
234 237
 		case '"':
235
-			sw.scanner.Next()
238
+			ch := sw.scanner.Next()
239
+			if sw.rawQuotes {
240
+				result.WriteRune(ch)
241
+			}
236 242
 			return result.String(), nil
237 243
 		case '$':
238 244
 			value, err := sw.processDollar()
... ...
@@ -269,7 +286,11 @@ func (sw *shellWord) processDollar() (string, error) {
269 269
 		if name == "" {
270 270
 			return "$", nil
271 271
 		}
272
-		return sw.getEnv(name), nil
272
+		value, found := sw.getEnv(name)
273
+		if !found && sw.skipUnsetEnv {
274
+			return "$" + name, nil
275
+		}
276
+		return value, nil
273 277
 	}
274 278
 
275 279
 	sw.scanner.Next()
... ...
@@ -285,7 +306,11 @@ func (sw *shellWord) processDollar() (string, error) {
285 285
 	switch ch {
286 286
 	case '}':
287 287
 		// Normal ${xx} case
288
-		return sw.getEnv(name), nil
288
+		value, found := sw.getEnv(name)
289
+		if !found && sw.skipUnsetEnv {
290
+			return fmt.Sprintf("${%s}", name), nil
291
+		}
292
+		return value, nil
289 293
 	case ':':
290 294
 		// Special ${xx:...} format processing
291 295
 		// Yes it allows for recursive $'s in the ... spot
... ...
@@ -301,19 +326,26 @@ func (sw *shellWord) processDollar() (string, error) {
301 301
 
302 302
 		// Grab the current value of the variable in question so we
303 303
 		// can use to to determine what to do based on the modifier
304
-		newValue := sw.getEnv(name)
304
+		newValue, found := sw.getEnv(name)
305 305
 
306 306
 		switch modifier {
307 307
 		case '+':
308 308
 			if newValue != "" {
309 309
 				newValue = word
310 310
 			}
311
+			if !found && sw.skipUnsetEnv {
312
+				return fmt.Sprintf("${%s:%s%s}", name, string(modifier), word), nil
313
+			}
311 314
 			return newValue, nil
312 315
 
313 316
 		case '-':
314 317
 			if newValue == "" {
315 318
 				newValue = word
316 319
 			}
320
+			if !found && sw.skipUnsetEnv {
321
+				return fmt.Sprintf("${%s:%s%s}", name, string(modifier), word), nil
322
+			}
323
+
317 324
 			return newValue, nil
318 325
 
319 326
 		default:
... ...
@@ -364,13 +396,13 @@ func isSpecialParam(char rune) bool {
364 364
 	return false
365 365
 }
366 366
 
367
-func (sw *shellWord) getEnv(name string) string {
367
+func (sw *shellWord) getEnv(name string) (string, bool) {
368 368
 	for key, value := range sw.envs {
369 369
 		if EqualEnvKeys(name, key) {
370
-			return value
370
+			return value, true
371 371
 		}
372 372
 	}
373
-	return ""
373
+	return "", false
374 374
 }
375 375
 
376 376
 func BuildEnvs(env []string) map[string]string {
... ...
@@ -24,7 +24,7 @@ const (
24 24
 // Dialer returns a connection that can be used by the session
25 25
 type Dialer func(ctx context.Context, proto string, meta map[string][]string) (net.Conn, error)
26 26
 
27
-// Attachable defines a feature that can be expsed on a session
27
+// Attachable defines a feature that can be exposed on a session
28 28
 type Attachable interface {
29 29
 	Register(*grpc.Server)
30 30
 }
... ...
@@ -66,7 +66,7 @@ func NewSession(ctx context.Context, name, sharedKey string) (*Session, error) {
66 66
 	return s, nil
67 67
 }
68 68
 
69
-// Allow enable a given service to be reachable through the grpc session
69
+// Allow enables a given service to be reachable through the grpc session
70 70
 func (s *Session) Allow(a Attachable) {
71 71
 	a.Register(s.grpcServer)
72 72
 }
... ...
@@ -171,6 +171,7 @@ func (sb *subBuilder) Build(ctx context.Context, e Edge) (CachedResult, error) {
171 171
 }
172 172
 
173 173
 func (sb *subBuilder) Context(ctx context.Context) context.Context {
174
+	ctx = session.NewContext(ctx, sb.state.getSessionID())
174 175
 	return opentracing.ContextWithSpan(progress.WithProgress(ctx, sb.mpw), sb.mspan)
175 176
 }
176 177
 
... ...
@@ -475,6 +476,7 @@ func (j *Job) Discard() error {
475 475
 }
476 476
 
477 477
 func (j *Job) Context(ctx context.Context) context.Context {
478
+	ctx = session.NewContext(ctx, j.SessionID)
478 479
 	return progress.WithProgress(ctx, j.pw)
479 480
 }
480 481
 
... ...
@@ -72,6 +72,7 @@ func (w *ingester) Writer(ctx context.Context, opts ...content.WriterOpt) (conte
72 72
 	}
73 73
 	writer, err := w.pusher.Push(ctx, wo.Desc)
74 74
 	if err != nil {
75
+		unlock()
75 76
 		return nil, err
76 77
 	}
77 78
 	return &lockedWriter{unlock: unlock, Writer: writer}, nil
... ...
@@ -1,113 +1,40 @@
1 1
 package specconv
2 2
 
3 3
 import (
4
-	"os"
5
-	"sort"
6 4
 	"strings"
7 5
 
8
-	"github.com/opencontainers/runc/libcontainer/system"
9
-	"github.com/opencontainers/runc/libcontainer/user"
10 6
 	"github.com/opencontainers/runtime-spec/specs-go"
11
-	"github.com/pkg/errors"
12 7
 )
13 8
 
14 9
 // ToRootless converts spec to be compatible with "rootless" runc.
15
-// * Adds userns (Note: since we are already in userns, ideally we should not need to do this. runc-side issue is tracked at https://github.com/opencontainers/runc/issues/1837)
16
-// * Fix up mount flags (same as above)
17
-// * Replace /sys with bind-mount (FIXME: we don't need to do this if netns is unshared)
10
+// * Remove /sys mount
11
+// * Remove cgroups
12
+//
13
+// See docs/rootless.md for the supported runc revision.
18 14
 func ToRootless(spec *specs.Spec) error {
19
-	if !system.RunningInUserNS() {
20
-		return errors.New("needs to be in user namespace")
21
-	}
22
-	uidMap, err := user.CurrentProcessUIDMap()
23
-	if err != nil && !os.IsNotExist(err) {
24
-		return err
25
-	}
26
-	gidMap, err := user.CurrentProcessUIDMap()
27
-	if err != nil && !os.IsNotExist(err) {
28
-		return err
29
-	}
30
-	return toRootless(spec, uidMap, gidMap)
31
-}
32
-
33
-// toRootless was forked from github.com/opencontainers/runc/libcontainer/specconv
34
-func toRootless(spec *specs.Spec, uidMap, gidMap []user.IDMap) error {
35
-	if err := configureUserNS(spec, uidMap, gidMap); err != nil {
36
-		return err
37
-	}
38
-	if err := configureMounts(spec); err != nil {
39
-		return err
40
-	}
41
-
42
-	// Remove cgroup settings.
43
-	spec.Linux.Resources = nil
44
-	spec.Linux.CgroupsPath = ""
45
-	return nil
46
-}
47
-
48
-// configureUserNS add suserns and the current ID map to the spec.
49
-// Since we are already in userns, ideally we should not need to add userns.
50
-// However, currently rootless runc always requires userns to be added.
51
-// https://github.com/opencontainers/runc/issues/1837
52
-func configureUserNS(spec *specs.Spec, uidMap, gidMap []user.IDMap) error {
53
-	spec.Linux.Namespaces = append(spec.Linux.Namespaces, specs.LinuxNamespace{
54
-		Type: specs.UserNamespace,
55
-	})
56
-
57
-	sort.Slice(uidMap, func(i, j int) bool { return uidMap[i].ID < uidMap[j].ID })
58
-	uNextContainerID := int64(0)
59
-	for _, u := range uidMap {
60
-		spec.Linux.UIDMappings = append(spec.Linux.UIDMappings,
61
-			specs.LinuxIDMapping{
62
-				HostID:      uint32(u.ID),
63
-				ContainerID: uint32(uNextContainerID),
64
-				Size:        uint32(u.Count),
65
-			})
66
-		uNextContainerID += int64(u.Count)
67
-	}
68
-	sort.Slice(gidMap, func(i, j int) bool { return gidMap[i].ID < gidMap[j].ID })
69
-	gNextContainerID := int64(0)
70
-	for _, g := range gidMap {
71
-		spec.Linux.GIDMappings = append(spec.Linux.GIDMappings,
72
-			specs.LinuxIDMapping{
73
-				HostID:      uint32(g.ID),
74
-				ContainerID: uint32(gNextContainerID),
75
-				Size:        uint32(g.Count),
76
-			})
77
-		gNextContainerID += int64(g.Count)
78
-	}
79
-	return nil
80
-}
81
-
82
-func configureMounts(spec *specs.Spec) error {
15
+	// Remove /sys mount because we can't mount /sys when the daemon netns
16
+	// is not unshared from the host.
17
+	//
18
+	// Instead, we could bind-mount /sys from the host, however, `rbind, ro`
19
+	// does not make /sys/fs/cgroup read-only (and we can't bind-mount /sys
20
+	// without rbind)
21
+	//
22
+	// PR for making /sys/fs/cgroup read-only is proposed, but it is very
23
+	// complicated: https://github.com/opencontainers/runc/pull/1869
24
+	//
25
+	// For buildkit usecase, we suppose we don't need to provide /sys to
26
+	// containers and remove /sys mount as a workaround.
83 27
 	var mounts []specs.Mount
84 28
 	for _, mount := range spec.Mounts {
85
-		// Ignore all mounts that are under /sys, because we add /sys later.
86 29
 		if strings.HasPrefix(mount.Destination, "/sys") {
87 30
 			continue
88 31
 		}
89
-
90
-		// Remove all gid= and uid= mappings.
91
-		// Since we are already in userns, ideally we should not need to do this.
92
-		// https://github.com/opencontainers/runc/issues/1837
93
-		var options []string
94
-		for _, option := range mount.Options {
95
-			if !strings.HasPrefix(option, "gid=") && !strings.HasPrefix(option, "uid=") {
96
-				options = append(options, option)
97
-			}
98
-		}
99
-		mount.Options = options
100 32
 		mounts = append(mounts, mount)
101 33
 	}
102
-
103
-	// Add the sysfs mount as an rbind, because we can't mount /sys unless we have netns.
104
-	// TODO: keep original /sys mount when we have netns.
105
-	mounts = append(mounts, specs.Mount{
106
-		Source:      "/sys",
107
-		Destination: "/sys",
108
-		Type:        "none",
109
-		Options:     []string{"rbind", "nosuid", "noexec", "nodev", "ro"},
110
-	})
111 34
 	spec.Mounts = mounts
35
+
36
+	// Remove cgroups so as to avoid `container_linux.go:337: starting container process caused "process_linux.go:280: applying cgroup configuration for process caused \"mkdir /sys/fs/cgroup/cpuset/buildkit: permission denied\""`
37
+	spec.Linux.Resources = nil
38
+	spec.Linux.CgroupsPath = ""
112 39
 	return nil
113 40
 }
... ...
@@ -6,7 +6,7 @@ github.com/davecgh/go-spew v1.1.0
6 6
 github.com/pmezard/go-difflib v1.0.0
7 7
 golang.org/x/sys 1b2967e3c290b7c545b3db0deeda16e9be4f98a2
8 8
 
9
-github.com/containerd/containerd d97a907f7f781c0ab8340877d8e6b53cc7f1c2f6
9
+github.com/containerd/containerd 1a5f9a3434ac53c0e9d27093ecc588e0c281c333
10 10
 github.com/containerd/typeurl a93fcdb778cd272c6e9b3028b2f42d813e785d40
11 11
 golang.org/x/sync 450f422ab23cf9881c94e2db30cac0eb1b7cf80c
12 12
 github.com/sirupsen/logrus v1.0.0
... ...
@@ -16,9 +16,9 @@ golang.org/x/net 0ed95abb35c445290478a5348a7b38bb154135fd
16 16
 github.com/gogo/protobuf v1.0.0
17 17
 github.com/gogo/googleapis b23578765ee54ff6bceff57f397d833bf4ca6869
18 18
 github.com/golang/protobuf v1.1.0
19
-github.com/containerd/continuity f44b615e492bdfb371aae2f76ec694d9da1db537
19
+github.com/containerd/continuity bd77b46c8352f74eb12c85bdc01f4b90f69d66b4
20 20
 github.com/opencontainers/image-spec v1.0.1
21
-github.com/opencontainers/runc 20aff4f0488c6d4b8df4d85b4f63f1f704c11abd
21
+github.com/opencontainers/runc a00bf0190895aa465a5fbed0268888e2c8ddfe85
22 22
 github.com/Microsoft/go-winio v0.4.11
23 23
 github.com/containerd/fifo 3d5202aec260678c48179c56f40e6f38a095738c
24 24
 github.com/opencontainers/runtime-spec eba862dc2470385a233c7507392675cbeadf7353 # v1.0.1-45-geba862d
... ...
@@ -28,8 +28,9 @@ google.golang.org/genproto d80a6e20e776b0b17a324d0ba1ab50a39c8e8944
28 28
 golang.org/x/text 19e51611da83d6be54ddafce4a4af510cb3e9ea4
29 29
 github.com/docker/go-events 9461782956ad83b30282bf90e31fa6a70c255ba9
30 30
 github.com/syndtr/gocapability db04d3cc01c8b54962a58ec7e491717d06cfcc16
31
-github.com/Microsoft/hcsshim v0.7.3
31
+github.com/Microsoft/hcsshim v0.7.9
32 32
 golang.org/x/crypto 0709b304e793a5edb4a2c0145f281ecdc20838a4
33
+github.com/containerd/cri 8506fe836677cc3bb23a16b68145128243d843b5 # release/1.2 branch
33 34
 
34 35
 github.com/urfave/cli 7bc6a0acffa589f415f88aca16cc1de5ffd66f9c
35 36
 github.com/morikuni/aec 39771216ff4c63d11f5e604076f9c45e8be1067b
... ...
@@ -40,8 +41,8 @@ golang.org/x/time f51c12702a4d776e4c1fa9b0fabab841babae631
40 40
 github.com/docker/docker 71cd53e4a197b303c6ba086bd584ffd67a884281
41 41
 github.com/pkg/profile 5b67d428864e92711fcbd2f8629456121a56d91f
42 42
 
43
-github.com/tonistiigi/fsutil f567071bed2416e4d87d260d3162722651182317
44
-github.com/hashicorp/go-immutable-radix 826af9ccf0feeee615d546d69b11f8e98da8c8f1 git://github.com/tonistiigi/go-immutable-radix.git
43
+github.com/tonistiigi/fsutil 2862f6bc5ac9b97124e552a5c108230b38a1b0ca
44
+github.com/hashicorp/go-immutable-radix 826af9ccf0feeee615d546d69b11f8e98da8c8f1 https://github.com/tonistiigi/go-immutable-radix
45 45
 github.com/hashicorp/golang-lru a0d98a5f288019575c6d1f4bb1573fef2d1fcdc4
46 46
 github.com/mitchellh/hashstructure 2bca23e0e452137f789efbc8610126fd8b94f73b
47 47
 github.com/docker/go-connections 3ede32e2033de7505e6500d6c868c2b9ed9f169d
... ...
@@ -66,6 +67,3 @@ github.com/opentracing-contrib/go-stdlib b1a47cfbdd7543e70e9ef3e73d0802ad306cc1c
66 66
 # used by dockerfile tests
67 67
 gotest.tools v2.1.0
68 68
 github.com/google/go-cmp v0.2.0
69
-
70
-# used by rootless spec conv test
71
-github.com/seccomp/libseccomp-golang 32f571b70023028bd57d9288c20efbcb237f3ce0
... ...
@@ -5,6 +5,7 @@ import (
5 5
 	"os"
6 6
 	"path/filepath"
7 7
 	"strings"
8
+	"syscall"
8 9
 	"time"
9 10
 
10 11
 	"github.com/docker/docker/pkg/fileutils"
... ...
@@ -71,7 +72,7 @@ func Walk(ctx context.Context, p string, opt *WalkOpt, fn filepath.WalkFunc) err
71 71
 			return err
72 72
 		}
73 73
 		defer func() {
74
-			if retErr != nil && os.IsNotExist(errors.Cause(retErr)) {
74
+			if retErr != nil && isNotExist(retErr) {
75 75
 				retErr = filepath.SkipDir
76 76
 			}
77 77
 		}()
... ...
@@ -216,3 +217,14 @@ func trimUntilIndex(str, sep string, count int) string {
216 216
 		}
217 217
 	}
218 218
 }
219
+
220
+func isNotExist(err error) bool {
221
+	err = errors.Cause(err)
222
+	if os.IsNotExist(err) {
223
+		return true
224
+	}
225
+	if pe, ok := err.(*os.PathError); ok {
226
+		err = pe.Err
227
+	}
228
+	return err == syscall.ENOTDIR
229
+}