fetching: S3 upload and download (#11117)
This extends Spack functionality so that it can fetch sources and binaries from-, push sources and binaries to-, and index the contents of- mirrors hosted on an S3 bucket.
High level to-do list:
- [x] Extend mirrors configuration to add support for `file://`, and `s3://` URLs.
- [x] Ensure all fetching, pushing, and indexing operations work for `file://` URLs.
- [x] Implement S3 source fetching
- [x] Implement S3 binary mirror indexing
- [x] Implement S3 binary package fetching
- [x] Implement S3 source pushing
- [x] Implement S3 binary package pushing
Important details:
* refactor URL handling to handle S3 URLs and mirror URLs more gracefully.
- updated parse() to accept already-parsed URL objects. an equivalent object
is returned with any extra s3-related attributes intact. Objects created with
urllib can also be passed, and the additional s3 handling logic will still be applied.
* update mirror schema/parsing (mirror can have separate fetch/push URLs)
* implement s3_fetch_strategy/several utility changes
* provide more feature-complete S3 fetching
* update buildcache create command to support S3
* Move the core logic for reading data from S3 out of the s3 fetch strategy and into
the s3 URL handler. The s3 fetch strategy now calls into `read_from_url()` Since
read_from_url can now handle S3 URLs, the S3 fetch strategy is redundant. It's
not clear whether the ideal design is to have S3 fetching functionality in a fetch
strategy, directly implemented in read_from_url, or both.
* expanded what can be passed to `spack buildcache` via the -d flag: In addition
to a directory on the local filesystem, the name of a configured mirror can be
passed, or a push URL can be passed directly.
2019-10-22 07:32:04 +00:00
|
|
|
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
|
|
|
|
|
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
|
|
|
|
#
|
|
|
|
|
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
|
|
|
|
|
2019-05-29 23:29:09 +00:00
|
|
|
generate ci jobs:
|
|
|
|
|
script:
|
|
|
|
|
- "./bin/generate-gitlab-ci-yml.sh"
|
|
|
|
|
tags:
|
2019-09-03 17:45:49 +00:00
|
|
|
- "spack-pre-ci"
|
2019-05-29 23:29:09 +00:00
|
|
|
artifacts:
|
|
|
|
|
paths:
|
|
|
|
|
- ci-generation
|
|
|
|
|
when: always
|