Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cached OCSP client support to Cert Auth #17093

Merged
merged 44 commits into from
Nov 21, 2022
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
8f34df9
wip
sgmiller Sep 8, 2022
5b63032
Add cached OCSP client support to Cert Auth
sgmiller Sep 9, 2022
b0798a7
->pointer
sgmiller Sep 9, 2022
87bd5a3
Code cleanup
sgmiller Sep 9, 2022
0b9f785
Fix unit tests
sgmiller Sep 9, 2022
fb65fd5
Use an LRU cache, and only persist up to 1000 of the most recently us…
sgmiller Sep 12, 2022
5d768c0
Fix caching, add fail open mode parameter to cert auth roles
sgmiller Sep 12, 2022
6e00633
reduce logging
sgmiller Sep 12, 2022
44d808d
Add the retry client and GET then POST logic
sgmiller Sep 12, 2022
2b73de7
Drop persisted cache, make cache size configurable, allow for paralle…
sgmiller Sep 12, 2022
0f9cda6
dead code
sgmiller Sep 12, 2022
fca24ee
Update builtin/credential/cert/path_certs.go
sgmiller Sep 13, 2022
59a3769
Hook invalidate to reinit the ocsp cache size
sgmiller Sep 13, 2022
ae8423c
Merge branch 'cert-auth-ocsp' of github.com:/hashicorp/vault into cer…
sgmiller Sep 13, 2022
ea1feb4
locking
sgmiller Sep 13, 2022
e7d790d
Conditionally init the ocsp client
sgmiller Sep 13, 2022
75244c9
Remove cache size config from cert configs, it's a backend global
sgmiller Sep 13, 2022
68c3308
Add field
sgmiller Sep 13, 2022
acc2c29
Remove strangely complex validity logic
sgmiller Sep 13, 2022
81e8d6a
Address more feedback
sgmiller Sep 13, 2022
ca7e8c0
Rework error returning logic
sgmiller Sep 14, 2022
b5f2b03
More edge cases
sgmiller Sep 14, 2022
6ba87f9
MORE edge cases
sgmiller Sep 14, 2022
4abdc5a
Merge branch 'main' into cert-auth-ocsp
sgmiller Sep 14, 2022
1be9cf5
Merge remote-tracking branch 'origin/main' into cert-auth-ocsp
sgmiller Oct 27, 2022
51db45f
Add a test matrix with a builtin responder
sgmiller Nov 3, 2022
8b39c75
Merge remote-tracking branch 'origin/main' into cert-auth-ocsp
sgmiller Nov 3, 2022
0642d7e
changelog
sgmiller Nov 3, 2022
0ff724c
Use an atomic for configUpdated
sgmiller Nov 4, 2022
7e50dd0
Actually use ocsp_enabled, and bind to a random port for testing
sgmiller Nov 4, 2022
222bc58
Update builtin/credential/cert/path_login.go
sgmiller Nov 7, 2022
f192134
Refactor unit tests
sgmiller Nov 8, 2022
b8fb6ed
Add status to cache
sgmiller Nov 8, 2022
9933c3b
Make some functions private
sgmiller Nov 8, 2022
6912f06
Rename for testing, and attribute
sgmiller Nov 8, 2022
bf4e630
Merge remote-tracking branch 'origin/main' into cert-auth-ocsp
sgmiller Nov 8, 2022
4021908
Up to date gofumpt
sgmiller Nov 8, 2022
8143803
remove hash from key, and disable the vault dependent unit test
sgmiller Nov 8, 2022
e91c2cf
Comment out TestMultiOCSP
sgmiller Nov 10, 2022
ff60c7d
imports
sgmiller Nov 15, 2022
65bb99a
more imports
sgmiller Nov 18, 2022
9a2acb3
Address semgrep results
sgmiller Nov 18, 2022
899f36d
Attempt to pass some sort of logging to test_responder
sgmiller Nov 18, 2022
ec3d5ff
fix overzealous search&replace
sgmiller Nov 21, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 22 additions & 3 deletions builtin/credential/cert/backend.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ package cert

import (
"context"
"github.com/hashicorp/go-hclog"
"github.com/hashicorp/vault/sdk/helper/ocsp"
"strings"
"sync"

Expand All @@ -14,6 +16,13 @@ func Factory(ctx context.Context, conf *logical.BackendConfig) (logical.Backend,
if err := b.Setup(ctx, conf); err != nil {
return nil, err
}
bConf, err := b.Config(ctx, conf.StorageView)
if err != nil {
return nil, err
}
if conf != nil {
cipherboy marked this conversation as resolved.
Show resolved Hide resolved
b.initOCSPClient(bConf.OcspCacheSize)
}
return b, nil
}

Expand All @@ -39,16 +48,18 @@ func Backend() *backend {
}

b.crlUpdateMutex = &sync.RWMutex{}

return &b
}

type backend struct {
*framework.Backend
MapCertId *framework.PathMap

crls map[string]CRLInfo
crlUpdateMutex *sync.RWMutex
crls map[string]CRLInfo
ocspDisabled bool
crlUpdateMutex *sync.RWMutex
ocspClientMutex sync.RWMutex
ocspClient *ocsp.Client
}

func (b *backend) invalidate(_ context.Context, key string) {
Expand All @@ -60,6 +71,14 @@ func (b *backend) invalidate(_ context.Context, key string) {
}
}

func (b *backend) initOCSPClient(cacheSize int) {
b.ocspClientMutex.Lock()
defer b.ocspClientMutex.Unlock()
b.ocspClient = ocsp.New(func() hclog.Logger {
return b.Logger()
}, cacheSize)
}

const backendHelp = `
The "cert" credential provider allows authentication using
TLS client certificates. A client connects to Vault and uses
Expand Down
58 changes: 57 additions & 1 deletion builtin/credential/cert/path_certs.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ import (
"github.com/hashicorp/vault/sdk/logical"
)

const defaultOCSPCacheSize = 100

func pathListCerts(b *backend) *framework.Path {
return &framework.Path{
Pattern: "certs/?",
Expand Down Expand Up @@ -47,7 +49,37 @@ Must be x509 PEM encoded.`,
EditType: "file",
},
},

"ocsp_enabled": {
cipherboy marked this conversation as resolved.
Show resolved Hide resolved
Type: framework.TypeBool,
Description: `Whether to attempt OCSP verification of certificates at login`,
},
"ocsp_ca_certificates": {
Type: framework.TypeString,
Description: `Any additional CA certificates needed to communicate with OCSP servers`,
DisplayAttrs: &framework.DisplayAttributes{
EditType: "file",
},
},
"ocsp_servers_override": {
Type: framework.TypeCommaStringSlice,
Description: `A comma-separated list of OCSP server addresses. If unset, the OCSP server is determined
from the AuthorityInformationAccess extension on the certificate being inspected.`,
},
"ocsp_fail_open": {
Type: framework.TypeBool,
Default: false,
Description: "If set to true, if an OCSP revocation cannot be made successfully, login will proceed rather. If false, failing to get an OCSP status fails the request.",
sgmiller marked this conversation as resolved.
Show resolved Hide resolved
},
"ocsp_query_all_servers": {
Type: framework.TypeBool,
Default: false,
Description: "If set to true, rather than accepting the first successful OCSP response, query all servers and consider the certificate valid only if all servers agree.",
},
"ocsp_cache_size": {
Type: framework.TypeInt,
Default: defaultOCSPCacheSize,
Description: "The size of the OCSP response cache.",
},
"allowed_names": {
Type: framework.TypeCommaStringSlice,
Description: `A comma-separated list of names.
Expand Down Expand Up @@ -294,6 +326,24 @@ func (b *backend) pathCertWrite(ctx context.Context, req *logical.Request, d *fr
if certificateRaw, ok := d.GetOk("certificate"); ok {
cert.Certificate = certificateRaw.(string)
}
if ocspCertificatesRaw, ok := d.GetOk("ocsp_ca_certificates"); ok {
cert.OcspCaCertificates = ocspCertificatesRaw.(string)
}
if ocspEnabledRaw, ok := d.GetOk("ocsp_enabled"); ok {
cert.OcspEnabled = ocspEnabledRaw.(bool)
}
if ocspServerOverrides, ok := d.GetOk("ocsp_servers_override"); ok {
cert.OcspServersOverride = ocspServerOverrides.([]string)
}
if ocspFailOpen, ok := d.GetOk("ocsp_fail_open"); ok {
cert.OcspFailOpen = ocspFailOpen.(bool)
}
if ocspQueryAll, ok := d.GetOk("ocsp_query_all_servers"); ok {
cert.OcspQueryAllServers = ocspQueryAll.(bool)
}
if ocspCacheSize, ok := d.GetOk("ocsp_cache_size"); ok {
cipherboy marked this conversation as resolved.
Show resolved Hide resolved
cert.OcspCacheSize = ocspCacheSize.(int)
}
cipherboy marked this conversation as resolved.
Show resolved Hide resolved
if displayNameRaw, ok := d.GetOk("display_name"); ok {
cert.DisplayName = displayNameRaw.(string)
}
Expand Down Expand Up @@ -424,6 +474,12 @@ type CertEntry struct {

Name string
Certificate string
OcspCaCertificates string
OcspEnabled bool
OcspServersOverride []string
OcspFailOpen bool
OcspCacheSize int
OcspQueryAllServers bool
DisplayName string
Policies []string
TTL time.Duration
Expand Down
5 changes: 4 additions & 1 deletion builtin/credential/cert/path_config.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,11 @@ func pathConfig(b *backend) *framework.Path {
func (b *backend) pathConfigWrite(ctx context.Context, req *logical.Request, data *framework.FieldData) (*logical.Response, error) {
disableBinding := data.Get("disable_binding").(bool)
enableIdentityAliasMetadata := data.Get("enable_identity_alias_metadata").(bool)

cacheSize := data.Get("ocsp_cache_size").(int)
entry, err := logical.StorageEntryJSON("config", config{
DisableBinding: disableBinding,
EnableIdentityAliasMetadata: enableIdentityAliasMetadata,
OcspCacheSize: cacheSize,
})
if err != nil {
return nil, err
Expand All @@ -46,6 +47,7 @@ func (b *backend) pathConfigWrite(ctx context.Context, req *logical.Request, dat
if err := req.Storage.Put(ctx, entry); err != nil {
return nil, err
}
b.initOCSPClient(cacheSize)
return nil, nil
}

Expand Down Expand Up @@ -85,4 +87,5 @@ func (b *backend) Config(ctx context.Context, s logical.Storage) (*config, error
type config struct {
DisableBinding bool `json:"disable_binding"`
EnableIdentityAliasMetadata bool `json:"enable_identity_alias_metadata"`
OcspCacheSize int `json:"ocsp_cache_size"`
}
72 changes: 60 additions & 12 deletions builtin/credential/cert/path_login.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (
"encoding/pem"
"errors"
"fmt"
"github.com/hashicorp/vault/sdk/helper/ocsp"
"strings"

"github.com/hashicorp/vault/sdk/framework"
Expand Down Expand Up @@ -224,8 +225,8 @@ func (b *backend) verifyCredentials(ctx context.Context, req *logical.Request, d
certName = d.Get("name").(string)
}

// Load the trusted certificates
roots, trusted, trustedNonCAs := b.loadTrustedCerts(ctx, req.Storage, certName)
// Load the trusted certificates and other details
roots, trusted, trustedNonCAs, verifyConf := b.loadTrustedCerts(ctx, req.Storage, certName)

// Get the list of full chains matching the connection and validates the
// certificate itself
Expand All @@ -234,16 +235,26 @@ func (b *backend) verifyCredentials(ctx context.Context, req *logical.Request, d
return nil, nil, err
}

var extraCas []*x509.Certificate
for _, t := range trusted {
extraCas = append(extraCas, t.Certificates...)
}

// If trustedNonCAs is not empty it means that client had registered a non-CA cert
// with the backend.
if len(trustedNonCAs) != 0 {
for _, trustedNonCA := range trustedNonCAs {
tCert := trustedNonCA.Certificates[0]
// Check for client cert being explicitly listed in the config (and matching other constraints)
if tCert.SerialNumber.Cmp(clientCert.SerialNumber) == 0 &&
bytes.Equal(tCert.AuthorityKeyId, clientCert.AuthorityKeyId) &&
b.matchesConstraints(clientCert, trustedNonCA.Certificates, trustedNonCA) {
return trustedNonCA, nil, nil
bytes.Equal(tCert.AuthorityKeyId, clientCert.AuthorityKeyId) {
matches, err := b.matchesConstraints(ctx, clientCert, trustedNonCA.Certificates, trustedNonCA, verifyConf)
if err != nil {
return nil, nil, err
}
if matches {
return trustedNonCA, nil, nil
}
}
}
}
Expand All @@ -260,10 +271,15 @@ func (b *backend) verifyCredentials(ctx context.Context, req *logical.Request, d
for _, tCert := range trust.Certificates { // For each certificate in the entry
for _, chain := range trustedChains { // For each root chain that we matched
for _, cCert := range chain { // For each cert in the matched chain
if tCert.Equal(cCert) && // ParsedCert intersects with matched chain
b.matchesConstraints(clientCert, chain, trust) { // validate client cert + matched chain against the config
// Add the match to the list
matches = append(matches, trust)
if tCert.Equal(cCert) { // ParsedCert intersects with matched chain
match, err := b.matchesConstraints(ctx, clientCert, chain, trust, verifyConf) // validate client cert + matched chain against the config
Copy link
Contributor

@cipherboy cipherboy Nov 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not quite sure this is correct down in the OCSP code, but its easier to explain here.

tCert comes via trusted off of the storage backend. Its order doesn't matter.

cCert comes via trustedChains off of the PeerCertificates:

func validateConnState(roots *x509.CertPool, cs *tls.ConnectionState) ([][]*x509.Certificate, error) {
certs := cs.PeerCertificates
if len(certs) == 0 {
return nil, nil
}
opts := x509.VerifyOptions{
Roots: roots,
Intermediates: x509.NewCertPool(),
KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth},
}
if len(certs) > 1 {
for _, cert := range certs[1:] {
opts.Intermediates.AddCert(cert)
}
}
chains, err := certs[0].Verify(opts)
if err != nil {
if _, ok := err.(x509.UnknownAuthorityError); ok {
return nil, nil
}
return nil, errors.New("failed to verify client's certificate: " + err.Error())
}
return chains, nil
}
via x509.Verify(...).

Per docs, trustedChains includes the leaf cert here, which shouldn't matter as we've hopefully excluded non-CA certs from chains before storage... But is in TLS order (leaf->[ICA->...]root)...

This is correct, as we might have some intermediates that aren't in trusted but are in trustedChains (confusing!)...

However, the call to checkForCertInOCSP is incorrect (and in particular, its [len(x) - 1] is incorrect!) -- because we're passing it trust coming from the TLS-sorted trustedChains. In particular, we're using the last cert in the chain, which isn't the leaf's issuer, but could be some arbitrary parent issuer. OCSP expects to get passed (cert, cert's issuer), not (cert, root) in particular.

This should fail verification, as the OCSP responder doesn't necessarily know about the ultimate Root CA this leaf is issued under, and may only know about the ICA that actually issued this leaf.

Hence I believe down there, given that trust includes the leaf in spot 0, the code should use chain[1].

To reproduce, I think if you use two separate CA mount points (not sure if you do in your tests), and don't tell the ICA mount about the root, you should see this fail OCSP with an error like not authoritative or unknown issuer or similar.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, great analysis. This was code I was very unsure about.

if err != nil {
return nil, nil, err
}
if match {
// Add the match to the list
matches = append(matches, trust)
}
}
}
}
Expand All @@ -279,15 +295,24 @@ func (b *backend) verifyCredentials(ctx context.Context, req *logical.Request, d
return matches[0], nil, nil
}

func (b *backend) matchesConstraints(clientCert *x509.Certificate, trustedChain []*x509.Certificate, config *ParsedCert) bool {
return !b.checkForChainInCRLs(trustedChain) &&
func (b *backend) matchesConstraints(ctx context.Context, clientCert *x509.Certificate, trustedChain []*x509.Certificate,
config *ParsedCert, conf *ocsp.VerifyConfig) (bool, error) {
soFar := !b.checkForChainInCRLs(trustedChain) &&
b.matchesNames(clientCert, config) &&
b.matchesCommonName(clientCert, config) &&
b.matchesDNSSANs(clientCert, config) &&
b.matchesEmailSANs(clientCert, config) &&
b.matchesURISANs(clientCert, config) &&
b.matchesOrganizationalUnits(clientCert, config) &&
b.matchesCertificateExtensions(clientCert, config)
if config.Entry.OcspEnabled {
ocspGood, err := b.checkForChainInOCSP(ctx, trustedChain, conf)
if err != nil {
return false, err
}
soFar = soFar && ocspGood
cipherboy marked this conversation as resolved.
Show resolved Hide resolved
}
return soFar, nil
}

// matchesNames verifies that the certificate matches at least one configured
Expand Down Expand Up @@ -478,7 +503,7 @@ func (b *backend) certificateExtensionsMetadata(clientCert *x509.Certificate, co
}

// loadTrustedCerts is used to load all the trusted certificates from the backend
func (b *backend) loadTrustedCerts(ctx context.Context, storage logical.Storage, certName string) (pool *x509.CertPool, trusted []*ParsedCert, trustedNonCAs []*ParsedCert) {
func (b *backend) loadTrustedCerts(ctx context.Context, storage logical.Storage, certName string) (pool *x509.CertPool, trusted []*ParsedCert, trustedNonCAs []*ParsedCert, conf *ocsp.VerifyConfig) {
pool = x509.NewCertPool()
trusted = make([]*ParsedCert, 0)
trustedNonCAs = make([]*ParsedCert, 0)
Expand All @@ -495,6 +520,7 @@ func (b *backend) loadTrustedCerts(ctx context.Context, storage logical.Storage,
}
}

conf = &ocsp.VerifyConfig{}
for _, name := range names {
entry, err := b.Cert(ctx, storage, strings.TrimPrefix(name, "cert/"))
if err != nil {
Expand All @@ -512,6 +538,8 @@ func (b *backend) loadTrustedCerts(ctx context.Context, storage logical.Storage,
b.Logger().Error("failed to parse certificate", "name", name)
continue
}
parsed = append(parsed, parsePEM([]byte(entry.OcspCaCertificates))...)

if !parsed[0].IsCA {
trustedNonCAs = append(trustedNonCAs, &ParsedCert{
Entry: entry,
Expand All @@ -528,10 +556,30 @@ func (b *backend) loadTrustedCerts(ctx context.Context, storage logical.Storage,
Certificates: parsed,
})
}
conf.OcspServersOverride = append(conf.OcspServersOverride, entry.OcspServersOverride...)
if entry.OcspFailOpen {
conf.OcspFailureMode = ocsp.FailOpenTrue
} else {
conf.OcspFailureMode = ocsp.FailOpenFalse
}
conf.QueryAllServers = conf.QueryAllServers || entry.OcspQueryAllServers
}
return
}

func (b *backend) checkForChainInOCSP(ctx context.Context, chain []*x509.Certificate, conf *ocsp.VerifyConfig) (bool, error) {
if b.ocspDisabled || len(chain) < 2 {
return true, nil
}
b.ocspClientMutex.RLock()
defer b.ocspClientMutex.RUnlock()
err := b.ocspClient.VerifyPeerCertificate(ctx, [][]*x509.Certificate{chain}, conf)
if err != nil {
return false, nil
}
return true, nil
}

func (b *backend) checkForChainInCRLs(chain []*x509.Certificate) bool {
badChain := false
for _, cert := range chain {
Expand Down
2 changes: 2 additions & 0 deletions sdk/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ require (
github.com/hashicorp/go-kms-wrapping/entropy v0.1.0
github.com/hashicorp/go-multierror v1.1.1
github.com/hashicorp/go-plugin v1.4.3
github.com/hashicorp/go-retryablehttp v0.5.3
github.com/hashicorp/go-secure-stdlib/base62 v0.1.1
github.com/hashicorp/go-secure-stdlib/mlock v0.1.1
github.com/hashicorp/go-secure-stdlib/parseutil v0.1.6
Expand Down Expand Up @@ -45,6 +46,7 @@ require (
github.com/fatih/color v1.7.0 // indirect
github.com/frankban/quicktest v1.10.0 // indirect
github.com/go-asn1-ber/asn1-ber v1.3.1 // indirect
github.com/hashicorp/go-cleanhttp v0.5.0 // indirect
github.com/hashicorp/yamux v0.0.0-20180604194846-3520598351bb // indirect
github.com/kr/text v0.2.0 // indirect
github.com/mattn/go-colorable v0.1.6 // indirect
Expand Down
2 changes: 2 additions & 0 deletions sdk/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFb
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=
github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/go-cleanhttp v0.5.0 h1:wvCrVc9TjDls6+YGAF2hAifE1E5U1+b4tH6KdvN3Gig=
github.com/hashicorp/go-cleanhttp v0.5.0/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80=
github.com/hashicorp/go-hclog v0.14.1/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39EeqMAyrmvFQ=
github.com/hashicorp/go-hclog v0.16.2 h1:K4ev2ib4LdQETX5cSZBG0DVLk1jwGqSPXBjdah3veNs=
Expand All @@ -102,6 +103,7 @@ github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+l
github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
github.com/hashicorp/go-plugin v1.4.3 h1:DXmvivbWD5qdiBts9TpBC7BYL1Aia5sxbRgQB+v6UZM=
github.com/hashicorp/go-plugin v1.4.3/go.mod h1:5fGEH17QVwTTcR0zV7yhDPLLmFX9YSZ38b18Udy6vYQ=
github.com/hashicorp/go-retryablehttp v0.5.3 h1:QlWt0KvWT0lq8MFppF9tsJGF+ynG7ztc2KIPhzRGk7s=
github.com/hashicorp/go-retryablehttp v0.5.3/go.mod h1:9B5zBasrRhHXnJnui7y6sL7es7NDiJgTc6Er0maI1Xs=
github.com/hashicorp/go-secure-stdlib/base62 v0.1.1 h1:6KMBnfEv0/kLAz0O76sliN5mXbCDcLfs2kP7ssP7+DQ=
github.com/hashicorp/go-secure-stdlib/base62 v0.1.1/go.mod h1:EdWO6czbmthiwZ3/PUsDV+UD1D5IRU4ActiaWGwt0Yw=
Expand Down
Loading