Implementation for minTTL #6808

hkiiita · 2024-11-13T17:29:50Z

This PR introduces a minTTL setting which would help address the problem of applications caching DNS response IPs indefinitely. Cluster administrators should be able to configure this value, ideally setting it to be equal to or greater than the maximum TTL value of the application's DNS cache.

This feature is a work towards resolving the issue of indefinite caching of DNS response IPs by certain applications.

Signed-off-by: Hemant <[email protected]>

antoninbas · 2024-11-13T21:09:08Z

pkg/config/agent/config.go

+	// The minTTL setting helps address the problem of applications caching DNS response IPs indefinitely.
+	// The Cluster administrators should configure this value, ideally setting it to be equal to or greater than the maximum TTL
+	// value of the application's DNS cache.
+	MinTTL uint32 `yaml:"minTTL,omitempty"`


We avoid unsigned integers in the config as far as I can tell, so maybe use int (personally I think int32 would be better, but we do use int consistently in the config apparently, so probably best to stick to int).

build/yamls/antrea.yml

Signed-off-by: Hemant <[email protected]>

antoninbas · 2024-11-14T18:45:21Z

cmd/antrea-agent/options.go

+	// If minTTL is greater than 1, it indicates that the value has been set externally by a cluster admin, and should be respected.
+	if o.config.MinTTL > 1 {


why 1 and not 0?

antoninbas · 2024-11-14T18:51:18Z

build/charts/antrea/conf/antrea-agent.conf

+# The minTTL setting helps address the problem of applications caching DNS response IPs indefinitely.
+# The Cluster administrators should configure this value, ideally setting it to be equal to or greater than the maximum TTL
+# value of the application's DNS cache.


I would remove "indefinitely". If the application does indeed cache the DNS entry forever, there is not much we can do.
We should also mention that this is for FQDN policy enforcement.
So maybe something like this:

The minTTL setting helps address the problem of applications caching DNS response IPs beyond the TTL value for the DNS record.
It is used to enforce FQDN policy rules, ensuring that resolved IPs are included in datapath rules for as long as the application is caching them.
This value should ideally be set to the maximum caching duration across all applications.

antoninbas · 2024-11-14T18:52:03Z

cmd/antrea-agent/options.go

@@ -90,6 +90,7 @@ type Options struct {
 	nplEndPort             int
 	dnsServerOverride      string
 	nodeType               config.NodeType
+	minTTL                 uint32


this field doesn't seem necessary?

antoninbas · 2024-11-14T18:53:37Z

pkg/agent/controller/networkpolicy/fqdn.go

+	getMaxTTL := func(ttl1, ttl2 uint32) uint32 {
+		if ttl1 > ttl2 {
+			return ttl1
+		} else {
+			return ttl2
+		}
+	}


there is a max built in in Golang: https://pkg.go.dev/builtin#max
It was introduced in Go 1.21

antoninbas · 2024-11-14T18:59:02Z

pkg/agent/controller/networkpolicy/fqdn.go

 	currentTime := f.clock.Now()
 	for _, ans := range msg.Answer {
 		switch r := ans.(type) {
 		case *dns.A:
 			if f.ipv4Enabled {
 				responseIPs[r.A.String()] = ipWithExpiration{
 					ip:             r.A,
-					expirationTime: currentTime.Add(time.Duration(r.Header().Ttl) * time.Second),
+					expirationTime: currentTime.Add(time.Duration(getMaxTTL(f.minTTL, r.Header().Ttl)) * time.Second),


I feel like technically, minTTL should only apply to DNS responses which were not initiated by Antrea, but intercepted by Antrea (responses to DNS queries generated by the application). However, this code applies to responses to DNS queries sent by Antrea (when an override DNS server is configured). Maybe we need to introduce a flag to distinguish between the 2 cases?

cc @tnqn @Dyanngg

We actually spent the length of last syncup meeting to discuss this, because I had the same suggestion but @tnqn’s opinion was that we don’t have to differentiate between the two cases. For security purpose we advise users to use FQDN rules only in allowlists for Antrea-native policies, and he thinks it’s okay that the clients’ TTL for a FQDN goes “out of sync” with the antrea agent since that’s not the gaurentee we want: we only want to enforce that client cannot access unintended addresses. So having an address for a domain which has longer TTL in antrea cache compared to the client is ok. I’ll let Quan chime in to see if I’m summarizing this correctly, but the end result of the discussion was we told Hemant to not worry about differentiating Antrea and client initiated dns queries

For security purpose we advise users to use FQDN rules only in allowlists for Antrea-native policies

Got it, makes sense to me

@hkiiita please ignore this comment

- Remove the redundant minTTL field from Options struct. - Use in built max function for comparison of max TTL values. - Improve descriptive comment about minTTL in config file. Signed-off-by: Hemant <[email protected]>

Signed-off-by: Hemant <[email protected]>

antoninbas · 2024-11-15T23:57:58Z

build/charts/antrea/conf/antrea-agent.conf

+# The minTTL setting helps address the problem of applications caching DNS response IPs beyond the TTL value for the DNS record.
+# It is used to enforce FQDN policy rules, ensuring that resolved IPs are included in datapath rules for as long as the application is caching them.
+# This value should ideally be set to the maximum caching duration across all applications.
+minTTL: {{ .Values.minTTL }}


the name is a bit generic, maybe we should go with fqdnCacheMinTTL?

antoninbas · 2024-11-16T00:00:45Z

cmd/antrea-agent/options.go

+	// Ensure that the minTTL is not negative.
+	o.config.MinTTL = max(o.config.MinTTL, 0)


it would be better to return an error if o.config.MinTTL < 0 IMO

while this code is in validateK8sNodeOptions, I am not sure this configuration parameter is specific to the "K8sNode" case. Maybe the check should be in the parent function? cc @tnqn

antoninbas · 2024-11-16T00:01:41Z

pkg/agent/controller/networkpolicy/fqdn.go

@@ -160,7 +161,7 @@ type fqdnController struct {
 	clock clock.Clock
 }

-func newFQDNController(client openflow.Client, allocator *idAllocator, dnsServerOverride string, dirtyRuleHandler func(string), v4Enabled, v6Enabled bool, gwPort uint32, clock clock.WithTicker) (*fqdnController, error) {
+func newFQDNController(client openflow.Client, allocator *idAllocator, dnsServerOverride string, dirtyRuleHandler func(string), v4Enabled, v6Enabled bool, gwPort uint32, clock clock.WithTicker, minTTL int) (*fqdnController, error) {


I wonder if minTTL should be uint32 here. This way we know it is a positive number. The conversion from int to uint32 can happen in cmd/antrea-agent.

Signed-off-by: Hemant <[email protected]>

antoninbas · 2024-11-18T19:04:29Z

build/charts/antrea/conf/antrea-agent.conf

 # It is used to enforce FQDN policy rules, ensuring that resolved IPs are included in datapath rules for as long as the application is caching them.
 # This value should ideally be set to the maximum caching duration across all applications.
-minTTL: {{ .Values.minTTL }}
+fqdnCacheMinTTL: {{ .Values.minTTL }}


Suggested change

fqdnCacheMinTTL: {{ .Values.minTTL }}

fqdnCacheMinTTL: {{ .Values.fqdnCacheMinTTL }}

This is why the manifests are not generated correctly (fqdnCacheMinTTL: instead of fqdnCacheMinTTL: 0)

Sorry , my bad. Will correct that.

antoninbas · 2024-11-18T19:05:08Z

cmd/antrea-agent/options.go

+	switch o.config.NodeType {
+	case config.ExternalNode.String():
 		o.nodeType = config.ExternalNode
 		return o.validateExternalNodeOptions()
-	} else if o.config.NodeType == config.K8sNode.String() {
+	case config.K8sNode.String():
 		o.nodeType = config.K8sNode
 		return o.validateK8sNodeOptions()
-	} else {
+	default:
 		return fmt.Errorf("unsupported nodeType %s", o.config.NodeType)


this is not a bad change, but I would avoid doing it in this PR as it is unrelated

antoninbas · 2024-11-18T19:06:25Z

cmd/antrea-agent/options.go

-	if o.config.NodeType == config.ExternalNode.String() {
+	// validate FqdnCacheMinTTL
+	if o.config.FqdnCacheMinTTL < 0 {
+		return fmt.Errorf("fqdnCacheMinTTL set to an invalid value, its must be a positive integer")


Suggested change

return fmt.Errorf("fqdnCacheMinTTL set to an invalid value, its must be a positive integer")

return fmt.Errorf("fqdnCacheMinTTL must be greater than or equal to 0")

antoninbas · 2024-11-18T19:07:13Z

pkg/config/agent/config.go

@@ -158,7 +158,7 @@ type AgentConfig struct {
 	// The minTTL setting helps address the problem of applications caching DNS response IPs indefinitely.
 	// The Cluster administrators should configure this value, ideally setting it to be equal to or greater than the maximum TTL
 	// value of the application's DNS cache.
-	MinTTL int `yaml:"minTTL,omitempty"`
+	FqdnCacheMinTTL int `yaml:"fqdnCacheMinTTL,omitempty"`


I think the field name here should be FQDNCacheMinTTL per our conventions

Dyanngg

I think we should also add unit test cases to verify that with fqdnController with minTTL set will set correct dnsEntry expiration times in cache

… per review. Signed-off-by: Hemant <[email protected]>

Signed-off-by: Hemant <[email protected]>

antoninbas · 2024-11-21T19:27:24Z

build/charts/antrea/conf/antrea-agent.conf

@@ -306,6 +306,11 @@ kubeAPIServerOverride: {{ .Values.kubeAPIServerOverride | quote }}
 # 10.96.0.10:53, [fd00:10:96::a]:53).
 dnsServerOverride: {{ .Values.dnsServerOverride | quote }}

+# The fqdnCacheMinTTL setting helps address the problem of applications caching DNS response IPs beyond the TTL value for the DNS record.


s/The fqdnCacheMinTTL setting helps address/fqdnCacheMinTTL helps address

antoninbas · 2024-11-21T19:43:22Z

pkg/agent/controller/networkpolicy/fqdn_test.go

@@ -744,3 +747,140 @@ func TestOnDNSResponse(t *testing.T) {
 		})
 	}
 }
+func TestFQDNCacheMinTTL(t *testing.T) {


I feel like in theory we only need to test parseDNSResponse, because this is the only place where minTTL is actually used. We just need to make sure that minTTL can override the TTL included in the DNS response. cc @Dyanngg

However, I don't feel super strongly about it, so if others think it is better to test onDNSResponseMsg "end-to-end", it is fine by me.

Signed-off-by: Hemant <[email protected]>

initial commit for implementing minTTL

346988f

Signed-off-by: Hemant <[email protected]>

antoninbas reviewed Nov 13, 2024

View reviewed changes

auto generated antrea yml files

66e72f1

Signed-off-by: Hemant <[email protected]>

antoninbas reviewed Nov 14, 2024

View reviewed changes

hkiiita added 3 commits November 15, 2024 22:08

Refactored minTTL confgiyration and use.

d92246f

- Remove the redundant minTTL field from Options struct. - Use in built max function for comparison of max TTL values. - Improve descriptive comment about minTTL in config file. Signed-off-by: Hemant <[email protected]>

Merge branch 'antrea-io:main' into fqdn-cache-minttl

2c58ad2

update final assertion and comment for the test.

e9e0286

Signed-off-by: Hemant <[email protected]>

antoninbas reviewed Nov 16, 2024

View reviewed changes

move minTTL validation to parent function and rename to fqdnCacheMinTTL

dd43d6b

Signed-off-by: Hemant <[email protected]>

antoninbas reviewed Nov 18, 2024

View reviewed changes

Dyanngg reviewed Nov 18, 2024

View reviewed changes

hkiiita added 2 commits November 21, 2024 23:05

add tests for FQDNCacheMinTTL functionality and made other changes as…

7de5ee7

… per review. Signed-off-by: Hemant <[email protected]>

re-grouped imports

fb28bba

Signed-off-by: Hemant <[email protected]>

antoninbas reviewed Nov 21, 2024

View reviewed changes

refactor to test just the parseDNSResponse

6c6dcc8

Signed-off-by: Hemant <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementation for minTTL #6808

Implementation for minTTL #6808

hkiiita commented Nov 13, 2024

antoninbas Nov 13, 2024

antoninbas Nov 14, 2024

antoninbas Nov 14, 2024

antoninbas Nov 14, 2024

antoninbas Nov 14, 2024

antoninbas Nov 14, 2024

Dyanngg Nov 14, 2024

antoninbas Nov 14, 2024

antoninbas Nov 15, 2024

antoninbas Nov 16, 2024

antoninbas Nov 16, 2024

antoninbas Nov 18, 2024

hkiiita Nov 19, 2024

antoninbas Nov 18, 2024

antoninbas Nov 18, 2024

antoninbas Nov 18, 2024

Dyanngg left a comment

antoninbas Nov 21, 2024

antoninbas Nov 21, 2024

		// If minTTL is greater than 1, it indicates that the value has been set externally by a cluster admin, and should be respected.
		if o.config.MinTTL > 1 {

		// Ensure that the minTTL is not negative.
		o.config.MinTTL = max(o.config.MinTTL, 0)

	fqdnCacheMinTTL: {{ .Values.minTTL }}
	fqdnCacheMinTTL: {{ .Values.fqdnCacheMinTTL }}

	return fmt.Errorf("fqdnCacheMinTTL set to an invalid value, its must be a positive integer")
	return fmt.Errorf("fqdnCacheMinTTL must be greater than or equal to 0")

Implementation for minTTL #6808

Are you sure you want to change the base?

Implementation for minTTL #6808

Conversation

hkiiita commented Nov 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dyanngg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment