High CPU and disk read/write, very large (2GB) state.db on k3s 1.22.9

[parkerfath]

We have many installations (around 200+) of k3s running on 4-CPU x86 hosts (single-node clusters) running Ubuntu 20.x, mostly at customer sites. Recently, one of the customers has been complaining of high CPU usage (around 80% consistently). I dove into it for a couple weeks, and have landed on some key symptoms:

1. High CPU, as mentioned. Also, running uptime yields load average typically around 8-9, vs. other customers' systems which usually sit between about 1.0 and 3.0. k3s as highest CPU user per top
2. Very high disk usage per dstat -d - reads of 30-60MB/s consistently, and write between 3-20MB consistently, vs. other systems consistently under 1 MB/s. k3s as highest disk user per top
3. /var/lib/rancher/k3s/server/db/state.db is > 2GB, vs. < 50 MB on other systems.
4. In journalctl -e -u k3s most of the logs are Trace, mixed with some errors such as "Compact failed" and "context deadline exceeded" (more detail on logs below)

I tried the vacuum steps mentioned in #1575 (comment) , which brought it from about 2.7GB down to 2.1GB, but otherwise didn't seem to benefit much.

I realize 1.22.9 is a few years old, but I'm hoping we can get it fixed without upgrading given the number of installations in the field, and I'm also hoping maybe this is just something simple I haven't realized yet.

Environmental Info:
K3s Version:
k3s version v1.22.9+k3s1 (8b0b50a)
go version go1.16.10

Node(s) CPU architecture, OS, and Version:
Linux QX7500B5 5.4.0-182-generic #202-Ubuntu SMP Fri Apr 26 12:29:36 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
Single node

Describe the bug:
See above

Steps To Reproduce:
Unsure exact steps. I don't think we're doing too much crazy with the install/config of k3s, but LMK if there is anything relevant to this issue and I can include details.

Expected behavior:
System should run at a reasonable CPU utilization level (e.g. 20-30%) and load average of 2-3. Disk usage per dstat -d under 1MB/s most of the time. Reasonably unpolluted system logs.

Actual behavior:
See above

Additional context / logs:
From journalctl -e -u k3s:

Jun 06 21:45:38 hostnameredacted k3s[11771]: I0606 21:45:38.893725   11771 trace.go:205] Trace[636222759]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/hostnameredacted,user-agent:k3s/v1.22.9+k3s1 (linux/amd64) kubernetes/8b0b50a,audit-id:dc061c09-b316-4d84-8bec-5a44affb1a75,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/1.1 (06-Jun-2024 21:45:37.907) (total time: 986ms):
Jun 06 21:45:38 hostnameredacted k3s[11771]: Trace[636222759]: ---"Object stored in database" 985ms (21:45:38.893)
Jun 06 21:45:38 hostnameredacted k3s[11771]: Trace[636222759]: [986.213986ms] [986.213986ms] END
Jun 06 21:45:39 hostnameredacted k3s[11771]: I0606 21:45:39.002407   11771 trace.go:205] Trace[920478471]: "Get" url:/api/v1/namespaces/ingress-nginx/configmaps/ingress-controller-leader,user-agent:nginx-ingress-controller/v1.1.3 (linux/amd64) ingress-nginx/9d3a285f19a704524439c75b947e2189406565ab,audit-id:c23eb4c6-63e2-4023-b0a3-6f4ab7250f53,client:10.42.0.31,accept:application/json, */*,protocol:HTTP/1.1 (06-Jun-2024 21:45:38.487) (total time: 515ms):
Jun 06 21:45:39 hostnameredacted k3s[11771]: Trace[920478471]: ---"About to write a response" 515ms (21:45:39.002)
Jun 06 21:45:39 hostnameredacted k3s[11771]: Trace[920478471]: [515.265455ms] [515.265455ms] END
Jun 06 21:45:45 hostnameredacted k3s[11771]: I0606 21:45:45.104616   11771 trace.go:205] Trace[1785314302]: "GuaranteedUpdate etcd3" type:*v1.Endpoints (06-Jun-2024 21:45:44.261) (total time: 842ms):
Jun 06 21:45:45 hostnameredacted k3s[11771]: Trace[1785314302]: ---"initial value restored" 393ms (21:45:44.654)
Jun 06 21:45:45 hostnameredacted k3s[11771]: Trace[1785314302]: ---"Transaction committed" 449ms (21:45:45.104)
Jun 06 21:45:45 hostnameredacted k3s[11771]: Trace[1785314302]: [842.863573ms] [842.863573ms] END

From journalctl -e u k3s | grep -v Trace:

Jun 06 21:41:36 hostnameredacted k3s[11771]: time="2024-06-06T21:41:36+05:30" level=error msg="error in txn: context deadline exceeded"
Jun 06 21:41:40 hostnameredacted k3s[11771]: time="2024-06-06T21:41:40+05:30" level=error msg="Compact failed: failed to record compact revision: sql: transaction has already been committed or rolled back"
Jun 06 21:41:40 hostnameredacted k3s[11771]: time="2024-06-06T21:41:40+05:30" level=error msg="error in txn: context canceled"
Jun 06 21:46:09 hostnameredacted k3s[11771]: E0606 21:46:09.074134   11771 status.go:71] apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded
Jun 06 21:46:09 hostnameredacted k3s[11771]: time="2024-06-06T21:46:09+05:30" level=error msg="error while range on /registry/events/qumulex/agent-66c685c88-qz6rq.17c4f6b6012a0980\x00 /registry/events/qumulex/: context deadline exceeded"
Jun 06 21:46:09 hostnameredacted k3s[11771]: E0606 21:46:09.087808   11771 writers.go:117] apiserver was unable to write a JSON response: http: Handler timeout
Jun 06 21:46:09 hostnameredacted k3s[11771]: E0606 21:46:09.088341   11771 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
Jun 06 21:46:09 hostnameredacted k3s[11771]: E0606 21:46:09.091877   11771 writers.go:130] apiserver was unable to write a fallback JSON response: http: Handler timeout
Jun 06 21:46:09 hostnameredacted k3s[11771]: E0606 21:46:09.092988   11771 timeout.go:139] post-timeout activity - time-elapsed: 4.684528ms, GET "/api/v1/namespaces/qumulex/events" result: <nil>
Jun 06 21:46:32 hostnameredacted k3s[11771]: time="2024-06-06T21:46:32+05:30" level=error msg="Compact failed: failed to record compact revision: sql: transaction has already been committed or rolled back"

dstat -d:

root@hostnameredacted:/# dstat -d
-dsk/total-
 read  writ
  38M   18M
 191M 2668k
  73M  900k
  90M 1232k
  72M 1024k
  53M   74M
  71M 1164k
  68M 1692k
  33M   73M
  41M 1164k
  63M 2256k

[brandond]

As you noted, v1.22.9 is a quite old release. There have been a number of improvements to the compact logic made in newer releases, so I suspect that things might be improved by newer versions. However, the messages you're seeing indicate that a combination of slow disk IO and high datastore write volume has filled to the database to the point where compaction can no longer run in the allotted time, and old rows are not being purged.

Follow the steps from this comment to figure out what's filling your database: #1575 (comment) - if you're able to share that data here, that would be great.

You can then run this query to manually compact the old rows:

delete from kine where id in (select id  from (select id,name from kine where id NOT IN (select max(id) as  id from kine group by name)) as t);

Note that you should do all of these things while k3s is not running.

[parkerfath]

Thanks for your help!

I was able to do the manual compaction, and while it didn't reduce the file much (still 2.1GB), the CPU load shown by uptime is better as is the read/write volume shown by dstat -d

As for the analysis output, here is what it yielded:

sqlite3_analyze:

/** Disk-Space Utilization Report For hostnameredacted-state.db

Page size in bytes................................ 4096      
Pages in the whole file (measured)................ 538148    
Pages in the whole file (calculated).............. 538148    
Pages that store data............................. 174888      32.5% 
Pages on the freelist (per header)................ 363259      67.5% 
Pages on the freelist (calculated)................ 363259      67.5% 
Pages of auto-vacuum overhead..................... 0            0.0% 
Number of tables in the database.................. 3         
Number of indices................................. 5         
Number of defined indices......................... 5         
Number of implied indices......................... 0         
Size of the file in bytes......................... 2204254208
Bytes of user payload stored...................... 469281206   21.3% 

*** Page counts for all tables with their indices *****************************

KINE.............................................. 174886      32.5% 
SQLITE_SCHEMA..................................... 1            0.0% 
SQLITE_SEQUENCE................................... 1            0.0% 

*** Page counts for all tables and indices separately *************************

KINE.............................................. 130155      24.2% 
KINE_NAME_ID_INDEX................................ 14108        2.6% 
KINE_NAME_PREV_REVISION_UINDEX.................... 13536        2.5% 
KINE_NAME_INDEX................................... 13234        2.5% 
KINE_ID_DELETED_INDEX............................. 2252         0.42% 
KINE_PREV_REVISION_INDEX.......................... 1601         0.30% 
SQLITE_SCHEMA..................................... 1            0.0% 
SQLITE_SEQUENCE................................... 1            0.0% 

*** All tables and indices ****************************************************

Percentage of total database......................  32.5%    
Number of entries................................. 3659576   
Bytes of storage consumed......................... 716341248 
Bytes of payload.................................. 615612477   85.9% 
Bytes of metadata................................. 16899315     2.4% 
Average payload per entry......................... 168.22    
Average unused bytes per entry.................... 22.91     
Average metadata per entry........................ 4.62      
Average fanout.................................... 115.00    
Maximum payload per entry......................... 33353     
Entries that use overflow......................... 241          0.007% 
Index pages used.................................. 1510      
Primary pages used................................ 172788    
Overflow pages used............................... 590       
Total pages used.................................. 174888    
Unused bytes on index pages....................... 1493773     24.2% 
Unused bytes on primary pages..................... 82317694    11.6% 
Unused bytes on overflow pages.................... 19385        0.80% 
Unused bytes on all pages......................... 83830852    11.7% 

*** All tables ****************************************************************

Percentage of total database......................  24.2%    
Number of entries................................. 609936    
Bytes of storage consumed......................... 533123072 
Bytes of payload.................................. 469282024   88.0% 
Bytes of metadata................................. 7213643      1.4% 
Average payload per entry......................... 769.40    
Average unused bytes per entry.................... 92.84     
Average metadata per entry........................ 11.83     
Average fanout.................................... 356.00    
Maximum payload per entry......................... 33353     
Entries that use overflow......................... 241          0.040% 
Index pages used.................................. 363       
Primary pages used................................ 129204    
Overflow pages used............................... 590       
Total pages used.................................. 130157    
Unused bytes on index pages....................... 190629      12.8% 
Unused bytes on primary pages..................... 56418787    10.7% 
Unused bytes on overflow pages.................... 19385        0.80% 
Unused bytes on all pages......................... 56628801    10.6% 

*** All indices ***************************************************************

Percentage of total database......................   8.3%    
Number of entries................................. 3049640   
Bytes of storage consumed......................... 183218176 
Bytes of payload.................................. 146330453   79.9% 
Bytes of metadata................................. 9685672      5.3% 
Average payload per entry......................... 47.98     
Average unused bytes per entry.................... 8.92      
Average metadata per entry........................ 3.18      
Average fanout.................................... 38.00     
Maximum payload per entry......................... 101       
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 1147      
Primary pages used................................ 43584     
Overflow pages used............................... 0         
Total pages used.................................. 44731     
Unused bytes on index pages....................... 1303144     27.7% 
Unused bytes on primary pages..................... 25898907    14.5% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 27202051    14.8% 

*** Table KINE and all its indices ********************************************

Percentage of total database......................  32.5%    
Number of entries................................. 3659568   
Bytes of storage consumed......................... 716333056 
Bytes of payload.................................. 615611648   85.9% 
Bytes of metadata................................. 16899166     2.4% 
Average payload per entry......................... 168.22    
Average unused bytes per entry.................... 22.91     
Average metadata per entry........................ 4.62      
Average fanout.................................... 115.00    
Maximum payload per entry......................... 33353     
Entries that use overflow......................... 241          0.007% 
Index pages used.................................. 1510      
Primary pages used................................ 172786    
Overflow pages used............................... 590       
Total pages used.................................. 174886    
Unused bytes on index pages....................... 1493773     24.2% 
Unused bytes on primary pages..................... 82310480    11.6% 
Unused bytes on overflow pages.................... 19385        0.80% 
Unused bytes on all pages......................... 83823638    11.7% 

*** Table KINE w/o any indices ************************************************

Percentage of total database......................  24.2%    
Number of entries................................. 609928    
Bytes of storage consumed......................... 533114880 
Bytes of payload.................................. 469281195   88.0% 
Bytes of metadata................................. 7213494      1.4% 
B-tree depth...................................... 3         
Average payload per entry......................... 769.40    
Average unused bytes per entry.................... 92.83     
Average metadata per entry........................ 11.83     
Average fanout.................................... 356.00    
Non-sequential pages.............................. 647          0.50% 
Maximum payload per entry......................... 33353     
Entries that use overflow......................... 241          0.040% 
Index pages used.................................. 363       
Primary pages used................................ 129202    
Overflow pages used............................... 590       
Total pages used.................................. 130155    
Unused bytes on index pages....................... 190629      12.8% 
Unused bytes on primary pages..................... 56411573    10.7% 
Unused bytes on overflow pages.................... 19385        0.80% 
Unused bytes on all pages......................... 56621587    10.6% 

*** Indices of table KINE *****************************************************

Percentage of total database......................   8.3%    
Number of entries................................. 3049640   
Bytes of storage consumed......................... 183218176 
Bytes of payload.................................. 146330453   79.9% 
Bytes of metadata................................. 9685672      5.3% 
Average payload per entry......................... 47.98     
Average unused bytes per entry.................... 8.92      
Average metadata per entry........................ 3.18      
Average fanout.................................... 38.00     
Maximum payload per entry......................... 101       
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 1147      
Primary pages used................................ 43584     
Overflow pages used............................... 0         
Total pages used.................................. 44731     
Unused bytes on index pages....................... 1303144     27.7% 
Unused bytes on primary pages..................... 25898907    14.5% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 27202051    14.8% 

*** Index KINE_ID_DELETED_INDEX of table KINE *********************************

Percentage of total database......................   0.42%   
Number of entries................................. 609928    
Bytes of storage consumed......................... 9224192   
Bytes of payload.................................. 7318050     79.3% 
Bytes of metadata................................. 1856804     20.1% 
B-tree depth...................................... 3         
Average payload per entry......................... 12.00     
Average unused bytes per entry.................... 0.08      
Average metadata per entry........................ 3.04      
Average fanout.................................... 187.00    
Non-sequential pages.............................. 15           0.67% 
Maximum payload per entry......................... 12        
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 12        
Primary pages used................................ 2240      
Overflow pages used............................... 0         
Total pages used.................................. 2252      
Unused bytes on index pages....................... 6467        13.2% 
Unused bytes on primary pages..................... 42871        0.47% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 49338        0.53% 

*** Index KINE_NAME_ID_INDEX of table KINE ************************************

Percentage of total database......................   2.6%    
Number of entries................................. 609928    
Bytes of storage consumed......................... 57786368  
Bytes of payload.................................. 46517180    80.5% 
Bytes of metadata................................. 1999076      3.5% 
B-tree depth...................................... 4         
Average payload per entry......................... 76.27     
Average unused bytes per entry.................... 15.20     
Average metadata per entry........................ 3.28      
Average fanout.................................... 35.00     
Non-sequential pages.............................. 1741        12.3% 
Maximum payload per entry......................... 101       
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 401       
Primary pages used................................ 13707     
Overflow pages used............................... 0         
Total pages used.................................. 14108     
Unused bytes on index pages....................... 458049      27.9% 
Unused bytes on primary pages..................... 8812063     15.7% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 9270112     16.0% 

*** Index KINE_NAME_INDEX of table KINE ***************************************

Percentage of total database......................   2.5%    
Number of entries................................. 609928    
Bytes of storage consumed......................... 54206464  
Bytes of payload.................................. 43468083    80.2% 
Bytes of metadata................................. 1988588      3.7% 
B-tree depth...................................... 4         
Average payload per entry......................... 71.27     
Average unused bytes per entry.................... 14.35     
Average metadata per entry........................ 3.26      
Average fanout.................................... 37.00     
Non-sequential pages.............................. 1589        12.0% 
Maximum payload per entry......................... 96        
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 352       
Primary pages used................................ 12882     
Overflow pages used............................... 0         
Total pages used.................................. 13234     
Unused bytes on index pages....................... 391318      27.1% 
Unused bytes on primary pages..................... 8358475     15.8% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 8749793     16.1% 

*** Index KINE_NAME_PREV_REVISION_UINDEX of table KINE ************************

Percentage of total database......................   2.5%    
Number of entries................................. 609928    
Bytes of storage consumed......................... 55443456  
Bytes of payload.................................. 44418099    80.1% 
Bytes of metadata................................. 1992212      3.6% 
B-tree depth...................................... 4         
Average payload per entry......................... 72.83     
Average unused bytes per entry.................... 14.81     
Average metadata per entry........................ 3.27      
Average fanout.................................... 36.00     
Non-sequential pages.............................. 1723        12.7% 
Maximum payload per entry......................... 101       
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 374       
Primary pages used................................ 13162     
Overflow pages used............................... 0         
Total pages used.................................. 13536     
Unused bytes on index pages....................... 438150      28.6% 
Unused bytes on primary pages..................... 8594995     15.9% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 9033145     16.3% 

*** Index KINE_PREV_REVISION_INDEX of table KINE ******************************

Percentage of total database......................   0.30%   
Number of entries................................. 609928    
Bytes of storage consumed......................... 6557696   
Bytes of payload.................................. 4609041     70.3% 
Bytes of metadata................................. 1848992     28.2% 
B-tree depth...................................... 3         
Average payload per entry......................... 7.56      
Average unused bytes per entry.................... 0.16      
Average metadata per entry........................ 3.03      
Average fanout.................................... 200.00    
Non-sequential pages.............................. 30           1.9% 
Maximum payload per entry......................... 11        
Entries that use overflow......................... 0            0.0% 
Index pages used.................................. 8         
Primary pages used................................ 1593      
Overflow pages used............................... 0         
Total pages used.................................. 1601      
Unused bytes on index pages....................... 9160        28.0% 
Unused bytes on primary pages..................... 90503        1.4% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 99663        1.5% 

*** Table SQLITE_SCHEMA *******************************************************

Percentage of total database......................   0.0%    
Number of entries................................. 7         
Bytes of storage consumed......................... 4096      
Bytes of payload.................................. 818         20.0% 
Bytes of metadata................................. 137          3.3% 
B-tree depth...................................... 1         
Average payload per entry......................... 116.86    
Average unused bytes per entry.................... 448.71    
Average metadata per entry........................ 19.57     
Maximum payload per entry......................... 260       
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 3141        76.7% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 3141        76.7% 

*** Table SQLITE_SEQUENCE *****************************************************

Percentage of total database......................   0.0%    
Number of entries................................. 1         
Bytes of storage consumed......................... 4096      
Bytes of payload.................................. 11           0.27% 
Bytes of metadata................................. 12           0.29% 
B-tree depth...................................... 1         
Average payload per entry......................... 11.00     
Average unused bytes per entry.................... 4073.00   
Average metadata per entry........................ 12.00     
Maximum payload per entry......................... 11        
Entries that use overflow......................... 0            0.0% 
Primary pages used................................ 1         
Overflow pages used............................... 0         
Total pages used.................................. 1         
Unused bytes on primary pages..................... 4073        99.44% 
Unused bytes on overflow pages.................... 0         
Unused bytes on all pages......................... 4073        99.44% 

*** Definitions ***************************************************************

(glossary omitted by Parker)

*******************************************************************************
The entire text of this report can be sourced into any SQL database
engine for further analysis.  All of the text above is an SQL comment.
The data used to generate this report follows:
*/
BEGIN;
CREATE TABLE space_used(
   name clob,        -- Name of a table or index in the database file
   tblname clob,     -- Name of associated table
   is_index boolean, -- TRUE if it is an index, false for a table
   is_without_rowid boolean, -- TRUE if WITHOUT ROWID table  
   nentry int,       -- Number of entries in the BTree
   leaf_entries int, -- Number of leaf entries
   depth int,        -- Depth of the b-tree
   payload int,      -- Total amount of data stored in this table or index
   ovfl_payload int, -- Total amount of data stored on overflow pages
   ovfl_cnt int,     -- Number of entries that use overflow
   mx_payload int,   -- Maximum payload size
   int_pages int,    -- Number of interior pages used
   leaf_pages int,   -- Number of leaf pages used
   ovfl_pages int,   -- Number of overflow pages used
   int_unused int,   -- Number of unused bytes on interior pages
   leaf_unused int,  -- Number of unused bytes on primary pages
   ovfl_unused int,  -- Number of unused bytes on overflow pages
   gap_cnt int,      -- Number of gaps in the page layout
   compressed_size int  -- Total bytes stored on disk
);
INSERT INTO space_used VALUES('sqlite_schema','sqlite_schema',0,0,7,7,1,818,0,0,260,0,1,0,0,3141,0,0,4096);
INSERT INTO space_used VALUES('kine','kine',0,0,739129,609928,3,469281195,2394895,241,33353,363,129202,590,190629,56411573,19385,647,533114880);
INSERT INTO space_used VALUES('sqlite_sequence','sqlite_sequence',0,0,1,1,1,11,0,0,11,0,1,0,0,4073,0,0,4096);
INSERT INTO space_used VALUES('kine_name_index','kine',1,0,609928,597047,4,43468083,0,0,96,352,12882,0,391318,8358475,0,1589,54206464);
INSERT INTO space_used VALUES('kine_name_prev_revision_uindex','kine',1,0,609928,596767,4,44418099,0,0,101,374,13162,0,438150,8594995,0,1723,55443456);
INSERT INTO space_used VALUES('kine_name_id_index','kine',1,0,609928,596222,4,46517180,0,0,101,401,13707,0,458049,8812063,0,1741,57786368);
INSERT INTO space_used VALUES('kine_id_deleted_index','kine',1,0,609928,607689,3,7318050,0,0,12,12,2240,0,6467,42871,0,15,9224192);
INSERT INTO space_used VALUES('kine_prev_revision_index','kine',1,0,609928,608336,3,4609041,0,0,11,8,1593,0,9160,90503,0,30,6557696);
COMMIT;

SQLite compact_rev_key stats:

pfath@pfath-sqlite-analyze:~$ ./sqlite3 hostnameredacted-state.db << EOF
.header on
.mode column
.dbinfo

SELECT 
  COUNT(*) AS rows, 
  (SELECT prev_revision FROM kine WHERE name = 'compact_rev_key' LIMIT 1) AS compact_rev,
  MAX(id) AS current_rev
FROM kine;

SELECT COUNT(*), name FROM kine GROUP BY name ORDER BY COUNT(*) DESC LIMIT 50;

EOF
database page size:  4096
write format:        2
read format:         2
reserved bytes:      0
file change counter: 12339922
database page count: 538148
freelist page count: 363259
schema cookie:       8
schema format:       4
default cache size:  0
autovacuum top root: 0
incremental vacuum:  0
text encoding:       1 (utf8)
user version:        0
application id:      0
software version:    3034001
number of tables:    2
number of indexes:   5
number of triggers:  0
number of views:     0
schema size:         565
data version         2
rows    compact_rev  current_rev
------  -----------  -----------
609928  33620222     33621222   
COUNT(*)  name                                                        
--------  ------------------------------------------------------------
176       /registry/configmaps/ingress-nginx/ingress-controller-leader
142       /registry/masterleases/192.168.1.175                        
134       /registry/leases/kube-node-lease/hostnameredacted                   
6         /registry/minions/hostnameredacted                                  
2         /registry/events/default/hostnameredacted.17d665f34206e6fc          
2         /registry/events/default/hostnameredacted.17d665f344ddee21          
2         /registry/events/default/hostnameredacted.17d665f34ba66b39          
2         /registry/events/default/hostnameredacted.17d665f34ba69c8a          
2         /registry/events/default/hostnameredacted.17d665f34ba6b2ed          
2         /registry/events/default/hostnameredacted.17d665f3a6246bf1          
2         /registry/events/default/hostnameredacted.17d6660b9f2fea7c          
2         /registry/events/default/hostnameredacted.17d66616c0192b47          
2         /registry/events/default/hostnameredacted.17d66637813ddb30          
2         /registry/events/default/hostnameredacted.17d66637827580e7          
2         /registry/events/default/hostnameredacted.17d666378b01cce7          
2         /registry/events/default/hostnameredacted.17d666378b020b65          
2         /registry/events/default/hostnameredacted.17d666378b023cd1          
2         /registry/events/default/hostnameredacted.17d6663842d6d017          
2         /registry/events/default/hostnameredacted.17d66650685c9c0c          
2         /registry/events/default/hostnameredacted.17d6665839db0c0e          
2         /registry/events/default/hostnameredacted.17d66682a12febb2          
2         /registry/events/default/hostnameredacted.17d66682a399e32c          
2         /registry/events/default/hostnameredacted.17d66682aadfa085          
2         /registry/events/default/hostnameredacted.17d66682aadfd8ba          
2         /registry/events/default/hostnameredacted.17d66682aadff06a          
2         /registry/events/default/hostnameredacted.17d666832d75bce2          
2         /registry/events/default/hostnameredacted.17d6669a9c2c2a13          
2         /registry/events/default/hostnameredacted.17d6693ed5e8939c          
2         /registry/events/default/hostnameredacted.17d6693ed6c2ad41          
2         /registry/events/default/hostnameredacted.17d6693ededb6081          
2         /registry/events/default/hostnameredacted.17d6693ededbb39f          
2         /registry/events/default/hostnameredacted.17d6693ededbccd8          
2         /registry/events/default/hostnameredacted.17d6693ee88a18d7          
2         /registry/events/default/hostnameredacted.17d66940c2f8ae93          
2         /registry/events/default/hostnameredacted.17d669556f1ead18          
2         /registry/events/default/hostnameredacted.17d6697289cb2302          
2         /registry/events/default/hostnameredacted.17d669728a224de4          
2         /registry/events/default/hostnameredacted.17d6697293c00588          
2         /registry/events/default/hostnameredacted.17d6697293c03778          
2         /registry/events/default/hostnameredacted.17d6697293c0716e          
2         /registry/events/default/hostnameredacted.17d66972dc961689          
2         /registry/events/default/hostnameredacted.17d66988484942c4          
2         /registry/events/default/hostnameredacted.17d66991e0751943          
2         /registry/events/ingress-nginx/ingress-nginx-controller-689c44d4d-8sl64.17d669958a6f1f29                                
2         /registry/events/ingress-nginx/ingress-nginx-controller-689c44d4d-8sl64.17d66998f85456e1                                
2         /registry/events/ingress-nginx/ingress-nginx-controller-689c44d4d-8sl64.17d66999051a388e                                
2         /registry/events/ingress-nginx/ingress-nginx-controller-689c44d4d-8sl64.17d66999359a39fe                                
2         /registry/events/ingress-nginx/ingress-nginx-controller-689c44d4d-8sl64.17d6699b688612c7                                
2         /registry/events/ingress-nginx/ingress-nginx-controller-689c44d4d-8sl64.17d674a75d3da74e                                
2         /registry/events/ingress-nginx/ingress-nginx-controller-689c44d4d-8sl64.17d674a762d627ec