Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rebuild for github.com/wtsi-ssg/wrstat@v6 #11

Draft
wants to merge 39 commits into
base: develop
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
2f18cc3
Add stats parser from github.com/sb10/stats-parse
mjkw31 Nov 18, 2024
02cf007
Copy basedirs, dguta, summary, and watch from https://github.com/wtsi…
mjkw31 Nov 20, 2024
34516a2
Corrected server test data
mjkw31 Nov 20, 2024
3477457
Generate stats output instead of relying on pre-build output
mjkw31 Nov 21, 2024
39d3d3e
Unquote paths in place and use new generated stat data
mjkw31 Nov 21, 2024
74ff155
Add summariser to walk stats file to produce summary and directory da…
mjkw31 Nov 21, 2024
5b68f85
Remove data embed
mjkw31 Nov 21, 2024
b214ac5
Add exported constants for EntryTypes
mjkw31 Nov 21, 2024
c77e7af
Rewrite Summarise method to use a heap instead of a linked-list for s…
mjkw31 Nov 21, 2024
b2710b2
Rework tests to compile
mjkw31 Nov 22, 2024
509e4bb
Move RealGIDAndUID func to own package to avoid import loop
mjkw31 Nov 22, 2024
08cd012
Add additional UserGroup tests
mjkw31 Nov 22, 2024
cefcb79
Reimplemented GroupUser
mjkw31 Nov 22, 2024
666ef15
Rewrite UserGroup to use less memory, and as a Directory Operator
mjkw31 Nov 22, 2024
0466404
Optimised directoryPath
mjkw31 Nov 25, 2024
3521fb5
Correct order that Output is called in
mjkw31 Nov 25, 2024
c99faf0
Generate userGroupDirectories throughout the Summariser run
mjkw31 Nov 25, 2024
d4bb878
Use DirectoryPath instead of byte-slice in data passed to Operator.Add
mjkw31 Nov 25, 2024
4e34be0
Move summary operations in sub-packages and rework dirguta into a Dir…
mjkw31 Nov 25, 2024
d1db6df
Add tests for DirGroupUserTypeAge
mjkw31 Nov 26, 2024
a6da2ad
Move database code to own package
mjkw31 Nov 27, 2024
4b8debc
Pull out dirguta db code into own package and correct tests
mjkw31 Nov 28, 2024
7c42dfc
Corrected server tests
mjkw31 Nov 28, 2024
57d8c1d
Start of base-dir determining code
mjkw31 Nov 29, 2024
6e7ca49
Removed need for splits fn and added DB interface to record basedirs
mjkw31 Dec 4, 2024
34df965
Add initial BaseDirs tests
mjkw31 Dec 4, 2024
1f18f13
Corrected nil-pointer deref
mjkw31 Dec 10, 2024
3397c77
Split intial DirectoryPath into multiple parts if it's not root
mjkw31 Dec 10, 2024
6a514d0
Root of DirectoryPath should be depth zero
mjkw31 Dec 11, 2024
f4eb805
Simplify append method
mjkw31 Dec 11, 2024
fd3a764
Update linter version
mjkw31 Dec 12, 2024
9d80956
Rewrite to have a single method that is used to determine whether a c…
mjkw31 Dec 12, 2024
81dd631
Add new types and restructure to take output from a summary.Summariser
mjkw31 Dec 12, 2024
af43582
Export InfoToType
mjkw31 Dec 12, 2024
2d5ad69
Capture more than just the Path of the BaseDir, but stats (atime, mti…
mjkw31 Dec 12, 2024
3a28f67
Increase go version in github action
mjkw31 Dec 12, 2024
ebe9f94
Delint
mjkw31 Dec 12, 2024
62c4a44
Correct basedir parent merging
mjkw31 Dec 13, 2024
ac41db1
Look at all returned data in test, not just paths
mjkw31 Dec 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/golangci-lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,10 @@ jobs:
steps:
- uses: actions/setup-go@v3
with:
go-version: 1.22.4
go-version: 1.23.3
- uses: actions/checkout@v3
- name: golangci-lint
uses: golangci/golangci-lint-action@v3
with:
version: v1.59.1
only-new-issues: true
version: v1.62.0
only-new-issues: true
96 changes: 96 additions & 0 deletions basedirs/basedirs.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
/*******************************************************************************
* Copyright (c) 2022, 2023 Genome Research Ltd.
*
* Authors:
* Sendu Bala <[email protected]>
* Michael Woolnough <[email protected]>
*
* Permission is hereby granted, free of charge, to any person obtaining
* a copy of this software and associated documentation files (the
* "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish,
* distribute, sublicense, and/or sell copies of the Software, and to
* permit persons to whom the Software is furnished to do so, subject to
* the following conditions:
*
* The above copyright notice and this permission notice shall be included
* in all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
* CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
* TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
******************************************************************************/

// package basedirs is used to summarise disk usage information by base
// directory, storing and retrieving the information from an embedded database.

package basedirs

import (
"github.com/ugorji/go/codec"
"github.com/wtsi-hgi/wrstat-ui/db"
)

// BaseDirs is used to summarise disk usage information by base directory and
// group or user.
type BaseDirs struct {
dbPath string
config Config
quotas *Quotas
ch codec.Handle
mountPoints mountPoints
}

// NewCreator returns a BaseDirs that lets you create a database summarising
// usage information by base directory, taken from the given tree and quotas.
//
// Choose splits and minDirs based on how many directories deep you expect data
// for different groups/users to appear. Eg. if your file structure is
// `/mounts/[group name]`, that's 2 directories deep and splits 1, minDirs 2
// might work well. If it's 5 directories deep, splits 4, minDirs 4 might work
// well.
func NewCreator(dbPath string, c Config, quotas *Quotas) (*BaseDirs, error) {
mp, err := getMountPoints()
if err != nil {
return nil, err
}

return &BaseDirs{
dbPath: dbPath,
config: c,
quotas: quotas,
ch: new(codec.BincHandle),
mountPoints: mp,
}, nil
}

// SetMountPoints can be used to manually set your mountpoints, if the automatic
// discovery of mountpoints on your system doesn't work.
func (b *BaseDirs) SetMountPoints(mountpoints []string) {
b.mountPoints = mountpoints
}

type SummaryWithChildren struct {
db.DirSummary
Children []*SubDir
}

type AgeDirs [len(db.DirGUTAges)][]SummaryWithChildren

type IDAgeDirs map[uint32]*AgeDirs

func (i IDAgeDirs) Get(id uint32) *AgeDirs {
ap := i[id]

if ap == nil {
ap = new(AgeDirs)

i[id] = ap
}

return ap
}
Loading
Loading