Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DLPX-92461 Add libkdumpfile10 and python3-drgn as dependencies of sdb to remove dependency on internal forks #344

Conversation

palash-gandhi
Copy link

@palash-gandhi palash-gandhi commented Oct 17, 2024

Problem

libkdumpfile and drgn are currently built from internal sources. These 2 repos
have had minimal changes authored by Delphix and all of them are build-related or
related to the python2-3 migration. Maintaining internal forks causes overheads.

Solution

To reduce the dependency of using internal forks, this change adds these 2 packages
as dependencies of sdb so that delphix/linux-pkg#325 can proceed

Testing Done

https://ops-jenkins.eng-tools-prd.aws.delphixcloud.com/job/linux-pkg/job/os-upgrade/job/build-package/job/sdb/job/pre-push/10/console
Some manual tests:

ubuntu@ip-10-110-219-58:~$ aws s3 cp s3://dev-de-images/builds/jenkins-ops/linux-pkg/os-upgrade/build-package/sdb/pre-push/10/sdb_1.0.0-1delphix.2024.10.17.19.19_amd64.deb .
download: s3://dev-de-images/builds/jenkins-ops/linux-pkg/os-upgrade/build-package/sdb/pre-push/10/sdb_1.0.0-1delphix.2024.10.17.19.19_amd64.deb to ./sdb_1.0.0-1delphix.2024.10.17.19.19_amd64.deb
ubuntu@ip-10-110-219-58:~$ sudo dpkg -i sdb_1.0.0-1delphix.2024.10.17.19.19_amd64.deb
Selecting previously unselected package sdb.
(Reading database ... 122844 files and directories currently installed.)
Preparing to unpack sdb_1.0.0-1delphix.2024.10.17.19.19_amd64.deb ...
Unpacking sdb (1.0.0-1delphix.2024.10.17.19.19) ...
dpkg: dependency problems prevent configuration of sdb:
 sdb depends on libkdumpfile10; however:
  Package libkdumpfile10 is not installed.
 sdb depends on python3-drgn; however:
  Package python3-drgn is not installed.

dpkg: error processing package sdb (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 sdb
ubuntu@ip-10-110-219-58:~$ sudo sdb
Traceback (most recent call last):
  File "/usr/bin/sdb", line 33, in <module>
    sys.exit(load_entry_point('sdb==0.1.0', 'console_scripts', 'sdb')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/bin/sdb", line 25, in importlib_load_entry_point
    return next(matches).load()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/metadata/__init__.py", line 205, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/usr/lib/python3/dist-packages/sdb/__init__.py", line 35, in <module>
    from sdb.target import (create_object, get_object, get_prog, get_type,
  File "/usr/lib/python3/dist-packages/sdb/target.py", line 46, in <module>
    import drgn
ModuleNotFoundError: No module named 'drgn'

After installing python3-drgn and libkdumpfile

ubuntu@ip-10-110-219-58:~$ sudo sdb
sdb: missing some debugging symbols (see https://drgn.readthedocs.io/en/latest/getting_debugging_symbols.html):
  kernel (could not find vmlinux for 6.8.0-1013-aws)
  kernel modules (could not find loaded kernel modules: could not find 'struct module')
sdb> find_task 1
sdb encountered an internal error due to a bug. Here's the
information you need to file the bug:
----------------------------------------------------------
Target Info:
	ProgramFlags.IS_LINUX_KERNEL|IS_LIVE|IS_LOCAL
	Platform(<Architecture.X86_64: 1>, <PlatformFlags.IS_64_BIT|IS_LITTLE_ENDIAN: 3>)

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sdb/internal/repl.py", line 107, in eval_cmd
    for obj in invoke([], input_):
  File "/usr/lib/python3/dist-packages/sdb/pipeline.py", line 152, in invoke
    yield from execute_pipeline(first_input, pipeline)
  File "/usr/lib/python3/dist-packages/sdb/pipeline.py", line 84, in execute_pipeline
    yield from massage_input_and_call(pipeline[-1], this_input)
  File "/usr/lib/python3/dist-packages/sdb/pipeline.py", line 43, in massage_input_and_call
    yield from cmd.call(objs)
  File "/usr/lib/python3/dist-packages/sdb/command.py", line 413, in call
    yield from self.__invalid_memory_objects_check(
  File "/usr/lib/python3/dist-packages/sdb/command.py", line 358, in __invalid_memory_objects_check
    for obj in objs:
  File "/usr/lib/python3/dist-packages/sdb/commands/linux/process.py", line 93, in _call
    yield find_task(sdb.get_prog(), pid)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/drgn/helpers/common/prog.py", line 213, in wrapper
    return f(args[0], None, *args[1:], **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/drgn/helpers/linux/pid.py", line 84, in find_task
    ns = prog["init_pid_ns"].address_of_()
         ~~~~^^^^^^^^^^^^^^^
KeyError: 'init_pid_ns'
----------------------------------------------------------
Link: https://github.com/delphix/sdb/issues/new

I think this is expected because the VM I used is a bootstrapped 24.04 VM without kernel debug symbols.

@palash-gandhi palash-gandhi force-pushed the dlpx/pr/palash-gandhi/9b88aacb-5cf3-4de4-ac21-60a4a8577039 branch from f7f53fb to d696202 Compare October 17, 2024 19:12
@palash-gandhi palash-gandhi marked this pull request as ready for review October 17, 2024 19:57
Copy link
Contributor

@prakashsurya prakashsurya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems reasonable.. I think we might want to use the latest drgn, as that's actively worked on and gets improvements.. but the folks that generally track that tooling/improvements aren't at delphix anymore, so this makes sense to me.. if we determine there's something we need that isn't in the ubuntu packages, we can do that later..

@palash-gandhi palash-gandhi enabled auto-merge (squash) October 23, 2024 21:52
@palash-gandhi palash-gandhi merged commit 68c962c into os-upgrade Oct 24, 2024
10 of 23 checks passed
@palash-gandhi palash-gandhi deleted the dlpx/pr/palash-gandhi/9b88aacb-5cf3-4de4-ac21-60a4a8577039 branch October 24, 2024 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants