Unhandled FormatField Exception

[rpm5099]

This is very likely caused by corruption in an elf file, but I thought you might want to look into the possible cause. Unfortunately I do not have a sample I can provide. Here is the traceback:

---------------------------------------------------------------------------
FieldError                                Traceback (most recent call last)
File ~/miniconda3/envs/py3114/lib/python3.11/site-packages/elftools/construct/core.py:351, in FormatField._parse(self, stream, context)
    350 try:
--> 351     return self.packer.unpack(_read_stream(stream, self.length))[0]
    352 except Exception as ex:

File ~/miniconda3/envs/py3114/lib/python3.11/site-packages/elftools/construct/core.py:293, in _read_stream(stream, length)
    292 if len(data) != length:
--> 293     raise FieldError("expected %d, found %d" % (length, len(data)))
    294 return data

FieldError: expected 4, found 0

During handling of the above exception, another exception occurred:

FieldError                                Traceback (most recent call last)
File ~/miniconda3/envs/py3114/lib/python3.11/site-packages/elftools/common/utils.py:41, in struct_parse(struct, stream, stream_pos)
     40         stream.seek(stream_pos)
---> 41     return struct.parse_stream(stream)
     42 except ConstructError as e:

File ~/miniconda3/envs/py3114/lib/python3.11/site-packages/elftools/construct/core.py:190, in Construct.parse_stream(self, stream)
    183 """
    184 Parse a stream.
    185 
    186 Files, pipes, sockets, and other streaming sources of data are handled
    187 by this method.
    188 """
--> 190 return self._parse(stream, Container())

File ~/miniconda3/envs/py3114/lib/python3.11/site-packages/elftools/construct/core.py:647, in Struct._parse(self, stream, context)
    646 else:
--> 647     subobj = sc._parse(stream, context)
    648     if sc.name is not None:

File ~/miniconda3/envs/py3114/lib/python3.11/site-packages/elftools/construct/core.py:353, in FormatField._parse(self, stream, context)
    352 except Exception as ex:
--> 353     raise FieldError(ex)

FieldError: expected 4, found 0

During handling of the above exception, another exception occurred:

ELFParseError                             Traceback (most recent call last)
Cell In[34], line 2
      1 eof = 0
----> 2 for i,seg in enumerate(elf.iter_segments()):
      3     print(i, seg.header.p_offset, seg.header.p_filesz)
      4     if seg.header.p_offset + seg.header.p_filesz > eof:

File ~/miniconda3/envs/py3114/lib/python3.11/site-packages/elftools/elf/elffile.py:207, in ELFFile.iter_segments(self, type)
    200 """ Yield all the segments in the file. If the optional |type|
    201     parameter is passed, this method will only yield segments of the
    202     given type. The parameter value must be a string containing the
    203     name of the type as defined in the ELF specification, e.g.
    204     'PT_LOAD'.
    205 """
    206 for i in range(self.num_segments()):
--> 207     segment = self.get_segment(i)
    208     if type is None or segment['p_type'] == type:
    209         yield segment

File ~/miniconda3/envs/py3114/lib/python3.11/site-packages/elftools/elf/elffile.py:197, in ELFFile.get_segment(self, n)
    194 """ Get the segment at index #n from the file (Segment object)
    195 """
    196 segment_header = self._get_segment_header(n)
--> 197 return self._make_segment(segment_header)

File ~/miniconda3/envs/py3114/lib/python3.11/site-packages/elftools/elf/elffile.py:604, in ELFFile._make_segment(self, segment_header)
    602     return InterpSegment(segment_header, self.stream)
    603 elif segtype == 'PT_DYNAMIC':
--> 604     return DynamicSegment(segment_header, self.stream, self)
    605 elif segtype == 'PT_NOTE':
    606     return NoteSegment(segment_header, self.stream, self)

File ~/miniconda3/envs/py3114/lib/python3.11/site-packages/elftools/elf/dynamic.py:247, in DynamicSegment.__init__(self, header, stream, elffile)
    239 def __init__(self, header, stream, elffile):
    240     # The string table section to be used to resolve string names in
    241     # the dynamic tag array is the one pointed at by the sh_link field
   (...)
    244     # segment, we do so by searching for the dynamic section whose content
    245     # is located at the same offset as the dynamic segment
    246     stringtable = None
--> 247     for section in elffile.iter_sections():
    248         if (isinstance(section, DynamicSection) and
    249                 section['sh_offset'] == header['p_offset']):
    250             stringtable = elffile.get_section(section['sh_link'])

File ~/miniconda3/envs/py3114/lib/python3.11/site-packages/elftools/elf/elffile.py:174, in ELFFile.iter_sections(self, type)
    167 """ Yield all the sections in the file. If the optional |type|
    168     parameter is passed, this method will only yield sections of the
    169     given type. The parameter value must be a string containing the
    170     name of the type as defined in the ELF specification, e.g.
    171     'SHT_SYMTAB'.
    172 """
    173 for i in range(self.num_sections()):
--> 174     section = self.get_section(i)
    175     if type is None or section['sh_type'] == type:
    176         yield section

File ~/miniconda3/envs/py3114/lib/python3.11/site-packages/elftools/elf/elffile.py:141, in ELFFile.get_section(self, n)
    137 """ Get the section at index #n from the file (Section object or a
    138     subclass)
    139 """
    140 section_header = self._get_section_header(n)
--> 141 return self._make_section(section_header)

File ~/miniconda3/envs/py3114/lib/python3.11/site-packages/elftools/elf/elffile.py:666, in ELFFile._make_section(self, section_header)
    664     return ARMAttributesSection(section_header, name, self)
    665 elif sectype == 'SHT_HASH':
--> 666     return self._make_elf_hash_section(section_header, name)
    667 elif sectype == 'SHT_GNU_HASH':
    668     return self._make_gnu_hash_section(section_header, name)

File ~/miniconda3/envs/py3114/lib/python3.11/site-packages/elftools/elf/elffile.py:740, in ELFFile._make_elf_hash_section(self, section_header, name)
    738 linked_symtab_index = section_header['sh_link']
    739 symtab_section = self.get_section(linked_symtab_index)
--> 740 return ELFHashSection(
    741     section_header, name, self, symtab_section
    742 )

File ~/miniconda3/envs/py3114/lib/python3.11/site-packages/elftools/elf/hash.py:80, in ELFHashSection.__init__(self, header, name, elffile, symboltable)
     78 def __init__(self, header, name, elffile, symboltable):
     79     Section.__init__(self, header, name, elffile)
---> 80     ELFHashTable.__init__(self, elffile, self['sh_offset'], symboltable)

File ~/miniconda3/envs/py3114/lib/python3.11/site-packages/elftools/elf/hash.py:33, in ELFHashTable.__init__(self, elffile, start_offset, symboltable)
     31 self.elffile = elffile
     32 self._symboltable = symboltable
---> 33 self.params = struct_parse(self.elffile.structs.Elf_Hash,
     34                            self.elffile.stream,
     35                            start_offset)

File ~/miniconda3/envs/py3114/lib/python3.11/site-packages/elftools/common/utils.py:43, in struct_parse(struct, stream, stream_pos)
     41     return struct.parse_stream(stream)
     42 except ConstructError as e:
---> 43     raise ELFParseError(str(e))

ELFParseError: expected 4, found 0

[rupran]

If you cannot provide the entire ELF file, can you maybe share the section header table (the output of readelf - S) and the program headers (readelf - l)? Additionally, can you check if the file has a dynamic symbol table but exports no symbols itself?

Thanks!

[rpm5099]

I don't have the data, but it was a fragment of an elf that was truncated well before the end of the program. The data after the truncation was essentially random as it was forensically carved based on the program headers. The traceback should get you close enough to find the code that did not handle the unexpected input well. If it was never intended to be able to safely parse potentially corrupt elf's that's understandable, but it happens and I figured I'd give you the option.

[sevaa]

The root cause of the error is that the stream ended prematurely. It's consistent with data corruption. I don't see how this is an issue with pyelftools and not with the ELF file.

In general, our preferred rule of thumb is - can the GNU or the LLVM tools (e. g. readelf) parse the binary with no errors? If they do, and pyelftools throws an error, it's issue with pyelftools. Else, it's the issue with the binary.

[rpm5099]

The root cause of the error is that the stream ended prematurely. I don't see how this is an issue with pyelftools and not with the ELF file.

The suggestion is to put in an exception indicating that the data is truncated or corrupt in place of the traceback above which is meaningless to users.

[sevaa]

But you already got an exception. Maybe not a very descriptive one, but providing a separate exception type for every possible kind of data corruption out there is not feasible.

Here is a very similar discussion: #367

[rpm5099]

Looked at #367, he is reporting the same thing - readelf provides some information about what went wrong and how the elf is malformed. Without that information you're left wondering whether the elf was corrupt or the tool just failed on a valid elf. I agree that the goal should not be to parse the file the same as the Linux kernel loader does, which does not use section headers at all. Again, if it's not a feature that you want to support - don't.

[sevaa]

Barring a clarification from @eliben, I'd say that providing diagnostics on the exact nature of file corruption is not the ambition of this effort.

[eliben]

I think this is a slightly different issue than #367, which was asking pyelftools to be forgiving and agree to parse a malformed file. That's a clear no.

This issue seems to be talking about producing a clearer error message in case this is simple; I don't have any fundamental objections to this, but the caveat is in the is simple.