Exception: string index out of range when fetching site lists with expand:fields #940

nicolas-lurkin-b12 · 2024-10-24T09:38:32Z

Describe the bug

I am trying to fetch the list of lists for a given site, with an expand on the fields.
This crashes during deserialisation with a "string index out of range".
The full stack trace is the following

File /usr/local/lib/python3.10/site-packages/msgraph/generated/sites/item/lists/item/items/items_request_builder.py:81, in ItemsRequestBuilder.get(self, request_configuration)
     78     raise Exception("Http core is null") 
     79 from ......models.list_item_collection_response import ListItemCollectionResponse
---> 81 return await self.request_adapter.send_async(request_info, ListItemCollectionResponse, error_mapping)

File /usr/local/lib/python3.10/site-packages/kiota_http/httpx_request_adapter.py:196, in HttpxRequestAdapter.send_async(self, request_info, parsable_factory, error_map)
    194     return None
    195 _deserialized_span = self._start_local_tracing_span("get_object_value", parent_span)
--> 196 value = root_node.get_object_value(parsable_factory)
    197 parent_span.set_attribute(DESERIALIZED_MODEL_NAME_KEY, value.__class__.__name__)
    198 _deserialized_span.end()

File /usr/local/lib/python3.10/site-packages/kiota_serialization_json/json_parse_node.py:222, in JsonParseNode.get_object_value(self, factory)
    220 if on_before := self.on_before_assign_field_values:
    221     on_before(result)
--> 222 self._assign_field_values(result)
    223 if on_after := self.on_after_assign_field_values:
    224     on_after(result)

File /usr/local/lib/python3.10/site-packages/kiota_serialization_json/json_parse_node.py:291, in JsonParseNode._assign_field_values(self, item)
    289         continue
    290     field_deserializer = field_deserializers[field_name]
--> 291     field_deserializer(JsonParseNode(field_value))
    292 elif item_additional_data is not None:
    293     item_additional_data[field_name] = self.try_get_anything(field_value)

File /usr/local/lib/python3.10/site-packages/msgraph/generated/models/list_item_collection_response.py:40, in ListItemCollectionResponse.get_field_deserializers.<locals>.<lambda>(n)
     36 from .base_collection_pagination_count_response import BaseCollectionPaginationCountResponse
     37 from .list_item import ListItem
     39 fields: Dict[str, Callable[[Any], None]] = {
---> 40     "value": lambda n : setattr(self, 'value', n.get_collection_of_object_values(ListItem)),
     41 }
     42 super_fields = super().get_field_deserializers()
     43 fields.update(super_fields)

File /usr/local/lib/python3.10/site-packages/kiota_serialization_json/json_parse_node.py:170, in JsonParseNode.get_collection_of_object_values(self, factory)
    165 """Gets the collection of type U values from the json node
    166 Returns:
    167     List[U]: The collection of model object values of the node
    168 """
    169 if isinstance(self._json_node, list):
--> 170     return list(
    171         map(
    172             lambda x: self._create_new_node(x).get_object_value(factory),  # type: ignore
    173             self._json_node,
    174         )
    175     )
    176 return []

File /usr/local/lib/python3.10/site-packages/kiota_serialization_json/json_parse_node.py:172, in JsonParseNode.get_collection_of_object_values.<locals>.<lambda>(x)
    165 """Gets the collection of type U values from the json node
    166 Returns:
    167     List[U]: The collection of model object values of the node
    168 """
    169 if isinstance(self._json_node, list):
    170     return list(
    171         map(
--> 172             lambda x: self._create_new_node(x).get_object_value(factory),  # type: ignore
    173             self._json_node,
    174         )
    175     )
    176 return []

File /usr/local/lib/python3.10/site-packages/kiota_serialization_json/json_parse_node.py:222, in JsonParseNode.get_object_value(self, factory)
    220 if on_before := self.on_before_assign_field_values:
    221     on_before(result)
--> 222 self._assign_field_values(result)
    223 if on_after := self.on_after_assign_field_values:
    224     on_after(result)

File /usr/local/lib/python3.10/site-packages/kiota_serialization_json/json_parse_node.py:291, in JsonParseNode._assign_field_values(self, item)
    289         continue
    290     field_deserializer = field_deserializers[field_name]
--> 291     field_deserializer(JsonParseNode(field_value))
    292 elif item_additional_data is not None:
    293     item_additional_data[field_name] = self.try_get_anything(field_value)

File /usr/local/lib/python3.10/site-packages/msgraph/generated/models/list_item.py:76, in ListItem.get_field_deserializers.<locals>.<lambda>(n)
     68 from .list_item_version import ListItemVersion
     69 from .sharepoint_ids import SharepointIds
     71 fields: Dict[str, Callable[[Any], None]] = {
     72     "analytics": lambda n : setattr(self, 'analytics', n.get_object_value(ItemAnalytics)),
     73     "contentType": lambda n : setattr(self, 'content_type', n.get_object_value(ContentTypeInfo)),
     74     "documentSetVersions": lambda n : setattr(self, 'document_set_versions', n.get_collection_of_object_values(DocumentSetVersion)),
     75     "driveItem": lambda n : setattr(self, 'drive_item', n.get_object_value(DriveItem)),
---> 76     "fields": lambda n : setattr(self, 'fields', n.get_object_value(FieldValueSet)),
     77     "sharepointIds": lambda n : setattr(self, 'sharepoint_ids', n.get_object_value(SharepointIds)),
     78     "versions": lambda n : setattr(self, 'versions', n.get_collection_of_object_values(ListItemVersion)),
     79 }
     80 super_fields = super().get_field_deserializers()
     81 fields.update(super_fields)

File /usr/local/lib/python3.10/site-packages/kiota_serialization_json/json_parse_node.py:222, in JsonParseNode.get_object_value(self, factory)
    220 if on_before := self.on_before_assign_field_values:
    221     on_before(result)
--> 222 self._assign_field_values(result)
    223 if on_after := self.on_after_assign_field_values:
    224     on_after(result)

File /usr/local/lib/python3.10/site-packages/kiota_serialization_json/json_parse_node.py:293, in JsonParseNode._assign_field_values(self, item)
    291     field_deserializer(JsonParseNode(field_value))
    292 elif item_additional_data is not None:
--> 293     item_additional_data[field_name] = self.try_get_anything(field_value)
    294 else:
    295     warnings.warn(
    296         f"Found additional property {field_name} to \
    297         deserialize but the model doesn't support additional data"
    298     )

File /usr/local/lib/python3.10/site-packages/kiota_serialization_json/json_parse_node.py:316, in JsonParseNode.try_get_anything(self, value)
    313 if self.__is_four_digit_number(value):
    314     return value
--> 316 datetime_obj = pendulum.parse(value)
    317 if isinstance(datetime_obj, pendulum.Duration):
    318     return datetime_obj.as_timedelta()

File /usr/local/lib/python3.10/site-packages/pendulum/parser.py:30, in parse(text, **options)
     26 def parse(text: str, **options: t.Any) -> Date | Time | DateTime | Duration:
     27     # Use the mock now value if it exists
     28     options["now"] = options.get("now")
---> 30     return _parse(text, **options)

File /usr/local/lib/python3.10/site-packages/pendulum/parser.py:43, in _parse(text, **options)
     40 if text == "now":
     41     return pendulum.now()
---> 43 parsed = base_parse(text, **options)
     45 if isinstance(parsed, datetime.datetime):
     46     return pendulum.datetime(
     47         parsed.year,
     48         parsed.month,
   (...)
     54         tz=parsed.tzinfo or options.get("tz", UTC),
     55     )

File /usr/local/lib/python3.10/site-packages/pendulum/parsing/__init__.py:78, in parse(text, **options)
     75 _options: dict[str, Any] = copy.copy(DEFAULT_OPTIONS)
     76 _options.update(options)
---> 78 return _normalize(_parse(text, **_options), **_options)

File /usr/local/lib/python3.10/site-packages/pendulum/parsing/__init__.py:116, in _parse(text, **options)
    113     return parse_iso8601(text)
    115 with contextlib.suppress(ValueError):
--> 116     return _parse_iso8601_interval(text)
    118 with contextlib.suppress(ParserError):
    119     return _parse_common(text, **options)

File /usr/local/lib/python3.10/site-packages/pendulum/parsing/__init__.py:217, in _parse_iso8601_interval(text)
    214 first, last = text.split("/")
    215 start = end = duration = None
--> 217 if first[0] == "P":
    218     # duration/end
    219     duration = parse_iso8601(first)
    220     end = parse_iso8601(last)

IndexError: string index out of range

The field that is being deserialised is 'WorkPhone' with a value of '/'. That is clearly not a date even though it is trying to decode it as a date.

Expected behavior

I expect to retrieve the decoded object

How to reproduce

from azure.identity import ClientSecretCredential
from msgraph import GraphServiceClient
import asyncio
from msgraph.generated.users.item.messages.messages_request_builder import  MessagesRequestBuilder

credentials = ClientSecretCredential(
            tenant_id=SHAREPOINT_TENANT_ID,
            client_id=SHAREPOINT_CLIENT_ID,
            client_secret=SHAREPOINT_SECRET_VALUE
        )
client = GraphServiceClient(credentials, scopes=["https://graph.microsoft.com/.default"])
loop = asyncio.get_event_loop()

site_id = <the_site_id>

query_params = MessagesRequestBuilder.MessagesRequestBuilderGetQueryParameters(expand=['fields'])
configuration = MessagesRequestBuilder.MessagesRequestBuilderGetRequestConfiguration(query_parameters=query_params)
fn = client.sites.by_site_id(site_id)lists.by_list_id("User Information List").items.get(request_configuration=configuration)
a = asyncio.run_coroutine_threadsafe(fn, loop)
a.result()

This obviously triggers the issue only when the data coming from the request contains the mentioned field with the specific value.

SDK Version

1.11.0

Latest version known to work for scenario above?

None

Known Workarounds

No response

Debug output

No response

Configuration

No response

Other information

No response

The text was updated successfully, but these errors were encountered:

nicolas-lurkin-b12 · 2024-10-24T13:46:23Z

After several tests, the last configuration that works is the following:

msgraph-core==1.1.6
msgraph-sdk==1.5.3
microsoft-kiota-abstractions==1.5.0
microsoft-kiota-authentication-azure==1.5.0
microsoft-kiota-http==1.5.0
microsoft-kiota-serialization-form==1.5.0
microsoft-kiota-serialization-json==1.2.0
microsoft-kiota-serialization-multipart==1.5.0
microsoft-kiota-serialization-text==1.5.0

with microsoft-kiota-serialization-json==1.2.0 pinned to 1.2.0

shemogumbe · 2024-10-26T09:26:32Z

Hello @nicolas-lurkin-b12 thanks for using the SDK and for raising this.

This looks like an issue we have been experiencing with our JSON serialization:
If I get you right, you mean the latest versions do not work for your scenario and only works for 1.2.0?

nicolas-lurkin-b12 · 2024-10-28T07:03:37Z

Hello @shemogumbe
Thanks for your answer. That is correct. As far as I could tell, using microsoft-kiota-serialization-json above 1.2.0 leads to the issue.

HsiehShuJeng · 2024-10-31T11:59:26Z

I encountered possibly the same issue here.

Main Logic

async def list_outlook_events(self, start_date: datetime, end_date: datetime):
    """
    Lists events created or participated in by the user between start_date and end_date.
    Ref: https://learn.microsoft.com/en-us/graph/api/event-get?view=graph-rest-1.0
    """
    events = []
    request_configuration = RequestConfiguration()
    request_configuration.headers.add("Prefer", 'outlook.timezone="Asia/Taipei"')

    # Instead of RequestConfiguration, manually set headers in the query
    result = await self.graph_client.me.events.get(
        request_configuration=request_configuration
    )
    old_event_count = 0
    stop_processing = False

    while True:
        for event in result.value:
            try:
                # transformation logic
            except Exception as e:
                print(f"Error processing event: {e}")
                traceback.print_exc()
                pyperclip.copy(str(event))
                sys.exit(0)

        if stop_processing:
            break

        # Handle pagination if there are more events to retrieve
        if result.odata_next_link:
            result = await self.graph_client.teams.with_url(result.odata_next_link).get()

    .
    .
    .
    return events

I have 2 MacBook Pros, that said, MacBook 1 and MacBook 2. The Python script on MacBook 1 works pretty much well and MacBook 2 the same UNTIL TODAY, i.e., MacBook 1 good job and MacBook 2 wanted to play a game with me, a.k.a, failed.

Error Message from Traceback

Traceback (most recent call last):
  File "/Users/shu-jenghsieh/Desktop/Automation Center/journal/jira/metrics/collect_stats.py", line 506, in <module>
    asyncio.run(collect_calendar_events(input_year, all=True))
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ~~~~~~~~~~^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/base_events.py", line 721, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "/Users/shu-jenghsieh/Desktop/Automation Center/journal/jira/metrics/collect_stats.py", line 477, in collect_calendar_events
    events = await lister.list_outlook_events(start_date_taipei, end_date_taipei)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shu-jenghsieh/Desktop/Automation Center/journal/jira/metrics/teams/list_outlook_events.py", line 546, in list_outlook_events
    result = await self.graph_client.teams.with_url(result.odata_next_link).get()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.13/site-packages/msgraph/generated/teams/teams_request_builder.py", line 69, in get
    return await self.request_adapter.send_async(request_info, TeamCollectionResponse, error_mapping)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.13/site-packages/kiota_http/httpx_request_adapter.py", line 196, in send_async
    value = root_node.get_object_value(parsable_factory)
  File "/opt/homebrew/lib/python3.13/site-packages/kiota_serialization_json/json_parse_node.py", line 222, in get_object_value
    self._assign_field_values(result)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/opt/homebrew/lib/python3.13/site-packages/kiota_serialization_json/json_parse_node.py", line 291, in _assign_field_values
    field_deserializer(JsonParseNode(field_value))
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.13/site-packages/msgraph/generated/models/team_collection_response.py", line 40, in <lambda>
    "value": lambda n : setattr(self, 'value', n.get_collection_of_object_values(Team)),
                                               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/opt/homebrew/lib/python3.13/site-packages/kiota_serialization_json/json_parse_node.py", line 170, in get_collection_of_object_values
    return list(
        map(
    ...<2 lines>...
        )
    )
  File "/opt/homebrew/lib/python3.13/site-packages/kiota_serialization_json/json_parse_node.py", line 172, in <lambda>
    lambda x: self._create_new_node(x).get_object_value(factory),  # type: ignore
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/opt/homebrew/lib/python3.13/site-packages/kiota_serialization_json/json_parse_node.py", line 222, in get_object_value
    self._assign_field_values(result)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/opt/homebrew/lib/python3.13/site-packages/kiota_serialization_json/json_parse_node.py", line 293, in _assign_field_values
    item_additional_data[field_name] = self.try_get_anything(field_value)
                                       ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.13/site-packages/kiota_serialization_json/json_parse_node.py", line 316, in try_get_anything
    datetime_obj = pendulum.parse(value)
  File "/opt/homebrew/lib/python3.13/site-packages/pendulum/parser.py", line 30, in parse
    return _parse(text, **options)
  File "/opt/homebrew/lib/python3.13/site-packages/pendulum/parser.py", line 43, in _parse
    parsed = base_parse(text, **options)
  File "/opt/homebrew/lib/python3.13/site-packages/pendulum/parsing/__init__.py", line 78, in parse
    return _normalize(_parse(text, **_options), **_options)
                      ~~~~~~^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.13/site-packages/pendulum/parsing/__init__.py", line 116, in _parse
    return _parse_iso8601_interval(text)
  File "/opt/homebrew/lib/python3.13/site-packages/pendulum/parsing/__init__.py", line 221, in _parse_iso8601_interval
    elif last[0] == "P":
         ~~~~^^^
IndexError: string index out of range

In the beginning, I tweaked try_get_anything in the module of json_parse_node to see what field cannot be parsed and there are quite a lot and NOT every (Outlook) fetching from odata_next_link failed yet still quite some link couldn't return results correctly. By conjecturing some issue in JSON parsing and asking Teach Goo (Google), I saw this discussion. Thanks to @nicolas-lurkin-b12 , my issue is resolved by adding microsoft-kiota-serialization-json==1.3.0 in requirements.txt in my project for automation.

I have tried the following versions for microsoft-kiota-serialization-json

microsoft-kiota-serialization-json==1.3.1
microsoft-kiota-serialization-json==1.3.2
microsoft-kiota-serialization-json==1.3.3
microsoft-kiota-serialization-json==1.5.0
microsoft-kiota-serialization-json==1.6.0

And all led me to the same error message.

Machine Info

ProductName:            macOS
ProductVersion:         14.6.1
BuildVersion:           23G93

Python Version

Python 3.13.0

It's installed by brew on the laptop where I encountered the issue.

Hopefully, my experience can help more developers.
cc @shemogumbe

lifegivesyoulemons · 2024-11-12T14:42:20Z

I'm experiencing the same problem with my Airflow DAG that connects to OneDrive file with Graph SDK.

What is confusing in my case is that I have multiple similar DAGs connecting to OneDrive, but only one of them has this error.
Also if I run the same pipeline manually on Windows, it works.

Our setup:
Ubuntu 22.04.5 LTS

microsoft-kiota-abstractions==1.6.0
microsoft-kiota-authentication-azure==1.0.0
microsoft-kiota-http==1.3.1
microsoft-kiota-serialization-form==1.6.0
microsoft-kiota-serialization-json==1.6.0
microsoft-kiota-serialization-multipart==1.6.0
microsoft-kiota-serialization-text==1.6.0

Python version:
Python 3.12.4

On Windows:
microsoft-kiota-serialization-json==1.3.0

nicolas-lurkin-b12 added status:waiting-for-triage An issue that is yet to be reviewed or assigned type:bug A broken experience labels Oct 24, 2024

shemogumbe removed the status:waiting-for-triage An issue that is yet to be reviewed or assigned label Oct 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exception: string index out of range when fetching site lists with expand:fields #940

Exception: string index out of range when fetching site lists with expand:fields #940

nicolas-lurkin-b12 commented Oct 24, 2024 •

edited

Loading

nicolas-lurkin-b12 commented Oct 24, 2024

shemogumbe commented Oct 26, 2024

nicolas-lurkin-b12 commented Oct 28, 2024

HsiehShuJeng commented Oct 31, 2024

lifegivesyoulemons commented Nov 12, 2024

Exception: string index out of range when fetching site lists with expand:fields #940

Exception: string index out of range when fetching site lists with expand:fields #940

Comments

nicolas-lurkin-b12 commented Oct 24, 2024 • edited Loading

Describe the bug

Expected behavior

How to reproduce

SDK Version

Latest version known to work for scenario above?

Known Workarounds

Debug output

Configuration

Other information

nicolas-lurkin-b12 commented Oct 24, 2024

shemogumbe commented Oct 26, 2024

nicolas-lurkin-b12 commented Oct 28, 2024

HsiehShuJeng commented Oct 31, 2024

lifegivesyoulemons commented Nov 12, 2024

nicolas-lurkin-b12 commented Oct 24, 2024 •

edited

Loading