No objects to concatenate error, run_scenicplus: "Formatting eGRNs"

[SidG13]

Hi,

I'm running into an issue related to the "Formatting eGRNs" step of run_scenicplus. Everything has run fine up to this point. I've run a previous sample before without issues as well, so this is new to me. Note that I'm using a custom species/annotation, so some fields are 'None'.

I ran:

from scenicplus.wrappers.run_scenicplus import run_scenicplus
try:
    run_scenicplus(
        scplus_obj = scplus_obj,
        variable = ['GEX_cell_cluster'],
        species = None,
        assembly = None,
        tf_file = '/home/administrator/Desktop/ExtraDrive1/Sid/data_manipulation/utoronto_human_tfs_v_1.01.txt',
        save_path = os.path.join(work_dir, 'scenicplus'),
        upstream = [1000, 150000],
        downstream = [1000, 150000],
        calculate_TF_eGRN_correlation = True,
        calculate_DEGs_DARs = True,
        export_to_loom_file = True,
        export_to_UCSC_file = False,
        path_bedToBigBed = '/home/administrator/Desktop/ExtraDrive1/Sid/data_manipulation',
        n_cpu = 8,
        _temp_dir = os.path.join(tmp_dir, 'ray_spill'))
except Exception as e:
    #in case of failure, still save the object
    dill.dump(scplus_obj, open(os.path.join(work_dir, 'scenicplus/scplus_obj.pkl'), 'wb'), protocol=4)
    raise(e)

And the eventual error is:

2023-03-16 18:06:42,551 GSEA         INFO     Subsetting on adjusted pvalue: 1, minimal NES: 0 and minimal leading edge genes 10
2023-03-16 18:06:42,552 GSEA         INFO     Merging eRegulons
2023-03-16 18:06:42,553 GSEA         INFO     Storing eRegulons in .uns[eRegulons].
2023-03-16 18:06:42,569 SCENIC+_wrapper INFO     Formatting eGRNs
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In [13], line 22
     19 except Exception as e:
     20     #in case of failure, still save the object
     21     dill.dump(scplus_obj, open(os.path.join(work_dir, 'scenicplus/scplus_obj.pkl'), 'wb'), protocol=4)
---> 22     raise(e)

Cell In [13], line 3
      1 from scenicplus.wrappers.run_scenicplus import run_scenicplus
      2 try:
----> 3     run_scenicplus(
      4         scplus_obj = scplus_obj,
      5         variable = ['GEX_cell_cluster'],
      6         species = None,
      7         assembly = None,
      8         tf_file = '/home/administrator/Desktop/ExtraDrive1/Sid/data_manipulation/utoronto_human_tfs_v_1.01.txt',
      9         save_path = os.path.join(work_dir, 'scenicplus'),
     10         upstream = [1000, 150000],
     11         downstream = [1000, 150000],
     12         calculate_TF_eGRN_correlation = True,
     13         calculate_DEGs_DARs = True,
     14         export_to_loom_file = True,
     15         export_to_UCSC_file = False,
     16         path_bedToBigBed = '/home/administrator/Desktop/ExtraDrive1/Sid/data_manipulation',
     17         n_cpu = 8,
     18         _temp_dir = os.path.join(tmp_dir, 'ray_spill'))
     19 except Exception as e:
     20     #in case of failure, still save the object
     21     dill.dump(scplus_obj, open(os.path.join(work_dir, 'scenicplus/scplus_obj.pkl'), 'wb'), protocol=4)

File ~/anaconda2/envs/py38/lib/python3.8/site-packages/scenicplus/wrappers/run_scenicplus.py:195, in run_scenicplus(scplus_obj, variable, species, assembly, tf_file, save_path, biomart_host, upstream, downstream, region_ranking, gene_ranking, simplified_eGRN, calculate_TF_eGRN_correlation, calculate_DEGs_DARs, export_to_loom_file, export_to_UCSC_file, tree_structure, path_bedToBigBed, n_cpu, _temp_dir, save_partial, **kwargs)
    193 if 'eRegulon_metadata' not in scplus_obj.uns.keys():
    194     log.info('Formatting eGRNs')
--> 195     format_egrns(scplus_obj,
    196                   eregulons_key = 'eRegulons',
    197                   TF2G_key = 'TF2G_adj',
    198                   key_added = 'eRegulon_metadata')
    201 if 'eRegulon_signatures' not in scplus_obj.uns.keys():
    202     log.info('Converting eGRNs to signatures')

File ~/anaconda2/envs/py38/lib/python3.8/site-packages/scenicplus/utils.py:764, in format_egrns(scplus_obj, eregulons_key, TF2G_key, key_added)
    761 tf2g_data = scplus_obj.uns[TF2G_key].copy()
    762 tf2g_data.columns = ['TF', 'Gene', 'TF2G_importance', 'TF2G_regulation',
    763                      'TF2G_rho', 'TF2G_importance_x_abs_rho', 'TF2G_importance_x_rho']
--> 764 egrn_metadata = pd.concat([pd.merge(r2g_data[x], tf2g_data[tf2g_data.TF == r2g_data[x].TF[0]], on=[
    765                           'TF', 'Gene']) for x in range(len(egrn_list))])
    766 scplus_obj.uns[key_added] = egrn_metadata

File ~/anaconda2/envs/py38/lib/python3.8/site-packages/pandas/util/_decorators.py:317, in deprecate_nonkeyword_arguments.<locals>.decorate.<locals>.wrapper(*args, **kwargs)
    311 if len(args) > num_allow_args:
    312     warnings.warn(
    313         msg.format(arguments=arguments),
    314         FutureWarning,
    315         stacklevel=find_stack_level(inspect.currentframe()),
    316     )
--> 317 return func(*args, **kwargs)

File ~/anaconda2/envs/py38/lib/python3.8/site-packages/pandas/core/reshape/concat.py:369, in concat(objs, axis, join, ignore_index, keys, levels, names, verify_integrity, sort, copy)
    147 @deprecate_nonkeyword_arguments(version=None, allowed_args=["objs"])
    148 def concat(
    149     objs: Iterable[NDFrame] | Mapping[HashableT, NDFrame],
   (...)
    158     copy: bool = True,
    159 ) -> DataFrame | Series:
    160     """
    161     Concatenate pandas objects along a particular axis.
    162 
   (...)
    367     1   3   4
    368     """
--> 369     op = _Concatenator(
    370         objs,
    371         axis=axis,
    372         ignore_index=ignore_index,
    373         join=join,
    374         keys=keys,
    375         levels=levels,
    376         names=names,
    377         verify_integrity=verify_integrity,
    378         copy=copy,
    379         sort=sort,
    380     )
    382     return op.get_result()

File ~/anaconda2/envs/py38/lib/python3.8/site-packages/pandas/core/reshape/concat.py:426, in _Concatenator.__init__(self, objs, axis, join, keys, levels, names, ignore_index, verify_integrity, copy, sort)
    423     objs = list(objs)
    425 if len(objs) == 0:
--> 426     raise ValueError("No objects to concatenate")
    428 if keys is None:
    429     objs = list(com.not_none(*objs))

ValueError: No objects to concatenate

My scenicplus object looks fine, everything is populated except eRegulons.

The two dataframes which I think are causing the issue look OK to me?

Does anyone have an idea what I could be missing? Thanks!

[SeppeDeWinter]

Hi @SidG13

can you show the output of

len(scplus_obj.uns['eRegulons'])

and

scplus_obj.uns['eRegulons'][0]

Best,

Seppe

[SidG13]

Hi @SeppeDeWinter, that slot is empty so there's nothing there. All the other slots look fine.

[SeppeDeWinter]

Unfortunately something went wrong with inferring eRegulons ...

You might want to try running build_grn again.

Best,

Seppe

[SidG13]

I reran the build_grn function

Still getting

[alexlenail]

@SidG13 Did you ultimately find out why this happened to you & how to resolve it?

[SeppeDeWinter]

Hi @alexlenail

For sid we figured eventually out that his gene expression matrix was very sparse. That's why he did not find any eRegulons at the end.

S