Calculating TF-eGRNs AUC correlation in the 10x multiome pbmc tutorial

[laurajohanne]

Hi,

Thank you for a very nice tutorial and the creation of Scenic+.
I am trying to follow our 10x multiome pbmc tutorial, but I am getting an issue when running "Calculating TF-eGRNs AUC correlation" within the scenicplus.wrapper function.

KeyError                                  Traceback (most recent call last)
Cell In[22], line 24
     21 except Exception as e:
     22     #in case of failure, still save the object
     23     dill.dump(scplus_obj, open(os.path.join(work_dir, 'scenicplus/scplus_obj.pkl'), 'wb'), protocol=-1)
---> 24     raise(e)

Cell In[22], line 3
      1 from scenicplus.wrappers.run_scenicplus import run_scenicplus
      2 try:
----> 3     run_scenicplus(
      4         scplus_obj = scplus_obj,
      5         variable = ['GEX_celltype'],
      6         species = 'hsapiens',
      7         assembly = 'hg38',
      8         tf_file = 'pbmc_tutorial/data/utoronto_human_tfs_v_1.01.txt',
      9         save_path = os.path.join(work_dir, 'scenicplus'),
     10         biomart_host = biomart_host,
     11         upstream = [1000, 150000],
     12         downstream = [1000, 150000],
     13         calculate_TF_eGRN_correlation = True,
     14         calculate_DEGs_DARs = True,
     15         export_to_loom_file = True,
     16         export_to_UCSC_file = True,
     17         path_bedToBigBed = 'pbmc_tutorial',
     18         n_cpu= None,
     19         address = 'auto',
     20         _temp_dir = os.path.join(tmp_dir, 'ray_spill')).options(num_cpus=60) # set the number of CPU cores to use
     21 except Exception as e:
     22     #in case of failure, still save the object
     23     dill.dump(scplus_obj, open(os.path.join(work_dir, 'scenicplus/scplus_obj.pkl'), 'wb'), protocol=-1)

File /work/SCENIC+/scenicplus/wrappers/run_scenicplus.py:253, in run_scenicplus(scplus_obj, variable, species, assembly, tf_file, save_path, biomart_host, upstream, downstream, region_ranking, gene_ranking, simplified_eGRN, calculate_TF_eGRN_correlation, calculate_DEGs_DARs, export_to_loom_file, export_to_UCSC_file, tree_structure, path_bedToBigBed, n_cpu, _temp_dir, save_partial, **kwargs)
    251 log.info('Calculating TF-eGRNs AUC correlation')
    252 for var in variable:
--> 253     generate_pseudobulks(scplus_obj, 
    254                      variable = var,
    255                      auc_key = 'eRegulon_AUC',
    256                      signature_key = 'Gene_based',
    257                      nr_cells = 5,
    258                      nr_pseudobulks = 100,
    259                      seed=555)
    260     generate_pseudobulks(scplus_obj, 
    261                              variable = var,
    262                              auc_key = 'eRegulon_AUC',
   (...)
    265                              nr_pseudobulks = 100,
    266                              seed=555)
    267     TF_cistrome_correlation(scplus_obj,
    268                     variable = var, 
    269                     auc_key = 'eRegulon_AUC',
    270                     signature_key = 'Gene_based',
    271                     out_key = var+'_eGRN_gene_based')

File /work/SCENIC+/scenicplus/cistromes.py:211, in generate_pseudobulks(scplus_obj, variable, normalize_expression, auc_key, signature_key, nr_cells, nr_pseudobulks, seed)
    190 """
    191 Generate pseudobulks based on the cistrome AUC matrix and gene expression
    192 
   (...)
    208     Seed to ensure that pseudobulk are reproducible.
    209 """
    210 cell_data = scplus_obj.metadata_cell
--> 211 cistromes_auc = scplus_obj.uns[auc_key][signature_key]
    212 cell_data = cell_data.loc[cistromes_auc.index, :]
    213 dgem = pd.DataFrame(
    214     scplus_obj.X_EXP, index=scplus_obj.cell_names, columns=scplus_obj.gene_names).copy()

KeyError: 'Gene_based'

I have looked though the documentation, but I cannot identidy a fix for this problem.

Is there anything I can try in order to fix this problem?

Thank you very much in advance.

[SeppeDeWinter]

Hi @laurajohanne

Thank you for the nice words.

Can you please show the output of

scplus_obj.uns['eRegulon_AUC']

Best,

Seppe

[laurajohanne]

Hi Seppe,

Thank you for your reply.

The output of scplus_obj.uns['eRegulon_AUC'] is just an empty dictionary.

scplus_obj.uns['eRegulon_AUC']
{}

The part of the output of scplus_obj.uns looks like this:
{'Cistromes': {'Unfiltered': {'ABL1_(568r)': +--------------+-----------+-----------+
| Chromosome | Start | End |
| (category) | (int32) | (int32) |
|--------------+-----------+-----------|
| chr1 | 89855679 | 89856179 |
| chr1 | 147541132 | 147541632 |
| chr1 | 84144085 | 84144585 |
| chr1 | 9559619 | 9560119 |
| ... | ... | ... |
| chrX | 357564 | 358064 |
| chrX | 11792779 | 11793279 |
| chrX | 107742032 | 107742532 |
| chrX | 1554870 | 1555370 |
+--------------+-----------+-----------+
Unstranded PyRanges object has 568 rows and 3 columns from 23 chromosomes.
For printing, the PyRanges was sorted on Chromosome.,

Thanks in advance.

Best regards Laura

[Citugulia40]

Hi, I am getting the same error, Did you find any solution for this? And also my
'scplus_obj.uns['eRegulon_AUC']' directory is empty.

Best

[SeppeDeWinter]

Hi @laurajohanne

You are running out of memory, how much do you have available?

All the best,

Seppe

[Citugulia40]

Hi, I have transferred the process on another server as well having 1.4 T of available memory and storage is also enough but I am getting the same error on that as well.

[SeppeDeWinter]

Hi @Citugulia40

Could you show the output of

print(len(scplus_obj.region_names))
print(len(scplus_obj.cell_names))

And does this produce the same error

import numpy as np
tmp = np.zeros((len(scplus_obj.region_names),
                            len(scplus_obj.cell_names)), dtype=np.int32)

All the best,

Seppe

[Citugulia40]

Thank you so much for your kind help.

print(len(scplus_obj.region_names))
print(len(scplus_obj.cell_names))

gives

799808
15036

and

import numpy as np
tmp = np.zeros((len(scplus_obj.region_names),
                            len(scplus_obj.cell_names)), dtype=np.int32)

is giving

---

MemoryError                               Traceback (most recent call last)
Cell In[60], line 2
      1 import numpy as np
----> 2 tmp = np.zeros((len(scplus_obj.region_names),
      3                             len(scplus_obj.cell_names)), dtype=np.int32)

MemoryError: Unable to allocate 44.8 GiB for an array with shape (799808, 15036) and data type int32

[SeppeDeWinter]

Hi @laurajohanne

You don't have an unreasonable large amount of cells nor regions.
Are you sure that your python process has access to the full 1.4 Tb of RAM? From the memory error it does not seem like it ...

All the best,

Seppe

[Citugulia40]

Hi,

This is my output now

free -h
              total        used        free      shared  buff/cache   available
Mem:          1.9Ti       518Gi       176Gi        44Gi       1.3Ti       1.4Ti
Swap:         8.0Gi       342Mi       7.7Gi

Thanks

[ghuls]

In general in you submit a job to a cluster you have to specify the amount of RAM that you want, else you probably get the default. e.g. 3G per requested core.