Segfault when attempting to 'import pytriqs.gf.local'

[opeil]

If I run 'a.out' (whether generated by main.c or by main.f90, does not matter) with 'test.py' containing the following line

from pytriqs.gf.local import *

I get a segfault.

Olivier, could you reproduce this?

Extra info: A closer look shows that the import chokes on '_import_array()'

#0 0x0000000000000000 in ?? ()
#1 0x00002aaab3913169 in PyEval_GetGlobals () at Python/ceval.c:3818
#2 0x00002aaab39310c3 in PyImport_Import (module_name=0x2aaab826b030) at Python/import.c:2855
#3 0x00002aaab39312fc in PyImport_ImportModule (name=) at Python/import.c:2129
#4 0x00002aaab36febd2 in import_array () from /home/opeil/Codes/triqs/triqs1.4.1/install/lib/python2.7/site-packages/pytriqs/gf/local/gf.so
#5 0x00002aaab3732341 in initgf () from /home/opeil/Codes/triqs/triqs1.4.1/install/lib/python2.7/site-packages/pytriqs/gf/local/gf.so
#6 0x00002aaaac5b02d5 in PyImport_LoadDynamicModule (name=0x63cb70 "pytriqs.gf.local.gf", pathname=0x6bca90 "/home/opeil/Codes/triqs/triqs1.4.1/install/lib/python2.7/site-packages/pytriqs/gf/local/gf.so", fp=)
at ./Python/importdl.c:53
#7 0x00002aaaac5ae951 in import_submodule (mod=0x2aaab2ffe0f8, subname=0x63cb81 "gf", fullname=0x63cb70 "pytriqs.gf.local.gf") at Python/import.c:2700
#8 0x00002aaaac5aebc4 in load_next (mod=0x2aaab2ffe0f8, altmod=0x2aaaac84b970, p_name=, buf=0x63cb70 "pytriqs.gf.local.gf", p_buflen=0x7fffffffd2f0) at Python/import.c:2515
#9 0x00002aaaac5af1f0 in import_module_level (name=, globals=, locals=, fromlist=0x2aaaacbea1d0, level=) at Python/import.c:2224
#10 PyImport_ImportModuleLevel (name=, globals=, locals=, fromlist=0x2aaaacbea1d0, level=) at Python/import.c:2288
#11 0x00002aaaac59127f in builtin___import (self=, args=, kwds=) at Python/bltinmodule.c:49
#12 0x00002aaaac4e8233 in PyObject_Call (func=0x2aaaacadc050, arg=, kw=) at Objects/abstract.c:2529
#13 0x00002aaaac591763 in PyEval_CallObjectWithKeywords (func=0x2aaaacadc050, arg=0x2aaaacbfa5d0, kw=) at Python/ceval.c:3902
#14 0x00002aaaac595c66 in PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2346
#15 0x00002aaaac599b1e in PyEval_EvalCodeEx (co=0x2aaaacc0dab0, globals=, locals=, args=, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
at Python/ceval.c:3265
#16 0x00002aaaac599c32 in PyEval_EvalCode (co=, globals=, locals=) at Python/ceval.c:673
#17 0x00002aaaac5ab462 in PyImport_ExecCodeModuleEx (name=0x6ba760 "pytriqs.gf.local", co=0x2aaaacc0dab0, pathname=0x63bb60 "/home/opeil/Codes/triqs/triqs1.4.1/install/lib/python2.7/site-packages/pytriqs/gf/local/init.pyc")
at Python/import.c:709
#18 0x00002aaaac5adb9e in load_source_module (name=0x6ba760 "pytriqs.gf.local", pathname=0x63bb60 "/home/opeil/Codes/triqs/triqs1.4.1/install/lib/python2.7/site-packages/pytriqs/gf/local/init.pyc", fp=)
at Python/import.c:1099
#19 0x00002aaaac5ae3fa in load_package (name=0x6ba760 "pytriqs.gf.local", pathname=) at Python/import.c:1166
#20 0x00002aaaac5ae951 in import_submodule (mod=0x2aaaacc085c8, subname=0x6ba76b "local", fullname=0x6ba760 "pytriqs.gf.local") at Python/import.c:2700
#21 0x00002aaaac5aebc4 in load_next (mod=0x2aaaacc085c8, altmod=0x2aaaacc085c8, p_name=, buf=0x6ba760 "pytriqs.gf.local", p_buflen=0x7fffffffd8c0) at Python/import.c:2515
#22 0x00002aaaac5af230 in import_module_level (name=, globals=, locals=, fromlist=0x2aaaacbd3310, level=) at Python/import.c:2232
#23 PyImport_ImportModuleLevel (name=, globals=, locals=, fromlist=0x2aaaacbd3310, level=) at Python/import.c:2288
#24 0x00002aaaac59127f in builtin___import__ (self=, args=, kwds=) at Python/bltinmodule.c:49
#25 0x00002aaaac4e8233 in PyObject_Call (func=0x2aaaacadc050, arg=, kw=) at Objects/abstract.c:2529
#26 0x00002aaaac591763 in PyEval_CallObjectWithKeywords (func=0x2aaaacadc050, arg=0x2aaaacbfa520, kw=) at Python/ceval.c:3902
#27 0x00002aaaac595c66 in PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2346
#28 0x00002aaaac599b1e in PyEval_EvalCodeEx (co=0x2aaaacbc2eb0, globals=, locals=, args=, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
at Python/ceval.c:3265
#29 0x00002aaaac599c32 in PyEval_EvalCode (co=, globals=, locals=) at Python/ceval.c:673
#30 0x00002aaaac5b99f0 in run_mod (fp=0x63a060, filename=, start=, globals=0x2aaaacaf6168, locals=0x2aaaacaf6168, closeit=0, flags=0x0) at Python/pythonrun.c:1377
#31 PyRun_FileExFlags (fp=0x63a060, filename=, start=, globals=0x2aaaacaf6168, locals=0x2aaaacaf6168, closeit=0, flags=0x0) at Python/pythonrun.c:1363
#32 0x00002aaaac5b9bcf in PyRun_SimpleFileExFlags (fp=0x63a060, filename=0x7fffffffe1a4 "test.py", closeit=0, flags=0x0) at Python/pythonrun.c:955
#33 0x0000000000400e0a in execute_python_file ()
#34 0x0000000000400bdd in main ()

[parcollet]

Can you retry with
RTLD_GLOBAL
instead of RTLD_LAZY
@mferrero : can you help here ? you solved a similar pb with boost long ago...

[opeil]

I added RTLD_GLOBAL to RTLD_LAZY (one must specify either LAZY or NOW).

It has solved the problem with the segfault. Good!

However, the next problem appeared: a peculiar segfault when the Python script is invoked the second time, i.e. main.c is modified as follows:

// launch python
execute_python_file(argv[1], argv[2]);
execute_python_file(argv[1], argv[2]);

The errors appears only if I try to import pytriqs.gf.local. Seems like some of the libraries (gf.so? libtriqs.so?) are not unloaded properly after PyFinalize().

The call-stack is huge. I put only several first and last lines.

#0 0x00002aaab85c33eb in PyArray_Item_INCREF (data=0x2aaaacbb4ef0 "\001", descr=0x2aaab8855ce0) at numpy/core/src/multiarray/refcount.c:35
#1 0x00002aaab85c748b in PyArray_FromScalar (scalar=, outcode=0x0) at numpy/core/src/multiarray/scalarapi.c:335
#2 0x00002aaab85c93dc in gentype_nonzero_number (m1=) at numpy/core/src/multiarray/scalartypes.c.src:337
#3 0x00002aaaac531af7 in PyObject_IsTrue (v=) at Objects/object.c:1578
#4 0x00002aaaac5956b9 in PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2396
#5 0x00002aaaac598a1e in fast_function (f=, throwflag=) at Python/ceval.c:4119
#6 call_function (f=, throwflag=) at Python/ceval.c:4054
#7 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2679
#8 0x00002aaaac599b1e in PyEval_EvalCodeEx (co=0x2aaabb48a430, globals=, locals=, args=, argcount=3, kws=0x2aaaba5c87e8, kwcount=0, defs=0x2aaaacb29068, defcount=1,
closure=0x0) at Python/ceval.c:3265
#9 0x00002aaaac597cd2 in fast_function (f=, throwflag=) at Python/ceval.c:4129
#10 call_function (f=, throwflag=) at Python/ceval.c:4054
#11 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2679
#12 0x00002aaaac599b1e in PyEval_EvalCodeEx (co=0x2aaabb48a2b0, globals=, locals=, args=, argcount=2, kws=0x623270, kwcount=0, defs=0x2aaab9cb0de8, defcount=2, closure=0x0)
at Python/ceval.c:3265
#13 0x00002aaaac597cd2 in fast_function (f=, throwflag=) at Python/ceval.c:4129
#14 call_function (f=, throwflag=) at Python/ceval.c:4054
#15 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2679
#16 0x00002aaaac599b1e in PyEval_EvalCodeEx (co=0x2aaaacb37db0, globals=, locals=, args=, argcount=2, kws=0x2aaab9ca2e18, kwcount=0, defs=0x2aaaacb29268, defcount=1,
closure=0x0) at Python/ceval.c:3265
#17 0x00002aaaac597cd2 in fast_function (f=, throwflag=) at Python/ceval.c:4129
#18 call_function (f=, throwflag=) at Python/ceval.c:4054
#19 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2679
#20 0x00002aaaac599b1e in PyEval_EvalCodeEx (co=0x2aaabcdea9b0, globals=, locals=, args=, argcount=2, kws=0x2aaabcde2ca0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
at Python/ceval.c:3265

.............

#120 0x00002aaaac5ae3fa in load_package (name=0xa93100 "pytriqs.gf.local", pathname=) at Python/import.c:1166
#121 0x00002aaaac5ae951 in import_submodule (mod=0x2aaac03a1e50, subname=0xa9310b "local", fullname=0xa93100 "pytriqs.gf.local") at Python/import.c:2700
#122 0x00002aaaac5aebc4 in load_next (mod=0x2aaac03a1e50, altmod=0x2aaac03a1e50, p_name=, buf=0xa93100 "pytriqs.gf.local", p_buflen=0x7fffffffd8c0) at Python/import.c:2515
#123 0x00002aaaac5af230 in import_module_level (name=, globals=, locals=, fromlist=0x2aaabcded6d0, level=) at Python/import.c:2232
#124 PyImport_ImportModuleLevel (name=, globals=, locals=, fromlist=0x2aaabcded6d0, level=) at Python/import.c:2288
#125 0x00002aaaac59127f in builtin___import__ (self=, args=, kwds=) at Python/bltinmodule.c:49
#126 0x00002aaaac4e8233 in PyObject_Call (func=0x2aaaacc13ab8, arg=, kw=) at Objects/abstract.c:2529
#127 0x00002aaaac591763 in PyEval_CallObjectWithKeywords (func=0x2aaaacc13ab8, arg=0x2aaabd2b0940, kw=) at Python/ceval.c:3902
#128 0x00002aaaac595c66 in PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2346
#129 0x00002aaaac599b1e in PyEval_EvalCodeEx (co=0x2aaabb4860b0, globals=, locals=, args=, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
at Python/ceval.c:3265
#130 0x00002aaaac599c32 in PyEval_EvalCode (co=, globals=, locals=) at Python/ceval.c:673
#131 0x00002aaaac5b99f0 in run_mod (fp=0x6456c0, filename=, start=, globals=0x2aaaacbe3910, locals=0x2aaaacbe3910, closeit=0, flags=0x0) at Python/pythonrun.c:1377
#132 PyRun_FileExFlags (fp=0x6456c0, filename=, start=, globals=0x2aaaacbe3910, locals=0x2aaaacbe3910, closeit=0, flags=0x0) at Python/pythonrun.c:1363
#133 0x00002aaaac5b9bcf in PyRun_SimpleFileExFlags (fp=0x6456c0, filename=0x7fffffffe1a4 "test.py", closeit=0, flags=0x0) at Python/pythonrun.c:955
#134 0x0000000000400e2a in execute_python_file ()
#135 0x0000000000400bfe in main ()

[parcollet]

Ok, I can not reproduce it here.
It works. There was just a LD_IBRARY_PATH pb to find triqs.so,
but that is another question.

Are you sure that you dlopened the same lib python as the one you compiled against ?

[parcollet]

PS : I was talking about the first error ... Did not refresh the navigator

[opeil]

Concerning the second error: it seems to have nothing to do with TRIQS. I can reproduce it by simply importing numpy in 'test.py'. Could you try it?

[parcollet]

Second error : I get the same here (still with RDT_LAZY, so nothing to do).
Stack is different, but the second call crashes

[parcollet]

yes, numpy is enough... strangly mpi4py does not cause a crash

[opeil]

Seems like a very similar (and very old) pb: https://bugs.launchpad.net/ubuntu/+source/python-numpy/+bug/184920

[opeil]

True: If I comment out PyFinalize() the segfault goes away. But this is not a solution.

[parcollet]

-yes, looks like it.

• well yes, that is not an option to remove PyFinalise...

Do we simply want to keep the interpreter open ??
It is a bit more complex (3 functions called as you first suggested)
but after all simpler ...

Let me look at Python C API ...

[parcollet]

ok, there is a
int Py_IsInitialized()
I will try now

[opeil]

Seems like we might have to do this, i.e. split Python embedder into two or three calls. I see these complaints about numpy not being able to clean-up properly everywhere. Incidentally, here is one answered by the author of mpi4py:
https://mail.scipy.org/pipermail/numpy-discussion/2009-March/040849.html

As far as I understand, one can safely keep Python open and call PyInitalize() multiple times. It is smart enough to do nothing when the interpreter is already there.
A compromise solution is to use two calls: One as it is right now but without PyFinalize(); and the second, finalizing Python, at the end of the host application.

[parcollet]

Done & pushed. Cf main.c

[parcollet]

It will never close the .so and the interpreter.
I use the handle of the lib python as a signal that the interpreter is initialized...
Ok, I am adding checks now ..

[opeil]

Ok. Preliminary the solution with PyFinalize() removed runs within VASP cycle. I will do more tests tomorrow.

[parcollet]

Ok. Cf the new pushed version, it is cleaner.
I am adding a few checks now.

[parcollet]

Pushed. Cf commit for a new version with 3 functions.