Skip to content

All record types fail when getRecord() is used for the second time #60

New issue
New issue
Closed
Closed
All record types fail when getRecord() is used for the second time#60
Assignees
pheest
Labels
bug

Description

@pheest
pheest
opened on Sep 16, 2026

I am adding new loopback tests (i.e. write to the aao record VAL field and scan using the OUT field to propagate to the aai record having the same FTVL).

The test fails usually with the values read back being discrepant.
Sometimes it passes.
Sometimes it delivers an access violation error code 0XC0000005.
Sometimes the test locks up.

Activity

  1. pheest
    self-assigned this
    on Sep 16, 2026
  2. pheest
    added
    bug
    on Sep 16, 2026

  3. pheest commented on Sep 21, 2026

    @pheest
    pheest
    on Sep 21, 2026
    CollaboratorAuthor

    This issue is caused by use of record caching in db.py.
    The issue is not restricted to use of array types, but is generic to all types.

    When a record instance is created by calling getRecord(), a reference to the record is stored in the global _rec_cache variable, and a reference is also returned.
    Subsequent calls to getRecord() then return the reference stored in _rec_cache, rather than creating a new instance from scratch.

    This usage is aimed at reducing the run-time overhead of re-creating records that are used repeatedly - typically those that are referenced at local scope.

    This concept fails because resources that are held by the record are de-allocated by the destructor when the record instance leaves scope.

    Any subsequent attempt to re-use the already-created record consequently fails, typically resulting in segmentation fault or unpredictable behaviour. This is reproducible on both Windows and Linux.

    There are no circumstances in which the cached record capability can be safely used.

  4. mdavidsaver commented on Sep 21, 2026

    @mdavidsaver
    mdavidsaver
    on Sep 21, 2026
    Collaborator

    This concept fails because resources that are held by the record are de-allocated by the destructor when the record instance leaves scope.

    I am having trouble parsing this statement. Which resources? (just memory?) Which destructor? Which scope?

    typically resulting in segmentation fault or unpredictable behaviour. This is reproducible on both Windows and Linux.

    Can you provide a stack trace?

    Do the test cases added in test_db.py trigger SIGSEGV if the change in db.py is reverted?

    I am wondering if a crash as you describe might actually be due to missing ref. count manipulation elsewhere.

  5. pheest commented on Sep 21, 2026

    @pheest
    pheest
    on Sep 21, 2026
    CollaboratorAuthor

    Hi @mdavidsaver,

    I have reliably reproduced the issue on Windows and Linux by copy'n'pasting a duplicate of the existing test 'def test_link(self):' and running the tests. This causes the 'src' and 'tgt' scalar records to be re-used in local scope, and fails on the second instance.
    I would invite you to do the same (using the existing code).

    I also reproduced the issue on Windows using my newly added array tests which re-use the 'rec:aai:l' and 'rec:aai:f' records. I did not get around to re-verifying the matter on Linux however.

    The change in db.py fixes the issue by removing the caching behaviour. This fully addresses the issue.

  6. mdavidsaver commented on Sep 22, 2026

    @mdavidsaver
    mdavidsaver
    on Sep 22, 2026
    Collaborator

    I am able to reproduce with https://github.com/mdavidsaver/pyDevSup-dev/tree/refs/heads/ci using python3-dbg from Debian 13. So the crash involves an array field feeding some invalid pointers into numpy..

    (gdb) py-bt
Traceback (most recent call first):
  File "/usr/lib/python3/dist-packages/numpy/_core/arrayprint.py", line 989, in fillFormat
    finite_vals = data[isfinite(data)]
  File "/usr/lib/python3/dist-packages/numpy/_core/arrayprint.py", line 985, in __init__
    self.fillFormat(data)
  File "/usr/lib/python3/dist-packages/numpy/_core/arrayprint.py", line 455, in <lambda>
    'float': lambda: FloatingFormat(
  File "/usr/lib/python3/dist-packages/numpy/_core/arrayprint.py", line 516, in _get_format_function
    return formatdict['float']()
  File "/usr/lib/python3/dist-packages/numpy/_core/arrayprint.py", line 583, in _array2string
    format_function = _get_format_function(data, **options)
  File "/usr/lib/python3/dist-packages/numpy/_core/arrayprint.py", line 557, in wrapper
    return f(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/numpy/_core/arrayprint.py", line 784, in array2string
    return _array2string(a, options, separator, prefix)
  File "/usr/lib/python3/dist-packages/numpy/_core/arrayprint.py", line 1591, in _array_repr_implementation
    lst = array2string(arr, max_line_width, precision, suppress_small,
  File "/usr/lib/python3/dist-packages/numpy/_core/arrayprint.py", line 1674, in array_repr
    return _array_repr_implementation(
  File "/usr/lib/python3/dist-packages/numpy/testing/_private/utils.py", line 243, in build_err_msg
    r = r_func(a)
  File "/usr/lib/python3/dist-packages/numpy/testing/_private/utils.py", line 801, in assert_array_compare
    msg = build_err_msg([x, y],
  File "/usr/lib/python3/dist-packages/numpy/testing/_private/utils.py", line 1172, in assert_array_almost_equal
    assert_array_compare(compare, actual, desired, err_msg=err_msg,
  File "/usr/lib/python3/dist-packages/numpy/_utils/__init__.py", line 86, in wrapper
    return fun(*args, **kwargs)
  File "/home/dev/epics/pyDevSup/python3.13d/linux-x86_64/devsup/test/test_db.py", line 120, in array_float_test
    assert_array_almost_equal(tgt.VAL, [])
  File "/home/dev/epics/pyDevSup/python3.13d/linux-x86_64/devsup/test/test_db.py", line 229, in test_aao_float
    self.array_float_test(src, tgt)
...

    Thread 1 "python3-dbg" received signal SIGSEGV, Segmentation fault.
Download failed: Invalid argument.  Continuing without source file ./string/../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S.
__memcpy_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:265
warning: 265    ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory
(gdb) bt
#0  __memcpy_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:265
#1  0x00007ffff56643c9 in memmove (__dest=<optimized out>, __src=<optimized out>, __len=<optimized out>)
    at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:36
#2  _contig_to_contig (context=<optimized out>, args=<optimized out>, dimensions=<optimized out>, __NPY_UNUSED_TAGGEDstrides=<optimized out>, 
    __NPY_UNUSED_TAGGEDdata=<optimized out>) at ../numpy/_core/src/multiarray/lowlevel_strided_loops.c.src:384
#3  0x00007ffff56e81b4 in raw_array_assign_array (ndim=<optimized out>, shape=<optimized out>, dst_dtype=<optimized out>, 
    dst_data=<optimized out>, dst_strides=<optimized out>, src_dtype=<optimized out>, src_data=<optimized out>, src_strides=0x7fffffff3e30)
    at ../numpy/_core/src/multiarray/array_assign_array.c:150
#4  0x00007ffff56e8ca7 in PyArray_AssignArray (dst=0x7ffff45269c0, src=0x7ffff4526940, wheremask=0x0, casting=NPY_NO_CASTING)
    at ../numpy/_core/include/numpy/ndarraytypes.h:1508
#5  0x00007ffff56fe4b0 in PyArray_CopyInto (dst=dst@entry=0x7ffff45269c0, src=src@entry=0x7ffff4526940)
    at ../numpy/_core/src/multiarray/ctors.c:2900
#6  0x00007ffff56f6d75 in PyArray_CastToType (arr=0x7ffff4526940, dtype=0x7ffff5f7aec0 <DOUBLE_Descr>, is_f_order=is_f_order@entry=0)
    at ../numpy/_core/src/multiarray/convert_datatype.c:317
#7  0x00007ffff57f4c4b in check_for_trivial_loop (ufuncimpl=0x7ffff553ce10, op=0x7fffffff4960, dtypes=0x7fffffff4d60, 
    casting=NPY_SAME_KIND_CASTING, buffersize=<optimized out>) at ../numpy/_core/src/umath/ufunc_object.c:808
#8  PyUFunc_GenericFunctionInternal (ufunc=0x7ffff6e439b0, ufuncimpl=0x7ffff553ce10, operation_descrs=0x7fffffff4d60, op=0x7fffffff4960, 
    casting=NPY_SAME_KIND_CASTING, order=NPY_KEEPORDER, wheremask=<optimized out>) at ../numpy/_core/src/umath/ufunc_object.c:2230
#9  ufunc_generic_fastcall (ufunc=<optimized out>, args=<optimized out>, len_args=<optimized out>, kwnames=<optimized out>, 
    outer=<optimized out>) at ../numpy/_core/src/umath/ufunc_object.c:4530
#10 0x00000000004b95d1 in _PyObject_VectorcallTstate (tstate=0xafbe80 <_PyRuntime+299040>, 
    callable=callable@entry=<numpy.ufunc at remote 0x7ffff6e439b0>, args=args@entry=0x7ffff7f929d8, nargsf=9223372036854775809, 
    kwnames=kwnames@entry=0x0) at ../Include/internal/pycore_call.h:168
#11 0x00000000004b96a1 in PyObject_Vectorcall (callable=callable@entry=<numpy.ufunc at remote 0x7ffff6e439b0>, 
    args=args@entry=0x7ffff7f929d8, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at ../Objects/call.c:327
#12 0x00000000005d2525 in _PyEval_EvalFrameDefault (tstate=0xafbe80 <_PyRuntime+299040>, frame=0x7ffff7f928f0, throwflag=0)
...

    Looking at C frame 4 I see an apparently bad data = 0x61 pointer in src.

    (gdb) p *(PyArrayObject_fields*)dst
$9 = {ob_base = {{ob_refcnt = 1, ob_refcnt_split = {1, 0}}, ob_type = 0x7ffff5f857a0 <PyArray_Type>}, 
  data = 0x12455d0 " \313\343\367\377\177", nd = 1, dimensions = 0x1243f30, strides = 0x1243f38, base = 0x0, 
  descr = 0x7ffff5f7aec0 <DOUBLE_Descr>, flags = 1287, weakreflist = 0x0, _buffer_info = 0x0, 
  mem_handler = <PyCapsule at remote 0x7ffff6ddc600>}

    (gdb) p *(PyArrayObject_fields*)src
$6 = {ob_base = {{ob_refcnt = 32, ob_refcnt_split = {32, 0}}, ob_type = 0x7ffff5f857a0 <PyArray_Type>}, 
  data = 0x61 <error: Cannot access memory at address 0x61>, nd = 1, dimensions = 0x1243fb0, strides = 0x1243fb8, base = 0x0, 
  descr = 0x7ffff5f7aec0 <DOUBLE_Descr>, flags = 3, weakreflist = 0x0, _buffer_info = 0x0, mem_handler = 0x0}

  7. mdavidsaver commented on Sep 22, 2026

    @mdavidsaver
    mdavidsaver
    on Sep 22, 2026
    Collaborator

    Ah, RR to the rescue. _rec_cache needs to be cleared by IOCHelper after each test cycle. testdbCleanup() frees all PDB records, which invalidates the DBADDR held by the python Record and Field objects.

  8. pheest commented on Oct 7, 2026

    @pheest
    pheest
    on Oct 7, 2026
    CollaboratorAuthor

    There is still a problem specifically with the AAO record type.
    When running an pyDevSup IOC that uses it, and then externally setting the value by means of e.g. caput, the IOC aborts without any message.

    The issue does not occur with any other record types.

    I have similarly tested this while using SoftIOC, and there was no issue.

  9. pheest commented on Oct 13, 2026

    @pheest
    pheest
    on Oct 13, 2026
    CollaboratorAuthor

    After code build, extensive testing has not been able to reproduce an issue with the AAO record type.
    I am sorry for raising a false alarm.

  10. pheest
    changed the title [-]Use of the aai and/or aao record types does not work correctly[/-] [+]All record types fail when getRecord() is used for the second time[/+] on Oct 20, 2026

  11. tynanford commented on Oct 26, 2026

    @tynanford
    tynanford
    on Oct 26, 2026
    Collaborator

    Fixed with #61

  12. tynanford
    closed this as completedon Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bug

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions