[BUG]: `cast_error` using `PYBIND11_OVERRIDE` with implicitly converted opaque vectors

[ManuelSchneid3r]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

3.0.4

Problem description

When invoking this

PYBIND11_OVERRIDE_PURE(std::vector<std::shared_ptr<Item>>, FallbackHandler, fallbacks, query);

where the return type is an opaque type with an implicit conversion from a Python list like this:

py::bind_vector<std::vector<std::shared_ptr<Item>>>(m, "ItemList");
py::implicitly_convertible<py::list, std::vector<std::shared_ptr<Item>>>();

and the actual value is a list, i.e. return [Item(…)], that has to be converted implicitly, pybind11 throws:

libc++abi: terminating due to uncaught exception of type pybind11::cast_error: When called outside a bound function, py::cast() cannot do Python -> C++ conversions which require the creation of temporary values

Root Cause (LLM hypothesis)

Because the call chain originates from native C++ rather than a pybind11 function dispatcher, no active frame context exists. type_caster_generic::load_impl calls loader_life_support::add_patient to handle the implicit conversion temporary, causing the lifecyle framework to crash.

Stack Trace

  * frame #0: 0x0000000100068cd0 python_test`pybind11::detail::loader_life_support::add_patient(h=handle @ 0x000000016fdfde90) at type_caster_base.h:93:13
    frame #1: 0x0000000100068508 python_test`bool pybind11::detail::type_caster_generic::load_impl<pybind11::detail::type_caster_generic>(this=0x000000016fdfe248, src=handle @ 0x000000016fdfe090, convert=true) at type_caster_base.h:1274:21
    frame #2: 0x0000000100068048 python_test`pybind11::detail::type_caster_generic::load(this=0x000000016fdfe248, src=handle @ 0x000000016fdfe0d8, convert=true) at type_caster_base.h:994:50
    frame #3: 0x00000001001142c4 python_test`pybind11::detail::type_caster<std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>, void>& pybind11::detail::load_type<std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>, void>(conv=0x000000016fdfe248, handle=0x000000016fdfe3b8) at cast.h:1647:15
    frame #4: 0x0000000100114278 python_test`pybind11::detail::type_caster<pybind11::detail::intrinsic_type<std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>>::type, void> pybind11::detail::load_type<std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>>(handle=0x000000016fdfe3b8) at cast.h:1666:5
    frame #5: 0x00000001001141d0 python_test`std::__1::enable_if<!detail::move_never<std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>>::value, std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>>::type pybind11::move<std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>>(obj=0x000000016fdfe3b8) at cast.h:1769:23
    frame #6: 0x0000000100113f2c python_test`std::__1::enable_if<!detail::is_pyobject<std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>>::value && detail::move_if_unreferenced<std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>>::value, std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>>::type pybind11::cast<std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>>(object=0x000000016fdfe3b8) at cast.h:1789:12
    frame #7: 0x00000001001260c0 python_test`std::__1::enable_if<detail::none_of<std::__1::integral_constant<bool, (std::is_reference<std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>>::value || std::is_pointer<std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>>::value) && !std::is_base_of<pybind11::detail::type_caster_generic, pybind11::detail::type_caster<pybind11::detail::intrinsic_type<std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>>::type, void>>::value && !std::is_same<pybind11::detail::intrinsic_type<std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>>::type, void>::value>, std::__1::is_same<__remove_cvref(std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>), _object*>, std::__1::is_void<std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>>>::value, std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>>::type pybind11::detail::cast_safe<std::__1::vector<std::__1::shared_ptr<albert::Item>, std::__1::allocator<std::__1::shared_ptr<albert::Item>>>>(o=0x000000016fdfe3b8) at cast.h:1902:12
    frame #8: 0x0000000100125bd8 python_test`PyFallbackHandler<albert::FallbackHandler>::fallbacks(this=0x00006000018a1200, query=0x000000016fdfe440) const at trampolineclasses.hpp:529:9
    frame #9: 0x0000000100172cd0 python_test`PythonTests::testFallbackQueryHandler(this=0x000000016fdfef70) at test.cpp:1139:32
[…]

Workaround (LLM suggestion that actually works)

Instantiating internal loader_life_support directly inside the trampoline resolves the issue, but relies on non-public APIs:

std::vector<std::shared_ptr<Item>> fallbacks(const QString &query) const override {
    pybind11::detail::loader_life_support life_support; 
    PYBIND11_OVERRIDE_PURE(std::vector<std::shared_ptr<Item>>, FallbackHandler, fallbacks, query);
}