Problems with execution info in Neo4J #8204

Pelayo-Reguera · 2026-08-31T21:53:30Z

Pelayo-Reguera
Aug 31, 2026

Hello everyone!

I'm using v2.19.0 with Neo4J v5 and I'm always getting a warning message saying that it wasn't able to register data and state into the database.

From the execution information perspective I can see the execution history and some details in each execution like logs and metrics. I've managed to know how they are working and search for errors, but I would appreciate it if I could get the execution lineage, error lineage and data tabs working.

In addition, I also noticed that many times when there is an error I can see in red the transform or action in the parent pipeline or workflow, but when I click on the drill down button to go to the referenced pipeline or workflow there is nothing marked.

So I would be grateful if someone could give me some advice of any type to try to solve this, because I'm not sure if it's something related to Apache HOP, the Neo4J configuration or anything else.

Answered by mattcasters

Sep 10, 2026

This feedback led to a thorough review of the Neo4j code in PR #8282

View full answer

bamaer · 2026-09-03T12:34:42Z

bamaer
Sep 3, 2026
Collaborator

do you get any errors or other information in the console you used to start Hop Gui?

4 replies

Pelayo-Reguera Sep 14, 2026
Author

Sorry for not answering before. Yes, after 5min or more I get the following error (details). It's like something related to the size of the commit, but I'm not sure how to split the execution information in more commits or at least select what information or content of the fields to store. Because for example, I'm retrieving JSON information from an API I'm working with, and I'm only interested in the extracted fields, not the JSON body that the API returns (because it's huge and I deduce it's the reason why the commit is so big).

Details

2026/09/14 09:27:06 [Error]    Volcado_logs - Error handling writing final pipeline state to location (non-fatal)
2026/09/14 09:27:06 [Error]    Volcado_logs - org.apache.hop.core.exception.HopException: 
2026/09/14 09:27:06 [Error]    Volcado_logs - Error getting execution from Neo4j
2026/09/14 09:27:06 [Error]    Volcado_logs - Error registering execution data to Neo4j
2026/09/14 09:27:06 [Error]    Volcado_logs - 
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.registerData(NeoExecutionInfoLocation.java:1109)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.apache.hop.pipeline.engines.local.LocalPipelineEngine.stopTransformExecutionInfoTimer(LocalPipelineEngine.java:527)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.apache.hop.pipeline.engines.local.LocalPipelineEngine.pipelineCompleted(LocalPipelineEngine.java:484)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.apache.hop.pipeline.Pipeline.fireExecutionFinishedListeners(Pipeline.java:1461)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.apache.hop.pipeline.Pipeline.lambda$startThreads$0(Pipeline.java:1259)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.apache.hop.pipeline.transform.BaseTransform.fireTransformFinishedListeners(BaseTransform.java:2978)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.apache.hop.pipeline.transform.BaseTransform.markStop(BaseTransform.java:2971)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.apache.hop.pipeline.transform.RunThread.run(RunThread.java:141)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at java.base/java.lang.Thread.run(Thread.java:1583)
2026/09/14 09:27:06 [Error]    Volcado_logs - Caused by: org.apache.hop.core.exception.HopRuntimeException: Error registering execution data to Neo4j
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.registerNeo4jData(NeoExecutionInfoLocation.java:1186)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.lambda$registerData$13(NeoExecutionInfoLocation.java:1107)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.neo4j.driver.internal.InternalSession.lambda$transaction$1(InternalSession.java:163)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.neo4j.driver.internal.retry.ExponentialBackoffRetryLogic.retry(ExponentialBackoffRetryLogic.java:119)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.neo4j.driver.internal.InternalSession.lambda$transaction$0(InternalSession.java:160)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.neo4j.driver.internal.observation.util.ObservationUtil.observe(ObservationUtil.java:40)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.neo4j.driver.internal.InternalSession.transaction(InternalSession.java:160)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.neo4j.driver.internal.InternalSession.execute(InternalSession.java:145)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.neo4j.driver.internal.InternalSession.executeWrite(InternalSession.java:123)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.neo4j.driver.Session.executeWrite(Session.java:123)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.registerData(NeoExecutionInfoLocation.java:1107)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	... 8 more
2026/09/14 09:27:06 [Error]    Volcado_logs - Caused by: org.apache.hop.core.exception.HopRuntimeException: Error saving rows and their metadata in Neo4j
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.saveNeo4jRowsAndMeta(NeoExecutionInfoLocation.java:1363)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.registerNeo4jData(NeoExecutionInfoLocation.java:1178)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	... 18 more
2026/09/14 09:27:06 [Error]    Volcado_logs - Caused by: org.neo4j.driver.exceptions.TransientException: The allocation of an extra 23.5 MiB would use more than the limit 21.0 GiB. Currently using 21.0 GiB. dbms.memory.transaction.total.max threshold reached
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.neo4j.driver.internal.util.Futures.blockingGet(Futures.java:63)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.neo4j.driver.internal.InternalTransaction.lambda$run$0(InternalTransaction.java:81)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.neo4j.driver.internal.observation.util.ObservationUtil.observe(ObservationUtil.java:40)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.neo4j.driver.internal.InternalTransaction.run(InternalTransaction.java:80)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.neo4j.driver.internal.AbstractQueryRunner.run(AbstractQueryRunner.java:32)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.neo4j.driver.internal.AbstractQueryRunner.run(AbstractQueryRunner.java:37)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.neo4j.driver.internal.DelegatingTransactionContext.run(DelegatingTransactionContext.java:41)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.execute(NeoExecutionInfoLocation.java:1747)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	at org.apache.hop.neo4j.execution.NeoExecutionInfoLocation.saveNeo4jRowsAndMeta(NeoExecutionInfoLocation.java:1331)
2026/09/14 09:27:06 [Error]    Volcado_logs - 	... 19 more
2026/09/14 09:27:06 [Error]    Volcado_logs - 	Suppressed: org.neo4j.driver.internal.util.ErrorUtil$InternalExceptionCause
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.adaptedbolt.ErrorMapper.mapToNeo4jException(ErrorMapper.java:224)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.adaptedbolt.ErrorMapper.mapBoltFailureException(ErrorMapper.java:165)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.adaptedbolt.RoutedErrorMapper.mapBoltFailureException(RoutedErrorMapper.java:64)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.adaptedbolt.ErrorMapper.map(ErrorMapper.java:81)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.adaptedbolt.AdaptingDriverResponseHandler.onError(AdaptingDriverResponseHandler.java:68)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.bolt.connection.routed.impl.RoutedBoltConnection$RoutedResponseHandler.onError(RoutedBoltConnection.java:160)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.bolt.connection.pooled.impl.PooledBoltConnection$PooledResponseHandler.onError(PooledBoltConnection.java:204)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.bolt.connection.netty.impl.BoltConnectionImpl$ResponseHandleImpl.lambda$onError$0(BoltConnectionImpl.java:677)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.bolt.connection.netty.impl.BoltConnectionImpl$ResponseHandleImpl.runIgnoringError(BoltConnectionImpl.java:809)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.bolt.connection.netty.impl.BoltConnectionImpl$ResponseHandleImpl.onError(BoltConnectionImpl.java:677)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.bolt.connection.netty.impl.BoltConnectionImpl$3.onError(BoltConnectionImpl.java:309)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.bolt.connection.netty.impl.messaging.v3.BoltProtocolV3.lambda$run$0(BoltProtocolV3.java:431)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.bolt.connection.netty.impl.handlers.RunResponseHandler.onFailure(RunResponseHandler.java:50)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.bolt.connection.netty.impl.async.inbound.InboundMessageDispatcher.handleFailureMessage(InboundMessageDispatcher.java:108)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.bolt.connection.netty.impl.messaging.v57.MessageReaderV57.unpackFailureMessage(MessageReaderV57.java:46)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.bolt.connection.netty.impl.messaging.common.CommonMessageReader.read(CommonMessageReader.java:53)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.bolt.connection.netty.impl.async.inbound.InboundMessageHandler.channelRead0(InboundMessageHandler.java:81)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.bolt.connection.netty.impl.async.inbound.InboundMessageHandler.channelRead0(InboundMessageHandler.java:34)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:361)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:325)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.bolt.connection.netty.impl.async.inbound.MessageDecoder.channelRead(MessageDecoder.java:40)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:361)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:348)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:470)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:288)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1429)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:176)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.handle(AbstractNioChannel.java:445)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioIoHandler$DefaultNioRegistration.handle(NioIoHandler.java:388)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioIoHandler.processSelectedKey(NioIoHandler.java:596)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioIoHandler.processSelectedKeysOptimized(NioIoHandler.java:571)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioIoHandler.processSelectedKeys(NioIoHandler.java:512)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioIoHandler.run(NioIoHandler.java:484)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.channel.SingleThreadIoEventLoop.runIo(SingleThreadIoEventLoop.java:225)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:196)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1195)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2026/09/14 09:27:06 [Error]    Volcado_logs - 		... 1 more

mattcasters Sep 14, 2026
Collaborator

Your reply is again quite lacking in details :-)
Are you saying you have a huge String, the result of an API call, that somehow gets logged?
You would like to have options to enable or disable storing various aspects of the execution information location then?

Pelayo-Reguera Sep 25, 2026
Author

Yeah, that would be great. But let me explain a bit more about the situation and what I have in mind.

The fact is that the execution information is not being stored correctly in Neo4j v5, as I mentioned in my first message with more details. Basically, I can see the logs, but not much more. I think that part is already fixed in PR #8282, as you mentioned.

From the second screenshot I sent, I understood that the problem is related to trying to send all the execution information at once, in a single transaction/commit to the database. As the error says:

Caused by: org.neo4j.driver.exceptions.TransientException: The allocation of an extra 23.5 MiB would use more than the limit 21.0 GiB. Currently using 21.0 GiB. dbms.memory.transaction.total.max threshold reached

The context is that I'm retrieving information from an API, and the response can be huge (and unfortunately, I cannot make it smaller). In other cases, the problem is that I end up accumulating a lot of different columns.

So I was wondering if there could be a possible solution for these cases where the information is split into several commits, or perhaps an option that allows the user to choose which columns they want to store in the database as execution information.

For example, let's say I have a column that contains the request to an API, another column containing the API response, and then several other columns (let's say a, b, and c) generated from that response using the JSON transform.

I need all of these columns to get the results I want from the Pipeline, but if something goes wrong and I need to inspect the execution afterwards, I may only be interested in seeing the information from columns a and c. In that case, there's no real need to store the API response in Neo4j, especially if each response can be around 500 KiB.

So, assuming something like this could be implemented, I would expect that when retrieving the execution information from the database and displaying it in the UI, the fields I selected to store would show the values they had when the pipeline was executed, while the columns I chose not to store (such as the API response) could simply show something like <no info>.

It might be quite a big feature, but I think it could be an interesting one. Anyway, let me know if I can help clarify the idea further or provide more details about the use case.

mattcasters Sep 25, 2026
Collaborator

Just to be clear: you don't have to store rows of data and not all of them either. A data profile is optional. If that is the issue, you can already solve it easily.

Otherwise: #8617

mattcasters · 2026-09-10T10:56:33Z

mattcasters
Sep 10, 2026
Collaborator

This feedback led to a thorough review of the Neo4j code in PR #8282

0 replies

Navigation Menu

Repository navigation

Problems with execution info in Neo4J #8204

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Problems with execution info in Neo4J #8204

Uh oh!

Pelayo-Reguera Aug 31, 2026

Replies: 2 comments · 4 replies

Uh oh!

bamaer Sep 3, 2026 Collaborator

Uh oh!

Pelayo-Reguera Sep 14, 2026 Author

Uh oh!

mattcasters Sep 14, 2026 Collaborator

Uh oh!

Pelayo-Reguera Sep 25, 2026 Author

Uh oh!

mattcasters Sep 25, 2026 Collaborator

Uh oh!

mattcasters Sep 10, 2026 Collaborator

Pelayo-Reguera
Aug 31, 2026

Replies: 2 comments 4 replies

bamaer
Sep 3, 2026
Collaborator

Pelayo-Reguera Sep 14, 2026
Author

mattcasters Sep 14, 2026
Collaborator

Pelayo-Reguera Sep 25, 2026
Author

mattcasters Sep 25, 2026
Collaborator

mattcasters
Sep 10, 2026
Collaborator