How to solve error: "State MISMATCH ..."

[vladislav-bakanov]

Hi!

My connector went to unrecoverable exception with State MISMATCH ... trace message.
I don't understand how to fix this issue.

Version: v1.0.17

How I caught to this error:

1. I added ClickHouse sink connector called connector-1 that consumes topic prod.dwh.debezium.crm.transactions and produce it to table sandbox.transactions
2. This connector failed, because created table mismatched some data types
3. I paused this connector
4. I fixed this problem (it took few hours + a night and topic min and max offsets were changed)
5. I renamed consumer group to connecto-2 and resumed connector

Eventually I got following error:

"trace": "org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
    at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:618)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:336)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:237)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:206)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:204)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259)
    at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:181)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.RuntimeException: Number of records: 1500
    at com.clickhouse.kafka.connect.util.Utils.handleException(Utils.java:116)
    at com.clickhouse.kafka.connect.sink.ClickHouseSinkTask.put(ClickHouseSinkTask.java:68)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:587)
    ... 11 more
Caused by: java.lang.RuntimeException: State MISMATCH given [1500] records for topic: [prod.dwh.debezium.crm.transactions], partition: [2], minOffset: [14344], maxOffset: [15843]
    at com.clickhouse.kafka.connect.sink.processing.Processing.doLogic(Processing.java:193)
    at com.clickhouse.kafka.connect.sink.ProxySinkTask.put(ProxySinkTask.java:99)
    at com.clickhouse.kafka.connect.sink.ClickHouseSinkTask.put(ClickHouseSinkTask.java:64)
    ... 12 more
"

This behaviour is happening here:

clickhouse-kafka-connect/src/main/java/com/clickhouse/kafka/connect/sink/processing/Processing.java

Line 181 in 7ca9b3f

throw new RuntimeException(String.format("State MISMATCH given [%s] records for topic: [%s], partition: [%s], minOffset: [%s], maxOffset: [%s]",

Is there any approach to fix the issue?

[Paultagoras]

Hi @vladislav-bakanov ! The connector state information is all stored in a table ("connect_state" by default) - if it has the old state information, you could truncate that table to clear out the old information. 🙂

As an alternative, if you don't need exactly-once you could disable it - it's not actually required for a lot of use cases, but you would know your situation better 🙂

[a0s]

Could you explain how to disable "exactly-once"? Sometimes i prefer to have two parallel connector (from the same mysql source) that writes into the same SharedReplacingMergeTree table.

[Paultagoras]

Simply set exactlyOnce=false in the connector properties (though it should default to false unless set otherwise)

[a0s]

After several weeks with exactlyOnce=false it starts showing State MISMATCH again.
Had to reset connect_state.

[a0s]

Just realized that tolerateStateMismatch setting exists.