Shrink NET::buff to reduce memory usage by SongLibing · Pull Request #714 · mysql/mysql-server

SongLibing · 2026-08-06T21:10:43Z

Problem:

NET::buff grows to accommodate large incoming packets but does not return to its initial size. Long-lived connections can therefore retain excess memory after processing occasional large requests.

Solution:

Add the net_buffer_shrink_threshold session variable. When the current packet uses no more than half of an expanded buffer, count a shrink request. After the configured number of consecutive requests, halve the buffer without shrinking below net_buffer_length or IO_SIZE. Preserve the existing network error state if the allocation fails.

Copyright (c) 2026, Oracle and/or its affiliates.

What does this change do?

Optimize the memory usage of net::buff
mysql/mysql-community#78

Why is it needed?

It is needed to reduce memory usage and remain the server stable.

How was it tested?

Added/updated MTR tests under mysql-test/
scripts/ci/mtr.sh passes locally
Ran the relevant full suite (name it): ______

Contributor checklist

I have signed the OCA with the email on these commits
Code is formatted (scripts/ci/format.sh)
Commits are focused with descriptive messages

AI assistance

I did not use AI assistance for this contribution
I used AI assistance for this contribution

If AI assistance was used, describe the tool(s) and extent of use:
codex, merge the code to mysql-trunk

Areas touched

NET, THD

phulakun · 2026-08-11T07:01:48Z

+                   FROM performance_schema.threads
+                   WHERE PROCESSLIST_ID = @con1_processlist_id);
+
+# =============================================================================


Kindly add test to verify behavior with small queries less than net_buffer_shrink_threshold.

For test2 and test3, I added query to show the allocated buffer, just after the forth SELECT 1. the allocated buffer should no change.

phulakun · 2026-08-11T07:04:52Z

+  uchar old_error = net->error;
+
+  if (net_realloc(net, target_buffer_capacity)) {
+    net->last_errno = old_last_errno;


If net_realloc() hits OOM:
my_realloc(..., MYF(MY_WME)) reports EE_OUTOFMEMORY through my_error().
This can populate the current THD diagnostics area.

Restoring net->error and net->last_errno repairs only the NET state. The THD diagnostics area can still hold OOM.

Perhaps shrink should call my_realloc(..., MYF(0)) instead. Something like,

// sql-common/net_serv.cc
// Note: net_shrink is a new function.
bool net_shrink(NET *net, size_t length) {
assert(net->write_pos == net->buff);
assert(length < net->max_packet);

const size_t target =
(length + IO_SIZE - 1) & ~(static_cast<size_t>(IO_SIZE) - 1);

uchar *buff = static_cast<uchar *>(my_realloc(
key_memory_NET_buff, net->buff,
target + NET_HEADER_SIZE + COMP_HEADER_SIZE, MYF(0)));

if (buff == nullptr) return true;

net->buff = buff;
net->buff_end = buff + target;
net->write_pos = buff;
net->read_pos = buff;
net->max_packet = static_cast(target);
return false;
}

What do you think?

Added net_shrink function into net_srv.cc, net_shrink doesn't report any error.

phulakun · 2026-08-11T07:07:32Z

+    "If required size of NET::buff currently is less than half the allocated "
+    "one, shrink will be requested. When number of consecutive requests is "
+    "equal or greater than this value, shrink will be triggered. 0 means "
+    "never do shrink.",


Perhaps its good to mention that NET::buff is not shrinked below net_buffer_length.

sure, added.

phulakun

Hello SongLibing!

Thank you for working on this. It looks good. Please find few comments.

Regards,
Praveen

SongLibing · 2026-08-12T06:13:37Z

Hello SongLibing!

Thank you for working on this. It looks good. Please find few comments.

Regards, Praveen

Hi Praveen,

Thanks for your review comments. I addressed all problems you mentioned. Please have a look.

Regards, Libing

phulakun

Hello Libing!

Thank you for addressing review comments.
Changes looks good to me. Kindly wait for review from Harin.

Regards,
Praveen

harinvadodaria

Hi Libing,

Thanks for submitting the patch.
Please see some comments below

harinvadodaria · 2026-08-13T06:19:56Z

  */
  void *extension;
+  /** Number of consecutive requests to shrink NET::buff. */
+  unsigned int shrink_requests;


Adding a new variable in NET would change sizeof(NET) and in turn sizeof(MYSQL) because NET is part of MYSQL structure. This is also captured by changes in mysql.h.pp. Such a change would need a bump in libmysql version.

Instead, please extend structure NET_SERVER in mysql_com_server.h and initialise in init_net_server_extension().

This way, ABI changes are avoided and no version bump is needed.

Got it, thanks! moved it into NET_SERVER.

harinvadodaria · 2026-08-13T06:32:37Z

    VALID_RANGE(1024, 1024 * 1024), DEFAULT(16384), BLOCK_SIZE(1024),
    NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(check_net_buffer_length));

+#define DEFAULT_NET_BUFFER_SHRINK_THRESHOLD 5


What is the rationale behind this default value? While I understand that shrinking a buffer reduces memory usage. It also means that every now and then, perf penalty is paid to realloc required buffer. Can you share some performance data based on your observations?

The target is to reduce memory usage for very large net buffer. The feature is enabled in production by default, we use a relative mild config value 5 with cautious. Because we don't want the config impact the instances in which queries are often large than net_buffer_length. And the value 5 works fine in production. Below is an real case we observed in production : you can see the memory usage is much less and the instance stays stable after the server is updated to the version with this feature.

harinvadodaria · 2026-08-13T06:34:17Z

 void net_clear(struct NET *net, bool check_buffer);
 void net_claim_memory_ownership(struct NET *net, bool claim);
 bool net_realloc(struct NET *net, size_t length);
+bool net_shrink(struct NET *net, size_t length);


Once shrink_request is moved out of NET, this function can be removed from mysql_com.h.
Instead, it can be part of server code itself.

Moved it into mysql_com_server.h

harinvadodaria · 2026-08-13T06:59:22Z

+    if (net->max_packet > thd->variables.net_buffer_length &&
+        net->max_packet > IO_SIZE &&
+        thd->variables.net_buffer_shrink_threshold != 0) {
+      thd->try_shrink_net_buffer(net, protocol->get_packet_length());


All of the above mentioned checks should be folded into try_shrink_net_buffer(). You already have assertions for same conditions in the function.

It is originally in the try function. I moved the check out try_shrink_net_buffer, because I think it will avoid a function call in most of the situation. So it is faster.

Folded into try_shrink_net_buffer(). The function is only called once in the query life, it is fine.

harinvadodaria · 2026-08-13T09:57:43Z

  net->last_errno = 0;
 #ifdef MYSQL_SERVER
  net->extension = nullptr;
+  net->shrink_requests = 0;


Can be removed once shrink_requests is moved as suggested in one of the previous comments.

harinvadodaria · 2026-08-13T09:58:15Z

+  @retval  true   failed to shrink
+  @retval  false  Shrunk to length successfully
+*/
+bool net_shrink(NET *net, size_t length) {


Please fold it into try_shrink_net_buffer() or define a separate function within server layer itself.

Added #ifdef MYSQL_SERVER to make it server only.

SongLibing · 2026-08-14T10:56:57Z

Hi @harinvadodaria ,
Thanks for the great suggestions. I addressed all problems. Please have a look.

Problem ======= NET::buff grows to accommodate large incoming packets but does not return to its initial size. Long-lived connections can therefore retain excess memory after processing occasional large requests. Solution ======== Add the net_buffer_shrink_threshold session variable. When the current packet uses no more than half of an expanded buffer, count a shrink request. After the configured number of consecutive requests, halve the buffer without shrinking below net_buffer_length or IO_SIZE. Preserve the existing network error state if the allocation fails.

harinvadodaria

Thank you for the updated patch, @SongLibing .
Approved.

SongLibing requested a review from a team August 6, 2026 21:10

SongLibing requested review from gopshank and seemasundara as code owners August 6, 2026 21:10

oracle-contributor-agreement Bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Aug 6, 2026

github-actions Bot added tests Changes touching test code or test data MTR Failed MTR suite failed Build Failed PR build failed labels Aug 6, 2026

gopshank requested review from harinvadodaria and phulakun and removed request for gopshank and seemasundara August 7, 2026 15:50

SongLibing force-pushed the net_buffer_shrink branch from 7c63231 to d2994e1 Compare August 10, 2026 17:08

github-actions Bot requested review from gopshank and seemasundara August 10, 2026 17:09

github-actions Bot added the Review Requested Review requested from code owners label Aug 10, 2026

phulakun reviewed Aug 11, 2026

View reviewed changes

SongLibing force-pushed the net_buffer_shrink branch from d2994e1 to f951c58 Compare August 12, 2026 06:08

SongLibing force-pushed the net_buffer_shrink branch from f951c58 to 6a514a9 Compare August 12, 2026 10:45

phulakun reviewed Aug 12, 2026

View reviewed changes

harinvadodaria reviewed Aug 13, 2026

View reviewed changes

matanbaruch mentioned this pull request Aug 14, 2026

Bug#102586: multi-table DELETE with ON DELETE CASCADE breaks row-based replication #719

Open

8 tasks

SongLibing force-pushed the net_buffer_shrink branch from 6a514a9 to 4e8855e Compare August 14, 2026 10:29

SongLibing force-pushed the net_buffer_shrink branch from 4e8855e to a5bcdde Compare August 14, 2026 15:39

harinvadodaria approved these changes Aug 19, 2026

View reviewed changes

Uh oh!

Conversation

SongLibing commented Aug 6, 2026

Problem:

Solution:

Copyright (c) 2026, Oracle and/or its affiliates.

What does this change do?

Why is it needed?

How was it tested?

Contributor checklist

AI assistance

Areas touched

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phulakun Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phulakun Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phulakun left a comment

Choose a reason for hiding this comment

Uh oh!

SongLibing commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

phulakun left a comment

Choose a reason for hiding this comment

Uh oh!

harinvadodaria left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SongLibing Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SongLibing commented Aug 14, 2026

Uh oh!

harinvadodaria left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

phulakun Aug 11, 2026 •

edited

Loading

phulakun Aug 11, 2026 •

edited

Loading

SongLibing commented Aug 12, 2026 •

edited

Loading

SongLibing Aug 14, 2026 •

edited

Loading