Reduce memory consumption in Cinder services
This patch reduces memory usage on the Cinder Volume and Backup services
by tuning glibc.
The specific tuning consist on disabling the per thread arenas and
disabling dynamic thresholds.
The Cinder Backup service suffers from high water mark memory usage and
uses excessive memory. As an example just after 10 restore operations
the service uses almost 1GB of RAM and does not ever free it afterwards.
With this patch the memory consumption of the service is reduced down to
almost 130MB. If we add a revert from Cinder (Change-Id
I43a20c8687f12bc52b014611cc6977c4c3ca212c) it goes down to 100MB during
my tests.
This glibc tuning is not applied to all Python services because I
haven't done proper testings on them and at first glance they don't seem
to have such great improvements.
Related-bug: #1908805
Change-Id: Ic9030d01468b3189350f83b04a8d1d346c489d3c
diff --git a/lib/cinder b/lib/cinder
index 52818a8..ca2c084 100644
--- a/lib/cinder
+++ b/lib/cinder
@@ -552,8 +552,13 @@
fi
run_process c-sch "$CINDER_BIN_DIR/cinder-scheduler --config-file $CINDER_CONF"
- run_process c-bak "$CINDER_BIN_DIR/cinder-backup --config-file $CINDER_CONF"
- run_process c-vol "$CINDER_BIN_DIR/cinder-volume --config-file $CINDER_CONF"
+ # Tune glibc for Python Services using single malloc arena for all threads
+ # and disabling dynamic thresholds to reduce memory usage when using native
+ # threads directly or via eventlet.tpool
+ # https://www.gnu.org/software/libc/manual/html_node/Memory-Allocation-Tunables.html
+ malloc_tuning="MALLOC_ARENA_MAX=1 MALLOC_MMAP_THRESHOLD_=131072 MALLOC_TRIM_THRESHOLD_=262144"
+ run_process c-bak "$CINDER_BIN_DIR/cinder-backup --config-file $CINDER_CONF" "" "" "$malloc_tuning"
+ run_process c-vol "$CINDER_BIN_DIR/cinder-volume --config-file $CINDER_CONF" "" "" "$malloc_tuning"
# NOTE(jdg): For cinder, startup order matters. To ensure that repor_capabilities is received
# by the scheduler start the cinder-volume service last (or restart it) after the scheduler