[Question] How is memory reclamation handled in RocksDB storage mode?

[cichengzi]

Problem Type (问题类型)

None

Before submit

• 我已经确认现有的 Issues 与 FAQ 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)

Environment (环境信息)

• Server Version: 1.5.0 (Apache Release Version)
• Backend: RocksDB
• OS: 4 CPUs, 32 G RAM, Ubuntu 22.0
• Data Size: 450w+ vertices, 1000w+ edges

Your Question (问题描述)

I'd like to know how HugeGraph 1.5.0 handles memory reclamation and control in RocksDB storage mode. After repeatedly creating graphs, adding nodes and edges, and performing Gremlin queries, memory usage remains consistently high. What can I do to get that memory reclaimed and released? Even after I executed a Gremlin query task with a large amount of data, all the HugeGraph backend interfaces became unusable. No matter which interface I called, it would eventually result in an InterruptedException or Timeout.

Vertex/Edge example (问题点 / 边数据举例)

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

[dosubot]

HugeGraph's memory management in RocksDB mode involves several layers, and high memory that doesn't get reclaimed is a known challenge. Here's what's happening and what you can do:

Why memory stays high:

1. Off-heap memory fragmentation: RocksDB allocates memtables and block cache off-heap. The default glibc ptmalloc allocator fragments memory and prevents arena reclamation [1]. A recommended mitigation is to replace it with jemalloc or tcmalloc via LD_PRELOAD [1].

2. Transaction listener leak: A known memory leak exists in CachedGraphTransaction where orphaned store event listeners accumulate over time when graphs are repeatedly created/closed [2]. This causes slow but persistent heap growth.

3. Vertex/edge caching: Large Gremlin queries can cause all results to be cached in heap memory. With vertex.cache_capacity and edge.cache_capacity set high, memory won't be freed until the cache expires or evicts entries [3].

Key RocksDB memory configuration options to tune [4]:

Setting	Default	Recommendation
rocksdb.block_cache_capacity	8MB	Set to 30-50% of available RAM for your workload
rocksdb.write_buffer_size	128MB	Reduce if memory constrained
rocksdb.max_write_buffer_number	6	Reduce to 3-4 to limit memtable memory
rocksdb.total_memory_size	48GB	Set to your actual available off-heap budget
rocksdb.write_buffer_ratio	0.66	Adjust based on read vs write workload

Actionable steps for your 32GB RAM environment:

1. Use jemalloc: Add LD_PRELOAD=/path/to/libjemalloc.so to your startup script to prevent memory fragmentation [1].

2. Reduce rocksdb.total_memory_size: The default is 48GB which exceeds your system RAM. Set it to something reasonable like 8-12GB given your 32GB machine also needs heap and OS memory [5].

3. Tune caches: Lower vertex.cache_capacity and edge.cache_capacity and set a reasonable cache_expire value [6].

4. JVM heap: Ensure your -Xmx isn't too large—leave room for RocksDB off-heap allocations. For 32GB total RAM with RocksDB, something like -Xms4g -Xmx8g is reasonable, depending on your query patterns.

5. Trigger compaction: After large data deletions or graph drops, call compactRange() via the API to reclaim SSTable space and reduce memory pinned by obsolete data [7].

Regarding the InterruptedException/Timeout after large queries—this is likely caused by GC pressure or RocksDB write stalls when memory is exhausted. The tuning above should help, but you may also want to monitor via /metrics/backend and /metrics/system endpoints (note: these can underreport actual RocksDB memory usage by a significant margin) [8].

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[cichengzi]

I currently have multiple machines, two with 32GB RAM and one with 64GB RAM. However, in actual use, as graphs are continuously created and tasks are executed, memory usage increases rapidly until it almost completely consumes all memory, and memory is not reclaimed for a long time. I've actually made some JVM memory settings in hugegraph-server.sh as follows:

JAVA_OPTIONS="-Xms16g -Xmx16g -XX:MetaspaceSize=1024m -XX:MaxMetaspaceSize=2047m -XX:MaxDirectMemorySize=2g -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=40"

Turn on security check

exec ${JAVA} -Dname="HugeGraphServer" ${JVM_OPTIONS} ${JAVA_OPTIONS} -cp ${CLASSPATH}: \

org.apache.hugegraph.dist.HugeGraphServer ${GREMLIN_SERVER_CONF} ${REST_SERVER_CONF}
>> ${OUTPUT} 2>&1

[dosubot]

Your JVM settings look reasonable for heap, but the issue is likely that most of the memory growth is happening off-heap in RocksDB's native allocations (memtables, block cache, table readers), which are outside of -Xmx control.

A few things to check and tune:

1. rocksdb.total_memory_size defaults to 48GB — this alone can consume more than your 32GB machines have available. On your 32GB machines with -Xmx16g, you only have ~14-16GB left for OS + RocksDB native memory. Set this to something like 8-10GB on 32GB machines [1].

2. Replace glibc's memory allocator with jemalloc. RocksDB's allocation pattern causes severe fragmentation with the default ptmalloc, preventing the OS from reclaiming freed memory [2]. Add to your startup:

export LD_PRELOAD=/path/to/libjemalloc.so

This is often the single most impactful fix for the "memory never gets reclaimed" symptom.

3. Tune write buffers. With the default max_write_buffer_number=6 and write_buffer_size=128MB, each column family can hold up to ~768MB in memtables. If you have many graphs (each with multiple column families), this adds up fast [3]. Consider reducing:

rocksdb.write_buffer_size=64m
rocksdb.max_write_buffer_number=3

4. Known transaction listener leak. Repeatedly creating graphs can trigger a memory leak in CachedGraphTransaction where orphaned event listeners accumulate and never get cleaned up [4]. This would explain why memory grows with each new graph creation and is never reclaimed.

5. Reduce vertex/edge cache sizes. If you're running large Gremlin queries, the in-heap vertex and edge caches can consume significant heap. Lower vertex.cache_capacity and edge.cache_capacity and set a shorter cache_expire [5].

To diagnose whether the problem is heap vs. off-heap, compare your JVM heap usage (via jstat -gc or GC logs) against total process RSS (via ps or /proc/<pid>/status). If heap is well under 16GB but RSS is 30GB+, the problem is squarely in RocksDB native memory and the allocator fragmentation issue.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[cichengzi]

When executing Gremlin tasks, memory reclamation (g1) is triggered if memory usage exceeds the 85% threshold, but this is insufficient. After execution, memory usage remains high and continues to increase with the accumulation of tasks.

[dosubot]

This behavior aligns with a known design limitation. Here's what's likely happening:

The 85% threshold triggers GC but can't reclaim everything because the memory isn't all garbage — much of it is actively held by:

1. Accumulated task results in backend storage. While completed tasks are removed from the in-memory ConcurrentHashMap, their results persist as ~task and ~taskresult vertices in the backend and get loaded back into memory when accessed [1]. These are never automatically cleaned up.

2. Vertex/edge caches (LRU, capacity-based only). HugeGraph's caches evict based on capacity limits and TTL — there is no reactive memory-pressure eviction that shrinks caches when heap is high [2] [3]. So even under memory pressure, caches remain full up to their configured capacity.

3. Off-heap RocksDB allocations (memtables, block cache) are completely invisible to G1 GC and the 85% heap monitor.

What you can do:

• Clean up old task results — periodically delete ~task and ~taskresult vertices for completed tasks, especially any with large result sets. This is currently a manual operation [1].
• Reduce cache capacities — lower vertex.cache_capacity and edge.cache_capacity so the LRU caches consume less heap. Also set a shorter cache_expire to let entries age out faster.
• Limit task result sizes — configure task.result_size_limit to a smaller value so individual tasks don't accumulate huge results in memory.
• For large data exports, avoid the task result mechanism entirely. Use shard/scan APIs or write results to external storage from within your Gremlin scripts [1].
• Tune InitiatingHeapOccupancyPercent — your current setting of 40 is reasonable, but with 16GB heap and long-lived objects, G1 mixed GC may not keep up. Consider adding -XX:G1HeapWastePercent=5 and -XX:G1MixedGCCountTarget=16 to be more aggressive about collecting old-gen regions.

The core issue is that the 85% monitor can trigger GC, but if the heap is full of live objects (cached data, task results, RocksDB JNI references), GC has nothing to collect. The fix has to come from reducing what's held alive at the application level.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[cichengzi]

Another problem is that I'm currently executing an extremely complex and computationally intensive query task, which has caused all my HugeGraph APIs to become unusable. They're interrupting my queries to ensure the task continues running. Even restarting HugeGraph or my machine hasn't fixed the issue. What can I do?

[dosubot]

The reason restarting doesn't help is that HugeGraph persists running/pending tasks as ~task vertices in backend storage and automatically restores them on server startup [1]. So your problematic task keeps getting re-queued every time the server comes back up.

To break out of this cycle, use the force-delete API:

DELETE http://{IP}:{PORT}/graphs/{GRAPH_NAME}/tasks/{TASK_ID}?force=true

This removes the task from both memory and backend storage, preventing it from being restored on the next restart [2] [3].

If all APIs are hanging, try this sequence:

1. Restart the server
2. Immediately issue the force-delete request before the restored task consumes all resources again
3. If you don't know the task ID, try GET /graphs/{graph}/tasks?status=RUNNING quickly after restart

If that still doesn't work (because the task re-queues too fast), you could try:

• Temporarily reduce the Gremlin task thread pool or add a short startup delay to give yourself a window
• As a last resort, if you have direct access to the RocksDB data directory, you could delete and recreate the problematic graph entirely using DELETE /graphs/{graph}/clear right after restart

To prevent this in the future:

• Set task.result_size_limit to a smaller value to prevent tasks from accumulating massive results
• For complex analytical queries, consider using the OLAP/algorithm APIs rather than raw Gremlin tasks, or break the query into smaller batches
• Note that HugeGraph only has 4 threads dedicated to Gremlin task execution [4], so a single resource-intensive task can significantly impact throughput

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[cichengzi]

It still doesn't work.

[dosubot]

If the force-delete API is also hanging (because the problematic task monopolizes resources too fast after restart), here are some more aggressive options:

Option 1: Switch to distributed scheduler to prevent task restoration

Before restarting, edit your graph configuration (e.g., conf/graphs/hugegraph.properties) and add:

task.scheduler_type=distributed

The distributed scheduler's restoreTasks() is effectively a no-op [1], so the stuck task won't immediately re-queue on startup. This gives you time to call the force-delete API. After cleanup, you can switch back to the standard scheduler.

Option 2: Delete and recreate the graph

If you can afford to lose the graph data, the fastest recovery is to:

1. Stop the server
2. Delete the RocksDB data directory for that graph (default path: rocksdb-data/ under your HugeGraph install) [2]
3. Restart the server and recreate the graph

Unfortunately, task vertices share the same column family ("V") as regular graph vertices [3], so there's no way to surgically remove just the task data from RocksDB files without custom code.

Option 3: Race the force-delete

Check your server logs for the task ID being restored (it should appear during startup). Then:

1. Restart the server
2. Immediately (within seconds) run:

   curl -X DELETE "http://{IP}:{PORT}/graphs/{GRAPH_NAME}/tasks/{TASK_ID}?force=true"

If the API responds at all in the first few seconds before the task fully loads, this can succeed.

Could you share what specifically happens when you try the force-delete? Does it time out, return an error, or never respond at all?

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[contrueCT]

Since this is on HugeGraph 1.5.0 and force=true still does not work, I think this is closely related to #3057, but there are two separate things to handle:

1. Recover the already stuck task.
2. Prevent future Gremlin tasks from writing huge results into the task result field again.

For the already stuck task, the normal task delete path may still be too heavy. In 1.5.0 local scheduler, a large result can be stored on the ~task vertex as ~result, and the normal query/delete path may still construct a HugeTask through HugeTask.fromVertex(). If the ~result payload is very large, even DELETE /tasks/{id}?force=true can hit the same memory pressure or be interrupted.

So for recovery, I think a temporary maintenance cleanup path is needed. A practical recovery flow could be:

start HugeGraph in a temporary maintenance/recovery mode
  -> skip automatic task restoration on startup
  -> delete the problematic task metadata/result by task id
     without calling HugeTask.fromVertex()
  -> restart HugeGraph normally

The cleanup code should remove the in-memory task entry first, then query the task vertex by id and delete the vertex directly. It should not deserialize the task, decompress the ~result payload, or return the full task result. For local scheduler, deleting the ~task vertex should also remove its ~result property. If separated task-result storage has been used, the corresponding ~taskresult record should be cleaned as well.

This should be done in a maintenance window and tested on a backup first. Deleting the whole RocksDB graph directory should only be the last resort because it removes graph data, not just the bad task.

For future large Gremlin results, the workaround from #3057 is the better direction: don't return a huge Map<String, String> or huge JSON string as the task result. Add a Gremlin-accessible chunk writer, write results incrementally during computation, and make the task return only a small manifest.

For example, add helper methods such as:

gremlinJob.writeResultChunk(String name, String content)
gremlinJob.appendResultLine(String name, String line)
gremlinJob.resultManifest()

Then the Gremlin script can flush partial output while it is computing, instead of building one huge result in memory and returning it through /tasks/{id}. The final task result should stay small, for example:

{
  "type": "chunked-result",
  "format": "jsonl",
  "chunk_count": 245,
  "total_count": 12345678,
  "result_uri": "..."
}

One important detail: chunking should happen during the computation. If the script still builds the whole Map<String, String> first and only writes chunks at the end, it may still cause the same memory pressure.

Separately, the memory-not-reclaimed part should be diagnosed by comparing JVM heap usage with process RSS. If heap is not high but RSS is high, RocksDB native memory, OS page cache, allocator fragmentation, and RocksDB buffer/cache settings are more relevant. If heap is high, task results, caches, or restored/running tasks are more likely.

Also, 1.5.0 is not the latest release. Upgrading to the latest HugeGraph release is recommended as a better baseline for existing fixes, but this large-single-task-result pattern should still be changed, because /tasks/{id} is not suitable as a bulk result download API.