SHARE
In high-traffic web applications, good management of memory is very important for the server to work smoothly. On the FTL Home platform, we faced a significant issue — the server's memory was running out very fast, causing crashes and frequent restarts. So, let’s take a look at the real reasons behind the problem and see how we managed to find practical solutions to make the system stable again
Nearly a year ago, we noticed that the server’s memory was being used up much quickly than before. At first, it was not a big problem because the server was restarted during releases, and the memory was cleared. However, over time, the situation worsened: the memory ran out in just a few days, then in two days, and eventually, the server would only work for a few hours before crashing.
Therefore, we needed a thorough analysis of memory usage. For this, we installed the stackprof gem on production and let it dump the memory report usage for a couple of days. After analyzing the reports, we found out that the cause of the leaks was large text strings (hundreds of thousands of lines) generated every time the server responded to a request.
Before that, we used the active_model_serializers gem, which was converting data into JSON using the as_json method. While converting to JSON, a large number of string objects were allocated in memory and were not garbage-collected. As a result, memory usage increased rapidly from about 500 KB to 10-20 MB per request. Since our server had only 2 GB of RAM, all the memory was quickly used up, leading to restarts and system freezes.
So, we identified that the main problems were inefficient serialization, large amounts of created unreleased string objects, and using old gems that were not designed for handling large data well. What should we do next?
The first and most important step to solving this was to switch from active_model_serializers to blueprinter. Blueprinter was chosen because it uses less memory and is faster at turning data into JSON. This change immediately reduced the number of strings created and lowered the memory load.
Next, we improved how we work with JSON by using the Oj gem. It is a fast library for parsing and generating JSON. Using Oj helped us create fewer strings and made the process of preparing responses more efficient.
We also added caching for the most resource-heavy endpoints. By saving results, we avoided recalculating the same data every time, reducing the number of objects created in memory.
Finally, we set up jemalloc, a better memory allocator, and tuned memory management settings.
All these steps together helped us cut down memory usage by about 10-15%. Now the servers runs smoothly and reliably without any issues and memory usage is solid and stable.
In conclusion, this experience demonstrated that the right approach to JSON and data serialization is crucial for maintaining the stability of high-load systems. Choosing the right tools, optimizing data processing, and utilizing caching can resolve memory issues and ensure systems run smoothly, even when many users are connected.