We have done tens of web and mobile online marketplaces in different countries and industries. Today we are going to tell about one of these marketplaces and explain why we had to rewrite the existing library to get things done during the development. In this Syndicode’s project developers had to create a Microsoft Azure data storage library and substantially rewrite the Ruby gem. More on why did it happen and how we organized the whole process read below.
We’ll start with the results.
Syndicode’s open-source MS Azure data storage library
MS Azure data storage is a cloud storage solution that possesses a huge object store for data objects, a file system service for the cloud, and a NoSQL store. The storage is:
- Safe and always available. There is a possibility to replicate data and get additional protection from any force major. All data are encrypted by the service and the one who has access to the data can provide you with control.
- Scalable. Azure Storage is very scalable, so it can store the data and performance of any need.
- Easily controlled. The storage is hardware maintained, easily updated, and contains critical issues.
- Accessible. Data that are stored in the library can be easily accessed from anywhere in the world over HTTP. Microsoft contains client libraries for Azure Storage in many languages (.NET, Java, Node.js, Python, PHP, Ruby, Go, REST API).
Azure Storage resources usually are accessed by any language that can make HTTP/HTTPS requests. The storage also offers programming libraries for several popular languages. That makes many working with Azure Storage a more simple process as it contains many handling details such as synchronous and asynchronous invocation, batching of operations, exception management, automatic retries, operational behavior, and so forth.
The reason why we had to rewrite the existing library
Shrine is a toolkit for handling file attachments in Ruby for how to build on-demand applications. Some highlights are:
- Modular design – the plugin system allows you to load only the functionality you need
- Memory friendly – streaming uploads and downloads make it work great with large files
- Cloud storage – store files on disk, AWS S3, Google Cloud, Cloudinary and others
- Persistence integrations – works with Sequel, ActiveRecord, ROM, Hanami and Mongoid and others
- Flexible processing – generate thumbnails eagerly or on-the-fly using ImageMagick or libvips
- Metadata validation – validate files based on extracted metadata
- Direct uploads – upload asynchronously to your app or to the cloud using Uppy
- Resumable uploads – make large file uploads resumable on S3 or tus
- Background jobs – built-in support for background processing that supports any backgrounding library
While working with Shrine we faced the following problem:
A .net core app running on an Azure web app kept restarting at random times. This appeared to be a result of either proactive self-healing or auto self-healing as the response time to a request exceeded the limits. The developers investigated the problem and thought it was the proactive side but after disabling, they experienced the same behavior. Private byte usage showed the classic sawtooth shape of a memory leak.
Taking into account that our client’s (Umico) corporate standard demanded working with Microsoft Azure data storage library only, to avoid memory leak problems Syndicode had to substantially rewrite the gem.
And we did it. For more than a month we were looking for an issue caused a memory leak. When the Shrine-storage gem was ready, we open-sourced it.
What’s Umico?
Let us tell you more about the project we were working on. Umico is an online marketplace which application gathers together all companies in Azerbaijan, where you can get cashback. With its help, you don’t need to carry lots of bonus cards anymore. All you need, it’s just download the Umico application, find and profitably buy what you need at the right moment.
There you can find companies in such categories as:
- food and drinks
- fashion and style
- devices and technologies
- health and beauty
- for children and moms
- home and household
- leisure and entertainment
- work and education
- trips and travel
- auto-, moto- and other transport
The whole procedure looks like this:
- You find a company with cashback in the application
- During purchase tell the cashier, that you have Umico card
- Present your plastic card or the card in the application
- Get you cashback credited to your account after the purchase payment
Every day the company pays for purchases. In this way, you can be rewarded and earn bonuses for each purchase with Umico and activity in the application. Bonuses can be used for purchases at Umico partners at a rate of 1 bonus = 1 AZN. The more bonuses you have, the more new purchases you make, for which you don’t have to spend your money. So, bonuses can be more valuable than money here.
How we contributed to the open-source
When we started the project for Umico, their corporate standard required not Amazon or Digital Ocean, but only MS Azure. The Shrine Gem was already in open-course but we had to significantly rewrite it since it had memory leaks and logical mistakes.
This leak occurred while users used Shrine together with AzureBlob. It happened because of the fact that there was no solution based on the Azure SDK for Shrine along with existing AWS S3, Memory and fileSystem solutions. Because of this, the application could not be used.
Because of the general trust in the rubygems-org and the difficulty of debugging an application that worked for Kubernetes, it was not possible to track the cause of a memory leak for a long time. It was especially difficult because there was any access to the cluster and it was impossible to reproduce the leak since it occurred while loading a large number of images. One iteration could take more than 3 hours.
It took the developer 1.5 months to exclude all possible options, search the cause of the problem and redesign the gem and the library. The problem appeared to be in the Shrine adapter to the gem “shrine-fog” and gem “fog-azure-rm”. They were the cause of the leak, so our developer decided to implement an adapter on the basis of standard interests of the S3 type that used the basic AzureBlob SDK. Finally, the leak was successfully removed. When this process was finally done, Syndicode posted the corrected gem in the community and now hundreds of people already downloaded our version of the gem.
Our hard work has not only helped the project to get started and function well but helped the community to get a high-quality open-source gem.
Find our previous issues about the most popular Rails repositories here.
And subscribe to our Telegram!