Switching from Tarantool 1.5 to Tarantool 1.6

Copyright 2009 Dmitry Kalinovsky

Hey! I’d like to tell you how one of our projects migrated from Tarantool 1.5 to Tarantool 1.6. Do you need to migrate to a newer version if everything works fine as it is? Is it an easy thing to pull off if you have thousands of lines of code? How to make sure live users aren’t affected? What difficulties lie ahead? What’s in it for us? If you want to know the answers, read on.

(more…)

Read More

Efficient storage: how we went down from 50 PB to 32 PB

efficient-storage-kat

As the Russian rouble exchange rate slumped two years ago, it drove us to think of cutting hardware and hosting costs for the Mail.Ru email service. To find ways of saving money, let’s first take a look at what emails consist of.

efficient-storage-1

Indexes and bodies account for only 15% of the storage size, whereas 85% is taken up by files. So, files (that is attachments) are worth exploring in more detail in terms of optimization. At that time, we didn’t have file deduplication in place, but we estimated that it could shrink the total storage size by 36% since many users receive the same messages, such as price lists from online stores or newsletters from social networks containing images and so on. In this article, I’m going to describe how we implemented a deduplication system under the supervision of Albert Galimov. (more…)

Read More

Implementing DMARC to Protect the Corporate Domain From Spoofing

kat

In this post, we will again discuss the problem of forged sender addresses (AKA ‘spoofing’). In recent years, such cases have become increasingly frequent: everything can be spoofed, whether it is an e-mail with utility bills, a message from a tax inspectorate, bank, etc. This problem can be addressed by configuring a strict DMARC policy. As an e-mail service, we have checked all incoming messages by using DMARC since February 2013. We were the first e-mail service in RuNet (in Russian) to support DMARC standards. Unfortunately, this is not enough to minimize the number of spoofed messages. The main thing is to ensure that a strict DMARC policy is supported on the sender side. This is why we are tirelessly emphasizing this issue through active explanatory campaigns and strongly urging everyone to implement a strict DMARC policy. (more…)

Read More

(Why) Mail.Ru Implements a Strict DMARC Policy

Competition of two kendo fighters. Japanese martial art of sword fighting

Recently, we announced (in Russian) a strict DMARC policy on all mail domains owned by Mail.Ru. On certain domains, including ‘bk.ru’ and ‘mail.ua’, a ‘p=reject’ policy is already enabled. In this article, we would like to explain some technical details of this change and provide recommendations to the owners of services, mail servers, and mailing lists.

DMARC (Domain-based Message Authentication, Reporting and Conformance, RFC 7489) is a protocol that allows administrators of domain zones to publish a policy for receiving and sending the reports of messages without authentication. For its authentication protocols, it uses SPF (RFC 7208) and DKIM (RFC 6376). (more…)

Read More

Technical Recommendations for Email Senders

postmaster

“Even if you get any letter, you won’t be able to read it”
(Mark Twain)

We have already explained how to create mailing lists, how to improve them and make them more effective. We presented metrics which help to build the sender’s reputation. We told you about the Postmaster Mail.Ru interface which allows you to track them. Many companies, both those in the first stages of development, and very successful ones, forget these rules, that causes them trouble with email delivery, problems with technical support, etc. But we hope that you are not among them. (more…)

Read More

The cat-and-mouse story of implementing anti-spam for Mail.Ru Group’s email service and what Tarantool has to do with this

Hey guys!

In this article, I’d like to tell you a story of implementing the anti-spam system for Mail.Ru Group’s email service and share our experience of using the Tarantool database within this project: what tasks Tarantool serves, what limitations and integration issues we faced, what pitfalls we fell into and how we finally arrived to a revelation. (more…)

Read More

Heavy workloads: our use cases of Tarantool

You have probably heard about Tarantool. This is a super fast DBMS with an application server inside. Tarantool is an open source project. It’s been around for more than 8 years. At Mail.Ru Group, we’re widely using it in more than half of our services like Email, Cloud, MyWorld, Mail.Ru Agent. We’re giving all the changes in Tarantool back to the community, so our users have the same version of Tarantool as we do. As of now, we have client libraries for nearly all popular languages. This was our priority over the last year. Libraries have been partially written by the community, and we immensely appreciate it. When we come across some really efficient library, we immediately include it into our public package repositories. We’re trying hard to deliver the DBMS and libraries right out of the box. (more…)

Read More

Asynchronous work with Tarantool in Python

7164887c961446a09aea38a920a4019f

You could have already seen some recent articles about the Tarantool NoSQL DBMS, how it’s used at its alma mater, Mail.Ru Group, and not only there. Yet no Tarantool recipes have been published for Pythoneers. In this article, I’ll talk about how we cook “Tarantool with the Python sauce” in our projects, what problems we encounter, the pitfalls, pros and cons, and will surely say “what’s the point” of this dish. :-) (more…)

Read More

The Bear, Dissection and February, 14th

9216b68feea344a3bbfc76a2364b93b9

A bear, a teddy bear – that’s the thing that makes girls’ pulse race and explodes in the depths of guys’ memories. Do you remember how not that long ago instead of a smartphone, one would get under a blanket with a shabby, one-eared, one-eyed but still oh-so-lovable little bear? In many countries, teddy bears are a symbol of love, care and sweet teenage feelings. So when the question of Valentine’s gifts for the colleagues came up, the answer was obvious – a teddy bear! But this one would be a huge bear robot since it was created by programmers. (more…)

Read More

Asyncio Tarantool Queue, get in the queue

funny-chickens-walking-in-line-on-road

In this article, I’m going to pay specific attention to information processing via Tarantool queues. My colleagues have recently published several articles in Russian on the benefits of queues (Queue processing infrastructure on My World social network and Push messages in REST API by the example of Target Mail.Ru system). Today I’d like to add some info on queues describing the way we solved our tasks and telling more about our work with Tarantool Queue in Python and asyncio.

(more…)

Read More