Design principles of Tarantool

I’m publishing a transcript of my talk at Highload Conference in Moscow in Spring 2015. It is actually the first part out of four (yes, I got a big conference slot back then :))

Here’s how I came to the idea of giving this talk. I don’t like speaking about new features, especially about upcoming features. While people enjoy listening to such talks, I don’t like spoiling the opportunity by presenting a feature before it’s ready. But people are also curious to learn how things work. So, this talk is about how it all works – or should work, from my perspective, – in a modern database management system (DBMS).

(more…)

Read More

Switching from Tarantool 1.5 to Tarantool 1.6

Hey! I’d like to tell you how one of our projects migrated from Tarantool 1.5 to Tarantool 1.6. Do you need to migrate to a newer version if everything works fine as it is? Is it an easy thing to pull off if you have thousands of lines of code? How to make sure live users aren’t affected? What difficulties lie ahead? What’s in it for us? If you want to know the answers, read on.

(more…)

Read More

Algorithm-Driven Design: How Artificial Intelligence Is Changing Design

Album covers processed through Prisma and Glitché (View large version)

I’ve been following the idea of algorithm-driven design for several years now and have collected some practical examples. The tools of the approach can help us to construct a UI, prepare assets and content, and personalize the user experience. The information, though, has always been scarce and hasn’t been systematic.

However, in 2016, the technological foundations of these tools became easily accessible, and the design community got interested in algorithms, neural networks and artificial intelligence (AI). Now is the time to rethink the modern role of the designer.

(more…)

Read More

Efficient storage: how we went down from 50 PB to 32 PB

As the Russian rouble exchange rate slumped two years ago, it drove us to think of cutting hardware and hosting costs for the Mail.Ru email service. To find ways of saving money, let’s first take a look at what emails consist of.

efficient-storage-1

Indexes and bodies account for only 15% of the storage size, whereas 85% is taken up by files. So, files (that is attachments) are worth exploring in more detail in terms of optimization. At that time, we didn’t have file deduplication in place, but we estimated that it could shrink the total storage size by 36% since many users receive the same messages, such as price lists from online stores or newsletters from social networks containing images and so on. In this article, I’m going to describe how we implemented a deduplication system under the supervision of Albert Galimov. (more…)

Read More

Implementing DMARC to Protect the Corporate Domain From Spoofing

In this post, we will again discuss the problem of forged sender addresses (AKA ‘spoofing’). In recent years, such cases have become increasingly frequent: everything can be spoofed, whether it is an e-mail with utility bills, a message from a tax inspectorate, bank, etc. This problem can be addressed by configuring a strict DMARC policy. As an e-mail service, we have checked all incoming messages by using DMARC since February 2013. We were the first e-mail service in RuNet (in Russian) to support DMARC standards. Unfortunately, this is not enough to minimize the number of spoofed messages. The main thing is to ensure that a strict DMARC policy is supported on the sender side. This is why we are tirelessly emphasizing this issue through active explanatory campaigns and strongly urging everyone to implement a strict DMARC policy. (more…)

Read More

(Why) Mail.Ru Implements a Strict DMARC Policy

Recently, we announced (in Russian) a strict DMARC policy on all mail domains owned by Mail.Ru. On certain domains, including ‘bk.ru’ and ‘mail.ua’, a ‘p=reject’ policy is already enabled. In this article, we would like to explain some technical details of this change and provide recommendations to the owners of services, mail servers, and mailing lists.

DMARC (Domain-based Message Authentication, Reporting and Conformance, RFC 7489) is a protocol that allows administrators of domain zones to publish a policy for receiving and sending the reports of messages without authentication. For its authentication protocols, it uses SPF (RFC 7208) and DKIM (RFC 6376). (more…)

Read More

Technical Recommendations for Email Senders

“Even if you get any letter, you won’t be able to read it”
(Mark Twain)

We have already explained how to create mailing lists, how to improve them and make them more effective. We presented metrics which help to build the sender’s reputation. We told you about the Postmaster Mail.Ru interface which allows you to track them. Many companies, both those in the first stages of development, and very successful ones, forget these rules, that causes them trouble with email delivery, problems with technical support, etc. But we hope that you are not among them. (more…)

Read More

The cat-and-mouse story of implementing anti-spam for Mail.Ru Group’s email service and what Tarantool has to do with this

Hey guys!

In this article, I’d like to tell you a story of implementing the anti-spam system for Mail.Ru Group’s email service and share our experience of using the Tarantool database within this project: what tasks Tarantool serves, what limitations and integration issues we faced, what pitfalls we fell into and how we finally arrived to a revelation. (more…)

Read More

Heavy workloads: our use cases of Tarantool

You have probably heard about Tarantool. This is a super fast DBMS with an application server inside. Tarantool is an open source project. It’s been around for more than 8 years. At Mail.Ru Group, we’re widely using it in more than half of our services like Email, Cloud, MyWorld, Mail.Ru Agent. We’re giving all the changes in Tarantool back to the community, so our users have the same version of Tarantool as we do. As of now, we have client libraries for nearly all popular languages. This was our priority over the last year. Libraries have been partially written by the community, and we immensely appreciate it. When we come across some really efficient library, we immediately include it into our public package repositories. We’re trying hard to deliver the DBMS and libraries right out of the box. (more…)

Read More