by LINE Engineer on 2016.8.19
Hello, my name is Yuto Kawamura. I’m a LINE server engineer in charge of developing and operating LINE’s core storage facilities such as HBase and Kafka.
Since the latter half of last year, I’ve been working on a new project called IMF, which stands for Internal Message Flow (or Fund). IMF has two main goals:
- Develop a data pipeline which provides a unified way of delivering events between our systems.
- Replace talk-dispatcher, a component in the LINE server system responsible for background task processing.
The two goals may seem unrelated, but we’re actually trying to adopt the same technologies for both; Apache Kafka and stream processing. Apache Kafka is a high-throughput distributed messaging system that was originally developed and used at LinkedIn. Although Kafka has various unique features, the most important ones are the following:
- Disk-based persistence with high throughput which is near in-memory through the use a page cache
- Ability to have multiple consumers consume messages from a topic (similar to a queue) multiple times because each client manages “offsets” which represents the position in the queue that consumption has caught up to
There are many interesting things that I could tell you about our work, such as the practices we’ve defined while working on Kafka engineering or the bigger picture of what the IMF project is about. But today, I’d like to focus on just one thing: How we implement stream processing.
by LINE Engineer on 2016.8.11
In New generation of safe messaging: “Letter Sealing”, we announced that end-to-end encryption (E2EE) has been made available on LINE messages.
But we’ve made even more improvements to safe messaging over the past few months, expanding Letter Sealing to features other than one-on-one chats. We’d like to share some of them here in this post.
Letter Sealing is now enabled on one-on-one chats by default
In the previous version of LINE, you had to opt in to use Letter Sealing. With the latest version we’ve made it so that it’s enabled by default. You can still opt out if you wish to do so.
One reason we couldn’t enable Letter Sealing by default for everyone was because of an issue we had with the iOS push notification system. LINE displays a small part of the messages you receive through push notifications. Unlike on Android, the push notifications on iOS gives apps much less room to work with. When someone sends a message to you, the servers normally only send part of the message to you through a push notification. The LINE server cannot read these Letter Sealing-enabled messages and thus can’t display the contents of the message on the notification.
Fortunately, a new type of notification called VoIP push notification was added to iOS 8. We were able to display the contents of messages using VoIP notifications to run the LINE app to decrypt encrypted messages. The team confirmed that VoIP notifications were very stable starting with iOS 9.3.1, and decided that Letter Sealing wass ready to be enabled as a default option.
Letter Sealing can now be enabled in group chats
Your messages are now safer even in group chats.
Messages inside a group chat with Letter Sealing enabled can only be encrypted by users in the same chat. When someone leaves the group, that user is no longer be able to decrypt the messages sent in the group.
Another thing to consider is that users in a group chat are not always online at any given time, and can’t always receive messages right away in real time. This limitation made applying Letter Sealing to groups much harder than applying it to one-on-one chats. We took these issues into consideration and spent a whole year tweaking and testing Letter Sealing so it is finally ready to be available on group chats in the latest version of the LINE app (LINE 6.5 for iOS and Android). For now, it’s only available for group chats that have up to 50 members, but we’re working on increasing this number to support larger group chats.
I’ll explain more on how LINE engineers dealt with these problems in an upcoming post.
Letter Sealing is now available on free calls (voice/video)
LINE has offered free voice and video calls since 2011. After discussing what features we should provide Letter Sealing to next, the team decided that we should start work on developing Letter Sealing for free calls.
Voice and video calls are now protected by an encryption key only known and exchanged between participants of the call. Even LINE employees cannot see what’s going on in these calls as the calls are protected by an encryption key, which is only stored on user devices and not on LINE servers.
No additional manual settings are required for using this feature. Users on LINE 6.5 (iOS, Android) or LINE 4.8 (Windows) and above can enjoy the extra protection provided by this security upgrade if they keep Letter Sealing enabled.
You can now easily tell if Letter Sealing is enabled
Your messages on LINE are still safe even without Letter Sealing, as they are encrypted when they travel between your device and our servers. No one can know about your messages even if they are monitoring your network. The reason we developed Letter Sealing was to add an extra layer of protection so that your messages can only be seen on your devices and kept safely encrypted on our servers.
After introducing Letter Sealing last year, the team hotly debated about adding a status indicator for Letter Sealing. We didn’t want to give our users the impression that chats without Letter Sealing are unsafe nor did we want to cause confusion on who you can use Letter Sealing with and who you cannot, since Letter Sealing was disabled by default when it was first introduced.
Now that Letter Sealing is enabled by default, we decided that it’s more important to give correct information to our users rather than dwell on these concerns. Starting with LINE 6.5 (iOS, Android) and LINE 4.8 (Windows, Mac), you will notice a padlock icon next to the name of the chat when you’re in chats or calls with Letter Sealing enabled.
The status indicator will be gradually rolled out to the various regions around the world. All users using supported versions of the LINE app should be able to see the status indicator displayed on Letter Sealing-enabled chats by the beginning of September.
We plan to bring all the security features mentioned above to platforms other than iOS, Android, Windows, and OS X1 as well. Our mission is to keep messaging safe on LINE, and we will continue to do so.
1: The status indicator will be displayed on one-on-one chats and group chats on LINE 4.8 for Windows and OS X. Status indicators for free calls will be available with the next update.
About the author
Shin, Ki Bin: A LINE engineer working on the messaging server.
by LINE Engineer on 2016.8.8
We’ve already announced LINE DEVELOPER DAY 2016. Today, we’re happy to announce that the official website and registration forms are also available.
- Date/Time: September 29, 2016 from 10 a.m. to 6:30 p.m.
- Place: Shibuya Hikarie, 9F, Hikarie Hall
- Invitees: Application and web engineers
- Admission: Free
- After-party: To take place in the Hikarie Hall after the event
You can register for the conference here. (Registrations are open until September 15)
We have 50 seats reserved for press and 450 seats for regular attendees. We would like to kindly remind you that the 50 seats reserved for press are meant for attendees that regularly post on their blogs or other social media, and are willing to write an article covering the event after the conference is over. We will send an email confirming your registration on September 22 if you are selected as an attendee. (Attendees will be randomly selected if the number of registrations exceed 450)
Check the schedule on the official LINE DEVELOPER DAY 2016 website
Scheduled to take place in the same venue with no additional fee required
Official Twitter hashtag: #linedevday
We are looking forward to meeting you all.
About the author
Kushii: In charge of actively and creatively promoting the technologies of LINE both in and out of the company. Also in charge of operations of LINE Developer Day 2016.
by LINE Engineer on 2016.8.5
If you have yet to read the introductory article to circuit breakers, I recommend you read the following article first: Circuit Breakers for distributed services
Applying CircuitBreaker to Channel Gateway
Channel Gateway servers provide various LINE server features to content providers. This is why Channel Gateway servers are highly affected by the servers they are connected to, with the effects easily spreading across all Channel Gateway servers.
It was when I was struggling to think of a solution to this problem that I first heard about circuit breakers. Circuit breakers seem to be what I was looking for since they are able to detect a problem in certain servers, blocking all requests that the server would have otherwise received. With that in mind, I decided to apply circuit breakers on Channel Gateway.
While I could have implemented my own circuit breakers on Channel Gateway, there were already excellent circuit breakers on Armeria. Armeria enables you to set various options on circuit breakers when you implement them using
CircuitBreakerBuilder, generating your own
CircuitBreaker object as a result. I was able to easily apply a customized circuit breaker on Channel Gateway thanks to this system.
by LINE Engineer on 2016.8.2
Hello, my name is Wataru Yukawa. I work at LINE as a data engineer.
As a data engineer, my daily duties include using Fluentd to collect logs, Hadoop to accumulate, and Hive to aggregate and analyze logs. Our Hadoop cluster is medium-sized, consisting of 40 units and approximately 370TB of DFS used space. Data from LINE family apps is smaller compared to the LINE app. While it’s nowhere near large enough to be considered as big data, it still has many types of different data, Fluentd tags, and over 400 Fluentd processes due to the various LINE family services tied to it. The Fluentd data flow amounts to 150 thousand messages per second during peak times.
As much as monitoring for Hadoop and Fluentd is crucial to us, the monitoring tools available to us were less than ideal. Prompting us to look for better solutions. That’s when I learned that the storage development team for the LINE app uses Prometheus and Grafana for monitoring. Prometheus is a next generation monitoring system with a pull-based architecture and powerful query language. I was so impressed with what Prometheus was capable of, I repeatedly listened to one of its developers speaking on a podcast about its fundamentals.
In the end, I decided to incorporate Prometheus into my working environment. Seeing how many teams were doing the same, I thought it would be a good idea to gather the people using Prometheus in a meetup where we can share information with each other. This is how we began Prometheus Casual Talks, the first Prometheus meetup event in Japan. While we still have to improve speakers diversity, as 4 out of 5 speakers were LINE employees, I think it’s a testament to how popular Prometheus is inside the company.
Through this blog post, I’d like to share my thoughts from the Prometheus meetup. And then briefly talk about the upcoming PromCon 2016.
by LINE Engineer on 2016.7.27
Last time I posted an article titled, “Analyzing Large Amounts of Security Data with Spark, Mesos, Zeppelin, and HDFS.” Today I will introduce how LINE applies cloud and stream processing technology to perform near-real-time processing on game data detected by AirArmor1.
1: AirArmor is a security solution for mobile games developed by LINE.
AirBorne DataCenter & Mesos (with DC/OS)
To analyze security data, we built our own system named AirBorne DataCenter. The system uses Apache Mesos as its base framework. And to process big data efficiently, the system implements open source components such as Kafka, Spark, Elasticsearch, Hadoop, Zeppelin, and Spring.
Launched in early 2015, AirBorne runs Spark on a Mesos framework, along with other tools like Zeppelin and Spring for querying and visualization. The goal of AirBorne is to query large amounts of data as quickly as possible. Traditional DBMS are not fit for large amounts of data. Likewise, it was difficult, or even impossible, to process large amounts of DB data or log data in JSON format. With AirBorne, we were able to overcome such difficulties. Also, as usability and analysis capability gradually evolved, it became necessary to automate the way we scale out physical nodes or monitor endpoints. The demand for enhanced usability and stability increased as well.
Most importantly, in addition to analyzing data on a regular basis, we wanted to process data in real-time, receive alerts immediately, and be able to monitor and control changes accordingly.
These new circumstances ultimately led us to re-design the existing AirBorne DataCenter and apply real-time processing.
Apache Mesos provides a framework where resources across multiple physical nodes can be combined and offered as a single logical resource. It can instantly allocate more resources on Spark executors when necessary while keeping the fixed amount of resources reserved for Kafka and Elasticsearch. This means we can improve the resource utilization by using otherwise idle resources. Also, Apache Mesos provides various ways to schedule resources using features such as Constraint, Role, and Quota.
DC/OS is a platform consisting of open source packages for the Mesos framework. Supported packages include Kafka, Elasticsearch, MySQL, Spark, Zeppelin, and more. You can easily install these packages and use various running tools that come with the platform. It has features for fault tolerance and horizontal scaling as well. We checked whether it would fit our purposes by conducting some trial tests and decided to bring it into our system.
by LINE Engineer on 2016.7.25
Hello, my name is Ono and I’m a LINE engineer. In this blog post, I’d like to talk about “circuit breakers” which we use with our LINE servers.
What is a Circuit Breaker?
The backend server systems for various web services and apps including LINE consist of networks that have several services connected with each other through APIs and RPCs.
What would happen if one of these networks suddenly failed to respond? The downed services would be blocked until they time-out, and all other services that rely on the blocked service would start a chain reaction of failures. If no one has been keeping an eye on the entire network, it will take a long time to figure out which service is the root cause.
These chain reaction failures should be prevented at all costs. At the least, the core features must be protected from these failures. To achieve that, we must block off access to the downed service before it affects another.
Circuit breakers are used to automate this whole process.