Introducing the LINE Game AirBorne DataCenter

by LINE Engineer on 2016.7.27

Last time I posted an article titled, “Analyzing Large Amounts of Security Data with Spark, Mesos, Zeppelin, and HDFS.” Today I will introduce how LINE applies cloud and stream processing technology to perform near-real-time processing on game data detected by AirArmor1.

1: AirArmor is a security solution for mobile games developed by LINE.

AirBorne DataCenter & Mesos (with DC/OS)

To analyze security data, we built our own system named AirBorne DataCenter. The system uses Apache Mesos as its base framework. And to process big data efficiently, the system implements open source components such as Kafka, Spark, Elasticsearch, Hadoop, Zeppelin, and Spring.


Launched in early 2015, AirBorne runs Spark on a Mesos framework, along with other tools like Zeppelin and Spring for querying and visualization. The goal of AirBorne is to query large amounts of data as quickly as possible. Traditional DBMS are not fit for large amounts of data. Likewise, it was difficult, or even impossible, to process large amounts of DB data or log data in JSON format. With AirBorne, we were able to overcome such difficulties. Also, as usability and analysis capability gradually evolved, it became necessary to automate the way we scale out physical nodes or monitor endpoints. The demand for enhanced usability and stability increased as well.

Most importantly, in addition to analyzing data on a regular basis, we wanted to process data in real-time, receive alerts immediately, and be able to monitor and control changes accordingly.

These new circumstances ultimately led us to re-design the existing AirBorne DataCenter and apply real-time processing.

Apache Mesos provides a framework where resources across multiple physical nodes can be combined and offered as a single logical resource. It can instantly allocate more resources on Spark executors when necessary while keeping the fixed amount of resources reserved for Kafka and Elasticsearch. This means we can improve the resource utilization by using otherwise idle resources. Also, Apache Mesos provides various ways to schedule resources using features such as Constraint, Role, and Quota.

DC/OS is a platform consisting of open source packages for the Mesos framework. Supported packages include Kafka, Elasticsearch, MySQL, Spark, Zeppelin, and more. You can easily install these packages and use various running tools that come with the platform. It has features for fault tolerance and horizontal scaling as well. We checked whether it would fit our purposes by conducting some trial tests and decided to bring it into our system.

More »

Circuit breakers for distributed services

by LINE Engineer on 2016.7.25

Hello, my name is Ono and I’m a LINE engineer. In this blog post, I’d like to talk about “circuit breakers” which we use with our LINE servers.

What is a Circuit Breaker?

The backend server systems for various web services and apps including LINE consist of networks that have several services connected with each other through APIs and RPCs.

What would happen if one of these networks suddenly failed to respond? The downed services would be blocked until they time-out, and all other services that rely on the blocked service would start a chain reaction of failures. If no one has been keeping an eye on the entire network, it will take a long time to figure out which service is the root cause.

These chain reaction failures should be prevented at all costs. At the least, the core features must be protected from these failures. To achieve that, we must block off access to the downed service before it affects another.

Circuit breakers are used to automate this whole process.

More »


by LINE Engineer on 2016.7.12

Hello everyone. My name is Kushii, and I am in charge of organizing conferences at LINE.
I am pleased to announce that LINE DEVELOPER DAY will be taking place once again this year.

At the event our engineering team plan to share some of the experiences, the technical challenges that exist in Japan and overseas, and introduce our latest products directly to you. Security-related topics, BOTs, and other themes will be explored during our individual sessions to explain how LINE strives to overcoming the various challenges ahead.

Event Summary

Date/Time: September 29, 2016 from 10 a.m. to 6:30 p.m.
Place: Shibuya Hikarie, 9F, Hikarie Hall
Invitees: Application and web engineers
Admission: Free
After-Party: To take place in the Hikarie Hall after the event

How to Sign Up

An application form will be made available towards the end of July.

Follow the LINE DEVELOPER DAY 2016 official account to be notified as soon as we begin accepting applications, and receive other latest news related to the event.

Please also check out the videos and information from last year’s event that are also available on our website.

Official Twitter Hashtag: #linedevday

How to provide stable “Loves”

by LINE Engineer on 2016.7.12

Hello, my name is Moznion and I’m part of the team working on LIVE: a video-based social media service that connects stars and celebrities to their fans.

On LIVE, users can watch their favorite stars participating in programs or performing concerts all in real-time. LIVE is currently available on iOS/Android and on PC web browsers, and quickly gaining popularity.

In this blog post, I’d like to explain how we handle large amounts of consecutive requests sent on LIVE.


On the iOS/Android version of LIVE, we have a feature that lets users tap on a “heart” to send support to the current broadcast that they are watching (No. 1 on the screenshot above). Users, or viewers in this case, can tap on the heart as much as they want, and the total number of hearts sent is represented on the screen in real-time (No. 2 on the screenshot above). Broadcasters can directly see how much support they are getting from their fans, and viewers can feel a sense of solidarity with other viewers while they immerse themselves in the experience.

Our engineers have a separate name for these “hearts.” We call them “loves,” and that is what I’ll refer to them as in this blog post.

As you can imagine, live broadcasts from popular idols receive a huge surge of “loves” instantaneously and require the back-end to efficiently distribute these requests so that the service can maintain itself. In the following sections of the post, I’ll elaborate on how we implemented a counter system on LIVE to have high capacity and near-instantaneous reactivity, with the ability to handle the amount of “loves” sent by a massive amount of users.

Read more

Fukuoka office, reopened with a whole new look

by LINE Engineer on 2016.7.8

Hello, my name is Kushii. I work in the development department in Japan and I’m in charge of promoting technology events at LINE.

LINE has quite a few development offices worldwide where many engineers write code every day. We have offices in Tokyo, Fukuoka, Fukushima in Japan and also in Korea, Taiwan, and Thailand.

The Fukuoka office was relocated recently, and I think this is a good opportunity to introduce the brand new, upgraded working place for our engineers.

The new office is located in the JRJP Hakata building, directly accessible from Hakata station. When you first set foot in the building, the LINE Friends characters greet you at the entrance.
Read more

The Road to Releasing LINE Trial Bot SDK

by LINE Engineer on 2016.6.15


Hello, my name is Matsuno and I work at LINE.

LINE BOT API Trial accounts were announced a few days ago, receiving a warm welcome from developers all over the world. Previously only available to developers working at companies who were under contract with LINE, now any individual developer can freely start developing their own project with their own LINE bot account.
In this post, I would like to talk about why we decided to start development on a LINE BOT API Trial account SDK and how we released it in less than a week.

Why the Bot SDK was created

As soon as LINE BOT API Trial accounts were announced, many of our own LINE engineers started developing for them. Which is when we began thinking…
What’s the most difficult part about creating a bot with this API??

Especially for static languages like Java, you need to create individual classes to map JSONs to. You have to implement a number of classes for things such as text messages, video chat, stickers, rich messages because in the case of LINE BOT API Trial accounts, there are many messages that send and receive API endpoints such as the ones mentioned above. Creating classes for each and every table available on LINE Developers is a painstaking endeavor and this prompted us to think,
this is actually a big deal!

We learned through several sources, mainly through company chats, that many people thought the same way.

That is how we came to the conclusion that:
Read more

try! Swift Presentation Report

by LINE Engineer on 2016.4.28

Hello everyone, my name is Inami (@inamiy) and I’m an iOS software engineer at LINE.

A Swift developer conference titled “try! Swift” was recently held in Shibuya, Tokyo from March 2 to March 4. LINE was a gold sponsor of the event, which was a gathering of over 500 people (30% of which were from overseas). There were many female presenters in the event as well, making it one of the most diverse gathering of developers I’ve ever seen; the likes of which I haven’t seen in any Swift/iOS study group! I was offered a chance to present at the event about “Parser Combinator in Swift”, a functional programming method.
Presentation slides
Live tweets during the event

Me speaking in front of the audience

Read more