by LINE Engineer on 2015.5.14
Hello, my name is K. I was in charge of organizing this year’s conference. As announced in the LINE DEVELOPER DAY_2015 Tokyo Announcement « LINE Engineers' Blog post, the LINE DEVELOPER DAY_2015 conference was held on April 28th, 2015.
There were so many people that signed up for the conference, we could not invite everyone. But thanks to all the engineers that showed interest in the event, the conference was a huge success. We would like to express our sincere gratitude!
I was in charge of planning and running the event, including the minute details. Planned as an opportunity to “let engineers outside the company have a deeper understanding of LINE,” it was certainly not an easy task. But the event was well received both during and after the conference, even trending on Twitter. We sincerely hope we can meet you all again with another conference in the near future. In-depth reports on the event are still in preparation. But in the meantime, here are the videos taken during the conference along with the presentation slides.
Opening Remarks Chief Executive Officer Takeshi Idezawa
LINE Global Culture Chief Technology Officer Euivin Park
LINE Messenger for the World Senior Executive Officer, Head of Service Development Ikebe Tomohiro
LINE Platform Development Chronicle Tom.T
Slides can be found here→ LINE Platform Development Chronicle
HBase and Redis: LINE’s Storage System for Processing Over 100 Million Messages Per Day Shunsuke.N
Slides can be found here→ HBase and Redis: LINE’s Storage System for Processing Over 100 Million Messages Per Day
Evolution of and Challenges in Developing LINE Android over the Last 4 Years Tsutomu.H
Slides can be found here→ Evolution of and Challenges in Developing LINE Android over the Last 4 Years
Burgeoning Sticker and Theme Sales Systems: A Chronicle of Risks and Revival Haruki.S
Slides can be found here→ Burgeoning Sticker and Theme Sales Systems: A Chronicle of Risks and Revival
How Should LINE Creators Market have been Developed Amidst the Internationalization of Web Services? Tokuhiro.M
Slides can be found here→ How Should LINE Creators Market have been Developed Amidst the Internationalization of Web Services?
Replacement of LINE’s Messaging Pipeline with Akka Actor and AMQP Yuichi.O
Slides can be found here→ Replacement of LINE’s Messaging Pipeline with Akka Actor and AMQP
LINE for Apple Watch Christopher.R
Slides can be found here→ LINE for Apple Watch
Changes to the Development Team Due to the Implementation of Swift Yosuke.I
Slides can be found here→ Changes to the Development Team Due to the Implementation of Swift
Development Strategies for a Global Network Environment and Multi-OS Supporting LINE Game Client Platform Heejun.K
TV TOKYO x LINE Business Connect Sato Nobuaki of TV Tokyo Communications Corporation
Slides can be found here→ TV TOKYO x LINE Business Connect
Platforms for Analyzing Big Data: Analytical Techniques Required to Aggregate Data Taiichi.H
Slides can be found here→ Platforms for Analyzing Big Data: Analytical Techniques Required to Aggregate Data
Developing Recommendation Engines that Utilize Bayesian Inference and Deep Learning Jun.N
Slides can be found here→ Developing a Recommendation Engine Based on Bayesian Inference and Deep Learning
by LINE Engineer on 2015.4.30
Hello, I am HT and I work in B612 for Android development. B612, which is named after the asteroid B-612 from the novella “the Little Prince,” is the world’s first selfie app to feature pre-filtered selfie and 3 or 6 second collaged video capture. In this post, I would like to discuss the process of creating an MP4 file by making a video collage using MediaCodec in B612.
Determining video size
B612 features video capture, but real-time video encoding is difficult to implement because it requires a large amount of data processing. Data must either be drawn on the screen by being sent from the GPU memory to the system memory and then being processed on the hardware encoder, or by processing each pixel individually on the CPU into video data. As this process will need to run in real-time handling large amounts of data, it is better if the size of the data can be as small as possible. In order to create video information with a small amount of data, the size of the screen being read must be minimized so that only the required information is used. To achieve this, the resulting video size should be determined first and then the size of each individual video will be calculated. The video size will be determined based on this data, and will be encoded based on the size from that result.
The size of the resulting video is determined by the collage chosen by the user. The image information from the camera will be arranged in the shape of the collage, matching the number of grids horizontally, and vertically. The size here will be the maximum size of the first resulting video. For example, if the pixel size of the image being sent from the camera is 320×480, and the collage is in the form of 1×2, the resulting image will be 320×960 with one image placed horizontally and two images placed vertically. The resulting video will then be reduced and resized to the maximum aspect ratio that the device is capable of displaying. Considering that this process needs to run in the graphic library, (GL) the maximum size of the video must be smaller than the maximum texture size that the GL can handle. Depending on the device, resolutions that are too large may not be able to be processed in real-time, so maximum resolutions should be reduced.
Once the size of the first resulting video is determined, the size of the original videos will be each determined based on this information. The resulting video will be horizontally and vertically divided by the number of the collage, and will be resized to be multiples of 16. This is because videos commonly use macro blocks of 16×16 pixels, and resolutions that are multiples of 16 generally work best on all devices.
Initializing MediaCodec and determining decoder type
MediaCodec can be initialized with the following code.
mediaCodec = MediaCodec.createEncoderByType("video/avc"); MediaFormat mediaFormat = MediaFormat.createVideoFormat("video/avc", width, height); mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, bitRate); mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, frameRate); mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface); mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, frameInterval); mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
As seen in the code above, the code used to generate encoders has been declared so that it will create an H.264 video with the appropriate “video/avc” MIME value. Also, the appropriate bitrate and framerate will be sent to the generated encoder so that the video can be created in the desired format. The color format sends COLOR_FormatSurface so that it can be encoded through Surface. While it is possible to add other color format options, only options provided in MediaCodec should be added if Surface is not used. However, with so many devices and so many color formats, the color format input data must be customized for each device. Since images with different color formats need the appropriate image data placement when they are combined into a collage, additional code is needed to accommodate for the color formats that an Android device supports. This leads to more development resources being used, and that is why we opted to generate MediaCodec encoders that use Surface instead.
The speed of the MediaCodec decoder will depend on the performance of the hardware it is running on. When combining collages that are small individually, but contains several videos, it may be more efficient to use software decoding more than hardware decoding. Also, using a hardware decoder codec may cause unexpected behavior on certain devices running a certain amount of instances of MediaCodec. That is why for devices that required more stability, we opted to use the OMX.google.h264.decoder option on the software decoder provided by Google. This way, multiple videos could be decoded more quickly in a stable environment.
Multi-threaded parallel decoding
MediaCodec is designed to encode and decode even on single threads, with timeout values provided from functions such as dequeueOutputBuffer. Setting a low timeout value will allow access to several decoders without threads being blocked for too long even on single threads. Although even when using this method, there will be an unavoidable slight delay, making it more effective to use multi-threaded video processing. That is why we chose to allocate a thread to each video decoding MediaCodec, so that video could be processed quickly even if there was a block in MediaCodec. Figure 2 below, illustrates the interaction process between the generated decoding threads and the internal thread of MediaCodec. While using individual threads in this way may have an advantage in performance, this may backfire by overloading the memory with too many images when the encoder cannot catch up with the speed of the decoder if all threads are working individually. To prevent this, we put in a buffer between the images being generated by the decoder and limited the size of these decoded images so that they would not overload the capacity. Through this method, decoding will only continue until the buffer is reached when encoding speed is slower than the decoding speed. After the encoding process empties the buffer, decoding will recommence. This way, memory can be prevented from being unnecessarily wasted.
Video synthesis using Surface
As mentioned in “Initializing MediaCodec and determining decoder type” above, the encoder is set to receive data from Surface during the encoding process. Before the data is sent to this encoder, the resulting image must be ready in the desired shape and design. To get this image, we rendered video in the shape that we wanted by allocating FrameBuffers. The result from the multi-threaded decoding from above is then updated as individual textures in the GL thread through the updateTexImage function. Later, we set vertex coordinates for these textures so that they would be arranged in the way we wanted in the FrameBuffer. Once the resulting video is drawn based on these vertex coordinates through the glDrawArrays function, the desired result can be achieved. While each individual video may slightly differ in length, they will be combined into one frame set to the desired frames per second (FPS).
Generating MP4 files using MediaMuxer
An encoded result that is only made of H.264 raw data does not contain information such as audio, and cannot be played in common players since additional information needed for playback cannot be included. In order to play the encoded result it the result screen or in another application, the encoded result must be inserted into a container format such as MP4. MediaMuxer, which is included in Android, can be used to generate MP4 files. The process can be achieved with the following code.
muxer = new MediaMuxer(path, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4); trackIndex = muxer.addTrack(newFormat); muxer.start(); // Repeatedly call the code below to record data on the MP4 file. muxer.writeSampleData(trackIndex, encodedData, bufferInfo); muxer.stop(); muxer.release();
After generating MediaMuxer with the code above, the video and audio must be inserted as separate tracks. However, since B612 does not need audio data during video synthesis, the audio data is only added if the user chooses to select audio to be added as a track while generating a file.
In this blog post, we took a look at how we used MediaCodec, how we improved speed with multiple threads, and how a collage is synthesized. With the process above, we were able to use hardware encoders instead of software codecs, to provide a more faster encoding speed to our users. It was our top priority to create an app that could maintain top performance without needlessly using up memory. And thanks to the multi-threaded process that minimizes resizing, we were able to achieve our goals.
by LINE Engineer on 2015.4.24
Hello, my name is Roy and I work in LINE iOS development. With the recent release of Apple Watch, I would like to introduce the features of Apple Watch, and how it was like developing LINE for Apple Watch.
Introducing Apple Watch
Apple Watch was first unveiled on September 9th, 2014 as “one more thing” following the iPhone 6, iPhone 6 Plus, and Apple Pay. It will be available in three different versions named Watch, Watch Sport, and Watch Edition with a choice between a 38mm or 42mm watch face. Apple Watch will be able to track the user’s movements, as well as receive notifications from an iPhone. An SDK was also released so that third parties could develop for Apple Watch as well. Apple Watch comes equipped with a display that supports Force Touch, and a Digital Crown that is used to zoom the display in or out. The side button below the Digital Crown launches Friends, giving you access to all of your contacts. On the bottom is the sensor, capable of detecting the heart rate of the user. Inside Apple Watch is the built-in speaker and Taptic Engine, which will alert users with sound or haptic feedback whenever they receive a notification.
Developing for Apple Watch
In order to develop for Apple Watch, your iOS app will need a WatchKit extension. While it was possible to have several extensions for a single iOS app, you are able to add only one WatchKit extension. Adding a WatchKit extension enables three functions: the WatchKit App, Glances, and custom notifications.
WatchKit app: The iOS app equivalent on Apple Watch. Apps can have various page layouts, multiple screens, and receive user input.
Glances: The widget equivalent on Apple Watch. Glances can be seen by swiping up from the bottom of the Apple Watch display. Apps that have been favorited by the user will appear here. For example, a glance of the calendar would show your schedule, and a glance of the weather app would show you the current weather situation.
Custom notifications: By default, all iOS notifications will show up on Apple Watch as well. You can change the look of how these notifications will actually appear, similar to how you would change the design of your WatchKit app. It is also possible to display related pictures.
Apps on Apple Watch can display images or text similar to how iOS apps work. Buttons, switches, and table views are available, but there is no screen rotation, or more advanced functions like video/music playback.
Apple Watch does not actually run any code on the device itself. Instead, the iPhone does. Apple Watch is only capable of displaying the the results of the code being run on the iPhone, and it only stores image resources and storyboards needed for the display. In other words, third party apps simply stop functioning if Apple Watch is separated from the iPhone that it is paired with. For example, if connection between Apple Watch and the paired iPhone is lost, the WatchKit app will stay stuck on a command without proceeding to the next step. Apple Watch merely acts as a screen that displays data sent from the paired iPhone.
In order to share data with an iOS app, the WatchKit app can use database, user configuration data or files. Alternatively, the WatchKit app can dynamically run the iOS app to send and receive data when needed. Queries can be sent to the iOS app through the openParentApplication function, and responses can be relayed back to the WatchKit app. While this function is convenient, it is slow to use because of the time it takes to communicate with the iOS app. Even a few seconds could be critical to user experience, since users would need to lift their arms up to look at the Watch display. That is why it is best to avoid using this function. However, if there is a feature that cannot be easily implemented by only using the WatchKit app, or if the schedule does not allow more time for development, it may be used sparingly.
Things that can and cannot be done on Apple Watch
- Third party apps cannot directly control hardware. Access to the Digital Crown (other than the default functions accessible when adding a table view), heart rate sensor, and GPS is restricted. However, built-in Apple Watch apps are able to access the hardware.
- Apple Watch can convert speech to text, but it can cannot record and save audio.
- Views cannot be stacked on top of each other.
- The way gestures are recognized cannot be changed.
- Force Touch can only be used to bring up a menu.
- Notifications only show up on Apple Watch if the display on the paired iPhone is off.
- By default, all notifications will show up on Apple Watch. You cannot choose specific notifications to appear on the screen.
Developing LINE for Apple Watch
LINE on Apple Watch uses two features: custom notifications and the WatchKit app.
Whenever the user receives a text, photo, or sticker they will receive a notification on their Apple Watch. In the case of the LINE iOS app, notifications will only let users know what they received through text such as “sticker” or “photo,” unable to see the actual contents until the app is open. On Apple Watch, all those notifications are directly displayed on-screen. Any other messages will be displayed in the form of text messages. The text had to limited to three lines since it would get cut off if it was too long.
For loading Stickers or photos by only using information from the payload, we used the aforementioned openParentApplication function. We were well aware that this method would be slower, but this way we could get it done fast and come up with a solution later. It was so slow in fact, that Apple requested that we change it before Apple Watch launches.
The reply button
We tried to include a reply button on the bottom of the message notification, but it was not as simple as it first seemed. In order to add a button to the notification meant that we had to branch the notification so that it can display buttons. This would require the notification payload being modified. We needed assistance from server-side, and then we had to deploy it on the actual server. There was another problem even on the server-side; the reply button would also show up on the iPhone notification as well. This was to accommodate for situations where an iPhone could be physically far away from an Apple Watch. Working on the WatchKit app more or less affected the iPhone LINE app in this way, so we had to get confirmation from the planning team. The confirmation process was a burden, and it could also potentially cause other issues as well. Ultimately, we had to exclude the reply button feature from the 1.0 version. The design team and planning team, even the people at Apple had asked us why we decided to do so. We answered their questions by telling them we would implement it in version 1.1.
The notifications system was one of the most difficult parts during development. Simply because there was no way to test them on an actual device. With the simulator, we were only able to test notifications under the scenario that they were already open on the device. Without access to the home screen or other apps, we were unable to test scenarios where a user might receive a notification while using another app, or receiving a notification while checking a notification, or closing the notification. As a result, we were unable to undergo any actual testing until we had the actual device.
Here are a few screenshots from the WatchKit app. LINE on Apple Watch lets users read unread messages and reply to them. The first screen will display a list of chat rooms with unread messages. Once the user taps on a chat room, they can read all the unread messages and reply to them by pressing the reply button. The user may reply with Stickers or Sticons, but even Sticons will still be sent in Sticker size. Once the user leaves the chat room, the room will be marked as read and removed from the list.
Loading the chat room list (refreshing the database)
There was an issue that prevented displaying chat rooms with new messages when the app was opened right after a notification was received. This was due to the iOS app not refreshing the database in time. Now when the user launches the Apple Watch LINE app they will see a loading screen while the iOS app refreshes the database. The loading screen was made to closely resemble the default Apple Watch screen so that users would not notice any huge differences. Although we knew that it would be slower, we used the openParentApplication function to refresh the chat room list. This was because we wanted to refresh the database even when LINE was open on the iPhone, and because it already took us long enough to implement this feature just for Apple Watch. The delay felt negligible once we added in the loading screen.
Scheduling issues during chat room implementation
The chat room did not present us with any serious issues, but implementing the chat room itself took a considerable length of time. Since it was arguably the most important part of the app, there were some design issues here and there, but thankfully there was no need for any research for new technology. The most noticeable difference with the iOS app is that the WatchKit app will not display profile pics, timestamps, and speech balloon tails for messages that were received in under a minute. This was a design choice to display as much information as we could on the small display. While it was aesthetically good, the minor bugs it caused took us a while to fix.
Delays in loading Stickers/Sticker lists
There was an issues with the speed in which Stickers and Sticons were being sent. Displaying a large number of Sticker and Sticon images simultaneously brought the speed down significantly. We were able to solve the speed issue by using the below function that is also recommended in Apple documentation.
This function had a huge advantage over the previous function:
[WKInterfaceImage setImage:[UIImage imageNamed:]]. The function I was using would send an image to the iPhone and then send it back to Apple Watch, while using
[WKInterfaceImage setImageNamed:] would load images stored on Apple Watch instantly. The new function offered a significant boost in image loading speed.
Lack of performance test tools
There were no available tools to test performance on Apple Watch. As of now, the only way to measure performance is to see how long a certain feature takes to run ourselves. We also could not perform any memory checks, although we were able to check if the classes we created were deallocated properly. The table view in particular, was unlike on iOS where cell could be re-used. We had to check if cell memory was deallocated properly. Other than that, we ran the app to see if there are any crashes.
The most important thing to know about Apple Watch was the specifications needed for development. We did not have any access to a test device, nor did the device have any prior versions that we could work with. There were limits to what we could learn no matter how thoroughly we read the documentation on Apple Watch development. Any further questions we had, we perused the forums to find the answers we needed.
As it was the first SDK to be released, there were many limitations to what we could do. It was unfortunate that third party apps were restricted from accessing many of the things Apple Watch had to offer. Third party apps are unable to access all the new fancy hardware in Apple Watch, or the UI effects that are apparently used in the built-in apps. At times it was confusing, because it was difficult to tell if it was my lack of research that was stopping me from using those features. Hopefully, the SDK might just be updated incrementally, and the next version is not long away.
Developing for iOS is always a convenient experience due to the detailed documentation available. Apple Watch development was also relatively painless due to my prior experience of developing a widget for iOS devices. Although without any tools to test performance or memory limitations, I cannot say for sure how the app will perform once it is actually distributed. With the Worldwide Developers Conference (WWDC) right around the corner, I sincerely hope development for Apple Watch is more improved down the line.
by LINE Engineer on 2015.4.1
The coming summer of 2015 marks the fourth anniversary since we have first announced the LINE mobile messenger app. Since its initial launch, LINE has gathered an overwhelmingly large number of users. Over the years we have also provided many LINE Family Apps to enable users to communicate by sharing pictures, playing games, keeping up with the news, and design wallpapers. To us, ensuring that users have an optimal mobile experience is one of our top priorities.
by LINE Engineer on 2015.3.20
Hello. My name is JY and I am an engineer on the LINE Rangers development team.
I wonder if the readers out there have ever played LINE Rangers? It is one of the many popular LINE Games serviced through LINE, while also being an in-house developed game.
I am currently working on server development on the LINE Rangers team; I still have much to learn. I’m also very new to the company since I have just joined in early 2014.
I used to wonder what kind of server engineers would be working at a company that provided services more than five hundred million users. I also wondered about things such as how skilled they would be, and what kind of environment they would be working in. Now, I’m working amongst those very engineers. I’ve learned a lot after I have started server development, and I would like to share some of my experiences in this post.
This post will be about “A Rookie Engineer’s Analysis on the Server of LINE Rangers.” And it will be divided into the two parts you see below.
- Development Environment
- Infrastructure Environment
First, I would like to talk about the LINE Rangers server development environment. I will introduce the tools we use, how we work, and how our source codes are managed, combined, and distributed. FYI, this post may not be very in-depth. Some readers may already know everything that I’m going to write down here. I’m not here to amaze you with never-before-seen technology or a deep understanding of the subject matter. Consider this post as a record of my struggles as a rookie engineer; an engineer who had to spend a full day just installing an IDE and getting the Local Server up and running.
Do you remember the tools you used when you first started learning how to develop? While there are many more, I believe most of you would be familiar with IDEs (Integrated Development Environments). These let you code, debug, compile, and distribute in an all-in-one package.
When talking about IDEs, Eclipse is probably the first thing that comes to mind. While being free, it’s as powerful as any paid software is. Anyone who studied Java would have used Eclipse at one point or another. The LINE Rangers server development team also uses Eclipse. I have always wondered about what kind of IDEs would be used in the actual field, just imagine my joy when I found out that the tool I’ve been using since my school days was used here as well. Eclipse is not the only IDE in use here. We also use other more recently developed IDEs as well, such as IntelliJ.
IntelliJ is a commercial IDE tool that has two different versions. The Community version is free for use and available to anyone, while the Ultimate version must be purchased. The Community version is open source, allowing engineers to download the source code and modify it to fit their needs. The Ultimate version is the Community version with more added functionality. Compared to the Community version, the Ultimate version near-perfectly supports more languages and frameworks.
You can’t talk about games without mentioning databases. With so many users that need to be managed, the importance of databases cannot be stressed enough. The DB of LINE Rangers is managed by professional DBAs (Database Administrators) who are more adept at managing DBs than the ordinary engineer. While the engineers still write query statements, they can request assistance from the DBAs if they have a query that needs to be more fine-tuned. The DBA will analyze the tables and fields of multiple DBs, revise the query and reflect the changes made to the Real DB. The LINE Rangers DB is divided into multiple phases: Alpha, Beta, and Real. We engineers handle the queries in the Alpha and Beta phases, but the Real DB is strictly restricted to the DBAs. There are different data and separate connected servers for each phase. Some of you might be wondering why we have multiple DBs when we only work on one game. Each DB phase closely relates to each development phase. The Alpha DB exists so that engineers can freely change anything during development. The Beta DB is where QA (Quality Assurance) testing is done in a realistic environment. Finally, the Real DB is where data for actual users is stored.
Our DBs are divided into three phases, but how about the servers? Servers have two more phases than the DBs, divided into five phases: Local, Alpha, Beta, Staging, and Real. The first phase is the Local Server, where each engineer can develop and test their code on their local computers. In other words, running a WAS (Web Application Server) on each engineer’s PC. While the Alpha Server is also used for development purposes, it differs from the Local Server. The Alpha Server is where code from each engineer is integrated into one and analyzed to find any overlaps. Also, the WAS is tested to see if it runs correctly before going to the Beta phase. The Beta Server is for QA testing, and communicating with the game client. Once the server engineers create an API, the client engineers can connect to the API to develop the client. Also, the Beta Server functions as a testing environment for QAEs (Quality Assurance Engineers) before actual release. The Staging Server provides a testing environment identical to the Real Server. Testing is done in the Staging Server before distributing to the Real Server. Since a single issue can lead to malfunction in the Real Server, a virtual version of the Real environment is used during testing. Lastly, the Real Server: the server that handles actual user data. Since this server is used to process massive amounts of data, it is physically divided into many parts. Due to the importance of this server, it is always monitored in real-time with a monitoring tool. We have gone over the databases and servers of Rangers, learning what they do in each phase. In a way, the two are inseparable. The diagram below depicts how each server is connected to each DB; five servers and three DBs forming the backend structure of LINE Rangers.
Numerous engineers work on the LINE Rangers server, and managing code is a very important matter. A history must be kept of who did what to which module. This is known as “revision control” or “versioning.” The LINE Rangers server uses SVN for versioning. We use an internal repository that allows each engineer to use SVN commands such as checkout, add, commit and revert directly from their IDE. Using the SVN features in IDE should be faster and easier than manually inputting SVN commands. Source code revisions can be safely kept under control this way.
Next, I would like to talk about the final destination of development: building and distribution. Rangers uses an internal automated system. The details are confidential so I will only describe them briefly.
Our internal automated system can handle the whole process of building to distribution with a press of a button. It also displays the progress of each step in real-time. I will briefly explain the workflow below. First, you must build the source code using Maven; a convenient all-in-one tool that can compile, package, test, and distribute.
The diagram above depicts each phase Maven goes through. These build phase clusters called “build life cycles” each present a set of tasks that need to be done.
Once the build is complete, it is distributed as the latest version. Below is a distribution scenario. Jenkins will create the final build going through the phases above through Maven. The build will then be uploaded to the automated system server. Jenkins is no longer required after this step as the automated system will go through the rest of the process with its predefined tasks. These tasks then divide the server distribution process into more detailed steps. The steps are as follows.
First, stop the web server and then the WAS on the machine that you want to distribute to. Second, package and distribute the build. This is the step where you will distribute the build package to a predetermined directory. Last, restart the WAS and then the web server.
Each step is written in a shell script and then sequentially run by the automated system. The results can be seen in real-time through a web browser Build/distribution is configured for all environments except for the Local Server that we discussed above in the five phases of the “Servers” section. Thanks to the automated build/distribution system, Rangers server engineers can save time and focus more on development.
Up to this point, I’ve explained about what kind of environment we LINE Game engineers work in. We’ve gone over how LINE Rangers is developed from the perspective of a server engineer. Next we will take a look at the the server structure of LINE Rangers.
Server Structure and Infrastructure Environment
As you can see in the diagram above, the LINE Rangers server can be divided into four parts: Front, Middle, Data, External. The External part on the right represents the external common infrastructure of LINE Rangers. LINE Rangers has a system that lets you purchase in-game currency called “Rubies” with real money, and this requires a payment infrastructure set up. We could have set up our own, but we could have much more easily just used one of the existing payments modules from LINE. Using an already existing system from our platform not only saved us time, but it was much safer. We also rely on external infrastructure for LINE authentication, push notifications, team leaderboard tabulation among others. The same infrastructure supports every other LINE game as well. In other words, for anything that needs to be repeatedly implemented with each new game, we utilize what we can from the LINE platform. That’s about it for the External part. Next, we’ll go over the Front, Middle, and Data parts.
The Front part of LINE Rangers can be divided into the Logic server and Static server. The Logic server is where requests from the client are calculated and returned or stored. Since it is often used by the client to call the API, it is sometimes called the API server. The Static server stores static files, such as images and sounds used in the game. These can also be processed in the Logic server, but LINE Rangers has a separate Static server because divvying it between the two ensures better performance. The Static server is in use when the progress bar fills up while LINE Rangers is loading the game.
LINE Rangers has an enormous player base. Everything that goes on in the game must be sent to and processed in the server, resulting in millions of requests being made if each player starts the game. That’s why the LINE Rangers server, especially the API server, was designed to resist the workload.
The diagram on the right is a simplified depiction of how the servers distribute the workload from numerous clients. A single server can only handle a finite number of requests at once, and that is why LINE Rangers utilizes several tens of servers. Providing a service on multiple servers like this is known as “load balancing” or “horizontal scaling.” This method improves performance by using several mid-tier systems rather than using a single high-tier system. The client does not need to know the individual addresses of each server in this case. An L4 Switch device, or Load Balancer, receives requests from the clients and then hands them off to the servers in the back-end. The client only needs to know the address of this L4 Switch. The requests are distributed in a round-robin method, sequentially allocating requests to servers registered in the Load Balancer. This is how we keep the server workload in check: by keeping a Load Balancer between the client and the servers. Exactly how many servers are being used by LINE Rangers? LINE Rangers is running 85 API servers. Then are all these 85 servers physical, you may ask. The LINE Rangers server is entirely virtualized, so a single physical device can run several Logic servers.
Simply put, server virtualization is a method of using a single physical device as several logic devices. As you can see in the diagram above, a single device can run several virtual machines. Only one actual device is needed to run several logically partitioned servers independently. Server virtualization ensures more expandability. In instances where an additional server is needed, you do not need any additional physical equipment to respond to that demand. The LINE Rangers API server is run by these 85 virtual machines. It was necessary to handle the millions of players, though it wasn’t always like this from the beginning. We did not need many servers immediately after launch since there were only a small amount of players. As we expanded and gained more users, we needed a way to handle all the requests. We were able to provide service for LINE Rangers without any major issues by using server virtualization. We used the same method for all servers running LINE Rangers, including the API servers.
This part is comprised of the Queue server, Batch server, and Callback server. The Queue server exists to deal with failover, which is a backup operational mode in which the functions of a system component are assumed by secondary system components when the primary component becomes unavailable through either failure or scheduled down time. For example, if an update to the DB cannot be processed because the DB equipment is malfunctioning, the data can be sent to the Queue server and where it will be sent back later one by one once the DB is in working order. It may not be processed in real-time, but accurate recovery is guaranteed. The Batch server is where various batch jobs needed in LINE Rangers are run. The Callback server is mostly used for payment. Once payments are processed through Google or Apple’s payment modules, the LINE Rangers Callback server is called, saving the results to the internal DB.
This is the server that stores various data created and used in LINE Rangers. Depending on the type, data can be stored in the RDBMS or the NoSQL DB. LINE Rangers uses MySQL as an RDBMS. Out of the 19 MySQL devices in total, one set is used to store metadata. Metadata refers to the basic information used in-game, such as unit attack power and health. The location where this data is stored is called the Info DB. Another ten sets are used to store information on each user. These sets are called the User DB. You should see the Info DB and User DB in the “LINE Rangers Server Structure” diagram above. Excluding these two DBs, there are an additional 8 sets that exist to store historical data. Historical data refers to things such as coin usage records or battle result records. LINE Rangers also uses NoSQL in addition to RDBMS; an open source server called redis in particular. To be more exact we are using a modified version of redis called nBase-ARC, courtesy of LINE’s parent company. ARC stands for Autonomous Redis Cluster. It is a distributed storage platform made by clustering sever redis servers, providing more ease of use. LINE Rangers puts nBase-ARC at the front-end of MySQL, using it as a cache. As seen in the “LINE Rangers Server Structure” diagram, the User Cache and Info Cache are what make up the nBase-ARC cache storage. The User Cache collects frequently used data from the User DB and then stores it on the memory. The Info Cache does the same, but it caches data from the Info DB instead.
Databases are obviously very important for data storage. However, the actual storage is done through servers, which is why the two are so closely connected. Then how are the DB and servers of LINE Rangers connected?
As you can see in the diagram above, there are multiple servers and databases. A single DB will inevitably have lower read/write performance as it needs to store data in one location, using multiple DBs can improve performance by distributing data across multiple locations. This method is known as DB Sharding, which I will go into more detail in my next post. For now, you only need to know that it’s a method of distributing and storing data.
Now, let’s think of a hypothetical worst-case scenario. What if one of the DB equipment malfunctions? What would happen if you cannot access any of the data stored on that DB? Obviously, the game wouldn’t run. A user that has their data on a DB that is still in working order would have no problems at all, but a user that has their data in the malfunctioning DB would experience server issues. That user would be greeted with strange errors or have their game crash. Since game servers should be kept out of the average player’s view as much as possible, the situation would be dire.
Replicating the DB to prepare for malfunctions is known as a Master-Slave structure. By having both a Master DB and a Slave DB, the Slave can replace the Master if any problems occur. Once the Master is back in working order, it can return. There can be one or several Slaves. The important part is that there must be one Master and it must be managed separately from the Slave DB. The Slave acts as a replacement for the Master, since it should have identical data. Of course, the Slave isn’t always idly waiting. The Slave DB is mostly used for data lookup. It is inefficient to read and write on a single DB since normally, it takes more time to write than read. That’s why the Master DB takes care of the writing while the Slave DB is used for reading. Data written to the Master is immediately replicated to the Slave, keeping the data synced between both databases. The reason I referred to the databases as “sets” is because they are sets of Masters and Slaves. The LINE Rangers MySQL DB is comprised of the same Master-Slave structure, the Info DB in particular having one Master paired with three Slaves. The reason to this is because a lot of SELECT queries are sent to the Info DB because it stores basic information used throughout the game. In order to ensure a more stable and reliable service, we decided to place three Slave databases dedicated to reading.
In this post, we have gone over the following parts of the LINE Rangers server development environment.
- How we code and what IDE we use
- How we deal with versioning
- How we have structured the servers and databases
- How we build and distribute
We have also seen how the LINE Rangers server and infrastructure is designed to handle hundreds and millions of players.
- External : LINE Rangers External Infrastructure
- Front : API & Static Servers
- Middle : Failover, Callback, Batch Servers
- Data : RDBMS, NoSQL DB
by LINE Engineer on 2015.3.12
LINE recruited its new employees publicly for the first time in 2015 and “LINE style” on-the-ground training was held for about 6 weeks for the newcomers. Here are the epilogues shared by 2 of the participants (1 male and 1 female).
Recruit Training Epilogue 1
Hello! My name is Jenny and I’ve just become one of the newest members of LINE. I have decided to share the experiences that I had during the 6 weeks of training. I wanted to let everyone know the struggles that my colleagues and I had to endure during the training course and, of course, to settle myself as a fresh new recruit. All the newly hired ones including myself have now been assigned with teams and positions, but just a few days ago we were going through the craziest schedule of 6-week training we’ve ever seen. The training course we went through required us to grow and improve ourselves with every step. Suffice to say; we will not forget it for a very long time. Upon completing the course, I soon figured out that we have not only grown as individuals because of the knowledge that each of us accumulated through the course, but also because of our unwavering pride as a part of LINE, affection towards LINE’s corporate culture and a firm reliability towards the colleagues who are on the same boat. Here, I am about to share with you my memoirs of the struggling challenges of becoming a LINE member and the valuable anecdotes I picked up during training. Here, I would like to talk about what I think was the 4 essential courses of our training. (Which were education, the workshop, visiting LINE headquarters in Japan, and our project.)
Let me begin with “education.” A large part of our training focused on education. Prior to the public recruiting of 2015, LINE only hired experienced engineers who already had the expertise in a specific part when there was an opening. As a new recruit to LINE without any expertise, having so much to learn, we had a long way to go to prepare ourselves to work in the field, let alone carry out the “mobile messenger development project” that I will introduce to you later. The training focused both on educating the newcomers on what LINE is and the hands-on knowledge of actual development. During the daily 10AM-7PM lectures, we were able to pick up LINE’s corporate culture, internal infrastructure for cooperation and development and other advanced technical details that cannot be easily learned outside. We were also granted an opportunity to have a glimpse on how LINE VoIP and LINE games were developed thanks to the precious time that our senior LINE engineers spared for us through the special courses. The technical education mostly focused on Java, Spring framework, Hadoop, Redis and Android; the tools that are widely used in LINE. We, as trainees, were able to develop the basic abilities to fulfill our roles in LINE and learned how to understand and communicate with each other even though we were to be separated into server and client development. Since we were given an opportunity to apply the technology that we learned from the messenger development project that went along with the training, the education surely felt very much practical and helpful for all of us.
My second part of the story is about the workshop that took place in the suburban training center. We had 2 wonderful days among our training period to stay in the suburbs, where we found peace, both mentally and physically, while enjoying the beautiful scenic view looking down the downtown area as well as the modern building design and interior. The accommodation that we used was full of eco-friendly spirit and cute elements and we were very much satisfied with the 2-day workshop. The sole tragedy was the fact that we only got to spend 2 hours in our accommodation since we stayed up all night at the Hackathon, coding until the break of dawn. In the training center, a so-called “Burning Room” greeted us. The room was a space for engineers to burn their energy while they concentrate on achieving the set goal of the day. We were surprised to find out this “Burning Room” was identified as “The Coding Pit” in the workshop schedule leaflet. However, we soon appreciated the existence of the Burning Room, although I have to admit it was quite scary and at the same time unique, since it was a space carefully prepared for the engineers or planners to accomplish their set goals in a comfortable manner. The Hackathon was surely a meaningful experience. It provided us with hands-on knowledge to materialize what we had learned in the books. We were able to complete the framework and foundation of the messenger app; which was a project given to all newcomers. Although we had to give up our sweet morning sleep to carry out the project due to the all-day trainings, words could not describe the satisfaction that we felt at the moment of accomplishing our goal at sunrise. Not only this, it felt as if I shared camaraderie with my colleagues with whom I worked with through many sleepless nights. We once had doubts of finishing our project, but on our march back to the living quarters we were filled with an unspeakable sense of fulfillment. Watching the rising sun, we had a newfound appreciation for the name, “Burning Room.”
Visiting the Japan Headquarters
As a new recruit to LINE, it was indeed a valuable experience having to be trained in Japan, the birthplace of LINE. I hurried my footsteps to Japan with high hopes since I am well aware that LINE is “The” messenger app in Japan used by almost everyone. Recently I heard from my acquaintance that life is impossible without LINE in Japan. We all stayed in Shibuya, Tokyo for 4 days where the Japan headquarters is located. LINE was in a marvelous high-rise building called Hikarie in Shibuya. The glass walls surrounding 4 sides of the building presented us with a magnificent day and night view of Shibuya. Such impression naturally led to my heart swelling with pride for LINE. We also got a chance to meet with those who contributed largely to the creation of LINE. All of the newcomers watched and listened with gleaming eyes to catch every word of the near-legendary development related stories in the initial stage of LINE and actual structure of LINE. There was also a Q&A session prepared for us, where we realized how global LINE was from the various nationalities of the employees. My heart pounded faster imagining myself working with these international engineers in such a global company. One of the most memorable things in Japan was the “Find LINE in Japan” mission. The mission was to track down traces of LINE in Japan, where LINE forms an important part of the everyday life. We found LINE Taxi, LINE@ and LINE Official Accounts (OA) in every corner of Shibuya where the mission took place and bumped into many Japanese people who were using LINE to communicate with their friends and families. Witnessing such a “LINE craze,” we saw how LINE had established itself as a service taking care of various aspects of everyday life in Japan, being beyond a mere messenger app. I still remember how proud we were when we visited the LINE Friends Store in Harajuku. The store was bustling with people with smiles on their faces as they took photos with LINE characters.
Last but not least, the project! It was the core essence of the training course, the alpha and omega. It served as a chance to think deeply and plan out a “messenger” worthy of the LINE namesake. Since there wasn’t much time to spend on the project due to our all-day training sessions, most of the new recruits devoted their weekends, sometimes heading off to the office one hour earlier and leaving 6 hours later than the regular sessions to focus on their projects. It pushed us to our limits, but my colleagues and I had the most amusing moments and learned the most valuable lessons through this project.
The theme of the project was to create a messenger with 3 of the other team members. We were able to use tools actually used in LINE such as NoSQL, internal infrastructure, Git, a Bug Tracking System and also Agile type development processes. We all aimed to create a messenger that would stand out, by infusing unique ideas so that it would be more than a conventional messenger app. At first, our mentors and even we doubted whether we could complete development of the messenger only using the leftover time we had within just 6 weeks. We had to kick off the process before getting familiar with the corporate culture or the infrastructure and it was quite awkward for us to work with team members who we had never met before. However, we were motivated to complete the project from what Euivin Park, CTO of LINE, told us, that there are various cases where a couple of LINE engineers gathered around a table to tackle internal development issues by their own accord. What we felt was more than just pressure forcing us to compete the project; it was a sense of great responsibility as a “LINE engineer.” The given task was not much different from the actual work we would be handling in the future.
To our relief, our challenge ended with success. On the last day of our training, all of the new engineers had a chance to present their awesome mobile messengers in front of many leaders of divisions and teams. We were very happy for the warm applaud that our seniors gave us. We may have been greenhorn novice engineers at the beginning of training, but we were now presenting the projects that we had been working on as genuine LINE engineers. The audience felt it, and my colleagues and I felt it as well. I completed the training with a firm belief in my mind: that “I can do anything!” The training course at LINE was far from being a one-way lecture where all the trainees would sit down while being told how great LINE is. It was indeed a valuable opportunity to have a hands-on experience on LINE’s culture, visiting places where LINE is widely used, learning about various technologies and infrastructure applied in the field. We were able to think of how to make LINE better, not only as rookie engineers, but in the shoes of the engineers already working in the company. I am very proud that I, together with all my colleagues, was able to witness a meaningful growth in my development capability as a novice engineer and thereby naturally becoming a real “LINE engineer.”
Recruit Training Epilogue 2
Hi, I am WH and I’ve just become a rookie engineer taking charge of LINE Camera development for iOS after completing a 6-week program of welcoming training. In 2015, LINE publicly recruited new employees for the first time. How do you imagine the excellent engineers of LINE planned their first training program for the new engineers? I would love to share the splendid experience that I had during the last 6 weeks.
I became part of the LINE crew in January, 2015. The training for newcomers was held for 6 weeks, beginning with an introduction on LINE services, and its organizational structure. After that, we were able to learn the basics of various development tools and processes used at LINE.
More in-depth training started from the second week. The overall topic was messenger server/ client development and I was provided with the tools, so to speak, required for fulfilling tasks from the company.
First of all, I received a notebook computer, a must-have item for engineering. The model I received was a mid 2014 13-inch MacBook Pro Retina installed with a 256GB SSD, which made the remainder of training much more enjoyable. The moment when I first booted up my computer is still vivid in my memory: Much to my surprise, the OSX login screen appeared in mere seconds, even though it was a cold boot. Moreover, my colleagues and I were permitted to freely use the conference room during the development period along with a 24-inch monitor.
Not only were the tools impressive, LINE also provided useful lectures in a timely manner. Our server used Redis for our database and Spring for framework. Since my colleagues and I all had different skillsets, we were provided with 8 hours of Java, Spring, Redis, and etc. lectures on a daily basis. We also had the pleasure of listening to special lectures given by our senior engineers who are contributing to the development of specific fields such as VoIP. We were able to feel and understand, albeit indirectly, the mental labor that our senior engineers experienced to come up with a service that they cared for.
Once the project theme and team were set, we had our first meeting as a team. Our aim during the first meeting was to establish the team’s development culture. We had hoped to achieve something greater than just gaining development knowledge by having a sense of purpose instead of just blindly developing.
As a result of our meeting, we summarized our team culture in 3 bullet points:
- Like LINE Engineers: First, instead of just jumping head-first into messenger development, we familiarize ourselves as much as we can with the tools and processes used in LINE.
- Team Improvement: Second, all team members should accumulate development experience across server and client through Pair & Mob programming.
- Effectiveness: Actively search for and adopt open sources for more effective development.
Moreover, upon realizing that many of the LINE teams remotely co-work with engineers in various nations, we decided to communicate not only in our mother language but also in English and Japanese. To prepare ourselves for a global era, we also agreed to use English when submitting issues to JIRA or committing to Git.
Most LINE teams apply Scrum during the development process (Using JIRA for issue management). Accordingly, our team segmented the training course to a total of 6 sprints for Scrum. We established a goal and drafted the backlog for the relevant sprint during the plan meeting held on the starting day of each sprint by gathering around a table at around 9 AM. The drafted backlog was, then, submitted as an issue to JIRA so that we can get hold of our progress at any time. Once the sprint began, we held a 15-minute meeting every day at 9:30 AM to share what had been done yesterday, what needs to be completed today and what other development issues were on the table.
After 2 weeks of listening to lectures and working on our project, we had 2 days of camp training. Honestly, we were not seeing much progress in our project due to lack of time from 8 hours of daily lectures. Not to mention the time we spent establishing our development environment. Understanding our situation, our mentors allowed us to be immersed in all-night development just like in a “hackathon.” We tried out Mob Programming for the first time during this occasion. Instead of splitting the roles and then collecting the outcome back again, we rather chose to have one computer and big monitor to work together. This was also in the same line with the ‘purpose of newbie training’ to accumulate experience on both server and client development. We were able to share a similar level of understanding on the structure by working hand in hand when forming a basic structure for the system. Such experience served as a foundation for our team to respond more flexibly to all sorts of issues that we encountered while implementing additional features afterwards.
Visiting the Japan Headquarters
On our 4th week of training, we joined a 4-day workshop at the LINE headquarters located in Shibuya, Tokyo in Japan. I can still remember the moment when I looked down at the beautiful, seemingly endless urban landscape through the glass walls of the LINE café located on the 27th floor of the Hikarie building. During the workshop in Japan, we had a chance to listen to the detailed description of the Hikarie office, introduction on LINE development organization as well as the lively stories told by engineers and planners who made the LINE of today. Contrary to rumors online, we realized that the success we witness today was certainly a collaboration of powerful technology and key marketing strategies based on thorough analysis of the characteristics of Japanese users.
On our third day of stay in Japan, we were given permission to freely roam Tokyo, while also being on a “LINE Treasure Hunt” of sorts. My colleagues and I visited the LINE Friends Store in Harajuku and bought LINE merchandise while observing the jolly-looking shoppers. We also found out that many stores used the LINE official account “LINE@” to stay in touch with their customers via LINE. It was a meaningful moment to actually feel LINE being firmly settled down as part of everyday life beyond in Japan. To them, it was more than a mere mobile messenger app.
Afterwards, some of my team members and I visited Odaiba on a monorail called the “Yurikamome.” We saw some cosplayers there and we jokingly said that we should wear LINE Friends costumes like Cony or Brown, next time when we visit again.
There were only 2 weeks of training left after our workshop in Japan. A mobile messenger app is generally equipped with many features other than message transferring functionality. Thanks to our mentor’s advice, we came to a consensus that rather than including all sorts of features to the messenger, we would only include one standout feature after we implemented the most basic functions of a mobile messenger app. Focusing on our core goal, we were able to complete our project in 2 weeks. Our unique messenger boasted a feature that let users enjoy anonymous chatting using the “LINE Friends” characters as their personal avatars.
On the last day of the training, we stood in front of the many leaders from the LINE development team and presented what we had been concentrating on during the past few weeks. After 6 weeks of training, all teams including ours were proudly presenting their expertise on par with an actual LINE engineer. We mentioned various development tools and processes used in LINE amazing the audience with how we had transformed from newbies. As much as we wanted to maintain the illusion of our professionalism during the presentations, when we actually demonstrated the applications we developed ourselves, the apps would malfunction or crash, drawing laughter from the audience.
Normally, when we think about training new recruits, it’s natural to imagine a chaotic schedule with endless lectures. Things are done differently at LINE. Our training only provided us with the big picture, teaching us only the fundamentals. The project we were given was like a huge empty canvas, what we would draw there was our own choice to make. I feel that my colleagues and I did what we set out to do, each of us becoming LINE engineers in our own right.
by LINE Engineer on 2015.3.11
Hello, I am SJ, an engineer working at LINE.
In this post, I would like to talk about counting characters. There are many places in various LINE services where the number of characters must be counted such as profile or group names, and status messages. Counting the characters on-screen is important for several reasons. The text must not be shorter or longer than necessary, and storage capacity must be allocated accordingly. As LINE is used worldwide, it is crucial that string length can be precisely calculated for various languages. One day, we encountered an issue where emojis would be counted as 2 when they should count as 1. Originating from Japan, emoji is a language comprised of images used all around the world today. 続きを読む »
by LINE Engineer on 2015.2.2
Hi, my name is James and I work for LINE Corporation as a junior software developer. I’m very pleased and even thrilled today because it’s such a great honor to share my experiences and thoughts from “LINE Developer Workshop & Hackathon 2015.”.
by Yoichiro on 2014.9.11
My name is Yoichiro, and I work in the Technology Strategy Department here at LINE. Thanks for taking the time to check out our blog. Today we’ll be introducing the system we use for the error detection and notification process for the gradually growing service “LINE BusinessConnect.”