2013年3月17日日曜日

fluentdとMongoDBでユーザー行動を見える化

このエントリーをはてなブックマークに追加

はじめに

エンジニアの@ryooo321です。
よろしくお願いします。
今回は弊社で運用中の全アプリで利用している行動分析プラットフォームについてご紹介したいと思います。
2012年の6月に作ってから、約9ヶ月ほど運用しています。

特徴

・手がかからないデータストア
・さまざまな問い合わせ対応で利用できる柔軟なクエリ
・機敏なMap/Reduceによる集計
・集計結果をCSVやグラフで可視化

目的

・ユーザーの問い合わせに効率的に対応し、アプリの企画・開発に集中するため
・ユーザーの行動を抽象化・可視化することでPDCAの質を向上させるため

行動ログのフロー

1. ユーザーからRuby on Rails製のソーシャルゲームにリクエスト

2. Railsからローカルのfluentdにログ出力(fluent-logger-ruby)

3. ローカルのfluentdから行動分析プラットフォームのfluentdにログ転送(fluentd)

4. 行動分析プラットフォームのfluentdに集約後、MongoDBにinsert(fluent-mongo)

5. MongoDB上でMap/Reduce集計し、集計結果をMongoDBに保存

6. 集計結果をCSVやグラフ出力


目次

1. アプリについて

2. fluentdの利用

3. MongoDBの利用

4. 行動をミクロな視点でみる

5. 行動をマクロな視点でみる

6. fluentdとmongo連携でのはまりどころ



1. アプリについて

現在メインで担当しているアプリは、全webサーバーで6,300,000req/dayほどの規模です。
アプリはRuby on Railsで作っており、Cassandra + MySQL + Memcacheを組み合わせて利用しています。


2. fluentdの利用

fluentdはイベントログの転送をするためのツールです。
起動やログを出力する方法は公式githubでご覧ください。

fluentdにはさまざまなプラグインが用意されていますので、
多種多様な言語・構成・目的を楽に乗り越えて活躍してくれるプロダクトだと思います。


起動/死活監視

GODを使っているのですが、fluentdは高負荷がかかっても落ちず高い可用性を発揮してくれます。

出力タイミング

サーバーのログからtailしつつ収集する方法もありますが、弊社ではRailsアプリから直接fluentdにログを送っています
Rubyのhashの形式のままfluentdを通じてMongoDBに保存できます。
fluentdは高負荷や落ちている場合でも処理をブロックしないので、アプリには影響が出ません。

弊社の場合Google analyticsも利用しているので、リクエスト統計などはそちらで収集しています。

出力するログの例

ユーザーのリクエスト毎(主にpost/delete系)に、下記のようなログを0〜数件出力しています。
document:
  _id: yyyymm_user_id_xxxxxxxxx
  user_id: xxxxxx
  category: ログ種別(user/payment/card/mission etc..)
  type: ログタイプ(in/out/use/get etc..)
  platform: xxx
  request:
    page: payment/finish
    viewer_id: xxxxxx
    request_id: 同じリクエストで複数ログを出すときは、先頭のログの_idを入れる
  device: xxx
  info:
    ここにログ種別ごとの詳細な情報をhashで出力
  condition:
    ログ出力時点でのユーザーの状態をすべてのログで出力
    level: xxx
    st: xxx
    bp: xxx
    max_st: xxx
    max_bp: xxx
    gacha_pt: xxx
    cards_count: xxx
    friends_count: xxx
    leader_card_id: xxx
    training_card_id: xxx
    joined_at: xxx
  time: ログ出力日時(fluentdが付与)
このようなログが1,000,000 rec/dayほど出力され、fluentdによりMongoDBに保存されます。


3. MongoDBの利用


Capped collectionを利用

MongoDBにはCapped collectionという固定サイズのコレクションを持つ仕組みがあります。
・固定サイズでアロケートされた領域にデータを書き込むため、通常のinsertより高速。
・固定サイズまでデータが溜まった場合、古いデータから自動で削除されていきます。

メリット

弊社ではこの仕組みを使うことで、チャンクの監視・手動のシャーディングを行わないで済むようにしています。
Capped collectionを使うことで運用から手が離れることのメリットが大きいです。

デメリット

一方で決して小さくないデメリットもあります。
・Capped collectionではデータのシャーディング(水平分割)ができませんので、Map/Reduceの分散処理の利点が十分に活かせません。
現在、1,000,000ほどの日次レコードを150ほどのMap/Reduceスクリプトで分析するのに、約30〜45分ほどかかっています。
レプリカセットという仕組みでの分散はできるのですが、MapReduceの利用には制限がかかります。(結果保存できずinlineでしか使えない)

・古いデータが削除されてしまう。
前述のログで600byte/レコードです。1日1,000,000レコードで0.6GBほどになります。
現在、100GBのCapped collectionを用意しているので、5ヶ月ほどは消えません。

また、古いデータが削除されてしまうため、MapReduceによる過去分の集計はできなくなります。
しかし、日次のMapReduceの結果を保存しておき、その結果は5ヶ月以上残し、MapReduceの結果に対してMapReduceで集計することで過去分の集計ができます。
そのため、データの持ち方や集計フローの設計が非常に重要になります。


4. 行動をミクロな視点でみる

行動調査

MongoDBでは、user_idとtimeにindexを貼っています。
これにより、ユーザーの行動は時系列で並べて調査できます。


出力したログはリクエストや対象モデルごとに絞り込みができるほか、json形式のログの全内容を確認することももちろん可能です。


行動ログの出力タイミング

ログはリクエストの最後にまとめて出力するのではなく、CassandraやMySQLに対してデータを処理した直後に都度出力しています。

これにより下記のようなメリットがあります。
・スキーマのデータ構造をシンプルに保て、検索や集計をシンプルに行えます。
・エラーなどによる出力漏れの可能性がほぼなくなりログの信頼性が大きく向上しました。


5. 行動をマクロな視点でみる

Map Reduceで集計

MongoDBのサーバー上にjavascriptで書いたMapReduceスクリプトを配置し、バッチで全scriptを実行しています。

共通ファンクション群のロード

下記のようにすることで、いくつかの便利なファンクション群をMapReduceスクリプトで使えるようにしています。
load(pwd() + '/map_reduces/lib.js')

MongoDBのmapReduceメソッドのscopeオプションを渡すことでこのメソッド群をmapファンクションやreduceファンクションでも利用できます。

集計の元ドキュメント

{
  category: 'payment',
  info: {
    price: 100,
    item_id: 20,
    count: 1
  },
  time: xxx,
  condition: {
    level: 10,
    joined_at: xxx,
  },・・・
}

一次集計されたドキュメント

_idが同じもののvalueを足し合わせてくれる単純なMapReduce集計を利用し、元ログを下記のように一次集計します。
一次集計することで、重い元データの集計を最低限にしています。
// 日付・購入物・属性の組み合わせごとに集計された状態です。
{
  _id: {
    ymd: '20130318',
    item_id: 20,
    level: 10,
    play_term: 20,
  },
  value: {
    count: 1,
    price: 100,
    arppu: 100,
  }
}

商品ごとの集計結果

一次集計した結果を元に、下記のように再集計します。
// 日付・購入物ごとにいくらの売上があったかの集計です。
{
  _id: {
    ymd: '20130318',
    item_id: 20,
  },
  value: {
    count: 1,
    price: 100,
    arppu: 100,
  }
}

プレイ日数ごとの集計結果

一次集計した結果を元に、下記のように再集計します。
※ レベルやプレイ日数は1単位ごとに集計してもよいのですが、あまり意味がないので5や10ごとに丸めて集計しています。
// 日付・プレイ日数ごとにいくらの売上があったかの集計です。
{
  _id: {
    ymd: '20130318',
    play_term: 20,
  },
  value: {
    count: 1,
    price: 100,
    arppu: 100,
  }
}

このような形で、さまざまな切り口での売上、商品のin/out、ユーザー数など、150ほどの集計を行っています。

さまざまなグラフによる可視化

日次で集計した結果を毎日必要なものをグラフやCSVで取得できます。




ルール

MapReduceした結果を簡単にCSVで出力したりグラフ化するに当たって、データを汎用的に取り扱うためいくつかのルールを決めました。
・_idとvalueの値は一次元のハッシュとすること
このルールにより、CSVやグラフの出力をシンプルに行えています。

・_idに必ずymdを持つこと
CSVの出力期間はymdでフィルターし、グラフ化のX軸はymdとし、Y軸はymd以外の_id値としています。
valueに複数のキーがあっても、valueのキーごとにグラフを出力します。
集計は日次単位でしか行っておらず、過去分の分析は通常行っていません。
※ 先ほどの例だと、X軸がymd、Y軸がプレイ日数、値がpriceのグラフなどが見れます。


6. fluentdとmongo連携でのはまりどころ

・fluent-mongoでMongoDBにデータを流すにあたって、MongoDBのRubyドライバでBSONエンコードできないデータを流すと「__broken_data」として登録されます。
例えばハッシュのキーがintだと「__broken_data」になります。

・fluentdからCapped collectionに流す
collectionに流す際は、fluentdの設定ファイルで「capped」指定と「capped_size」指定が必要です。


一緒に働きたい方、絶賛 募集中

京都でスキルアップしたいエンジニアの皆さん、ご応募お待ちしています!
京都でスキルアップしたい学生さん、アルバイトも可能なのでご応募お待ちしています!
大阪、滋賀、神戸から通勤実績あり


以上、長文にお付き合い下さいましてありがとうございました。

64 件のコメント:

  1. Good post! I am very glad to see this article.I appreciate you for sharing this informative article. Anyway we have a writing services company, Here is our social network community how much is edubirdie. You may follow our writers community for any kind of writing help in future. Thanks

    返信削除
  2. Thanks for sharing, nice post! Post really provice useful information!

    Hệ thống website máy đưa võng TS của An Thái Sơn gồm: Mayduavong.me, Mayduavong.biz, Mayduavongtudong.org, Mayduavongts.com, Treem.org cung cấp máy đưa võng giá rẻ tốt nhất.

    返信削除
  3. I am sure that education is really important. On https://writemypaper4me.org/blog/research-paper-conclusion you can read advices that will help you a lot in your academic life.

    返信削除
  4. What are the most important features of our company?
    Taif Furniture Transport Company depends on a number of factors to distinguish it and make it always the first in the field of transport of clothes, including:

    Our company transfers furniture by using sophisticated modern transport vehicles with covered boxes to protect furniture from damage caused by exposure to weather conditions.
    Al-Aafash Transport Company uses special furniture carriers to easily transport furniture from the windows of the house to the outside and vice versa with complete ease and complete safety in the case of narrow stairs, which do not allow the area to enter the furniture.
    All the technical staff of our company trained at the highest levels of training and have the best experience of providing the best results aspiring to reach them as soon as possible.
    The transport method followed by the furniture transfer company in Taif depends on the type and size of the furniture to be moved. Each piece of furniture we use to transport the appropriate types of transport machines and wagons, as well as vehicles equipped for each type separately, so as to assure the client of Afsha who wants to transfer it completely and not expose it to any From fractures, scratches or damage.
    The cheapest prices of the transfer company Abaf Taif and the transfer of furniture with jaw and installation and packaging
    The cheapest prices of the transfer company Abaf Taif and the transfer of furniture with jaw and installation and packaging
    Best Furniture Moving Company in Taif
    شركة نقل عفش ببريدة
    شركة نقل عفش بتبوك
    شركة نقل عفش بالجبيل
    شركة نقل عفش بابها

    شركة نقل عفش

    返信削除
  5. Assignment Help online
    Try to add only relevant data and information. If you feel that you are stuck, you can take assignment help online to get a proper idea. There are several web services which can help you to find out a suitable solution to your problems.

    返信削除
  6. Thank you for such an interesting blog. If you are looking for more such informative blogs, visit Great Assignment Help website. Here you will not only find information, but also Online Assignment Help from professionals. It is a one-stop solution for all your assignment writing-related queries.

    返信削除
  7. If you are seeking for Maths Homework Help services then do not waste your time and simply visit at StudentsAssignmentHelp.com. We have a team of professional homework helpers that deliver quality assignment writing help to students at an affordable price.

    返信削除
  8. We have been into the education industry from past few years. We are GoAssignmentHelp which offers online assignment help, assignment writing service, assignment help in Sydney for students and learning professionals. Take the advantage of nursing assignment help or all assignment help only at our one-stop destination for assignment needs.

    返信削除
  9. My Assignment Services, is well-known for its 24-hour online Assignment Help on the WhatsApp platform. Students can reach out to us with their queries at any point of the day or night and get the most effective solutions for them. University assessments have to follow specific referencing and citation styles like Harvard, APA, Chicago and MLA. Also, they should be referenced only from credible academic sources. Our best online academic help experts talk about how to select and evaluate the right academic sources, which will help you clear all your academic-related doubts. Nursing Assignment Help by My Assignment Services has been trusted by millions of students worldwide for over a decade. We have been providing complete academic assistance to students struggling with their college and university tasks. We have a team of dedicated subject matter experts who maintain a 100% record of submitting orders well before the deadline. This also gives the students some time to review them and ask for revisions, if needed. You can trust our academic ghostwriters completely to get best quality write-ups including case studies, research proposals, dissertations and Essay Help, and more.

    返信削除
  10. このコメントは投稿者によって削除されました。

    返信削除
  11. This informative blog was really amazing. Visit ogeninfo for the best SEO Services and Website Designing Companies.
    PPC Company in Delhi

    返信削除
  12. For Shipping and Logistics Services, visit Y & H Cargo India and get a better service for you.
    Logistics Company in India

    返信削除
  13. If you learn about music, Spin Gurus is one of the best place to get the best opportunity to you.
    Sound Engineering Courses

    返信削除
  14. Thank you so much for this valuable information. Visit for Mutual Fund Investment Schemes.
    Mutual Fund Agent

    返信削除
  15. Usually, I never comment on blogs but your article is so convincing that I never stop myself to say something about it. You’re doing a great job Man, Keep it up.strategic management assignment help

    返信削除
  16. Thanks for this informative content. It’s really good. Actually, I want to share some thoughts and reviews about an assignment help company in USA.

    AllAssignmenthelp reviews

    返信削除
  17. Thank You! For sharing such a great article, It’s been a amazing article.
    It’s provide lot’s of information, I really enjoyed to read this.
    Mobile app development company in toronto

    返信削除
  18. The experts in our team have knowledge in various academic disciplines such as English, economics, history, management, IT, law, arts, nursing, hospitality, etc. So grab the best assignment offers today. At StudentsAssignmentHelp.com we have the best and high-qualified team of Essay Writers.

    返信削除
  19. Students Assignment Help provides the best write my assignments for all students. Our professional expert writers provide academic assistance services to all students. Students can get help from our online assignment writers 24*7.

    返信削除
  20. We are providing the best assignment help Melbourne. Best service for your academic-related writing works. You achieve the best grade in exams. Directly hire at Ausrtaliaassignmnethelp.com.

    返信削除
  21. Thanks for the article may be useful for everything
    App Developers in Noida

    返信削除

  22. Thanks for sharing your wealthy information. This is one of the excellent posts which I have seen. I go through your all of your blog, but this blog is the best one. It is really what I wanted to see hope in future you will continue for sharing such an excellent post.
    We also provide blog
    cdr writing services
    cdr engineers australia sample

    返信削除
  23. Thank you so much for sharing such an amazing blog. Visit Ogen Infosystem for the best Website Designing Services in Delhi.
    Website Designing Company in Delhi

    返信削除
  24. Nice blog, Y&H Cargo India for best Shipping Services and Freight Forwarding Companies.
    Logistics Company in India

    返信削除
  25. Mutual fund wala is the best Investment Advisor in Delhi and provide best mutual fund schemes.
    Investment Advisor in Delhi

    返信削除

  26. Hi, I think that I noticed you visited my website thus I got here to go back the favor?. I am attempting to find issues to improve my website! I guess its good enough to make use of some of your ideas!!

    cdr report for civil engineer
    cpd engineers australia sample

    返信削除
  27. It is nice to read such high-quality content. It is a good article that discusses the topic at hand quite well. I am looking forward to read more articles from your site. Keep up the good work.
    assignment help

    返信削除
  28. Thanks for sharing valuable Information, I really very impressive on your blog. I hope you continue on blogging job.
    Mobile App Development Company in Mumbai

    返信削除
  29. ABC Assignment Help provides professional Perdisco assignment help in various subjects for students studying in colleges and Universities across Australia, UK, USA, New Zealand and Canada. We deliver quality solutions for assignments in over 100 subjects. Our tutors focus on providing step-by-step solution to every assignment problem. As a trusted brand among students, we strive to provide 100% unique solutions at an affordable price.

    返信削除
  30. Students want to instant assignment help? Then, hire over best experts. We have best experts they are hold master and Ph.D. degrees. Solve any writing problems and provide best work. Book our experts at australiaassignmenthelp.com.

    返信削除
  31. Your blog is filled with unique good articles! I was impressed how well you express your thoughts. Many times students are tensed for their projects and assignments, so at that time they can take online business ethics assignments to complete it on time. There are experts available online who prepare the perfect assignment assistance for you.

    返信削除
  32. StudentsAssignmentHelp.com experts are here for your help. They will finish your academic pressure and present amazing assignment help experts services online. Our writers hold masters and Ph.D. degrees from the eminent universities across the world.

    返信削除
  33. I feel happy about and learning more about this topic. keep sharing your information regularly for my future reference. This content creates new hope and inspiration within me. Thanks for sharing an article like this. the information which you have provided is better than another blog.
    IELTS Coaching in Dwarka
    Best IELTS Coaching in Dwarka

    返信削除
  34. このコメントは投稿者によって削除されました。

    返信削除
  35. このコメントは投稿者によって削除されました。

    返信削除
  36. Write My Assignment In A Click| Online Assignment Help 24/7 Live
    Worry for who will 'make my assignment'? A perfect platform for writing your assignment at the very cheapest price possible. 1000+ PhD experts , 100% satisfaction, 0% plagi, 24/7 online support. Best writing service for assignment help online in your pocket budget. Book your order now. Get flat 20% off on your first order. Also get assignment help Canada.

    返信削除
  37. It is fantastic post. I'll be waiting for more posts. Stop the collection calls

    返信削除
  38. このコメントは投稿者によって削除されました。

    返信削除
  39. Thanks for the great share and keep up this good work!. Queen Creek Arizona Real Estate

    返信削除
  40. Thanks for sharing this valuable information, I am here to share some benificial information for stduents looking for assignment help. We are assignment providers providing , mechanical engineering assignment help, electronics assignment help and management assignment help.

    返信削除
  41. Don't waste your more time because students assignment help is working for students, so you can easily get their math assignment help searvice and easily collect our more best services from the professional experts of StudentsAssignmentHelp.com. Students who are demadning for homework service for them we are always here with out best team. that's why you can collect top homework service easily.

    返信削除
  42. Thank you so much for sharing this valuable information with us. Get Painting Services by Kalakutir for Fleet Painting and School Bus Painting and more.
    School Bus Painting

    返信削除
  43. Great blog. I really appreciate the information I have gathered from your blog which really helped me a lot. Thanks a lot for sharing this. Waiting eagerly to see your next post. thesis editing services

    返信削除
  44. We are proud of our best Assignment Help Singapore experts because of their dedication providing continuous support to students by helping them meet deadlines and scoring better grades. We understand how important academic assessments are in developing a student's career and future opportunities, this is why we take extreme measures to ensure that all our solutions are best-in-class. All students looking for assignment help Singapore can reach us and avail the best online nursing assignment help from our academic experts.
    Our primary motive is not only to provide you with the best help but also to make your journey smooth. The increasing importance of writing assignments has prompted students to take a helping hand from assignment writing services. During higher studies in colleges, students often have to prepare multiple documents, quizzes, and surprise tests. This is the main reason why most students search for assignment help online over the internet and choose only the most proficient and trusted academic writing experts.

    返信削除
  45. Assignment help is the online service where students can get full knowledge in academic session. assignment help

    返信削除
  46. Our masters and Ph.D. degree professionals that understand students demands and finish the task properly. Our experts of native writers from the USA, Ireland, Australia, Singapore, UK, New Zealand, etc are working for students.
    At students assignment help experts are ready to offer you business management assignment help.

    返信削除
  47. Our masters and Ph.D. degree professionals that understand students demands and finish the task properly. Our experts of native writers from the USA, Ireland, Australia, Singapore, UK, New Zealand, etc are working for students.
    At students assignment help experts are ready to offer you business management assignment help.

    返信削除
  48. ABC Assignment Help is an incomparable online Engineering assignment help company delivering excellent academic assignments, essays, coursework and reports. Through a team of over 3000 subject experts we ensure individual attention to every student making the assignment help experience completely personalized in nature. With our round the clock services, you can be assured of high grades every time.

    返信削除
  49. Management students are worried about management assignment help. Don’t worry about them, we provide al type of assignment writing work. We have the best solution for you. We have 10+ years of experienced experts to help you. Hire Now at Australiaassignmenthelp.com.

    返信削除
  50. We are proud of our best Assignment Help Australia by experts because of their dedication towards providing continuous support to students by helping them meet deadlines and scoring better grades. Hiring an expert to write your work is easy but hiring the RIGHT expert to get it done is a challenge. This is because there are a multitude of assignment help australia services and picking the best is always going to be a trial and error method. In case of engineering and pure sciences, the data analysis’ initial three/four steps remain the same. We have some of the best Engineering Assignment Help who have intensive knowledge of wide range of topics as discussed above and beyond. They feel completely pressurized. He starts disliking his school or college. But may come, he has to submit his assignment before the deadline, therefore he chooses to buy assignment services available to him. To help students complete their assignments Gotoassignmenthelp is here to serve them with the best assignment help.

    返信削除
  51. Gotoessayhelp is a round the clock essay help service which caters solutions request to various subjects’ tools & methodology in a multi environment learning concept for the subjects like essay help online. We are a renowned service provider of essay assignment help and have been receiving an overwhelmed response globally.

    返信削除
  52. あなたが私たちと共有した情報はとても役に立ちました、ありがとうございました。
    stick rpg 2

    返信削除
  53. Nursing is a vast subject and I got to know about the scope of this vast subject after reading this blog. Now, I got to know why students need nursing assignment help  from experts. My friends have recommended me to use the services of Online Assignment Expert. This is because their assignment experts handle even the trickiest programming codes with ease!
    Thus, from now on whenever I would require Online Assignment Help, I would knock the doors of the it assignment help experts of online assignment expert.

    返信削除
  54. Looking for the best writing service online but lost in the number of fake reviews and don’t know who to believe? We at ScamFighter publish only genuine reviews on all writing service from the web. Look at the last review on: https://scamfighter.net/review/speedypaper.com

    返信削除
  55. Nice blog, Ogen Infosystem provides best Website Development and Digital Marketing Services in Delhi. For more information visit the website.
    SEO Service in Delhi

    返信削除
  56. Students need assignment help Adelaide? Don’t worry, we provide at any location in Australia our online service. Our experts write plagiarism free content. Hire now at Australiaassignmenthelp.com.

    返信削除
  57. Thanks for sharing informative post. Soulblu is a leading vape store Melbourne offering wide range of e liquids, e electronic cigarette, vape supplies, DIY mixing tools, etc.

    返信削除
  58. My Assignment Help is an incomparable online Assignment help company delivering excellent academic assignments, essays, coursework and reports. Through a team of over 3000 subject experts we ensure individual attention to every student making the assignment help experience completely personalized in nature. With our round the clock services, you can be assured of high grades every time.

    返信削除
  59. Nice blog. Dealing with tough assignment deadline? Don't worry Student Assignment Help experts are here to serve you best solutions.

    返信削除