{"id":302,"date":"2018-11-10T09:19:12","date_gmt":"2018-11-10T09:19:12","guid":{"rendered":"http:\/\/www.sofadeve.com\/?p=302"},"modified":"2018-06-04T10:05:20","modified_gmt":"2018-06-04T10:05:20","slug":"loading-half-a-billion-rows-into-mysql","status":"publish","type":"post","link":"https:\/\/sofadeve.com\/?p=302","title":{"rendered":"Loading half a billion rows into MySQL"},"content":{"rendered":"<p>Interesting post on the derwiki blog &#8230; Especially the commenting is quite entertaining!<br \/>\nAmazing how ignorance produces patronizing statements (-> Morg).<br \/>\nSee belwo the top of teh post &#8230;<\/p>\n<p>Background<br \/>\nWe have a legacy system in our production environment that keeps track of when a user takes an action on Causes.com (joins a Cause, recruits a friend, etc). I say legacy, but I really mean a prematurely-optimized system that I\u2019d like to make less smart. This 500m record database is split across monthly sharded tables. Seems like a great solution to scaling (and it is) \u2014 except that we don\u2019t need it. And based on our usage pattern (e.g. to count a user\u2019s total number of actions, we need to do query N tables), this leads to pretty severe performance degradation issues. Even with memcache layer sitting in front of old month tables, new features keep discovering new N-query performance problems. Noticing that we have another database happily chugging along with 900 million records, I decided to migrate the existing system into a single table setup. The goals were:<br \/>\nreduce complexity. Querying one table is simpler than N tables.<br \/>\npush as much complexity as possible to the database. The wrappers around the month-sharding logic in Rails are slow and buggy.<br \/>\nincrease performance. Also related to one table query being simpler than N.<br \/>\n&#8230;<\/p>\n<p><a href=\"http:\/\/derwiki.tumblr.com\/post\/24490758395\/loading-half-a-billion-rows-into-mysql\" target=\"_blank\">>> Go to Source Article<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Interesting post on the derwiki blog &#8230; Especially the commenting is quite entertaining! Amazing how ignorance produces patronizing statements (-> Morg). See belwo the top of teh post &#8230; Background We have a legacy system in our production environment that keeps track of when a user takes an action on Causes.com (joins a Cause, recruits [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"chat","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[27,5,26,10],"tags":[],"class_list":["post-302","post","type-post","status-publish","format-chat","hentry","category-data-management","category-database","category-etl","category-mysql","post_format-post-format-chat"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5jq4Y-4S","_links":{"self":[{"href":"https:\/\/sofadeve.com\/index.php?rest_route=\/wp\/v2\/posts\/302","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sofadeve.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sofadeve.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sofadeve.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sofadeve.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=302"}],"version-history":[{"count":1,"href":"https:\/\/sofadeve.com\/index.php?rest_route=\/wp\/v2\/posts\/302\/revisions"}],"predecessor-version":[{"id":303,"href":"https:\/\/sofadeve.com\/index.php?rest_route=\/wp\/v2\/posts\/302\/revisions\/303"}],"wp:attachment":[{"href":"https:\/\/sofadeve.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=302"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sofadeve.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=302"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sofadeve.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=302"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}