{"id":1019,"date":"2020-07-13T16:06:59","date_gmt":"2020-07-13T16:06:59","guid":{"rendered":"https:\/\/nilg.ai\/?p=1019"},"modified":"2022-05-25T11:41:37","modified_gmt":"2022-05-25T11:41:37","slug":"experiment-management-and-reproducible-research","status":"publish","type":"post","link":"https:\/\/nilg.ai\/pt\/202007\/experiment-management-and-reproducible-research\/","title":{"rendered":"Experiment Management and Reproducible Research"},"content":{"rendered":"<p>In this tutorial, we will discuss how can we achieve reproducible data pipelines and research while keeping track of the experiments that lead to reproducible production Machine Learning models. We will go over all the <a href=\"https:\/\/nilg.ai\/pt\/technology-stack\/\">popular tools we use<\/a> available and do a small demo of how we can use these tools (e.g., <a href=\"https:\/\/dvc.org\/\">DVC<\/a>, <a href=\"https:\/\/www.pachyderm.com\/\">Pachiderm<\/a>, <a href=\"http:\/\/neptune.ml\/\">Neptune<\/a>, <a href=\"https:\/\/www.comet.ml\/site\/\">Comet,<\/a>\u00a0<a href=\"https:\/\/www.wandb.com\/\">Weights &amp; Biases<\/a> e <a href=\"https:\/\/mlflow.org\/\">MLFlow<\/a>) to get a seamless workflow with a good balance between production and experimentation.<\/p>\n<p>Check our video below, share and subscribe if you like it.<\/p>\n<div align=\"center\"><div class=\"responsive-embed widescreen\"><iframe src=\"https:\/\/www.youtube.com\/embed\/yumDu6pxnd0\" width=\"840\" height=\"472\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/div><\/div>\n<h1><\/h1>\n<p>&nbsp;<\/p>","protected":false},"excerpt":{"rendered":"<p>In this tutorial, we will discuss how can we achieve reproducible data pipelines and research while keeping track of the experiments that lead to reproducible production Machine Learning models. We will go over all the popular tools we use available and do a small demo of how we can use these tools (e.g., DVC, Pachiderm, [&hellip;]<\/p>\n","protected":false},"author":44,"featured_media":1020,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[79],"tags":[48,45,76],"class_list":["post-1019","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-webinar","tag-ai4tech","tag-machine-learning","tag-tools"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v20.8 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Experiment Management and Reproducible Research - NILG.AI<\/title>\n<meta name=\"description\" content=\"A tutorial on how to achieve reproducible data pipelines and keeping track of the experiments that lead to production Machine Learning models\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/nilg.ai\/pt\/202007\/experiment-management-and-reproducible-research\/\" \/>\n<meta property=\"og:locale\" content=\"pt_PT\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Experiment Management and Reproducible Research - NILG.AI\" \/>\n<meta property=\"og:description\" content=\"A tutorial on how to achieve reproducible data pipelines and keeping track of the experiments that lead to production Machine Learning models\" \/>\n<meta property=\"og:url\" content=\"https:\/\/nilg.ai\/pt\/202007\/experiment-management-and-reproducible-research\/\" \/>\n<meta property=\"og:site_name\" content=\"NILG.AI\" \/>\n<meta property=\"article:published_time\" content=\"2020-07-13T16:06:59+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-05-25T11:41:37+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/nilg.ai\/wp-content\/uploads\/2022\/05\/pexels-chokniti-khongchum-2280571-e1653476248932.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"2048\" \/>\n\t<meta property=\"og:image:height\" content=\"1367\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Tiago Freitas\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@nilg_ai\" \/>\n<meta name=\"twitter:site\" content=\"@nilg_ai\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Tiago Freitas\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minuto\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/nilg.ai\/202007\/experiment-management-and-reproducible-research\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/nilg.ai\/202007\/experiment-management-and-reproducible-research\/\"},\"author\":{\"name\":\"Tiago Freitas\",\"@id\":\"https:\/\/nilg.ai\/#\/schema\/person\/d688d9a2049419d524b4990bf74f705d\"},\"headline\":\"Experiment Management and Reproducible Research\",\"datePublished\":\"2020-07-13T16:06:59+00:00\",\"dateModified\":\"2022-05-25T11:41:37+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/nilg.ai\/202007\/experiment-management-and-reproducible-research\/\"},\"wordCount\":93,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/nilg.ai\/#organization\"},\"keywords\":[\"AI4tech\",\"Machine Learning\",\"Tools\"],\"articleSection\":[\"Webinar\"],\"inLanguage\":\"pt-PT\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/nilg.ai\/202007\/experiment-management-and-reproducible-research\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/nilg.ai\/202007\/experiment-management-and-reproducible-research\/\",\"url\":\"https:\/\/nilg.ai\/202007\/experiment-management-and-reproducible-research\/\",\"name\":\"Experiment Management and Reproducible Research - NILG.AI\",\"isPartOf\":{\"@id\":\"https:\/\/nilg.ai\/#website\"},\"datePublished\":\"2020-07-13T16:06:59+00:00\",\"dateModified\":\"2022-05-25T11:41:37+00:00\",\"description\":\"A tutorial on how to achieve reproducible data pipelines and keeping track of the experiments that lead to production Machine Learning models\",\"inLanguage\":\"pt-PT\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/nilg.ai\/202007\/experiment-management-and-reproducible-research\/\"]}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/nilg.ai\/#website\",\"url\":\"https:\/\/nilg.ai\/\",\"name\":\"NILG.AI\",\"description\":\"Create ever-improving businesses with AI\",\"publisher\":{\"@id\":\"https:\/\/nilg.ai\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/nilg.ai\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"pt-PT\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/nilg.ai\/#organization\",\"name\":\"NILG.AI\",\"url\":\"https:\/\/nilg.ai\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"pt-PT\",\"@id\":\"https:\/\/nilg.ai\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/nilg.ai\/wp-content\/uploads\/2022\/03\/logo.svg\",\"contentUrl\":\"https:\/\/nilg.ai\/wp-content\/uploads\/2022\/03\/logo.svg\",\"caption\":\"NILG.AI\"},\"image\":{\"@id\":\"https:\/\/nilg.ai\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/twitter.com\/nilg_ai\",\"https:\/\/youtube.com\/@nilg_ai\",\"https:\/\/www.linkedin.com\/company\/nilg-ai\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/nilg.ai\/#\/schema\/person\/d688d9a2049419d524b4990bf74f705d\",\"name\":\"Tiago Freitas\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"pt-PT\",\"@id\":\"https:\/\/nilg.ai\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/60b85a85ae710d384bbe4f3f47ddc38ccdfdced12ac8db54e98c0279afc03365?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/60b85a85ae710d384bbe4f3f47ddc38ccdfdced12ac8db54e98c0279afc03365?s=96&d=mm&r=g\",\"caption\":\"Tiago Freitas\"},\"url\":\"https:\/\/nilg.ai\/pt\/author\/tiago\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Experiment Management and Reproducible Research - NILG.AI","description":"A tutorial on how to achieve reproducible data pipelines and keeping track of the experiments that lead to production Machine Learning models","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/nilg.ai\/pt\/202007\/experiment-management-and-reproducible-research\/","og_locale":"pt_PT","og_type":"article","og_title":"Experiment Management and Reproducible Research - NILG.AI","og_description":"A tutorial on how to achieve reproducible data pipelines and keeping track of the experiments that lead to production Machine Learning models","og_url":"https:\/\/nilg.ai\/pt\/202007\/experiment-management-and-reproducible-research\/","og_site_name":"NILG.AI","article_published_time":"2020-07-13T16:06:59+00:00","article_modified_time":"2022-05-25T11:41:37+00:00","og_image":[{"width":2048,"height":1367,"url":"https:\/\/nilg.ai\/wp-content\/uploads\/2022\/05\/pexels-chokniti-khongchum-2280571-e1653476248932.jpg","type":"image\/jpeg"}],"author":"Tiago Freitas","twitter_card":"summary_large_image","twitter_creator":"@nilg_ai","twitter_site":"@nilg_ai","twitter_misc":{"Written by":"Tiago Freitas","Est. reading time":"1 minuto"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/nilg.ai\/202007\/experiment-management-and-reproducible-research\/#article","isPartOf":{"@id":"https:\/\/nilg.ai\/202007\/experiment-management-and-reproducible-research\/"},"author":{"name":"Tiago Freitas","@id":"https:\/\/nilg.ai\/#\/schema\/person\/d688d9a2049419d524b4990bf74f705d"},"headline":"Experiment Management and Reproducible Research","datePublished":"2020-07-13T16:06:59+00:00","dateModified":"2022-05-25T11:41:37+00:00","mainEntityOfPage":{"@id":"https:\/\/nilg.ai\/202007\/experiment-management-and-reproducible-research\/"},"wordCount":93,"commentCount":0,"publisher":{"@id":"https:\/\/nilg.ai\/#organization"},"keywords":["AI4tech","Machine Learning","Tools"],"articleSection":["Webinar"],"inLanguage":"pt-PT","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/nilg.ai\/202007\/experiment-management-and-reproducible-research\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/nilg.ai\/202007\/experiment-management-and-reproducible-research\/","url":"https:\/\/nilg.ai\/202007\/experiment-management-and-reproducible-research\/","name":"Experiment Management and Reproducible Research - NILG.AI","isPartOf":{"@id":"https:\/\/nilg.ai\/#website"},"datePublished":"2020-07-13T16:06:59+00:00","dateModified":"2022-05-25T11:41:37+00:00","description":"A tutorial on how to achieve reproducible data pipelines and keeping track of the experiments that lead to production Machine Learning models","inLanguage":"pt-PT","potentialAction":[{"@type":"ReadAction","target":["https:\/\/nilg.ai\/202007\/experiment-management-and-reproducible-research\/"]}]},{"@type":"WebSite","@id":"https:\/\/nilg.ai\/#website","url":"https:\/\/nilg.ai\/","name":"NILG.AI","description":"Create ever-improving businesses with AI","publisher":{"@id":"https:\/\/nilg.ai\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/nilg.ai\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"pt-PT"},{"@type":"Organization","@id":"https:\/\/nilg.ai\/#organization","name":"NILG.AI","url":"https:\/\/nilg.ai\/","logo":{"@type":"ImageObject","inLanguage":"pt-PT","@id":"https:\/\/nilg.ai\/#\/schema\/logo\/image\/","url":"https:\/\/nilg.ai\/wp-content\/uploads\/2022\/03\/logo.svg","contentUrl":"https:\/\/nilg.ai\/wp-content\/uploads\/2022\/03\/logo.svg","caption":"NILG.AI"},"image":{"@id":"https:\/\/nilg.ai\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/twitter.com\/nilg_ai","https:\/\/youtube.com\/@nilg_ai","https:\/\/www.linkedin.com\/company\/nilg-ai\/"]},{"@type":"Person","@id":"https:\/\/nilg.ai\/#\/schema\/person\/d688d9a2049419d524b4990bf74f705d","name":"Tiago Freitas","image":{"@type":"ImageObject","inLanguage":"pt-PT","@id":"https:\/\/nilg.ai\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/60b85a85ae710d384bbe4f3f47ddc38ccdfdced12ac8db54e98c0279afc03365?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/60b85a85ae710d384bbe4f3f47ddc38ccdfdced12ac8db54e98c0279afc03365?s=96&d=mm&r=g","caption":"Tiago Freitas"},"url":"https:\/\/nilg.ai\/pt\/author\/tiago\/"}]}},"_links":{"self":[{"href":"https:\/\/nilg.ai\/pt\/wp-json\/wp\/v2\/posts\/1019","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nilg.ai\/pt\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nilg.ai\/pt\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nilg.ai\/pt\/wp-json\/wp\/v2\/users\/44"}],"replies":[{"embeddable":true,"href":"https:\/\/nilg.ai\/pt\/wp-json\/wp\/v2\/comments?post=1019"}],"version-history":[{"count":3,"href":"https:\/\/nilg.ai\/pt\/wp-json\/wp\/v2\/posts\/1019\/revisions"}],"predecessor-version":[{"id":1023,"href":"https:\/\/nilg.ai\/pt\/wp-json\/wp\/v2\/posts\/1019\/revisions\/1023"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/nilg.ai\/pt\/wp-json\/wp\/v2\/media\/1020"}],"wp:attachment":[{"href":"https:\/\/nilg.ai\/pt\/wp-json\/wp\/v2\/media?parent=1019"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nilg.ai\/pt\/wp-json\/wp\/v2\/categories?post=1019"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nilg.ai\/pt\/wp-json\/wp\/v2\/tags?post=1019"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}