{"id":1682,"date":"2020-02-19T09:25:54","date_gmt":"2020-02-19T14:25:54","guid":{"rendered":"https:\/\/muutaa.com\/?p=1682"},"modified":"2021-04-13T09:26:00","modified_gmt":"2021-04-13T13:26:00","slug":"ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning","status":"publish","type":"post","link":"https:\/\/muutaa.com\/fr\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\/","title":{"rendered":"AI predictive modeling in clinical settings: Expert- augmented machine learning"},"content":{"rendered":"<div data-draftjs-conductor-fragment=\"{&quot;blocks&quot;:[{&quot;key&quot;:&quot;1h1d9&quot;,&quot;text&quot;:&quot;A reference and snapshot of an article published named 'Expert-augmented machine learning' in PNAS by researchers Efstathios D. Gennatas, Jerome H. Friedman, Lyle H. Ungar, Romain Pirracchio, Eric Eaton, Lara G. Reichmann, Yannet Interian, Jos\u00e9 Marcio Luna, Charles B. Simone II, Andrew Auerbach, Elier Delgado, Mark J. van der Laan, Timothy D. Solberg, and Gilmer Valdes.&quot;,&quot;type&quot;:&quot;unstyled&quot;,&quot;depth&quot;:0,&quot;inlineStyleRanges&quot;:[{&quot;offset&quot;:55,&quot;length&quot;:35,&quot;style&quot;:&quot;UNDERLINE&quot;}],&quot;entityRanges&quot;:[{&quot;offset&quot;:55,&quot;length&quot;:35,&quot;key&quot;:0}],&quot;data&quot;:{}},{&quot;key&quot;:&quot;8phl4&quot;,&quot;text&quot;:&quot;&quot;,&quot;type&quot;:&quot;unstyled&quot;,&quot;depth&quot;:0,&quot;inlineStyleRanges&quot;:[],&quot;entityRanges&quot;:[],&quot;data&quot;:{}},{&quot;key&quot;:&quot;37ldr&quot;,&quot;text&quot;:&quot;At MUUTAA we are digesting these interesting insights and discussion points raised, as we are currently undertaking development efforts to increase the performance of algorithms through human interaction to get to faster, more precise results requiring less data. &quot;,&quot;type&quot;:&quot;unstyled&quot;,&quot;depth&quot;:0,&quot;inlineStyleRanges&quot;:[],&quot;entityRanges&quot;:[],&quot;data&quot;:{}},{&quot;key&quot;:&quot;8lig2&quot;,&quot;text&quot;:&quot;&quot;,&quot;type&quot;:&quot;unstyled&quot;,&quot;depth&quot;:0,&quot;inlineStyleRanges&quot;:[],&quot;entityRanges&quot;:[],&quot;data&quot;:{}},{&quot;key&quot;:&quot;4hm7p&quot;,&quot;text&quot;:&quot;The below is a copy of the significance section and the discussion points raised by the researchers. Refer to the full article on PNAS for additional information. &quot;,&quot;type&quot;:&quot;unstyled&quot;,&quot;depth&quot;:0,&quot;inlineStyleRanges&quot;:[],&quot;entityRanges&quot;:[],&quot;data&quot;:{}},{&quot;key&quot;:&quot;6qffv&quot;,&quot;text&quot;:&quot;&quot;,&quot;type&quot;:&quot;unstyled&quot;,&quot;depth&quot;:0,&quot;inlineStyleRanges&quot;:[],&quot;entityRanges&quot;:[],&quot;data&quot;:{}},{&quot;key&quot;:&quot;am6m7&quot;,&quot;text&quot;:&quot;\\&quot;Machine learning is increasingly used across fields to derive insights from data, which further our understanding of the world and help us anticipate the future. The performance of predictive modeling is dependent on the amount and quality of available data. In practice, we rely on human experts to perform certain tasks and on machine learning for others. However, the optimal learning strategy may involve combining the complementary strengths of humans and machines. We present expert-augmented machine learning, an automated way to automatically extract problem-specific human expert knowledge and integrate it with machine learning to build robust, dependable, and data-efficient predictive models.&quot;,&quot;type&quot;:&quot;unstyled&quot;,&quot;depth&quot;:0,&quot;inlineStyleRanges&quot;:[],&quot;entityRanges&quot;:[],&quot;data&quot;:{}},{&quot;key&quot;:&quot;13a1b&quot;,&quot;text&quot;:&quot;\\nDespite increasing success and growing popularity, ML algorithms can be data inefficient and often generalize poorly to unseen cases. We have introduced EAML, the first methodology to automatically extract problem-specific clinical prior knowledge from experts and incorporate it into ML models. Related previous work had attempted to predict risk based on clinicians\u2019 assessment of individual cases using all available patient characteristics with limited success (20). Here, in contrast, we transformed the raw physiologic data into a set of simple rules and asked clinicians to assess the risk of subpopulations defined by those rules relative to the whole sample. We showed that utilizing this extracted prior knowledge allows: 1) discovery of hidden confounders and limitations of clinicians\u2019 knowledge, 2) better generalization to changes in the underlying feature distribution, 3) improved accuracy in the face of time decay, 4) training with less data, and 5) illustrating the limitations of models chosen using cross-validation estimated from the empirical distribution. We used the MIMIC dataset from the PhysioNet project (9) (10), a large dataset of intensive-care patients, to predict hospital mortality. We showed that EAML allowed the discovery of a hidden confounder (intubation) that can change the interpretation of common variables used to model ICU mortality in multiple available clinical scoring systems\u2014APACHE (11), SAPS II (21), or SOFA (13). Google Scholar lists over 10,000 citations of PhysioNet\u2019s MIMIC dataset as of December 2018, with \u223c1,600 new papers published every year. Conclusions on treatment effect or variable importance using this dataset should be taken with caution, especially since intubation status can be implicitly learned from the data, as shown in this study, even though the variable was not recorded. Moreover, we identified areas where clinicians\u2019 knowledge may need evaluation and possibly further training, such as the case where clinicians overestimated the mortality risk of old age in the absence of other strong risk factors. Further investigation is warranted to establish whether clinicians\u2019 perceived risk is negatively impacting treatment decisions.We have built EAML to incorporate clinicians\u2019 knowledge along with its uncertainty into the final ML model. EAML is not merely a different way of regularizing a machine-learned model but is designed to extract domain knowledge not necessarily present in the training data. We have shown that incorporating this prior knowledge helps the algorithm generalize better to changes in the underlying variable distributions, which, in this case, happened after a rebuilding of the database by the PhysioNet Project. We have also demonstrated that we can train models more robust to accuracy decay with time. Preferentially using those rules where clinicians agree with the empirical data not only produces models that generalize better, but it does so with considerably less data (n = 400 vs. n = 800). This result can be of high value in multiple fields where data are scarce and\/or expensive to collect. We also demonstrated the limitation of selecting models using cross-validated estimation from within the empirical distribution. We showed that there is no advantage in incorporating clinicians\u2019 knowledge if the test set is drawn from the same distribution as the training. However, when the same model was tested in a population whose variables had changed or that were acquired at a later time, including clinicians\u2019 answers improved performance and made the algorithm more data efficient.The MIMIC dataset offered a great opportunity to demonstrate the concept and potential of EAML. A major strength of the dataset is the large number of cases, while one of the main weaknesses is that all cases originated from a single hospital. We were able to show the benefit of EAML in the context of feature coding changes and time decay (MIMIC-III1 and MIMIC-III2). However, proper application of EAML requires independent training, validation, and testing sets, ideally from different institutions. Crucially, an independent validation set is required in order to choose the best subset of rules (hard EAML) or the lambda hyperparameter (soft EAML). If the validation set has the same correlation structure between the covariates and outcome as the training set, cross-validation will choose a lambda of 0 provided there are enough data points. However, if the validation set is different from the training set, then incorporating expert knowledge will help and the tuning will result in lambda greater than 0. This is the same for any ML model training where hyperparameter tuning cannot be effectively performed by cross-validation of the training set if that set is not representative of the whole population of interest, which is most commonly the case in clinical datasets. One of the biggest contributions of this paper is showing the risk of using a validation set that has been randomly subsampled from the empirical distribution and as such contains the same correlations as the training data. Our team is preparing a multiinstitutional EAML study to optimize the algorithm for real-world applications.Finally, this work also has implications on the interpretability and quality assessment of ML algorithms. It is often considered that a trade-off exists between interpretability and accuracy of ML models (22, 23). However, as shown by Friedman and Popescu (24), rule ensembles, and therefore EAML, are on average more accurate than Random Forest and slightly more accurate than Gradient Boosting in a variety of complex problems. EAML builds on RuleFit to address the accuracy\u2013interpretability trade-off in ML and allows one to examine all of the model\u2019s rule ahead of deployment, which is essential to building trust in predictive models.\\&quot;&quot;,&quot;type&quot;:&quot;unstyled&quot;,&quot;depth&quot;:0,&quot;inlineStyleRanges&quot;:[{&quot;offset&quot;:2986,&quot;length&quot;:1,&quot;style&quot;:&quot;ITALIC&quot;},{&quot;offset&quot;:2998,&quot;length&quot;:1,&quot;style&quot;:&quot;ITALIC&quot;}],&quot;entityRanges&quot;:[{&quot;offset&quot;:467,&quot;length&quot;:2,&quot;key&quot;:1},{&quot;offset&quot;:1135,&quot;length&quot;:1,&quot;key&quot;:2},{&quot;offset&quot;:1139,&quot;length&quot;:2,&quot;key&quot;:3},{&quot;offset&quot;:1435,&quot;length&quot;:2,&quot;key&quot;:4},{&quot;offset&quot;:1449,&quot;length&quot;:2,&quot;key&quot;:5},{&quot;offset&quot;:1463,&quot;length&quot;:2,&quot;key&quot;:6},{&quot;offset&quot;:5423,&quot;length&quot;:2,&quot;key&quot;:7},{&quot;offset&quot;:5427,&quot;length&quot;:2,&quot;key&quot;:8},{&quot;offset&quot;:5475,&quot;length&quot;:2,&quot;key&quot;:9}],&quot;data&quot;:{}}],&quot;entityMap&quot;:{&quot;0&quot;:{&quot;type&quot;:&quot;LINK&quot;,&quot;mutability&quot;:&quot;MUTABLE&quot;,&quot;data&quot;:{&quot;url&quot;:&quot;https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117&quot;,&quot;target&quot;:&quot;_top&quot;,&quot;rel&quot;:&quot;noopener noreferrer&quot;}},&quot;1&quot;:{&quot;type&quot;:&quot;LINK&quot;,&quot;mutability&quot;:&quot;MUTABLE&quot;,&quot;data&quot;:{&quot;url&quot;:&quot;https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117#ref-20&quot;,&quot;target&quot;:&quot;_top&quot;,&quot;rel&quot;:&quot;noopener noreferrer&quot;}},&quot;2&quot;:{&quot;type&quot;:&quot;LINK&quot;,&quot;mutability&quot;:&quot;MUTABLE&quot;,&quot;data&quot;:{&quot;url&quot;:&quot;https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117#ref-9&quot;,&quot;target&quot;:&quot;_top&quot;,&quot;rel&quot;:&quot;noopener noreferrer&quot;}},&quot;3&quot;:{&quot;type&quot;:&quot;LINK&quot;,&quot;mutability&quot;:&quot;MUTABLE&quot;,&quot;data&quot;:{&quot;url&quot;:&quot;https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117#ref-10&quot;,&quot;target&quot;:&quot;_top&quot;,&quot;rel&quot;:&quot;noopener noreferrer&quot;}},&quot;4&quot;:{&quot;type&quot;:&quot;LINK&quot;,&quot;mutability&quot;:&quot;MUTABLE&quot;,&quot;data&quot;:{&quot;url&quot;:&quot;https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117#ref-11&quot;,&quot;target&quot;:&quot;_top&quot;,&quot;rel&quot;:&quot;noopener noreferrer&quot;}},&quot;5&quot;:{&quot;type&quot;:&quot;LINK&quot;,&quot;mutability&quot;:&quot;MUTABLE&quot;,&quot;data&quot;:{&quot;url&quot;:&quot;https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117#ref-21&quot;,&quot;target&quot;:&quot;_top&quot;,&quot;rel&quot;:&quot;noopener noreferrer&quot;}},&quot;6&quot;:{&quot;type&quot;:&quot;LINK&quot;,&quot;mutability&quot;:&quot;MUTABLE&quot;,&quot;data&quot;:{&quot;url&quot;:&quot;https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117#ref-13&quot;,&quot;target&quot;:&quot;_top&quot;,&quot;rel&quot;:&quot;noopener noreferrer&quot;}},&quot;7&quot;:{&quot;type&quot;:&quot;LINK&quot;,&quot;mutability&quot;:&quot;MUTABLE&quot;,&quot;data&quot;:{&quot;url&quot;:&quot;https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117#ref-22&quot;,&quot;target&quot;:&quot;_top&quot;,&quot;rel&quot;:&quot;noopener noreferrer&quot;}},&quot;8&quot;:{&quot;type&quot;:&quot;LINK&quot;,&quot;mutability&quot;:&quot;MUTABLE&quot;,&quot;data&quot;:{&quot;url&quot;:&quot;https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117#ref-23&quot;,&quot;target&quot;:&quot;_top&quot;,&quot;rel&quot;:&quot;noopener noreferrer&quot;}},&quot;9&quot;:{&quot;type&quot;:&quot;LINK&quot;,&quot;mutability&quot;:&quot;MUTABLE&quot;,&quot;data&quot;:{&quot;url&quot;:&quot;https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117#ref-24&quot;,&quot;target&quot;:&quot;_top&quot;,&quot;rel&quot;:&quot;noopener noreferrer&quot;}}}}\">\n<div class=\"jwLWP _2hXa7 _30PMG blog-post-text-font blog-post-text-color public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr\" data-block=\"true\" data-editor=\"editor\" data-offset-key=\"91gl7-0-0\">\n<div class=\"public-DraftStyleDefault-block public-DraftStyleDefault-ltr\" data-offset-key=\"91gl7-0-0\"><span data-offset-key=\"91gl7-0-0\">A reference and snapshot of an article published named <\/span><a class=\"_2qJYG blog-link-hashtag-color iPHwd _3Ul6g\" href=\"https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117\" target=\"_top\" rel=\"noopener noreferrer\"><span data-offset-key=\"91gl7-1-0\">&#8216;Expert-augmented machine learning&#8217;<\/span><\/a><span data-offset-key=\"91gl7-2-0\"> in PNAS by researchers Efstathios D. Gennatas, Jerome H. Friedman, Lyle H. Ungar, Romain Pirracchio, Eric Eaton, Lara G. Reichmann, Yannet Interian, Jos\u00e9 Marcio Luna, Charles B. Simone II, Andrew Auerbach, Elier Delgado, Mark J. van der Laan, Timothy D. Solberg, and Gilmer Valdes.<\/span><\/div>\n<\/div>\n<div class=\"jwLWP _2hXa7 _30PMG blog-post-text-font blog-post-text-color public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr\" data-block=\"true\" data-editor=\"editor\" data-offset-key=\"89j1i-0-0\">\n<div class=\"public-DraftStyleDefault-block public-DraftStyleDefault-ltr\" data-offset-key=\"89j1i-0-0\"><span data-offset-key=\"89j1i-0-0\">\u00a0<\/span><\/div>\n<\/div>\n<div class=\"jwLWP _2hXa7 _30PMG blog-post-text-font blog-post-text-color public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr\" data-block=\"true\" data-editor=\"editor\" data-offset-key=\"dl59c-0-0\">\n<div class=\"public-DraftStyleDefault-block public-DraftStyleDefault-ltr\" data-offset-key=\"dl59c-0-0\"><span data-offset-key=\"dl59c-0-0\">At MUUTAA we are digesting these interesting insights and discussion points raised, as we are currently undertaking development efforts to increase the performance of algorithms through human interaction to get to faster, more precise results requiring less data. <\/span><\/div>\n<\/div>\n<div class=\"jwLWP _2hXa7 _30PMG blog-post-text-font blog-post-text-color public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr\" data-block=\"true\" data-editor=\"editor\" data-offset-key=\"1sjqc-0-0\">\n<div class=\"public-DraftStyleDefault-block public-DraftStyleDefault-ltr\" data-offset-key=\"1sjqc-0-0\"><span data-offset-key=\"1sjqc-0-0\">\u00a0<\/span><\/div>\n<\/div>\n<div class=\"jwLWP _2hXa7 _30PMG blog-post-text-font blog-post-text-color public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr\" data-block=\"true\" data-editor=\"editor\" data-offset-key=\"l2h7-0-0\">\n<div class=\"public-DraftStyleDefault-block public-DraftStyleDefault-ltr\" data-offset-key=\"l2h7-0-0\"><span data-offset-key=\"l2h7-0-0\">The below is a copy of the significance section and the discussion points raised by the researchers. Refer to the full article on PNAS for additional information. <\/span><\/div>\n<\/div>\n<div class=\"jwLWP _2hXa7 _30PMG blog-post-text-font blog-post-text-color public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr\" data-block=\"true\" data-editor=\"editor\" data-offset-key=\"6avk3-0-0\">\n<div class=\"public-DraftStyleDefault-block public-DraftStyleDefault-ltr\" data-offset-key=\"6avk3-0-0\"><span data-offset-key=\"6avk3-0-0\">\u00a0<\/span><\/div>\n<\/div>\n<div class=\"jwLWP _2hXa7 _30PMG blog-post-text-font blog-post-text-color public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr\" data-block=\"true\" data-editor=\"editor\" data-offset-key=\"745lv-0-0\">\n<div class=\"public-DraftStyleDefault-block public-DraftStyleDefault-ltr\" data-offset-key=\"745lv-0-0\"><span data-offset-key=\"745lv-0-0\">&#8220;Machine learning is increasingly used across fields to derive insights from data, which further our understanding of the world and help us anticipate the future. The performance of predictive modeling is dependent on the amount and quality of available data. In practice, we rely on human experts to perform certain tasks and on machine learning for others. However, the optimal learning strategy may involve combining the complementary strengths of humans and machines. We present expert-augmented machine learning, an automated way to automatically extract problem-specific human expert knowledge and integrate it with machine learning to build robust, dependable, and data-efficient predictive models.<\/span><\/div>\n<\/div>\n<div class=\"jwLWP _2hXa7 _30PMG blog-post-text-font blog-post-text-color public-DraftStyleDefault-block-depth0 public-DraftStyleDefault-text-ltr\" data-block=\"true\" data-editor=\"editor\" data-offset-key=\"a8c56-0-0\">\n<div class=\"public-DraftStyleDefault-block public-DraftStyleDefault-ltr\" data-offset-key=\"a8c56-0-0\"><span data-offset-key=\"a8c56-0-0\"> Despite increasing success and growing popularity, ML algorithms can be data inefficient and often generalize poorly to unseen cases. We have introduced EAML, the first methodology to automatically extract problem-specific clinical prior knowledge from experts and incorporate it into ML models. Related previous work had attempted to predict risk based on clinicians\u2019 assessment of individual cases using all available patient characteristics with limited success (<\/span><a class=\"_2qJYG blog-link-hashtag-color iPHwd _3Ul6g\" href=\"https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117#ref-20\" target=\"_top\" rel=\"noopener noreferrer\"><span data-offset-key=\"a8c56-1-0\">20<\/span><\/a><span data-offset-key=\"a8c56-2-0\">). Here, in contrast, we transformed the raw physiologic data into a set of simple rules and asked clinicians to assess the risk of subpopulations defined by those rules relative to the whole sample. We showed that utilizing this extracted prior knowledge allows: 1) discovery of hidden confounders and limitations of clinicians\u2019 knowledge, 2) better generalization to changes in the underlying feature distribution, 3) improved accuracy in the face of time decay, 4) training with less data, and 5) illustrating the limitations of models chosen using cross-validation estimated from the empirical distribution. We used the MIMIC dataset from the PhysioNet project (<\/span><a class=\"_2qJYG blog-link-hashtag-color iPHwd _3Ul6g\" href=\"https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117#ref-9\" target=\"_top\" rel=\"noopener noreferrer\"><span data-offset-key=\"a8c56-3-0\">9<\/span><\/a><span data-offset-key=\"a8c56-4-0\">) (<\/span><a class=\"_2qJYG blog-link-hashtag-color iPHwd _3Ul6g\" href=\"https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117#ref-10\" target=\"_top\" rel=\"noopener noreferrer\"><span data-offset-key=\"a8c56-5-0\">10<\/span><\/a><span data-offset-key=\"a8c56-6-0\">), a large dataset of intensive-care patients, to predict hospital mortality. We showed that EAML allowed the discovery of a hidden confounder (intubation) that can change the interpretation of common variables used to model ICU mortality in multiple available clinical scoring systems\u2014APACHE (<\/span><a class=\"_2qJYG blog-link-hashtag-color iPHwd _3Ul6g\" href=\"https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117#ref-11\" target=\"_top\" rel=\"noopener noreferrer\"><span data-offset-key=\"a8c56-7-0\">11<\/span><\/a><span data-offset-key=\"a8c56-8-0\">), SAPS II (<\/span><a class=\"_2qJYG blog-link-hashtag-color iPHwd _3Ul6g\" href=\"https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117#ref-21\" target=\"_top\" rel=\"noopener noreferrer\"><span data-offset-key=\"a8c56-9-0\">21<\/span><\/a><span data-offset-key=\"a8c56-10-0\">), or SOFA (<\/span><a class=\"_2qJYG blog-link-hashtag-color iPHwd _3Ul6g\" href=\"https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117#ref-13\" target=\"_top\" rel=\"noopener noreferrer\"><span data-offset-key=\"a8c56-11-0\">13<\/span><\/a><span data-offset-key=\"a8c56-12-0\">). Google Scholar lists over 10,000 citations of PhysioNet\u2019s MIMIC dataset as of December 2018, with \u223c1,600 new papers published every year. Conclusions on treatment effect or variable importance using this dataset should be taken with caution, especially since intubation status can be implicitly learned from the data, as shown in this study, even though the variable was not recorded. Moreover, we identified areas where clinicians\u2019 knowledge may need evaluation and possibly further training, such as the case where clinicians overestimated the mortality risk of old age in the absence of other strong risk factors. Further investigation is warranted to establish whether clinicians\u2019 perceived risk is negatively impacting treatment decisions.We have built EAML to incorporate clinicians\u2019 knowledge along with its uncertainty into the final ML model. EAML is not merely a different way of regularizing a machine-learned model but is designed to extract domain knowledge not necessarily present in the training data. We have shown that incorporating this prior knowledge helps the algorithm generalize better to changes in the underlying variable distributions, which, in this case, happened after a rebuilding of the database by the PhysioNet Project. We have also demonstrated that we can train models more robust to accuracy decay with time. Preferentially using those rules where clinicians agree with the empirical data not only produces models that generalize better, but it does so with considerably less data (<\/span><span data-offset-key=\"a8c56-12-1\">n<\/span><span data-offset-key=\"a8c56-12-2\"> = 400 vs. <\/span><span data-offset-key=\"a8c56-12-3\">n<\/span><span data-offset-key=\"a8c56-12-4\"> = 800). This result can be of high value in multiple fields where data are scarce and\/or expensive to collect. We also demonstrated the limitation of selecting models using cross-validated estimation from within the empirical distribution. We showed that there is no advantage in incorporating clinicians\u2019 knowledge if the test set is drawn from the same distribution as the training. However, when the same model was tested in a population whose variables had changed or that were acquired at a later time, including clinicians\u2019 answers improved performance and made the algorithm more data efficient.The MIMIC dataset offered a great opportunity to demonstrate the concept and potential of EAML. A major strength of the dataset is the large number of cases, while one of the main weaknesses is that all cases originated from a single hospital. We were able to show the benefit of EAML in the context of feature coding changes and time decay (MIMIC-III1 and MIMIC-III2). However, proper application of EAML requires independent training, validation, and testing sets, ideally from different institutions. Crucially, an independent validation set is required in order to choose the best subset of rules (hard EAML) or the lambda hyperparameter (soft EAML). If the validation set has the same correlation structure between the covariates and outcome as the training set, cross-validation will choose a lambda of 0 provided there are enough data points. However, if the validation set is different from the training set, then incorporating expert knowledge will help and the tuning will result in lambda greater than 0. This is the same for any ML model training where hyperparameter tuning cannot be effectively performed by cross-validation of the training set if that set is not representative of the whole population of interest, which is most commonly the case in clinical datasets. One of the biggest contributions of this paper is showing the risk of using a validation set that has been randomly subsampled from the empirical distribution and as such contains the same correlations as the training data. Our team is preparing a multiinstitutional EAML study to optimize the algorithm for real-world applications.Finally, this work also has implications on the interpretability and quality assessment of ML algorithms. It is often considered that a trade-off exists between interpretability and accuracy of ML models (<\/span><a class=\"_2qJYG blog-link-hashtag-color iPHwd _3Ul6g\" href=\"https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117#ref-22\" target=\"_top\" rel=\"noopener noreferrer\"><span data-offset-key=\"a8c56-13-0\">22<\/span><\/a><span data-offset-key=\"a8c56-14-0\">, <\/span><a class=\"_2qJYG blog-link-hashtag-color iPHwd _3Ul6g\" href=\"https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117#ref-23\" target=\"_top\" rel=\"noopener noreferrer\"><span data-offset-key=\"a8c56-15-0\">23<\/span><\/a><span data-offset-key=\"a8c56-16-0\">). However, as shown by Friedman and Popescu (<\/span><a class=\"_2qJYG blog-link-hashtag-color iPHwd _3Ul6g\" href=\"https:\/\/www.pnas.org\/content\/early\/2020\/02\/14\/1906831117#ref-24\" target=\"_top\" rel=\"noopener noreferrer\"><span data-offset-key=\"a8c56-17-0\">24<\/span><\/a><span data-offset-key=\"a8c56-18-0\">), rule ensembles, and therefore EAML, are on average more accurate than Random Forest and slightly more accurate than Gradient Boosting in a variety of complex problems. EAML builds on RuleFit to address the accuracy\u2013interpretability trade-off in ML and allows one to examine all of the model\u2019s rule ahead of deployment, which is essential to building trust in predictive models.&#8221;<\/span><\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>A reference and snapshot of an article published named &#8216;Expert-augmented machine learning&#8217; in PNAS by researchers Efstathios D. Gennatas, Jerome H. Friedman, Lyle H. Ungar, Romain Pirracchio, Eric Eaton, Lara G. Reichmann, Yannet Interian, Jos\u00e9 Marcio Luna, Charles B. Simone II, Andrew Auerbach, Elier Delgado, Mark J. van der Laan, Timothy D. Solberg, and Gilmer [&hellip;]<\/p>\n","protected":false},"author":203168979,"featured_media":1683,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","site-sidebar-layout":"default","site-content-layout":"default","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"disabled","footer-sml-layout":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_wpcom_ai_launchpad_first_post":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"{title}\n\n{excerpt}\n\n{url}","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_wpas_customize_per_network":false,"jetpack_post_was_ever_published":false},"categories":[1372],"tags":[],"class_list":["post-1682","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>AI predictive modeling in clinical settings: Expert- augmented machine learning - MUUTAA<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/muutaa.com\/fr\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\/\" \/>\n<meta property=\"og:locale\" content=\"fr_CA\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"AI predictive modeling in clinical settings: Expert- augmented machine learning - MUUTAA\" \/>\n<meta property=\"og:description\" content=\"A reference and snapshot of an article published named &#8216;Expert-augmented machine learning&#8217; in PNAS by researchers Efstathios D. Gennatas, Jerome H. Friedman, Lyle H. Ungar, Romain Pirracchio, Eric Eaton, Lara G. Reichmann, Yannet Interian, Jos\u00e9 Marcio Luna, Charles B. Simone II, Andrew Auerbach, Elier Delgado, Mark J. van der Laan, Timothy D. Solberg, and Gilmer [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/muutaa.com\/fr\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\/\" \/>\n<meta property=\"og:site_name\" content=\"MUUTAA\" \/>\n<meta property=\"article:published_time\" content=\"2020-02-19T14:25:54+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-04-13T13:26:00+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/i2.wp.com\/muutaa.com\/wp-content\/uploads\/2021\/04\/predictiveModel.png?fit=560%2C347&ssl=1\" \/>\n\t<meta property=\"og:image:width\" content=\"560\" \/>\n\t<meta property=\"og:image:height\" content=\"347\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"muutaa\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u00c9crit par\" \/>\n\t<meta name=\"twitter:data1\" content=\"muutaa\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimation du temps de lecture\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/muutaa.com\\\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/muutaa.com\\\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\\\/\"},\"author\":{\"name\":\"muutaa\",\"@id\":\"https:\\\/\\\/muutaa.com\\\/#\\\/schema\\\/person\\\/0ad84933a30e32d1556461df9ad22c5e\"},\"headline\":\"AI predictive modeling in clinical settings: Expert- augmented machine learning\",\"datePublished\":\"2020-02-19T14:25:54+00:00\",\"dateModified\":\"2021-04-13T13:26:00+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/muutaa.com\\\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\\\/\"},\"wordCount\":1113,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/muutaa.com\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/muutaa.com\\\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/i0.wp.com\\\/muutaa.com\\\/wp-content\\\/uploads\\\/2021\\\/04\\\/predictiveModel.png?fit=560%2C347&ssl=1\",\"articleSection\":[\"blog\"],\"inLanguage\":\"fr-CA\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/muutaa.com\\\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/muutaa.com\\\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\\\/\",\"url\":\"https:\\\/\\\/muutaa.com\\\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\\\/\",\"name\":\"AI predictive modeling in clinical settings: Expert- augmented machine learning - MUUTAA\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/muutaa.com\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/muutaa.com\\\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/muutaa.com\\\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/i0.wp.com\\\/muutaa.com\\\/wp-content\\\/uploads\\\/2021\\\/04\\\/predictiveModel.png?fit=560%2C347&ssl=1\",\"datePublished\":\"2020-02-19T14:25:54+00:00\",\"dateModified\":\"2021-04-13T13:26:00+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/muutaa.com\\\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\\\/#breadcrumb\"},\"inLanguage\":\"fr-CA\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/muutaa.com\\\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"fr-CA\",\"@id\":\"https:\\\/\\\/muutaa.com\\\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\\\/#primaryimage\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/muutaa.com\\\/wp-content\\\/uploads\\\/2021\\\/04\\\/predictiveModel.png?fit=560%2C347&ssl=1\",\"contentUrl\":\"https:\\\/\\\/i0.wp.com\\\/muutaa.com\\\/wp-content\\\/uploads\\\/2021\\\/04\\\/predictiveModel.png?fit=560%2C347&ssl=1\",\"width\":560,\"height\":347},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/muutaa.com\\\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/muutaa.com\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"AI predictive modeling in clinical settings: Expert- augmented machine learning\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/muutaa.com\\\/#website\",\"url\":\"https:\\\/\\\/muutaa.com\\\/\",\"name\":\"MUUTAA\",\"description\":\"Decision Support for Healthcare Supply Chain\",\"publisher\":{\"@id\":\"https:\\\/\\\/muutaa.com\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/muutaa.com\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"fr-CA\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/muutaa.com\\\/#organization\",\"name\":\"MUUTAA\",\"url\":\"https:\\\/\\\/muutaa.com\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"fr-CA\",\"@id\":\"https:\\\/\\\/muutaa.com\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/i2.wp.com\\\/muutaa.com\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/Final-Logo-1.png?fit=2033%2C614&ssl=1\",\"contentUrl\":\"https:\\\/\\\/i2.wp.com\\\/muutaa.com\\\/wp-content\\\/uploads\\\/2021\\\/03\\\/Final-Logo-1.png?fit=2033%2C614&ssl=1\",\"width\":2033,\"height\":614,\"caption\":\"MUUTAA\"},\"image\":{\"@id\":\"https:\\\/\\\/muutaa.com\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.linkedin.com\\\/company\\\/muutaa\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/muutaa.com\\\/#\\\/schema\\\/person\\\/0ad84933a30e32d1556461df9ad22c5e\",\"name\":\"muutaa\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"fr-CA\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/efa337b4782c364cd3657acc73030a21ec128eb1c5e919973b502c99b4258818?s=96&d=identicon&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/efa337b4782c364cd3657acc73030a21ec128eb1c5e919973b502c99b4258818?s=96&d=identicon&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/efa337b4782c364cd3657acc73030a21ec128eb1c5e919973b502c99b4258818?s=96&d=identicon&r=g\",\"caption\":\"muutaa\"},\"sameAs\":[\"http:\\\/\\\/muutaa.wordpress.com\"],\"url\":\"https:\\\/\\\/muutaa.com\\\/fr\\\/author\\\/muutaa\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"AI predictive modeling in clinical settings: Expert- augmented machine learning - MUUTAA","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/muutaa.com\/fr\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\/","og_locale":"fr_CA","og_type":"article","og_title":"AI predictive modeling in clinical settings: Expert- augmented machine learning - MUUTAA","og_description":"A reference and snapshot of an article published named &#8216;Expert-augmented machine learning&#8217; in PNAS by researchers Efstathios D. Gennatas, Jerome H. Friedman, Lyle H. Ungar, Romain Pirracchio, Eric Eaton, Lara G. Reichmann, Yannet Interian, Jos\u00e9 Marcio Luna, Charles B. Simone II, Andrew Auerbach, Elier Delgado, Mark J. van der Laan, Timothy D. Solberg, and Gilmer [&hellip;]","og_url":"https:\/\/muutaa.com\/fr\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\/","og_site_name":"MUUTAA","article_published_time":"2020-02-19T14:25:54+00:00","article_modified_time":"2021-04-13T13:26:00+00:00","og_image":[{"width":560,"height":347,"url":"https:\/\/i2.wp.com\/muutaa.com\/wp-content\/uploads\/2021\/04\/predictiveModel.png?fit=560%2C347&ssl=1","type":"image\/png"}],"author":"muutaa","twitter_card":"summary_large_image","twitter_misc":{"\u00c9crit par":"muutaa","Estimation du temps de lecture":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/muutaa.com\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\/#article","isPartOf":{"@id":"https:\/\/muutaa.com\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\/"},"author":{"name":"muutaa","@id":"https:\/\/muutaa.com\/#\/schema\/person\/0ad84933a30e32d1556461df9ad22c5e"},"headline":"AI predictive modeling in clinical settings: Expert- augmented machine learning","datePublished":"2020-02-19T14:25:54+00:00","dateModified":"2021-04-13T13:26:00+00:00","mainEntityOfPage":{"@id":"https:\/\/muutaa.com\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\/"},"wordCount":1113,"commentCount":0,"publisher":{"@id":"https:\/\/muutaa.com\/#organization"},"image":{"@id":"https:\/\/muutaa.com\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/muutaa.com\/wp-content\/uploads\/2021\/04\/predictiveModel.png?fit=560%2C347&ssl=1","articleSection":["blog"],"inLanguage":"fr-CA","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/muutaa.com\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/muutaa.com\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\/","url":"https:\/\/muutaa.com\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\/","name":"AI predictive modeling in clinical settings: Expert- augmented machine learning - MUUTAA","isPartOf":{"@id":"https:\/\/muutaa.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/muutaa.com\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\/#primaryimage"},"image":{"@id":"https:\/\/muutaa.com\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/muutaa.com\/wp-content\/uploads\/2021\/04\/predictiveModel.png?fit=560%2C347&ssl=1","datePublished":"2020-02-19T14:25:54+00:00","dateModified":"2021-04-13T13:26:00+00:00","breadcrumb":{"@id":"https:\/\/muutaa.com\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\/#breadcrumb"},"inLanguage":"fr-CA","potentialAction":[{"@type":"ReadAction","target":["https:\/\/muutaa.com\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\/"]}]},{"@type":"ImageObject","inLanguage":"fr-CA","@id":"https:\/\/muutaa.com\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\/#primaryimage","url":"https:\/\/i0.wp.com\/muutaa.com\/wp-content\/uploads\/2021\/04\/predictiveModel.png?fit=560%2C347&ssl=1","contentUrl":"https:\/\/i0.wp.com\/muutaa.com\/wp-content\/uploads\/2021\/04\/predictiveModel.png?fit=560%2C347&ssl=1","width":560,"height":347},{"@type":"BreadcrumbList","@id":"https:\/\/muutaa.com\/ai-predictive-modeling-in-clinical-settings-expert-augmented-machine-learning\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/muutaa.com\/"},{"@type":"ListItem","position":2,"name":"AI predictive modeling in clinical settings: Expert- augmented machine learning"}]},{"@type":"WebSite","@id":"https:\/\/muutaa.com\/#website","url":"https:\/\/muutaa.com\/","name":"MUUTAA","description":"Decision Support for Healthcare Supply Chain","publisher":{"@id":"https:\/\/muutaa.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/muutaa.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"fr-CA"},{"@type":"Organization","@id":"https:\/\/muutaa.com\/#organization","name":"MUUTAA","url":"https:\/\/muutaa.com\/","logo":{"@type":"ImageObject","inLanguage":"fr-CA","@id":"https:\/\/muutaa.com\/#\/schema\/logo\/image\/","url":"https:\/\/i2.wp.com\/muutaa.com\/wp-content\/uploads\/2021\/03\/Final-Logo-1.png?fit=2033%2C614&ssl=1","contentUrl":"https:\/\/i2.wp.com\/muutaa.com\/wp-content\/uploads\/2021\/03\/Final-Logo-1.png?fit=2033%2C614&ssl=1","width":2033,"height":614,"caption":"MUUTAA"},"image":{"@id":"https:\/\/muutaa.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.linkedin.com\/company\/muutaa"]},{"@type":"Person","@id":"https:\/\/muutaa.com\/#\/schema\/person\/0ad84933a30e32d1556461df9ad22c5e","name":"muutaa","image":{"@type":"ImageObject","inLanguage":"fr-CA","@id":"https:\/\/secure.gravatar.com\/avatar\/efa337b4782c364cd3657acc73030a21ec128eb1c5e919973b502c99b4258818?s=96&d=identicon&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/efa337b4782c364cd3657acc73030a21ec128eb1c5e919973b502c99b4258818?s=96&d=identicon&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/efa337b4782c364cd3657acc73030a21ec128eb1c5e919973b502c99b4258818?s=96&d=identicon&r=g","caption":"muutaa"},"sameAs":["http:\/\/muutaa.wordpress.com"],"url":"https:\/\/muutaa.com\/fr\/author\/muutaa\/"}]}},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/muutaa.com\/wp-content\/uploads\/2021\/04\/predictiveModel.png?fit=560%2C347&ssl=1","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pcTFyz-r8","_links":{"self":[{"href":"https:\/\/muutaa.com\/fr\/wp-json\/wp\/v2\/posts\/1682","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/muutaa.com\/fr\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/muutaa.com\/fr\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/muutaa.com\/fr\/wp-json\/wp\/v2\/users\/203168979"}],"replies":[{"embeddable":true,"href":"https:\/\/muutaa.com\/fr\/wp-json\/wp\/v2\/comments?post=1682"}],"version-history":[{"count":1,"href":"https:\/\/muutaa.com\/fr\/wp-json\/wp\/v2\/posts\/1682\/revisions"}],"predecessor-version":[{"id":1684,"href":"https:\/\/muutaa.com\/fr\/wp-json\/wp\/v2\/posts\/1682\/revisions\/1684"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/muutaa.com\/fr\/wp-json\/wp\/v2\/media\/1683"}],"wp:attachment":[{"href":"https:\/\/muutaa.com\/fr\/wp-json\/wp\/v2\/media?parent=1682"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/muutaa.com\/fr\/wp-json\/wp\/v2\/categories?post=1682"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/muutaa.com\/fr\/wp-json\/wp\/v2\/tags?post=1682"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}