multilevel modeling using r

Thus, the random-intercept model appears to be the best fit. Call for Papers/Proposals/Nominations (4), © 2021 American Psychological Association. In the presented example, we find an estimated intercept of 622.11 and a (nonsignificant) slope of -24.89, and these numbers apply to all cases. Multilevel Modeling Tutorial 3 The Department of Statistics and Data Sciences, The University of Texas at Austin Introduction This document serves to compare the procedures and output for two-level hierarchical linear models from six different statistical software programs: SAS, Stata, HLM, R, SPSS, and Mplus. A psychologist recruits three bilingual participants (Adam, Betty and Carol) who are equally fluent in English and Spanish. A different intercept is estimated for each participant (dotted lines), assuming the same slope for all participants. In the left-hand portion of the figure below, we can see a graphical representation of this random-intercepts model. To do so, we create a base model that is identical to the original model minus the effect of interest. Multilevel data occur when observations are nested within groups, for example, when students are nested within schools in a district. SO, Let’s try running this as a multilevel, random effects model: Using a multi-level model allows us to separate the within-group effects from the between-group effects, whereas regular regression blends them together into a single coefficient. We estimate the variability for each random-effect and use that to control for the variance when estimating the significance our fixed-effects. By controlling for the variability in the time to speak due to each subject’s different baselines (i.e., random-intercepts), the remaining variability indicated a significant fixed-effect of language across individuals. Journal of Personality, 79(1), 2-50. But what happens when we separate the schools by rich or poor status? In other words, this formula indicates that there are multiple responses per subject, and that each response will depend on that subject’s baseline. We looked at 6 schools (3 rich and 3 poor) with 40 students in each rich school and 160 students in each poor school, and we measured them on Happiness, number of Friends, and GPA. These are considered fixed because they take on a predetermined set of values. West, S.G., Ryu, E., Kwok, O.M., & Cham, H. (2011). Another way to put this, the regular (lm) regression indicates that the more friends a student has, the happier they are, but looking closer this is not the case in all schools (and is, in fact, the opposite in some). We examine the intra-class correlation (ICC) to determine if multi-level modeling is the correct choice for our analysis. Our model would then include a (B0│Subject)+(B0 |Item) for random intercepts by-subject and by-item. The goal of multi-level modeling is to draw a conclusion about the general sample that you have while controlling for differences you are not trying to explain (in this example, rich vs. poor). If we knew this information, we could put SES as another Level 2 predictor. Multilevel models (MLMs, also known as linear mixed models, hierarchical linear models or mixed-effect models) have become increasingly popular in psychology for analyzing data with repeated measurements or data organized in nested levels (e.g., students in classrooms). Our model will control for this within-cluster variance and test the fixed-effect estimates against the remaining between-cluster variance. Thus, we will compare whether the addition of the random slopes to the random-intercepts model results in an improved fit. This will be important. Now we have a positive, significant effect of Friends and no effect of GPA. The Quantitative Methods for Psychology, 10(1): 13- 28. Returning to our example, we are going to add a random-effect for “Subject” that characterizes each participant’s idiosyncratic variation from the fixed-effect estimates. In a multilevel (random effects) model, the effects of both types of variable can be estimated. Human Communication Research, 32(4), 385-410. Remember! Our ICC is greater than 0, meaning we were correct to think of this as a multi-level problem. He is a doctoral candidate in the cognitive psychology program at Stony Brook University. Now that we specified our random effects, we can test whether the fixed effect of language is significant. (2010). Here is a snippet of what the output would look like in R: Random Intercepts: Timei = B0 + B1 * Languagei + (B0|Subject) + ei, Random Slopes: Timei = B0 + B1 * Languagei + (B0 + B1 * Language|Subject) + ei. Now let’s add in Student ID’s and the Schools as variables. Cohen, J., Cohen, P., West, S.G., & Aiken, L.S. Base Model: Timei = B0 + (B0|Subject) +ei, Random Intercepts: Timei = B0 + B1* Languagei + (B0|Subject) +ei. This is a mixed model add-on package to calculate p-values based on a certain set of assumptions. Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli. Cross-referencing the documentation When reading this manual, you will find references to other Stata manuals. That is, we might also wish to include random slopes in addition to the random intercepts. Introductions to using MLMs are available for R (Winters, 2013), SPSS (Hayes, 2006), and SAS (Peugh, 2010). Other examples need to be modified to not … (2008) and West et al. Therefore, students who received initial instruction in SEM with lavaan should have little di culty using other (commercial) SEM programs in the future. New York: Routledge. SAS/SPSS uses Satterthwaite approximations. For instance, individuals may be nested within workgroups, or repeated measures may be nested within individuals. Collapsing across school types in this case was not ideal because different things were happening within each school type, compromising the generalizability of the findings. Instead of one general random-effect that captures how each observation deviates from the predicted fixed-effects, there will be multiple random-effects that capture how observations deviate within a cluster, and how each cluster deviates from the overall group. Although mathematically sophisticated, MLMs are easy to use once familiar with some basic concepts. P-values are hotly disputed in Mixed Models. Data analysis using regression and multilevel/hierarchical models. (2013). Huta, V. (2014). Each ddf is a different method of attaining p-values, so you can choose which to run. If you were trying to generalize your findings or use them to argue for/show a need for an intervention, your results would be misleading and could cause problems. Chapter 6: Growth modeling, survival analysis, and N=1 time series analysis view examples. This is a special type of data where there is only one measurement per student (i.e., there is no within subject variable), and students are nested in schools so we must control for the effects of schools. We find an intercept of 522.11 and a slope of -24.89, suggesting that participants took about 522 ms to begin speaking when responding in English, but began speaking 25 ms faster when responding in Spanish. The expression (B0 + B1 * Language|Subject) tells the model to estimate different intercepts and slopes for each individual, as shown in the right-hand portion of the figure below. Mixed-effects modeling with crossed random effects for subjects and items. We can easily update our model by adding a term for the regression slope: Timei = B0 + B1 * Languagei + (B0 + B1 * Language|Subject) + ei. In our regular (lm) regression, Friends had a significant effect, b = 1.40 (p < .001). It is read by psychologists, students, academic administrators, journalists and policymakers in Congress and federal science agencies. (2011) provide excellent in-depth discussions. The fixed effects include the intercept (B0) and the slope (B1) for the dichotomous independent variable "Language." She is interested in whether a matching or mismatching response language will affect the time it takes to begin the utterance. Making a number of assumptions, these outputs will mirror the results of a traditional ANOVA fairly closely. Notice how the correlation between Friends and Happiness has changed when you collapse across all rich and poor schools. Although I am telling you that some schools are rich and some are poor, you may not know this information when you come in and try to do the analysis. Violating the assumption of independence does not bias coefficient estimates, but does tend to underestimate standard errors. And this estimate is based on the most optimistic assumptions about the R factor, vaccine efficacy and vaccine uptake. This flexibility makes MLM an attractive option for researchers. Let’s test if these formulas work with an example from each set of schools (one rich and one poor). (2008). Tutorials in Quantitative Methods for Psychology, 8(1): 52-69. We wondered if happiness could be predicted by number of friends and/or GPA. Great, our correlation between Friends and Happiness is -.76 and we have a correlation between GPA and Happiness of -.05. She shows them three color words printed in a nonmatching color (e.g., the word “red” printed in green) and asks the participants to name the color of the ink instead of the word presented. There are also Kenward-Roger approximations (see Westfall, Kenny, & Judd, 2014). Baayen et al. We may expect each participant has some individual differences that affects the intercept but not the slope. For example, since we only used three color words out of the entire population of possible color words, we might use the items as another random-effect to control for “by-item” variability. The results indicate that a student’s GPA does not have an effect on their Happiness when controlling for Level 2 fluctuations in Happiness. Also notice that the correlation between GPA and Happiness is .05, which is the same as for the rich school - the only one that had showed a correlation. When to use hierarchical linear modeling. Uh-oh. Thus, we will create a model with only an intercept as a fixed effect, and compare it to our random-intercepts model. If you want to report main effects and interactions instead of slopes, you might convert your mixed model into an ANOVA. The issue with this approach is that we repeatedly sampled from the same individual, and this violates the OLS assumption that observations are independent from each other. What could bring R below 1, short of a full-scale lockdown? Our Level 2 refers to the groups in which the level 1 units are nested; in this case Level 2 is the schools. Most of the examples found in the Mplus section can be run using the demo version. Observant readers will recognize that this is the same numbers we obtained OLS regression. Multilevel data occur when observations are nested within groups, for example, when students are nested within schools in a district. Micah Mumper is the cognitive science representative on the APA Student Science Council. These are considered “random” because we assume there is some idiosyncratic variation in each case that cannot be accounted for by the fixed effects. install.packages(“effects”) # Helps you visualize interactions, install.packages(“corrplot”) # Helps you to visualize correlations, install.packages(“stargazer”) # Makes tables, install.packages(“car”) # Helps with Diagnostics, install.packages(“MASS”) # For stepwise regression, install.packages(“ggplot2”) # To create plots. (2008) and West et al. Multilevel modeling: Current and future applications in personality research. Also, I am only using six schools to demonstrate, but in a more “normal” case of nested design there may be way too many schools to treat SES as a fixed factor in the design. Notice, we have a negative, significant effect for Friends but no effect for GPA. The psychologist carefully records the time for the participant to begin speaking in milliseconds (ms). A practical guide to multilevel modeling. Note, we would not use a MLM for this exact data in practice as estimates of random effects with only three groups are very unreliable. Modeling Longitudinal Data by Robert Weiss; Logistic Regression and Related Methods Applied Logistic Regression (Second Edition) by David Hosmer and Stanley Lemeshow; An Introduction to Categorical Data Analysis by Alan Agresti; Multilevel Modeling Introduction to Multilevel Modeling by … It is a positive correlation as well, but only poor schools (those with more students) had a positive correlation. In the present example, our MLM gave us a more accurate interpretation - that no main effect of Friends existed for all schools generally. Baayen, R.H., Davidson, D.J., & Bates, D.M. Thus, we can model our data at the observation level (micro-level) and at the cluster level (macro-level). Our Level 1 will be the lowest unit of observation - those nested in groups. Let’s see what our correlations look like when lumping all the rich and poor schools together. For a comprehensive look at how to run diagnostics, please see Chapter 12 and Chapter 18. Happy modeling. If we knew this information, we would put SES as another Level 2 predictor. Applied multiple regression/correlation analysis for the behavioral sciences (3rd Ed.). (2006). Not properly controlling for these differences, which you may often not know are there, will increase your chance of Type I error. First, let’s plot with Friends as our IV. Rich schools have r = -.76 and poor schools have r = .77. (2011) provide excellent in-depth discussions. Putting all schools together in a regular regression, we have a positive, significant effect of Friends and no effect of GPA. Journal of School Psychology, 48(1), 85-112. An introduction to hierarchical linear modeling. Thus, while this 25 ms difference was not significant using OLS regression, it is significant in a MLM. arXiv:1308.5499. Snijders, T.A.B., & Bosker, R.J. (2012). Our regression equation now looks like: Timei = B0 + B1 * Languagei + (B0|Subject) + ei. London: Sage. The general error term (ei) captures the error in prediction for case i. Linear models and linear mixed effects models in R with linguistic applications. It gave us a positive effect (Friends) when only poor schools had a positive effect. In this way, we resolve the issue of having multiple responses per subject. There was no correlation between GPA and Happiness. Schools 1-3 are richand schools 4-6 are poor. Mumper, M. (2017, January). We may also have reason to believe that the relationship between time to speak and the response language might differ for each individual, in addition to the baseline differences. Missing data in R and Bugs In R, missing values are indicated by NA’s. This model allows the variable Friends to vary between schools. In ordinary least squares (OLS) regression, we would model this data using the following formula: For each case, the “Time” score can be separated into fixed and random effects. This is accomplished using an LR test too. Advancing psychology to benefit society and improve lives. For example, to see some of the data from five respondents in the data file for the Social Indicators Survey (arbitrarily picking rows 91–95), we type cbind (sex, race, educ_r, r_age, earnings, police)[91:95,] R code and get sex race educ_r r_age earnings police R output Either are acceptable. The result is nonsignificant, suggesting that addition of the random slopes did not improve the fit of the model. Retrieved from http://arxiv.org/pdf/1308.5499.pdf (PDF, 2.69MB). Our simple story - We looked at 6 schools (3 rich and 3 poor) with 40 students in each rich school and 160 students in each poor school, and we measured them on Happiness, number of Friends, and GPA. (2008). The purpose of this primer was to demonstrate some of the fundamentals of MLMs and to encourage researchers to consider them further. Great, our correlation between Friends and Happiness is .77, which is higher AND in the opposite direction than in rich schools. With the fixed-effects and random-effects specified, we can interpret the fixed-effects similarly to an OLS regression. (2013). Gelman, A., & Hill, J. 4 lavaan: An R Package for Structural Equation Modeling Finally, the mimic option makes a smooth transition possible from lavaan to one of the major commercial programs, and back. Beyond Multiple Linear Regression: Applied Generalized Linear Models and Multilevel Models in R (R Core Team 2020) is intended to be accessible to undergraduate students who have successfully completed a regression course through, for example, a textbook like Stat2 (Cannon et al. From this simple example, we can easily scale up our model. install.packages(“texreg”) # Allows you to save tables. Westfall, J., Kenny, D. A., & Judd, C. M. (2014). Now that we have our equations, we’re going to create the data frame for each school. . In this case, Level 1 is the students. Using a fixed effects model, inferences cannot be made beyond the groups in the sample. We can also have nested random effects. Hayes, A.F. These include: cont3, cont6, cont10, mix1-mix6, mix8-mix11, mix14, and penn1-penn7. (2006). That’s a huge reduction, but 1.2 is still bigger than 1: it means that the epidemic will double every 3 weeks. While this accounts for the individual by-subject variation, we still have our general error term ei to capture the random deviations within the same subject. Winter, B. NOTE: Although I am telling you that some schools are rich and some are poor, you may not know this information when you come in and try to do the analysis. The ICC measures the degree of clustering in our data and answers the question, “How much does my Level 2 predict the total variance of my study?” If your ICC is greater than 0, you have a multi-level study. For GPA and Happiness, only the rich school had a correlation. Now that we know they work, let’s put all schools together into one dataset. However, in our mixed (lmer) regression, Friends had a larger (2.16), but non-significant effect. Because the effect of Friends was different in different schools, it makes sense that the multi-level model (MLM) did not show a significant effect. The data is presented to the right. Let’s try running a normal regression on all the data at once. This is known as a random-intercepts model, because we are going to estimate a different intercept for each subject. To assess whether the addition of the random-slopes improved the fit of our model, we can use a goodness of fit test known as a likelihood ratio (LR) chi-square difference test (i.e., nested model test; Snijders & Bosker, 2012). PSA is the monthly e-newsletter of the APA Science Directorate. COVID-19 resources for psychologists, health-care workers and the public. This combination of different “levels” of analysis gives rise to the term multi-level modeling. The correlation is now .24 (which is between Rich and Poor schools). In this primer, I use a simple example to demonstrate the fundamental principles of MLMs. Now let’s see what happens if you run a normal regression on rich and poor schools separately. Multilevel Models in R 5 1 Introduction This is an introduction to how R can be used to perform a wide variety of multilevel analyses. Baayen et al. Multilevel modelling. The result is significant ((X(1) = 3.96, p < .05), suggesting that the addition of the fixed-effect of language improves model fit. A primer on multilevel modeling. In addition, there is also the fixed-effect regression (solid line) that captures the overall group effect. I give you three examples below. Pay attention to the fact that the schools have different correlation values for Friends and Happiness depending on if they’re rich or poor. However, participants are cued to respond in English for three trials and Spanish for the other three. . Chapter 7: Mixture modeling with cross-sectional data view examples. One solution is to estimate degrees of freedom to get p-values. Multilevel analyses are applied to data that have some form of a nested structure. The regular regression did not reflect what was happening in each school type. you will need to refit the models with the package lmerTest installed and loaded. Chapter 8: Mixture modeling with longitudinal data view examples. We’ve also made the equations different so that our X (number of friends) coefficient is negative in rich schools but positive in the poor schools. LS0tDQp0aXRsZTogIkNoYXB0ZXIgMTY6IE11bHRpbGV2ZWwgTW9kZWxpbmciDQphdXRob3I6ICJUZXJlc2EgRy4gQm9yb3dza2kiDQpvdXRwdXQ6DQogIGh0bWxfZG9jdW1lbnQ6DQogICAgdGhlbWU6IGNlcnVsZWFuDQogICAgaGlnaGxpZ2h0OiB0ZXh0bWF0ZQ0KICAgIGZvbnRzaXplOiA4cHQNCiAgICB0b2M6IHRydWUNCiAgICBudW1iZXJfc2VjdGlvbnM6IHRydWUNCiAgICBjb2RlX2Rvd25sb2FkOiB0cnVlDQogICAgdG9jX2Zsb2F0Og0KICAgICAgY29sbGFwc2VkOiBmYWxzZQ0KLS0tDQoNCmBgYHtyIHNldHVwLCBpbmNsdWRlPUZBTFNFfQ0Ka25pdHI6Om9wdHNfY2h1bmskc2V0KGVjaG8gPSBUUlVFKQ0Ka25pdHI6Om9wdHNfY2h1bmskc2V0KG1lc3NhZ2UgPSBGQUxTRSkNCmtuaXRyOjpvcHRzX2NodW5rJHNldCh3YXJuaW5nID0gRkFMU0UpDQprbml0cjo6b3B0c19jaHVuayRzZXQoZmlnLndpZHRoPTUpDQprbml0cjo6b3B0c19jaHVuayRzZXQoZmlnLmhlaWdodD0zLjc1KQ0Ka25pdHI6Om9wdHNfY2h1bmskc2V0KGZpZy5hbGlnbj0nY2VudGVyJykgDQpgYGANCg0KIyBQcmVwcGluZyB5b3VyIFINCg0KIyNQYWNrYWdlcyB5b3Ugd2lsbCBuZWVkOg0KDQotIGluc3RhbGwucGFja2FnZXMoImxtZTQiKSAjIEFsbG93cyB5b3UgdG8gZml0IGxpbmVhciBtaXhlZC1lZmZlY3RzIG1vZGVscy4NCi0gaW5zdGFsbC5wYWNrYWdlcygiZWZmZWN0cyIpICMgSGVscHMgeW91IHZpc3VhbGl6ZSBpbnRlcmFjdGlvbnMNCi0gaW5zdGFsbC5wYWNrYWdlcygiY29ycnBsb3QiKSAjIEhlbHBzIHlvdSB0byB2aXN1YWxpemUgY29ycmVsYXRpb25zIA0KLSBpbnN0YWxsLnBhY2thZ2VzKCJzdGFyZ2F6ZXIiKSAjIE1ha2VzIHRhYmxlcw0KLSBpbnN0YWxsLnBhY2thZ2VzKCJjYXIiKSAjIEhlbHBzIHdpdGggRGlhZ25vc3RpY3MNCi0gaW5zdGFsbC5wYWNrYWdlcygiTUFTUyIpICMgRm9yIHN0ZXB3aXNlIHJlZ3Jlc3Npb24NCi0gaW5zdGFsbC5wYWNrYWdlcygiZ2dwbG90MiIpICMgVG8gY3JlYXRlIHBsb3RzDQotIGluc3RhbGwucGFja2FnZXMoInRleHJlZyIpICMgQWxsb3dzIHlvdSB0byBzYXZlIHRhYmxlcy4NCg0KIyNMb2FkIHRoZSBsaWJyYXJpZXMgeW91IG5lZWQuDQoNCmBgYHtyfQ0KbGlicmFyeShsbWU0KQ0KbGlicmFyeShlZmZlY3RzKQ0KbGlicmFyeShjb3JycGxvdCkNCmxpYnJhcnkoc3RhcmdhemVyKQ0KbGlicmFyeShjYXIpDQpsaWJyYXJ5KE1BU1MpDQpsaWJyYXJ5KGdncGxvdDIpDQpsaWJyYXJ5KHRleHJlZykNCmBgYA0KDQoNCiMgVGhlIERhdGEgU2V0Og0KDQpUaGlzIGlzIGEgc3BlY2lhbCB0eXBlIG9mIGRhdGEgd2hlcmUgdGhlcmUgaXMgb25seSBvbmUgbWVhc3VyZW1lbnQgcGVyIHN0dWRlbnQgKGkuZS4sIHRoZXJlIGlzIG5vIHdpdGhpbiBzdWJqZWN0IHZhcmlhYmxlKSwgYW5kIHN0dWRlbnRzIGFyZSBuZXN0ZWQgaW4gc2Nob29scyBzbyB3ZSBtdXN0IGNvbnRyb2wgZm9yIHRoZSBlZmZlY3RzIG9mIHNjaG9vbHMuIA0KDQpNdWx0aWxldmVsIGRhdGEgb2NjdXIgd2hlbiBvYnNlcnZhdGlvbnMgYXJlIG5lc3RlZCB3aXRoaW4gZ3JvdXBzLCBmb3IgZXhhbXBsZSwgd2hlbiBzdHVkZW50cyBhcmUgbmVzdGVkIHdpdGhpbiBzY2hvb2xzIGluIGEgZGlzdHJpY3QuDQoNCk91ciBzaW1wbGUgc3RvcnkgLQ0KDQpXZSBsb29rZWQgYXQgNiBzY2hvb2xzICgzIHJpY2ggYW5kIDMgcG9vcikgd2l0aCA0MCBzdHVkZW50cyBpbiBlYWNoIHJpY2ggc2Nob29sIGFuZCAxNjAgc3R1ZGVudHMgaW4gZWFjaCBwb29yIHNjaG9vbCwgYW5kIHdlIG1lYXN1cmVkIHRoZW0gb24gSGFwcGluZXNzLCBudW1iZXIgb2YgRnJpZW5kcywgYW5kIEdQQS4gV2Ugd29uZGVyZWQgaWYgaGFwcGluZXNzIGNvdWxkIGJlIHByZWRpY3RlZCBieSBudW1iZXIgb2YgZnJpZW5kcyBhbmQvb3IgR1BBLg0KDQpPdXIgTGV2ZWwgMSB3aWxsIGJlIHRoZSBsb3dlc3QgdW5pdCBvZiBvYnNlcnZhdGlvbiAtIHRob3NlIG5lc3RlZCBpbiBncm91cHMuIEluIHRoaXMgY2FzZSwgTGV2ZWwgMSBpcyB0aGUgc3R1ZGVudHMuDQoNCk91ciBMZXZlbCAyIHJlZmVycyB0byB0aGUgZ3JvdXBzIGluIHdoaWNoIHRoZSBsZXZlbCAxIHVuaXRzIGFyZSBuZXN0ZWQ7IGluIHRoaXMgY2FzZSBMZXZlbCAyIGlzIHRoZSBzY2hvb2xzLg0KDQojIyBTaW11bGF0aW5nIHRoZSBEYXRhc2V0Og0KDQojIyMgTWFrZSB5b3VyIGVxdWF0aW9ucw0KQ3JlYXRlIGEgZGlzdHJpYnV0aW9uIGZvciBlYWNoIHNjaG9vbCwgbWFraW5nIHRoZSB2YXJpYW5jZSBvZiByaWNoIHNjaG9vbHMgc21hbGwgYW5kIHRoZSB2YXJpYW5jZSBvZiBwb29yIHNjaG9vbHMgbGFyZ2UuIFdlJ3ZlIGFsc28gbWFkZSB0aGUgZXF1YXRpb25zIGRpZmZlcmVudCBzbyB0aGF0IG91ciBYIChudW1iZXIgb2YgZnJpZW5kcykgY29lZmZpY2llbnQgaXMgbmVnYXRpdmUgaW4gcmljaCBzY2hvb2xzIGJ1dCBwb3NpdGl2ZSBpbiB0aGUgcG9vciBzY2hvb2xzLiBXZSBrZWVwIG91ciBaIChHUEEpIGNvZWZmaWNpZW50IHRoZSBzYW1lIGZvciBhbGwgc2Nob29scy4gDQoNCmBgYHtyfQ0Kc2V0LnNlZWQoNDIpDQpucmljaD00MA0KbnBvb3I9MTYwDQojUGFyYW1hdGVycw0KUy5GLlJpY2g9LTIgIyBzZXR0aW5nIHBhcmFtYWV0ZXJzIGZvciB0aGUgRnJpZW5kcyB2YXJpYWJsZSBpbiB0aGUgUmljaCBzY2hvb2xzDQpTLkYuUG9vcj02ICMgc2V0dGluZyBwYXJhbWV0ZXJzIGZvciB0aGUgRnJpZW5kcyB2YXJpYWJsZSBpbiB0aGUgUG9vciBzY2hvb2xzDQoNClMuRy5SaWNoPS43ICMgc2V0dGluZyBwYXJhbWV0ZXJzIGZvciB0aGUgR1BBIHZhcmlhYmxlIGluIHRoZSBSaWNoIHNjaG9vbHMNClMuRy5Qb29yPS43ICMgc2V0dGluZyBwYXJhbWV0ZXJzIGZvciB0aGUgR1BBIHZhcmlhYmxlIGluIHRoZSBQb29yIHNjaG9vbHMNCiMgUmljaCBTY2hvb2xzDQojIFNjaG9vbCAxDQpYMSA8LSBybm9ybShucmljaCwgMTAsIDIpICMgbnVtYmVyIG9mIGZyaWVuZHMNCloxIDwtIHJ1bmlmKG5yaWNoLCAxLjAsIDQuMCkgIyBHUEENClkxIDwtIFMuRi5SaWNoKlgxICsgUy5HLlJpY2gqWjEgKyA4MCArIHJub3JtKG5yaWNoLCBzZD0gNSkgIyBPdXIgZXF1YXRpb24gdG8gY3JlYXRlIFkNCg0KIyBTY2hvb2wgMg0KWDIgPC0gcm5vcm0obnJpY2gsIDEwLCAyKSAjIG51bWJlciBvZiBmcmllbmRzDQpaMiA8LSBydW5pZihucmljaCwgMS4wLCA0LjApICMgR1BBDQpZMiA8LSBTLkYuUmljaCpYMiArIFMuRy5SaWNoKloyICsgNzUgKyBybm9ybShucmljaCwgc2Q9IDUpDQoNCiMgU2Nob29sIDMNClgzIDwtIHJub3JtKG5yaWNoLCAxMCwgMikgIyBudW1iZXIgb2YgZnJpZW5kcw0KWjMgPC0gcnVuaWYobnJpY2gsIDEuMCwgNC4wKSAjIEdQQQ0KWTMgPC0gUy5GLlJpY2gqWDMgKyBTLkcuUmljaCpaMyArOTAgKyBybm9ybShucmljaCwgc2Q9IDUpDQoNCiMgUG9vciBTY2hvb2xzDQojIFNjaG9vbCA0DQpYNCA8LSBybm9ybShucG9vciwgNSwgMikgI251bWJlciBvZiBmcmllbmRzDQpaNCA8LSBydW5pZihucG9vciwgMS4wLCA0LjApICNHUEENClk0IDwtIFMuRi5Qb29yKlg0ICsgUy5HLlBvb3IqWjQgKyAzNSArIHJub3JtKG5wb29yLCBzZCA9IDEwKQ0KDQojIFNjaG9vbCA1DQpYNSA8LSBybm9ybShucG9vciwgNSwgMikgI251bWJlciBvZiBmcmllbmRzDQpaNSA8LSBydW5pZihucG9vciwgMS4wLCA0LjApICNHUEENClk1IDwtIFMuRi5Qb29yKlg1ICsgUy5HLlBvb3IqWjUgKyA0MCArIHJub3JtKG5wb29yLCBzZCA9IDEwKQ0KDQojIFNjaG9vbCA2DQpYNiA8LSBybm9ybShucG9vciwgNSwgMikgI251bWJlciBvZiBmcmllbmRzDQpaNiA8LSBydW5pZihucG9vciwgMS4wLCA0LjApICNHUEENClk2IDwtIFMuRi5Qb29yKlg2ICsgUy5HLlBvb3IqWjYgKyA1MCArIHJub3JtKG5wb29yLCBzZCA9IDEwKQ0KDQpgYGAgDQoNCiMjIyBNYWtlIHlvdXIgZGF0YSBmcmFtZXMNCk5vdyB0aGF0IHdlIGhhdmUgb3VyIGVxdWF0aW9ucywgd2UncmUgZ29pbmcgdG8gY3JlYXRlIHRoZSBkYXRhIGZyYW1lIGZvciBlYWNoIHNjaG9vbC4NCg0KYGBge3J9DQojIFRoZSAzIFJpY2ggU2Nob29sczoNClN0dWRlbnQuRGF0YS5TY2hvb2wuMTwtZGF0YS5mcmFtZShIYXBwaW5lc3M9WTEsIEZyaWVuZHM9WDEsIEdQQT1aMSkNClN0dWRlbnQuRGF0YS5TY2hvb2wuMjwtZGF0YS5mcmFtZShIYXBwaW5lc3M9WTIsIEZyaWVuZHM9WDIsIEdQQT1aMikNClN0dWRlbnQuRGF0YS5TY2hvb2wuMzwtZGF0YS5mcmFtZShIYXBwaW5lc3M9WTMsIEZyaWVuZHM9WDMsIEdQQT1aMykNCg0KIyBUaGUgMyBQb29yIFNjaG9vbHM6DQpTdHVkZW50LkRhdGEuU2Nob29sLjQ8LWRhdGEuZnJhbWUoSGFwcGluZXNzPVk0LCBGcmllbmRzPVg0LCBHUEE9WjQpDQpTdHVkZW50LkRhdGEuU2Nob29sLjU8LWRhdGEuZnJhbWUoSGFwcGluZXNzPVk1LCBGcmllbmRzPVg1LCBHUEE9WjUpDQpTdHVkZW50LkRhdGEuU2Nob29sLjY8LWRhdGEuZnJhbWUoSGFwcGluZXNzPVk2LCBGcmllbmRzPVg2LCBHUEE9WjYpDQoNCmBgYA0KDQojIyMgVGVzdGluZyB0aGUgZm9ybXVsYXMNCkxldCdzIHRlc3QgaWYgdGhlc2UgZm9ybXVsYXMgd29yayB3aXRoIGFuIGV4YW1wbGUgZnJvbSBlYWNoIHNldCBvZiBzY2hvb2xzIChvbmUgcmljaCBhbmQgb25lIHBvb3IpLg0KDQpgYGB7cn0NCiMgUmljaCAtIFNjaG9vbCAxDQpjb3JyLnN0dWRlbnQgPSBjb3IoU3R1ZGVudC5EYXRhLlNjaG9vbC4xKQ0KY29ycnBsb3QoY29yci5zdHVkZW50LCBtZXRob2QgPSAibnVtYmVyIixkaWFnID0gRkFMU0UsdHlwZSA9ICJsb3dlciIpDQpgYGANCg0KR3JlYXQsIG91ciBjb3JyZWxhdGlvbiBiZXR3ZWVuIEZyaWVuZHMgYW5kIEhhcHBpbmVzcyBpcyAtLjc2IGFuZCB3ZSBoYXZlIGEgY29ycmVsYXRpb24gYmV0d2VlbiBHUEEgYW5kIEhhcHBpbmVzcyBvZiAtLjA1Lg0KDQpgYGB7cn0NCiMgUG9vciAtIFNjaG9vbCA0DQpjb3JyLnN0dWRlbnQgPSBjb3IoU3R1ZGVudC5EYXRhLlNjaG9vbC40KQ0KY29ycnBsb3QoY29yci5zdHVkZW50LCBtZXRob2QgPSAibnVtYmVyIixkaWFnID0gRkFMU0UsdHlwZSA9ICJsb3dlciIpDQpgYGANCg0KR3JlYXQsIG91ciBjb3JyZWxhdGlvbiBiZXR3ZWVuIEZyaWVuZHMgYW5kIEhhcHBpbmVzcyBpcyAuNzcsIHdoaWNoIGlzIGhpZ2hlciBBTkQgaW4gdGhlIG9wcG9zaXRlIGRpcmVjdGlvbiB0aGFuIGluIHJpY2ggc2Nob29scy4gVGhlcmUgd2FzIG5vIGNvcnJlbGF0aW9uIGJldHdlZW4gR1BBIGFuZCBIYXBwaW5lc3MuDQoNClBheSBhdHRlbnRpb24gdG8gdGhlIGZhY3QgdGhhdCB0aGUgc2Nob29scyBoYXZlIGRpZmZlcmVudCBjb3JyZWxhdGlvbiB2YWx1ZXMgZm9yIEZyaWVuZHMgYW5kIEhhcHBpbmVzcyBkZXBlbmRpbmcgb24gaWYgdGhleSdyZSByaWNoIG9yIHBvb3IuIFJpY2ggc2Nob29scyBoYXZlIHIgPSAtLjc2IGFuZCBwb29yIHNjaG9vbHMgaGF2ZSByID0gLjc3LiBGb3IgR1BBIGFuZCBIYXBwaW5lc3MsIG9ubHkgdGhlIHJpY2ggc2Nob29sIGhhZCBhIGNvcnJlbGF0aW9uLiBUaGlzIHdpbGwgYmUgaW1wb3J0YW50LiANCg0KTk9URTogQWx0aG91Z2ggSSBhbSB0ZWxsaW5nIHlvdSB0aGF0IHNvbWUgc2Nob29scyBhcmUgcmljaCBhbmQgc29tZSBhcmUgcG9vciwgeW91IG1heSBub3Qga25vdyB0aGlzIGluZm9ybWF0aW9uIHdoZW4geW91IGNvbWUgaW4gYW5kIHRyeSB0byBkbyB0aGUgYW5hbHlzaXMuIElmIHdlIGtuZXcgdGhpcyBpbmZvcm1hdGlvbiwgd2Ugd291bGQgcHV0IFNFUyBhcyBhbm90aGVyIExldmVsIDIgcHJlZGljdG9yLiANCg0KDQojIyMgUHV0ICdlbSB0b2dldGhlciBhbmQgd2hhdCBoYXZlIHlvdSBnb3QhPw0KTm93IHRoYXQgd2Uga25vdyB0aGV5IHdvcmssIGxldCdzIHB1dCBhbGwgc2Nob29scyB0b2dldGhlciBpbnRvIG9uZSBkYXRhc2V0LiANCg0KYGBge3J9DQpBbGwuU2Nob29scy5EYXRhIDwtIHJiaW5kKFN0dWRlbnQuRGF0YS5TY2hvb2wuMSwgU3R1ZGVudC5EYXRhLlNjaG9vbC4yLCBTdHVkZW50LkRhdGEuU2Nob29sLjMsIFN0dWRlbnQuRGF0YS5TY2hvb2wuNCwgU3R1ZGVudC5EYXRhLlNjaG9vbC41LCBTdHVkZW50LkRhdGEuU2Nob29sLjYpIA0KaGVhZChBbGwuU2Nob29scy5EYXRhKQ0KYGBgDQoNCiMjIyBBZGRpbmcgU3R1ZGVudCBhbmQgU2Nob29sIFZhcmlhYmxlcw0KTm93IGxldCdzIGFkZCBpbiBTdHVkZW50IElEJ3MgYW5kIHRoZSBTY2hvb2xzIGFzIHZhcmlhYmxlcy4NCg0KYGBge3J9DQojIEFkZGluZyB0aGUgc3ViamVjdCB2YXJpYWJsZSAoU3R1ZGVudCBJRCkNCkFsbC5TY2hvb2xzLkRhdGEkU3R1ZGVudElEPC1zZXEoMTpucm93KEFsbC5TY2hvb2xzLkRhdGEpKQ0KIyBEaWQgaXQgd29yaz8NCmhlYWQoQWxsLlNjaG9vbHMuRGF0YSkNCiMgWWVzIQ0KDQojIEFkZGluZyB0aGUgU2Nob29sIHZhcmlhYmxlLiANCkFsbC5TY2hvb2xzLkRhdGEkU2Nob29sPC1jKHJlcCgxLCBucmljaCksIHJlcCgyLG5yaWNoKSwgcmVwKDMsIG5yaWNoKSwgcmVwKDQsIG5wb29yKSwgDQogICAgICAgICAgICAgICAgICAgICAgICAgICByZXAoNSwgbnBvb3IpLCByZXAoNiwgbnBvb3IpKQ0KYGBgDQoNCiMjIyBUZXN0IHRoZSBmdWxsIGRhdGFzZXQNCkxldCdzIHNlZSB3aGF0IHRoZSBzY2F0dGVyIHBsb3RzIGZvciBlYWNoIHNjaG9vbC4NCg0KLSBTY2hvb2xzIDEtMyBhcmUgcmljaGFuZCBzY2hvb2xzIDQtNiBhcmUgcG9vci4NCg0KDQojIFBsb3R0aW5nIA0KDQpOb3cgbGV0J3MgcGxvdCB1cCByYXcgZGF0YS4NCg0KRmlyc3QsIGxldCdzIHBsb3Qgd2l0aCBGcmllbmRzIGFzIG91ciBJVi4NCg0KYGBge3J9DQp0aGVtZV9zZXQodGhlbWVfYncoYmFzZV9zaXplID0gMTIsIGJhc2VfZmFtaWx5ID0gIiIpKSANCg0KIyBGcmllbmRzDQpNb2RlbC5QbG90LkZyaWVuZHMgPC1nZ3Bsb3QoZGF0YSA9IEFsbC5TY2hvb2xzLkRhdGEsIGFlcyh4ID0gRnJpZW5kcywgeT1IYXBwaW5lc3MsZ3JvdXA9U2Nob29sKSkrCQ0KICBmYWNldF9ncmlkKCB+IFNjaG9vbCkrCQ0KICBnZW9tX3BvaW50KGFlcyhjb2xvdXIgPSBTY2hvb2wpKSsJDQogIGdlb21fc21vb3RoKG1ldGhvZCA9ICJsbSIsIHNlID0gVFJVRSwgYWVzKGNvbG91ciA9IFNjaG9vbCkpKwkNCiAgeGxhYigiRnJpZW5kcyIpK3lsYWIoIkhhcHBpbmVzcyIpKwkNCiAgdGhlbWUobGVnZW5kLnBvc2l0aW9uID0gIm5vbmUiKQkNCk1vZGVsLlBsb3QuRnJpZW5kcwkNCmBgYA0KDQpOb3csIGxldCdzIHBsb3Qgd2l0aCBHUEEgYXMgb3VyIElWLg0KDQpgYGB7cn0NCk1vZGVsLlBsb3QuR1BBIDwtZ2dwbG90KGRhdGEgPSBBbGwuU2Nob29scy5EYXRhLCBhZXMoeCA9R1BBLCB5PUhhcHBpbmVzcyxncm91cD1TY2hvb2wpKSsJDQogIGZhY2V0X2dyaWQoIH4gU2Nob29sKSsJDQogIGdlb21fcG9pbnQoYWVzKGNvbG91ciA9IFNjaG9vbCkpKwkNCiAgZ2VvbV9zbW9vdGgobWV0aG9kID0gImxtIiwgc2UgPSBUUlVFLCBhZXMoY29sb3VyID0gU2Nob29sKSkrCQ0KICB4bGFiKCJHUEEiKSt5bGFiKCJIYXBwaW5lc3MiKSsJDQogIHRoZW1lKGxlZ2VuZC5wb3NpdGlvbiA9ICJub25lIikJDQpNb2RlbC5QbG90LkdQQQkNCmBgYA0KDQpMZXQncyBzZWUgd2hhdCBvdXIgY29ycmVsYXRpb25zIGxvb2sgbGlrZSB3aGVuIGx1bXBpbmcgYWxsIHRoZSByaWNoIGFuZCBwb29yIHNjaG9vbHMgdG9nZXRoZXIuDQoNCmBgYHtyfQ0KY29yci5zdHVkZW50ID0gY29yKEFsbC5TY2hvb2xzLkRhdGFbLDE6M10pDQpjb3JycGxvdChjb3JyLnN0dWRlbnQsIG1ldGhvZCA9ICJudW1iZXIiLGRpYWcgPSBGQUxTRSx0eXBlID0gImxvd2VyIikNCmBgYA0KDQpOb3RpY2UgaG93IHRoZSBjb3JyZWxhdGlvbiBiZXR3ZWVuIEZyaWVuZHMgYW5kIEhhcHBpbmVzcyBoYXMgY2hhbmdlZCB3aGVuIHlvdSBjb2xsYXBzZSBhY3Jvc3MgYWxsIHJpY2ggYW5kIHBvb3Igc2Nob29scy4gVGhlIGNvcnJlbGF0aW9uIGlzIG5vdyAuMjQgKHdoaWNoIGlzIGJldHdlZW4gUmljaCBhbmQgUG9vciBzY2hvb2xzKS4gSXQgaXMgYSBwb3NpdGl2ZSBjb3JyZWxhdGlvbiBhcyB3ZWxsLCBidXQgb25seSBwb29yIHNjaG9vbHMgKHRob3NlIHdpdGggbW9yZSBzdHVkZW50cykgaGFkIGEgcG9zaXRpdmUgY29ycmVsYXRpb24uDQpBbHNvIG5vdGljZSB0aGF0IHRoZSBjb3JyZWxhdGlvbiBiZXR3ZWVuIEdQQSBhbmQgSGFwcGluZXNzIGlzIC4wNSwgd2hpY2ggaXMgdGhlIHNhbWUgYXMgZm9yIHRoZSByaWNoIHNjaG9vbCAtIHRoZSBvbmx5IG9uZSB0aGF0IGhhZCBzaG93ZWQgYSBjb3JyZWxhdGlvbi4gDQoNCg0KIyBSZWd1bGFyIFJlZ3Jlc3Npb24NCg0KIyMgQWxsIERhdGENCkxldCdzIHRyeSBydW5uaW5nIGEgbm9ybWFsIHJlZ3Jlc3Npb24gb24gYWxsIHRoZSBkYXRhIGF0IG9uY2UuDQoNCkZpcnN0LCBsZXQncyBjZW50ZXIgdGhlIHZhcmlhYmxlcy4NCg0KYGBge3J9DQpBbGwuU2Nob29scy5EYXRhJEZyaWVuZHMuQzwtc2NhbGUoQWxsLlNjaG9vbHMuRGF0YSRGcmllbmRzLCBzY2FsZSA9IEZBTFNFKVssXQ0KQWxsLlNjaG9vbHMuRGF0YSRHUEEuQzwtc2NhbGUoQWxsLlNjaG9vbHMuRGF0YSRHUEEsIHNjYWxlID0gRkFMU0UpWyxdDQpgYGANCg0KTm93IHdlIGNhbiBydW4gdGhlIHJlZ3Jlc3Npb24uDQoNCmBgYHtyfQ0KUmVnLk1vZGVsPC1sbShIYXBwaW5lc3MgfiBGcmllbmRzLkMgKyBHUEEuQywgZGF0YSA9IEFsbC5TY2hvb2xzLkRhdGEpDQpzdW1tYXJ5KFJlZy5Nb2RlbCkNCiNwbG90KFJlZy5Nb2RlbCkNCmBgYA0KDQpQdXR0aW5nIGFsbCBzY2hvb2xzIHRvZ2V0aGVyIGluIGEgcmVndWxhciByZWdyZXNzaW9uLCB3ZSBoYXZlIGEgcG9zaXRpdmUsIHNpZ25pZmljYW50IGVmZmVjdCBvZiBGcmllbmRzIGFuZCBubyBlZmZlY3Qgb2YgR1BBLiBCdXQgd2hhdCBoYXBwZW5zIHdoZW4gd2Ugc2VwYXJhdGUgdGhlIHNjaG9vbHMgYnkgcmljaCBvciBwb29yIHN0YXR1cz8NCg0KIyMgQnkgU2Nob29sDQpOb3cgbGV0J3Mgc2VlIHdoYXQgaGFwcGVucyBpZiB5b3UgcnVuIGEgbm9ybWFsIHJlZ3Jlc3Npb24gb24gcmljaCBhbmQgcG9vciBzY2hvb2xzIHNlcGFyYXRlbHkuDQoNCmBgYHtyfQ0KIyBSaWNoIFNjaG9vbHMNClNjaG9vbC5SaWNoLlJlZy5Nb2RlbDwtbG0oSGFwcGluZXNzIH4gRnJpZW5kcy5DICsgR1BBLkMsIGRhdGEgPSBzdWJzZXQoQWxsLlNjaG9vbHMuRGF0YSwgU2Nob29sPDQpKQ0Kc3VtbWFyeShTY2hvb2wuUmljaC5SZWcuTW9kZWwpDQpgYGANCg0KTm90aWNlLCB3ZSBoYXZlIGEgbmVnYXRpdmUsIHNpZ25pZmljYW50IGVmZmVjdCBmb3IgRnJpZW5kcyBidXQgbm8gZWZmZWN0IGZvciBHUEEuDQoNCmBgYHtyfQ0KIyBQb29yIFNjaG9vbHMNClNjaG9vbC5Qb29yLlJlZy5Nb2RlbDwtbG0oSGFwcGluZXNzIH4gRnJpZW5kcy5DICsgR1BBLkMsIGRhdGEgPSBzdWJzZXQoQWxsLlNjaG9vbHMuRGF0YSwgU2Nob29sPjMpKQ0Kc3VtbWFyeShTY2hvb2wuUG9vci5SZWcuTW9kZWwpDQpgYGANCg0KVWgtb2guIE5vdyB3ZSBoYXZlIGEgcG9zaXRpdmUsIHNpZ25pZmljYW50IGVmZmVjdCBvZiBGcmllbmRzIGFuZCBubyBlZmZlY3Qgb2YgR1BBLg0KDQojIyBXaGF0J3MgdGhlIHByb2JsZW0/DQoNClJlbWVtYmVyISBBbHRob3VnaCBJIGFtIHRlbGxpbmcgeW91IHRoYXQgc29tZSBzY2hvb2xzIGFyZSByaWNoIGFuZCBzb21lIGFyZSBwb29yLCB5b3UgbWF5IG5vdCBrbm93IHRoaXMgaW5mb3JtYXRpb24gd2hlbiB5b3UgY29tZSBpbiBhbmQgdHJ5IHRvIGRvIHRoZSBhbmFseXNpcy4gSWYgd2Uga25ldyB0aGlzIGluZm9ybWF0aW9uLCB3ZSBjb3VsZCBwdXQgU0VTIGFzIGFub3RoZXIgTGV2ZWwgMiBwcmVkaWN0b3IuIEFsc28sIEkgYW0gb25seSB1c2luZyBzaXggc2Nob29scyB0byBkZW1vbnN0cmF0ZSwgYnV0IGluIGEgbW9yZSAibm9ybWFsIiBjYXNlIG9mIG5lc3RlZCBkZXNpZ24gdGhlcmUgbWF5IGJlIHdheSB0b28gbWFueSBzY2hvb2xzIHRvIHRyZWF0IFNFUyBhcyBhIGZpeGVkIGZhY3RvciBpbiB0aGUgZGVzaWduLg0KDQpUaGUgcmVndWxhciByZWdyZXNzaW9uIGRpZCBub3QgcmVmbGVjdCB3aGF0IHdhcyBoYXBwZW5pbmcgaW4gZWFjaCBzY2hvb2wgdHlwZS4gSXQgZ2F2ZSB1cyBhIHBvc2l0aXZlIGVmZmVjdCAoRnJpZW5kcykgd2hlbiBvbmx5IHBvb3Igc2Nob29scyBoYWQgYSBwb3NpdGl2ZSBlZmZlY3QuIENvbGxhcHNpbmcgYWNyb3NzIHNjaG9vbCB0eXBlcyBpbiB0aGlzIGNhc2Ugd2FzIG5vdCBpZGVhbCBiZWNhdXNlIGRpZmZlcmVudCB0aGluZ3Mgd2VyZSBoYXBwZW5pbmcgd2l0aGluIGVhY2ggc2Nob29sIHR5cGUsIGNvbXByb21pc2luZyB0aGUgZ2VuZXJhbGl6YWJpbGl0eSBvZiB0aGUgZmluZGluZ3MuDQoNCkFub3RoZXIgd2F5IHRvIHB1dCB0aGlzLCB0aGUgcmVndWxhciAobG0pIHJlZ3Jlc3Npb24gaW5kaWNhdGVzIHRoYXQgdGhlIG1vcmUgZnJpZW5kcyBhIHN0dWRlbnQgaGFzLCB0aGUgaGFwcGllciB0aGV5IGFyZSwgYnV0IGxvb2tpbmcgY2xvc2VyIHRoaXMgaXMgbm90IHRoZSBjYXNlIGluIGFsbCBzY2hvb2xzIChhbmQgaXMsIGluIGZhY3QsIHRoZSBvcHBvc2l0ZSBpbiBzb21lKS4gSWYgeW91IHdlcmUgdHJ5aW5nIHRvIGdlbmVyYWxpemUgeW91ciBmaW5kaW5ncyBvciB1c2UgdGhlbSB0byBhcmd1ZSBmb3Ivc2hvdyBhIG5lZWQgZm9yIGFuIGludGVydmVudGlvbiwgeW91ciByZXN1bHRzIHdvdWxkIGJlIG1pc2xlYWRpbmcgYW5kIGNvdWxkIGNhdXNlIHByb2JsZW1zLiANCg0KU08sDQpMZXQncyB0cnkgcnVubmluZyB0aGlzIGFzIGEgbXVsdGlsZXZlbCwgcmFuZG9tIGVmZmVjdHMgbW9kZWw6DQoNCg0KIyBSYW5kb20gRWZmZWN0cyBNb2RlbA0KDQpVc2luZyBhIG11bHRpLWxldmVsIG1vZGVsIGFsbG93cyB1cyB0byBzZXBhcmF0ZSB0aGUgd2l0aGluLWdyb3VwIGVmZmVjdHMgZnJvbSB0aGUgYmV0d2Vlbi1ncm91cCBlZmZlY3RzLCB3aGVyZWFzIHJlZ3VsYXIgcmVncmVzc2lvbiBibGVuZHMgdGhlbSB0b2dldGhlciBpbnRvIGEgc2luZ2xlIGNvZWZmaWNpZW50LiANCg0KDQojIyBSdW5uaW5nIHRoZSBSYW5kb20gRWZmZWN0cyBNb2RlbCANCg0KDQojIyMgRmlyc3QsIHdlIHJ1biB0aGUgbnVsbCBtb2RlbC4NCg0KYGBge3J9DQpOdWxsPC1sbWVyKEhhcHBpbmVzcyB+IDEgIyBUaGlzIHNpbXBseSBtZWFucyBIYXBwaW5lc3MgcHJlZGljdGVkIGJ5IHRoZSBpbnRlcmNlcHQNCiAgICAgICAgICAgICAgICAgICsoMXxTY2hvb2wpLCAjIGVhY2ggc2Nob29sIGdldHMgaXRzIG93biBpbnRlcmNlcHQgDQogICAgICAgICAgICAgICAgICBkYXRhPUFsbC5TY2hvb2xzLkRhdGEsIFJFTUwgPSBGQUxTRSkNCnN1bW1hcnkoTnVsbCkNCmBgYA0KDQpXZSBleGFtaW5lIHRoZSBpbnRyYS1jbGFzcyBjb3JyZWxhdGlvbiAoSUNDKSB0byBkZXRlcm1pbmUgaWYgbXVsdGktbGV2ZWwgbW9kZWxpbmcgaXMgdGhlIGNvcnJlY3QgY2hvaWNlIGZvciBvdXIgYW5hbHlzaXMuIFRoZSBJQ0MgbWVhc3VyZXMgdGhlIGRlZ3JlZSBvZiBjbHVzdGVyaW5nIGluIG91ciBkYXRhIGFuZCBhbnN3ZXJzIHRoZSBxdWVzdGlvbiwgIkhvdyBtdWNoIGRvZXMgbXkgTGV2ZWwgMiBwcmVkaWN0IHRoZSB0b3RhbCB2YXJpYW5jZSBvZiBteSBzdHVkeT8iIElmIHlvdXIgSUNDIGlzIGdyZWF0ZXIgdGhhbiAwLCB5b3UgaGF2ZSBhIG11bHRpLWxldmVsIHN0dWR5LiANCg0KYGBge3J9DQpJQ0MuTW9kZWw8LWZ1bmN0aW9uKE1vZGVsLk5hbWUpIHsNCiAgdGF1Lk51bGw8LWFzLm51bWVyaWMobGFwcGx5KHN1bW1hcnkoTW9kZWwuTmFtZSkkdmFyY29yLCBkaWFnKSkNCiAgc2lnbWEuTnVsbCA8LSBhcy5udW1lcmljKGF0dHIoc3VtbWFyeShNb2RlbC5OYW1lKSR2YXJjb3IsICJzYyIpXjIpDQogIElDQy5OdWxsIDwtIHRhdS5OdWxsLyh0YXUuTnVsbCtzaWdtYS5OdWxsKQ0KICByZXR1cm4oSUNDLk51bGwpDQp9DQoNCklDQy5Nb2RlbChOdWxsKQ0KYGBgDQoNCk91ciBJQ0MgaXMgZ3JlYXRlciB0aGFuIDAsIG1lYW5pbmcgd2Ugd2VyZSBjb3JyZWN0IHRvIHRoaW5rIG9mIHRoaXMgYXMgYSBtdWx0aS1sZXZlbCBwcm9ibGVtLg0KDQoNCiMjIyBOZXh0LCB3ZSBydW4gdGhlIExldmVsIDEgcHJlZGljdG9yIChHUEEpLg0KDQpgYGB7cn0NClRoZS5Nb2RlbC4xPC1sbWVyKEhhcHBpbmVzcyB+IEdQQS5DIA0KICAgICAgICAgICAgICAgICAgKygxfFNjaG9vbCksDQogICAgICAgICAgICAgICAgZGF0YT1BbGwuU2Nob29scy5EYXRhLCBSRU1MID0gRkFMU0UpDQpzdW1tYXJ5KFRoZS5Nb2RlbC4xKQ0KYGBgDQoNClRoZSByZXN1bHRzIGluZGljYXRlIHRoYXQgYSBzdHVkZW50J3MgR1BBIGRvZXMgbm90IGhhdmUgYW4gZWZmZWN0IG9uIHRoZWlyIEhhcHBpbmVzcyB3aGVuIGNvbnRyb2xsaW5nIGZvciBMZXZlbCAyIGZsdWN0dWF0aW9ucyBpbiBIYXBwaW5lc3MuIA0KDQoNCiMjIyBGaW5hbGx5LCB3ZSBydW4gdGhlIExldmVsIDIgcHJlZGljdG9yIChGcmllbmRzKS4NCg0KVGhpcyBtb2RlbCBhbGxvd3MgdGhlIHZhcmlhYmxlIEZyaWVuZHMgdG8gdmFyeSBiZXR3ZWVuIHNjaG9vbHMuDQoNCg0KYGBge3J9DQpUaGUuTW9kZWwuMjwtbG1lcihIYXBwaW5lc3MgfiBGcmllbmRzLkMgKyBHUEEuQyANCiAgICAgICAgICAgICAgICAgICsoMStGcmllbmRzLkN8U2Nob29sKSwgIyBlYWNoIHNjaG9vbCBnZXRzIGl0cyBvd24gaW50ZXJjZXB0LCBhbmQgRnJpZW5kcyBjYW4gdmFyeSBhcyBhIGZ1bmN0aW9uIG9mIHRoZSBzY2hvb2wuDQogICAgICAgICAgICAgICAgZGF0YT1BbGwuU2Nob29scy5EYXRhLCBSRU1MID0gRkFMU0UpDQpzdW1tYXJ5KFRoZS5Nb2RlbC4yKQ0KYGBgDQoNClRoZSByZXN1bHRzIGluZGljYXRlIHRoYXQgdGhlIG51bWJlciBvZiBGcmllbmRzIGEgc3R1ZGVudCBoYXMgZG9lcyBub3QgaGF2ZSBhbiBlZmZlY3Qgb24gSGFwcGluZXNzIHdoZW4gY29udHJvbGxpbmcgZm9yIHRoZSByYW5kb20gZWZmZWN0cyBvZiBMZXZlbCAyIGluZmx1ZW5jZXMuDQoNCg0KIyMjIFdoYXQgZG8gdGhlIHJlc3VsdHMgbWVhbj8NCg0KSW4gb3VyIHJlZ3VsYXIgKGxtKSByZWdyZXNzaW9uLCBGcmllbmRzIGhhZCBhIHNpZ25pZmljYW50IGVmZmVjdCwgYiA9IDEuNDAgKHAgPCAuMDAxKS4gSG93ZXZlciwgaW4gb3VyIG1peGVkIChsbWVyKSByZWdyZXNzaW9uLCBGcmllbmRzIGhhZCBhIGxhcmdlciAoMi4xNiksIGJ1dCBub24tc2lnbmlmaWNhbnQgZWZmZWN0Lg0KDQpXaHkgaXMgdGhpcyBpbXBvcnRhbnQ/IFRoZSBnb2FsIG9mIG11bHRpLWxldmVsIG1vZGVsaW5nIGlzIHRvIGRyYXcgYSBjb25jbHVzaW9uIGFib3V0IHRoZSBnZW5lcmFsIHNhbXBsZSB0aGF0IHlvdSBoYXZlIHdoaWxlIGNvbnRyb2xsaW5nIGZvciBkaWZmZXJlbmNlcyB5b3UgYXJlIG5vdCB0cnlpbmcgdG8gZXhwbGFpbiAoaW4gdGhpcyBleGFtcGxlLCByaWNoIHZzLiBwb29yKS4gTm90IHByb3Blcmx5IGNvbnRyb2xsaW5nIGZvciB0aGVzZSBkaWZmZXJlbmNlcywgd2hpY2ggeW91IG1heSBvZnRlbiBub3Qga25vdyBhcmUgdGhlcmUsIHdpbGwgaW5jcmVhc2UgeW91ciBjaGFuY2Ugb2YgVHlwZSBJIGVycm9yLiBCZWNhdXNlIHRoZSBlZmZlY3Qgb2YgRnJpZW5kcyB3YXMgZGlmZmVyZW50IGluIGRpZmZlcmVudCBzY2hvb2xzLCBpdCBtYWtlcyBzZW5zZSB0aGF0IHRoZSBtdWx0aS1sZXZlbCBtb2RlbCAoTUxNKSBkaWQgbm90IHNob3cgYSBzaWduaWZpY2FudCBlZmZlY3QuIEluIHRoZSBwcmVzZW50IGV4YW1wbGUsIG91ciBNTE0gZ2F2ZSB1cyBhIG1vcmUgYWNjdXJhdGUgaW50ZXJwcmV0YXRpb24gLSB0aGF0IG5vIG1haW4gZWZmZWN0IG9mIEZyaWVuZHMgZXhpc3RlZCBmb3IgYWxsIHNjaG9vbHMgZ2VuZXJhbGx5LiANCg0KDQojIyBDaGVja2luZyB5b3VyIEFzc3VtcHRpb25zDQoNCkZvciBhIGNvbXByZWhlbnNpdmUgbG9vayBhdCBob3cgdG8gcnVuIGRpYWdub3N0aWNzLCBwbGVhc2Ugc2VlIFtDaGFwdGVyIDEyXShodHRwOi8vYWRlbW9zLnBlb3BsZS51aWMuZWR1L0NoYXB0ZXIxMi5odG1sKSBhbmQgW0NoYXB0ZXIgMThdKGh0dHA6Ly9hZGVtb3MucGVvcGxlLnVpYy5lZHUvQ2hhcHRlcjE4Lmh0bWwpICANCg0KDQojIFAtVmFsdWVzDQoNCipQKi12YWx1ZXMgYXJlIGhvdGx5IGRpc3B1dGVkIGluIE1peGVkIE1vZGVscy4gDQoNCk9uZSBzb2x1dGlvbiBpcyB0byBlc3RpbWF0ZSBkZWdyZWVzIG9mIGZyZWVkb20gdG8gZ2V0IHAtdmFsdWVzLiBTQVMvU1BTUyB1c2VzIFNhdHRlcnRod2FpdGUgYXBwcm94aW1hdGlvbnMuIFRoZXJlIGFyZSBhbHNvIEtlbndhcmQtUm9nZXIgYXBwcm94aW1hdGlvbnMgKHNlZSBXZXN0ZmFsbCwgS2VubnksICYgSnVkZCwgMjAxNCkuIEVpdGhlciBhcmUgYWNjZXB0YWJsZS4gTWFraW5nIGEgbnVtYmVyIG9mIGFzc3VtcHRpb25zLCB0aGVzZSBvdXRwdXRzIHdpbGwgbWlycm9yIHRoZSByZXN1bHRzIG9mIGEgdHJhZGl0aW9uYWwgQU5PVkEgZmFpcmx5IGNsb3NlbHkuIA0KDQppbnN0YWxsLnBhY2thZ2VzKCJsbWVyVGVzdCIpDQoNClRoaXMgaXMgYSBtaXhlZCBtb2RlbCBhZGQtb24gcGFja2FnZSB0byBjYWxjdWxhdGUgcC12YWx1ZXMgYmFzZWQgb24gYSBjZXJ0YWluIHNldCBvZiBhc3N1bXB0aW9ucy4NCkVhY2ggZGRmIGlzIGEgZGlmZmVyZW50IG1ldGhvZCBvZiBhdHRhaW5pbmcgcC12YWx1ZXMsIHNvIHlvdSBjYW4gY2hvb3NlIHdoaWNoIHRvIHJ1bi4gSSBnaXZlIHlvdSB0aHJlZSBleGFtcGxlcyBiZWxvdy4gDQp5b3Ugd2lsbCBuZWVkIHRvIHJlZml0IHRoZSBtb2RlbHMgd2l0aCB0aGUgcGFja2FnZSBgbG1lclRlc3RgIGluc3RhbGxlZCBhbmQgbG9hZGVkLg0KDQpgYGB7cn0NCmxpYnJhcnkobG1lclRlc3QpDQpUaGUuTW9kZWwuMjwtbG1lcihIYXBwaW5lc3MgfiBGcmllbmRzLkMgKyBHUEEuQyANCiAgICAgICAgICAgICAgICAgICsoMStGcmllbmRzLkN8U2Nob29sKSwNCiAgICAgICAgICAgICAgICBkYXRhPUFsbC5TY2hvb2xzLkRhdGEsIFJFTUwgPSBGQUxTRSkNCg0Kc3VtbWFyeShUaGUuTW9kZWwuMixkZGYgPSAibG1lNCIpDQpzdW1tYXJ5KFRoZS5Nb2RlbC4yLCBkZGYgPSAiU2F0dGVydGh3YWl0ZSIpICMgU0FTIG1ldGhvZA0KYGBgDQoNCklmIHlvdSB3YW50IHRvIHJlcG9ydCBtYWluIGVmZmVjdHMgYW5kIGludGVyYWN0aW9ucyBpbnN0ZWFkIG9mIHNsb3BlcywgeW91IG1pZ2h0IGNvbnZlcnQgeW91ciBtaXhlZCBtb2RlbCBpbnRvIGFuIEFOT1ZBLg0KDQpgYGB7cn0NCmFub3ZhKFRoZS5Nb2RlbC4yLGRkZiA9ICJLZW53YXJkLVJvZ2VyIikgIyBLZW5ueSBldCBhbCBzdWdnZXN0ZWQgbWV0aG9kDQpgYGANCg0KIyBSZWZlcmVuY2VzDQoNCkh1dGEsIFYuICgyMDE0KS4gV2hlbiB0byB1c2UgaGllcmFyY2hpY2FsIGxpbmVhciBtb2RlbGluZy4gVGhlIFF1YW50aXRhdGl2ZSBNZXRob2RzIGZvciBQc3ljaG9sb2d5LCAxMCgxKTogMTMtIDI4Lg0KDQpXZXN0ZmFsbCwgSi4sIEtlbm55LCBELiBBLiwgJiBKdWRkLCBDLiBNLiAoMjAxNCkuIFN0YXRpc3RpY2FsIHBvd2VyIGFuZCBvcHRpbWFsIGRlc2lnbiBpbiBleHBlcmltZW50cyBpbiB3aGljaCBzYW1wbGVzIG9mIHBhcnRpY2lwYW50cyByZXNwb25kIHRvIHNhbXBsZXMgb2Ygc3RpbXVsaS4gSm91cm5hbCBvZiBFeHBlcmltZW50YWwgUHN5Y2hvbG9neTogR2VuZXJhbCwgMTQzKDUpLCAyMDIwLTIwNDUuDQoNCldvbHRtYW4sIEguLCBGZWxkc3RhaW4sIEEuLCBNYWNLYXksIEouIEMuLCAmIFJvY2NoaSwgTS4gKDIwMTIpLiBBbiBpbnRyb2R1Y3Rpb24gdG8gaGllcmFyY2hpY2FsIGxpbmVhciBtb2RlbGluZy4gVHV0b3JpYWxzIGluIFF1YW50aXRhdGl2ZSBNZXRob2RzIGZvciBQc3ljaG9sb2d5LCA4KDEpOiA1Mi02OS4NCg0KDQo8c2NyaXB0Pg0KICAoZnVuY3Rpb24oaSxzLG8sZyxyLGEsbSl7aVsnR29vZ2xlQW5hbHl0aWNzT2JqZWN0J109cjtpW3JdPWlbcl18fGZ1bmN0aW9uKCl7DQogIChpW3JdLnE9aVtyXS5xfHxbXSkucHVzaChhcmd1bWVudHMpfSxpW3JdLmw9MSpuZXcgRGF0ZSgpO2E9cy5jcmVhdGVFbGVtZW50KG8pLA0KICBtPXMuZ2V0RWxlbWVudHNCeVRhZ05hbWUobylbMF07YS5hc3luYz0xO2Euc3JjPWc7bS5wYXJlbnROb2RlLmluc2VydEJlZm9yZShhLG0pDQogIH0pKHdpbmRvdyxkb2N1bWVudCwnc2NyaXB0JywnaHR0cHM6Ly93d3cuZ29vZ2xlLWFuYWx5dGljcy5jb20vYW5hbHl0aWNzLmpzJywnZ2EnKTsNCg0KICBnYSgnY3JlYXRlJywgJ1VBLTk4ODc4NzkzLTEnLCAnYXV0bycpOw0KICBnYSgnc2VuZCcsICdwYWdldmlldycpOw0KDQo8L3NjcmlwdD4NCg0K, A Language, not a Letter: Learning Statistics in R. install.packages(“lme4”) # Allows you to fit linear mixed-effects models.

Pod Terbaik 2021 Malaysia, Pa State Police Hiring Process, English Pottery History, Wolkskool Week 5, Maklike Potbrood Resep, Apartments In Ferndale, Wa,