Research Methodology

The analysis was done using the R Statistical language (v4.0.3; R Core Team, 2020) on Windows 10 x64, using the packages qqplotr (v0.0.4), gridExtra (v2.3), xtable (v1.8.4), rpivotTable (v0.3.0), ggplot2 (v3.3.0), tidyr (v1.1.2), dplyr (v0.8.5), rmarkdown (v2.6), sp (v1.4.5), rworldmap (v1.3.6) and knitr (v1.28).

Research Questions

This study investigated product management practices in general with the ISPMA (2020) reference model and then roadmapping in particular with the DEEP v1.1 maturity model (Münch, Trieflinger, and Lang, 2019). It aims at understanding how SPM practices described by in the literature fit together with Software Product Manager (SPM) practices used in real life.This study addresses following research questions:

RQ1 How mature are roadmap processes used by SPMs?

RQ1.1 Are there factors that predict the maturity level?

RQ1.2 What recommendations can be made to improve material for SPM?

And using the theoretical DEEP framework as a lens to compare the practices and industry bodies of knowledge. From the background literature review in forming these questions there are two soft hypothesis to help frame up the research approach:

  1. Maturity will have some kind of link to industry, org size, and country the software product manager is based in.
  2. The theories supported, in the different professional organisations and their Product Management Bodies of Knowledge (ProdBOK), are drawn from clusters of researchers particular to each organisation. i.e. there is fragmentation in transmission of theory to practice

This is assuming a linear, or successionist, explanation of causality (Horrocks and Fowles, 2012, section 2.3) that certain behaviours and processes lead to related outcomes, although the context that the actors are operating in - that is to say the Software Product Managers - may also have an influence on the outcome.

Study Design

The factors for consideration in the research questions falls into two broad topics, the respondent and the organisation where they work:

  • Respondent, e.g.
    • Job title
    • Membership of professional bodies
    • Which areas of the SPM reference model they have responsibility for
    • Where they look for information on professional development
    • Do they feel happy with the process and their responsible areas
  • Respondent’s organisation where they perform the role, e.g.
    • Location
    • Industrial sector
    • Size of org and product team
    • Update frequencies of releases and product launches
    • Maturity of roadmapping process

The three forms of data collection will be: + Questionnaire around SPM practices, this will go to a wide distribution and expect in the region of 100-1000 responses based on similar survey’s results. The structure is in Appendix B (primary quantitative) + Semi-structured interview based on same subject areas but more depth and an exploratory element if any factors or concerns were missed. 5-10 interviews. The structure is in Appendix C (primary qualitative) + Theoretical literature review to track dimensions of maturity model present. About 25 sources (secondary qualitative)

The target audience for the survey is Software Product Managers, or at least those that take part in online Product Management discussions. Rather than members of any particular organisation, such as ISPMA. To get a good cross-section of the global industry it will be shared in the following locations:

  • Mind the Product (2021) Slack channel ~42,000 members
  • Accidental Product Manager LinkedIn (2021a) group ~35,000 members
  • Product Management LinkedIn (2021b) group ~130,000 members
  • LinkedIn personal post with snowball seeding
  • Product Unleashed (2021) Community
  • Twitter using #prdmgmt hashtag with snowball seeding
  • Twitter ads

The snowball sampling (Goodman 2011, Groves 2004) is to attempt to reach practitioners who use general social media but are not as active in specialist forums. A drawback here is that the initial respondents may not be typical as “influencers”, but this should be countered by the more general groups with large membership figures.

The questionnaire will be hosted in Google Forms, with the results stored in Google Sheets to allow for reproducible data analysis using R.

Code book

Section Question Question Type Column Coding
Research notification (Introductory text about the research)
I have obtained sufficient information about the study and the processing of my personal data. I have understood the information I have obtained and want to participate in the study. Single-answer single choice (checkbox) removed from analysis file
About You Job title Single-answer multiple choice + Open answer Job.title
Are you a member of any of these professional bodies? Single-answer multiple choice + Open answer
About where you work Which industry does your organisation primarily work in? Single-answer multiple choice
What kind of Product do you work on? (Select all that apply) Multi-answer multiple choice (checkboxes) + Open answer Split into category indicators if contains “B2B” product.b2b = True if contains “B2C” product.b2c = True if contains “Mobile” product.mobile =True if contains “Web” product.web = True if contains “Desktop” product.Desktop = True if contains “Service” product.Service = True
Number of employees in your organisation Single-answer multiple choice
Time to market Single-answer multiple choice
Release heart beat Single-answer multiple choice
Product Team Size Single-answer multiple choice
Your primary location Single-answer multiple choice
Our product roadmap helps us deliver our strategy Single-answer multiple choice (Likert-type scale) roadmap.happiness 1 = “Highly unhappy”, 2 =“Unhappy”,3 = “Neutral”, 4 = “Happy”,5 = “Highly happy”
About your role Strategic Management Multi-answer multiple choice (checkboxes) + Open answer
Product Strategy Multi-answer multiple choice (checkboxes) + Open answer
Product Planning Multi-answer multiple choice (checkboxes) + Open answer
Development Multi-answer multiple choice (checkboxes) + Open answer
Marketing Multi-answer multiple choice (checkboxes) + Open answer
Sales & Distribution Multi-answer multiple choice (checkboxes) + Open answer
Service & Support Multi-answer multiple choice (checkboxes) + Open answer
I have appropriate responsibility to achieve my goals Single-answer multiple choice (Likert-type scale) role.happiness 1 = “Highly unhappy”, 2 =“Unhappy”,3 = “Neutral”, 4 = “Happy”, 5 = “Highly happy”
About your roadmap Roadmap detailing - How adequate is your roadmap detailed with respect to the timeline? Single-answer multiple choice roadmap.detailing clean_responses\(roadmap.detailing == "Next steps are planned ad-hoc and there is no mid- to long term planning. Only short-term planning exists." ~ 1, clean_responses\)roadmap.detailing == “All tasks are planned and worked out in detail for short-,mid- and long term.” ~ 3, clean_responses\(roadmap.detailing == "There is some correlation between time and level of detail but the detailing of the items is not done systematically." ~ 8, clean_responses\)roadmap.detailing == “There is a clear correlation between time and level of detail. The timelier items are more detailed.” ~ 15, clean_responses\(roadmap.detailing == "Short-term items are detailed, prioritized, estimated and validated. Mid-term items are under validation or being discovered. The long-term timeframe contains themes." ~ 20 | | | | Roadmap items - Which items are on your product roadmap? | Single-answer multiple choice | roadmap.item | clean_responses\)roadmap.items == “Mainly products” ~ 1, clean_responses\(roadmap.items == "Mainly products, features" ~ 3, clean_responses\)roadmap.items == “Mainly business goals, products, features” ~ 10, clean_responses\(roadmap.items == "Mainly customer and business goals, products, features and for the long-term timeframe topics (e.g., smart home)" ~ 12, clean_responses\)roadmap.items == “Mainly product vision, customer and business goals, products, features and for the long-term timeframe themes (i.e., high-level customer needs)” ~ 20,
Reliability - How often do you adjust your product roadmap? Single-answer multiple choice roadmap.reliability clean_responses\(roadmap.reliability == "Permanent ad-hoc adjustments." ~ 1, clean_responses\)roadmap.reliability == “Frequent ad-hoc adjustments.” ~ 3, clean_responses\(roadmap.reliability == "Mainly in regular review cycles (e.g., every 3 months)" ~ 10, clean_responses\)roadmap.reliability == “Adjustments are mainly done reactively on demand.” ~ 10, clean_responses\(roadmap.reliability == "Adjustments are mainly done proactively." ~ 16 | | | | Confidence - How confident are you that the roadmap items have the expected impact on goals? | Single-answer multiple choice | roadmap.confidence | clean_responses\)roadmap.confidence == “The impacts are not considered.” ~ 1, clean_responses\(roadmap.confidence == "The impacts are mainly estimated by experts" ~ 4, clean_responses\)roadmap.confidence == “The impacts are mainly determined based on data from the past (e.g., statistics).” ~ 7, clean_responses\(roadmap.confidence == "The impacts are partly validated" ~ 10, clean_responses\)roadmap.confidence == “The impacts are systematically validated.” ~ 14
Discovery – How do you conduct product discovery? Single-answer multiple choice roadmap.discovery clean_responses\(roadmap.discovery == "No discovery activities. Typically, a manager is defining the roadmap items." ~ 1, clean_responses\)roadmap.discovery == “Product roadmap items are mainly defined based on expert knowledge.” ~ 2, clean_responses\(roadmap.discovery == "Product roadmap items are mainly defined based on customer requests." ~ 4, clean_responses\)roadmap.discovery == “Several discovery activities are conducted (e.g., user research) but they are not or only loosely integrated with delivery activities.” ~ 8, clean_responses\(roadmap.discovery == "Close integration of discovery and delivery activities" ~ 10 | | | | Responsibility – Who is responsible for placing items on the roadmap? | Single-answer multiple choice | roadmap.responsibility | clean_responses\)roadmap.responsibility == “Tools are used to decide if items are placed on the roadmap (e.g., decision matrix).” ~ 1, clean_responses\(roadmap.responsibility == "Management" ~ 2, clean_responses\)roadmap.responsibility == “Specific roles (e.g., portfolio manager)” ~ 2, clean_responses\(roadmap.responsibility == "Product Management" ~ 3, clean_responses\)roadmap.responsibility == “Product Management with cross-functional product teams in liaison with key stakeholders.” ~ 4
Prioritization - How do you prioritise items? Single-answer multiple choice roadmap.prioritization clean_responses\(roadmap.prioritization == "First in, first out" ~ 1, clean_responses\)roadmap.prioritization == “Opinions determine priority” ~ 2, clean_responses\(roadmap.prioritization == "Prioritization is based on the capability to deliver (e.g., low hanging fruits)" ~ 3, clean_responses\)roadmap.prioritization == “Prioritization is based on short term benefit (e.g., shareholder value).” ~ 3, clean_responses\(roadmap.prioritization == "Prioritization is done with an established process and focuses on delivering value to customers and the business." ~ 7 | | | | Extent of alignment - How do you align your stakeholders? | Single-answer multiple choice | roadmap.alignment | clean_responses\)roadmap.alignment == “No alignment. No one or only one stakeholder such as high level management has a product roadmap that is not communicated to others.” ~ 1, clean_responses\(roadmap.alignment == "Several loosely connected product roadmaps for internal stakeholders exist." ~ 2, clean_responses\)roadmap.alignment == “Several loosely connected product roadmaps for internal and external stakeholders exist.” ~ 2, clean_responses\(roadmap.alignment == "One central product roadmap exists for different internal and external stakeholders." ~ 3, clean_responses\)roadmap.alignment == “One central product roadmap exists that allows to derive different representations for different stakeholders. A process for achieving alignment and buy-in is in place.” ~ 5
Ownership of the product roadmap - Who owns the product roadmap and is accountable? Single-answer multiple choice roadmap.ownership clean_responses\(roadmap.ownership == "No owner defined" ~ 1, clean_responses\)roadmap.ownership == “Managers” ~ 2, clean_responses\(roadmap.ownership == "Ownership is shared between multiple roles" ~ 3, clean_responses\)roadmap.ownership == “Strategy or portfolio planning” ~ 3, clean_responses$roadmap.ownership == “Product management or product teams” ~ 4
Tools - Which tool(s) do you use to manage your roadmap process? Multi-answer multiple choice (checkboxes) + Open answer roadmap.tools
Finally Where do you look for information about Software Product Management? (tick all that apply) Multi-answer multiple choice (checkboxes) + Open answer info.sources
Is there anything else you’d like to say about product management, strategy, and innovation in your working life? (the answers to this question will not be in the publicly available data set, and only published in aggregate analysis or quoted in an anonymised form) Open answer removed from analysis file

Descriptive Statistics

Demographics - Organisational Context

What is the make up of the organisations that the respondents are working in?

org.industry org.employees
Software / IT :38 < 10 :13
Banking / Finance :12 10-49 :19
Private / Consumer : 6 50-249 :13
Retail / Wholesale : 3 250-4499:16
Medical / Health Care: 2 >= 4500 :10
aquamarine : 1
(Other) : 9
org.TTM org.releases org.prodteamsize
Less than 4.5 months :28 More than 12 releases a year :28 < 4 :25
4.5 months to < 9 months :18 5-12 releases a year :19 4-9 :21
9 Months to < 18 months :12 3-4 releases a year :11 10-19 :14
More than 18 months : 5 About 2 releases per year : 5 20-49 : 5
Don’t know and cannot estimate: 8 About 1 release per year : 3 50-249: 3
Less than one release per year: 0 > 250 : 3
No release so far : 5

Demographics - Respondents

And where are they located?

Map

Location of respondents to the survey

Location data

clean_responses %>%
  group_by(org.location) %>%
  summarise(n = n()) %>%
  kable()
org.location n
Antigua and Barbuda 1
Brazil 1
Canada 2
Cyprus 1
Germany 1
Greece 1
India 7
Ireland 2
Japan 2
Mauritius 1
Mexico 1
Netherlands 2
New Zealand 1
Nigeria 2
Norway 4
Pakistan 1
Poland 1
Portugal 2
Qatar 1
Romania 1
Singapore 1
Spain 1
Sri Lanka 1
Sweden 2
Turkey 1
Ukraine 1
United Arab Emirates 1
United Kingdom 19
United States 8
Vietnam 1

Org and team size by location

          ggplot(clean_responses, aes(x=org.employees, y=org.prodteamsize)) + 
            geom_jitter(shape=16, position=position_jitter(0.2)) +
            facet_wrap(~org.location)

Role - Perceptions

How well did the respondents view their roadmap process and level of responsibility in the role?

Chart

Perception of role/roadmap

Data by job title

Job.title n roadmap.happiness role.happiness
Associate/Junior Product Manager 5 3.6
CEO 6 3.5 4.5
Continuous improvement specialist 1 2.0 3.0
CTO 5 4.8 5.0
Director of Product Design 1 5.0 4.0
Engineering Manager 2 4.0 5.0
FP&A 1 4.0 5.0
Freelancer ( Products & Growth ) 1 4.0 4.0
Head of Product/VP/Director/CPO 11 3.5 4.2
Lead Product Manager 6 4.3 4.3
Operational Manager 2 5.0 4.0
Product Consultant 1 4.0 4.0
Product Manager 15 3.5 3.8
Product Marketing Manager 1 3.0 5.0
Senior Product Manager 12 3.7 3.8
Senior Product Marketing Manager 1 5.0 5.0

Role - Responsibilities ISPMA model

Next we asked which activities we part of the respondent’s responsibilities based on the ISPMA’s v1.3 of the Framework

ISPMA’s Framework v1.3

All tables number of respondents who answered yes to the specified activities.

Strategic management

practices %>%
  select(Job.title, sm.corpstrat, sm.portfoliomgmt, sm.innovationmgmt, sm.resmgmt, sm.marketanalysis, sm.prodanalysis) %>%
  group_by(Job.title) %>%
  summarise("n" = n(),
            "Corporate Strategy" = sum(sm.corpstrat),
            "Portfolio Management" = sum(sm.portfoliomgmt),
            "Innovation Management" = sum(sm.innovationmgmt),
            "Resource Management" = sum(sm.resmgmt),
            "Market Analysis" = sum(sm.marketanalysis),
            "Product Analysis" = sum(sm.prodanalysis)) %>%
  kable()
Job.title n Corporate Strategy Portfolio Management Innovation Management Resource Management Market Analysis Product Analysis
Associate/Junior Product Manager 5 1 0 1 3 1 2
CEO 6 1 2 4 2 4 4
Continuous improvement specialist 1 1 0 1 0 0 0
CTO 5 3 2 5 4 2 2
Director of Product Design 1 0 1 1 1 0 0
Engineering Manager 2 1 0 0 1 0 2
FP&A 1 0 0 0 0 1 0
Freelancer ( Products & Growth ) 1 0 0 1 0 1 1
Head of Product/VP/Director/CPO 11 3 5 7 6 6 11
Lead Product Manager 6 3 4 5 3 4 4
Operational Manager 2 2 0 0 2 0 2
Product Consultant 1 0 0 1 0 1 1
Product Manager 15 9 7 10 9 11 13
Product Marketing Manager 1 0 0 1 0 0 1
Senior Product Manager 12 4 5 6 6 5 8
Senior Product Marketing Manager 1 1 0 0 0 1 1

Product Strategy

practices %>%
  select(roadmap.mat_level, ps.position, ps.deliverymodel, ps.sourcing, ps.bizcase, ps.pricing, ps.ecosystem, ps.legalandpr, ps.perfandrisk) %>%
  group_by(roadmap.mat_level) %>%
  summarise("n" = n(),
            "Positioning and product definition" = sum(ps.position),
            "Delivery model & Service strategy" = sum(ps.deliverymodel),
            "Sourcing" = sum(ps.sourcing),
            "Business case and costing" = sum(ps.bizcase),
            "Pricing" = sum(ps.pricing),
            "Ecosystem management" = sum(ps.ecosystem),
            "Legal & PR management" = sum(ps.legalandpr),
            "Performance & Risk management" = sum(ps.perfandrisk)) %>%
  kable()
roadmap.mat_level n Positioning and product definition Delivery model & Service strategy Sourcing Business case and costing Pricing Ecosystem management Legal & PR management Performance & Risk management
1 1 0 1 0 0 0 0 0 0
2 2 1 1 0 1 1 0 0 1
3 25 15 13 8 12 5 5 3 11
4 35 28 20 5 28 13 12 12 18
5 8 6 5 1 5 5 2 2 6

Product Planning

practices %>%
  select(roadmap.mat_level, pp.lifecycle, pp.roadmapping, pp.releaseplanning, pp.prodRE) %>%
  group_by(roadmap.mat_level) %>%
  summarise("n" = n(),
            "Product life-cycle management" = sum(pp.lifecycle),
            "Roadmapping" = sum(pp.roadmapping),
            "Release planning" = sum(pp.releaseplanning),
            "Product requirements engineering" = sum(pp.prodRE)) %>%
  kable()
roadmap.mat_level n Product life-cycle management Roadmapping Release planning Product requirements engineering
1 1 0 0 0 0
2 2 2 2 2 1
3 25 17 19 16 14
4 35 24 32 24 24
5 8 7 6 6 7

Development

practices %>%
            select(roadmap.mat_level, dev.engmgmt,dev.projmgmt, dev.projRE, dev.ux, dev.qual ) %>%
            group_by(roadmap.mat_level) %>%
            summarise("n" = n(),
                      "Engineering Management" = sum(dev.engmgmt),
                      "Project Management" = sum(dev.projmgmt),
                      "Project requirements engineering" = sum(dev.projRE),
                      "User experience design" = sum(dev.ux),
                      "Quality Management" = sum(dev.qual)) %>%
  kable()
roadmap.mat_level n Engineering Management Project Management Project requirements engineering User experience design Quality Management
1 1 0 0 0 0 1
2 2 1 0 0 0 1
3 25 9 16 9 18 13
4 35 9 22 16 18 12
5 8 4 6 3 4 3

Marketing

          practices %>%
            select(roadmap.mat_level, mar.plan,mar.cust, mar.oppomgmt, mar.mix, mar.gtm,mar.ops ) %>%
            group_by(roadmap.mat_level) %>%
            summarise("n" = n(),
                      "Marketing planning" = sum(mar.plan),
                      "Customer analysis" = sum(mar.cust),
                      "Opportunity Management" = sum(mar.oppomgmt),
                      "Marketing mix optimisation" = sum(mar.mix),
                      "Product launches (GTM)" = sum(mar.gtm),
                      "Operational marketing" = sum(mar.ops)) %>%
  kable()
roadmap.mat_level n Marketing planning Customer analysis Opportunity Management Marketing mix optimisation Product launches (GTM) Operational marketing
1 1 0 1 0 0 0 0
2 2 1 1 1 0 0 0
3 25 8 16 8 4 0 6
4 35 12 21 12 8 0 5
5 8 2 5 1 0 0 0

Sales and Distribution

          practices %>%
            select(roadmap.mat_level, sd.salesplan, sd.chanelprep, sd.crm, sd.salesops, sd.opdistro ) %>%
            group_by(roadmap.mat_level) %>%
            summarise("n" = n(),
                      "Sales planning" = sum(sd.salesplan),
                      "Channel prep" = sum(sd.chanelprep),
                      "CRM" = sum(sd.crm),
                      "Operational sales" = sum(sd.salesops),
                      "Operational distribution" = sum(sd.opdistro)) %>%
  kable()
roadmap.mat_level n Sales planning Channel prep CRM Operational sales Operational distribution
1 1 1 1 0 0 0
2 2 2 1 1 0 0
3 25 3 6 12 3 4
4 35 8 9 12 5 2
5 8 1 1 5 0 0

Service and Support

         practices %>%
            select(roadmap.mat_level, ss.serviceplan, ss.serviceprov, ss.techsupport, ss.marsupport, ss.salessupport) %>%
            group_by(roadmap.mat_level) %>%
            summarise("n" = n(),
                      "Service planning and preparation" = sum(ss.serviceplan),
                      "Service provisioning" = sum(ss.serviceprov),
                      "Technical support" = sum(ss.techsupport),
                      "Marketing support" = sum(ss.marsupport),
                      "Sales support" = sum(ss.salessupport)) %>%
  kable()
roadmap.mat_level n Service planning and preparation Service provisioning Technical support Marketing support Sales support
1 1 0 0 0 1 1
2 2 0 1 0 0 1
3 25 14 8 7 5 4
4 35 9 4 19 12 15
5 8 4 2 3 2 1

Information Sources

Looking at the transmission of theory to practice, where do people go to get answers? Which communities are they a part of?

Sources

Where do people go to get their information?

Job.title n Events and conferences Blogs Books Online Communities Professional Bodies Professional Certification Professional Training Tool Vendor material Google
Associate/Junior Product Manager 5 2 3 1 2 0 2 1 0 0
CEO 6 3 2 1 3 1 0 1 1 0
Continuous improvement specialist 1 0 0 0 0 0 0 0 0 1
CTO 5 3 3 3 3 1 1 1 2 0
Director of Product Design 1 0 0 0 1 0 0 0 0 0
Engineering Manager 2 1 1 0 1 0 0 0 0 0
FP&A 1 0 0 0 0 0 0 0 1 0
Freelancer ( Products & Growth ) 1 0 1 1 1 0 0 0 0 0
Head of Product/VP/Director/CPO 11 2 9 5 9 0 0 0 1 0
Lead Product Manager 6 4 5 4 3 0 0 0 2 0
Operational Manager 2 2 0 0 2 0 2 0 0 0
Product Consultant 1 0 1 1 1 0 0 0 1 0
Product Manager 15 6 13 10 12 1 1 4 5 0
Product Marketing Manager 1 0 1 0 0 0 0 0 0 0
Senior Product Manager 12 7 8 10 9 1 1 0 5 0
Senior Product Marketing Manager 1 0 1 1 1 0 0 0 1 0

Professional bodies

Which professional bodies are they members of?

Job.title n ACM AIPMM BCS ISPMA PDMA MTP WiP None
Associate/Junior Product Manager 5 0 0 0 0 0 3 0 2
CEO 6 0 0 0 1 1 1 0 4
Continuous improvement specialist 1 0 0 0 0 0 0 0 1
CTO 5 0 0 1 0 0 0 0 4
Director of Product Design 1 0 0 0 0 0 0 0 1
Engineering Manager 2 0 0 0 0 0 0 0 2
FP&A 1 0 0 0 0 0 0 0 1
Freelancer ( Products & Growth ) 1 0 0 0 0 0 0 0 1
Head of Product/VP/Director/CPO 11 0 1 0 0 0 0 0 10
Lead Product Manager 6 0 0 1 0 0 0 0 5
Operational Manager 2 0 0 0 0 0 0 0 2
Product Consultant 1 0 0 0 0 0 0 0 1
Product Manager 15 0 0 1 0 0 3 1 11
Product Marketing Manager 1 1 0 0 0 0 0 0 0
Senior Product Manager 12 0 0 0 0 0 2 2 9
Senior Product Marketing Manager 1 0 0 0 0 0 0 0 1

DEEP Roadmap Maturity

There were a spread of responses, but generally skewing to the right. In the charts below the least “mature” practices are to the left and the highest scoring, mature processes to the right. (According to DEEP Product Maturity Model V1.1 By Munch, Trieflinger and Lang.)

All responses

Product responses

By job title

How mature were the roadmap processes by job title of respondent?

Job.title n roadmap.DEEPScore roadmap.mat_level
Associate/Junior Product Manager 5 47 3.0
CEO 6 50 3.2
Continuous improvement specialist 1 62 4.0
CTO 5 60 3.6
Director of Product Design 1 64 4.0
Engineering Manager 2 65 4.0
FP&A 1 43 3.0
Freelancer ( Products & Growth ) 1 69 4.0
Head of Product/VP/Director/CPO 11 66 3.8
Lead Product Manager 6 78 4.3
Operational Manager 2 32 3.0
Product Consultant 1 80 4.0
Product Manager 15 61 3.8
Product Marketing Manager 1 61 4.0
Senior Product Manager 12 56 3.5
Senior Product Marketing Manager 1 68 4.0

Summary Stats

Distribution of DEEP scores.

cat("All responses")

All responses

xtable(t(summary(clean_responses[,"roadmap.DEEPScore"])))
Min. 1st Qu. Median Mean 3rd Qu. Max.
9 48 61 60 74 94 
cat("Product responses")

Product responses

xtable(t(summary(prod_responses[,"roadmap.DEEPScore"])))
Min. 1st Qu. Median Mean 3rd Qu. Max.
9 52 63 62 77 92 
cat("Non-product responses")

Non-product responses

xtable(t(summary(nonprod_responses[,"roadmap.DEEPScore"])))
Min. 1st Qu. Median Mean 3rd Qu. Max.
20 36 52 53 62 94

Tool usage

What kind of tools did the respondents use?

All

Percentage of tools types used

    tools %>%
            select(custom , office , product , project ) %>%
            summarise("Custom Tools" = sum(custom)/n()*100,
                      "Office Tools" = sum(office)/n()*100,
                      "Specialist Product Tools" = sum(product)/n()*100,
                      "Project management tools" = sum(project)/n()*100)  %>% 
  kable(digits = 2, title = "Percentage of tools types used")
Custom Tools Office Tools Specialist Product Tools Project management tools
5.6 42 24 46 
cat("count of tools broken out by category")

count of tools broken out by category

 toolsBreakdown <- data.frame(category = "Custom",
                             tool = "custom",
                             count = sum(tools[,"custom"]))
toolsBreakdown <- rbind(toolsBreakdown,data.frame(category = "Project",
                             tool = "ATLASSIAN: Jira/Confluence",
                             count = length(grep("Jira", tools[,7], ignore.case = TRUE))))
toolsBreakdown <- rbind(toolsBreakdown,data.frame(category = "Office",
                                                  tool = "Google Sheets",
                                                  count = length(grep("sheets", tools[,7], ignore.case = TRUE))))
toolsBreakdown <- rbind(toolsBreakdown,data.frame(category = "Office",
                                                  tool = "Microsoft Excel",
                                                  count = length(grep("Excel", tools[,7], ignore.case = TRUE))))
toolsBreakdown <- rbind(toolsBreakdown,data.frame(category = "Office",
                                                  tool = "Google Slides",
                                                  count = length(grep("Google Slides", tools[,7], ignore.case = TRUE))))
toolsBreakdown <- rbind(toolsBreakdown,data.frame(category = "Office",
                                                  tool = "MICROSOFT: PowerPoint",
                                                  count = length(grep("PowerPoint", tools[,7], ignore.case = TRUE))))

toolsBreakdown <- rbind(toolsBreakdown,data.frame(category = "Project",
                                                  tool = "MICROSOFT: Project Server",
                                                  count = length(grep("MICROSOFT: Project Server", tools[,7], ignore.case = TRUE))))
toolsBreakdown <- rbind(toolsBreakdown,data.frame(category = "Project",
                                                  tool = "taskstreamer",
                                                  count = length(grep("taskstreamer", tools[,7], ignore.case = TRUE))))
toolsBreakdown <- rbind(toolsBreakdown,data.frame(category = "Project",
                                                  tool = "ServiceNow",
                                                  count = length(grep("ServiceNowr", tools[,7], ignore.case = TRUE))))
toolsBreakdown <- rbind(toolsBreakdown,data.frame(category = "Project",
                                                  tool = "Planview",
                                                  count = length(grep("Planview", tools[,7], ignore.case = TRUE))))
toolsBreakdown <- rbind(toolsBreakdown,data.frame(category = "Project",
                                                  tool = "Trello",
                                                  count = length(grep("Trello", tools[,7], ignore.case = TRUE))))
toolsBreakdown <- rbind(toolsBreakdown,data.frame(category = "Product",
                                                  tool = "ITONICS: Roadmapping-Engine",
                                                  count = length(grep("roadmap", tools[,7], ignore.case = TRUE))))
toolsBreakdown <- rbind(toolsBreakdown,data.frame(category = "Product",
                                                  tool = "AHA!",
                                                  count = length(grep("AHA", tools[,7], ignore.case = TRUE))))
toolsBreakdown <- rbind(toolsBreakdown,data.frame(category = "Product",
                                                  tool = "Roadmunk",
                                                  count = length(grep("Roadmunk", tools[,7], ignore.case = TRUE))))
toolsBreakdown <- rbind(toolsBreakdown,data.frame(category = "Product",
                                                  tool = "ProductPlan",
                                                  count = length(grep("ProductPlan", tools[,7], ignore.case = TRUE))))
toolsBreakdown <- rbind(toolsBreakdown,data.frame(category = "Product",
                                                  tool = "Productboard",
                                                  count = length(grep("Productboard", tools[,7], ignore.case = TRUE))))
toolsBreakdown <- rbind(toolsBreakdown,data.frame(category = "Product",
                                                  tool = "Pendo",
                                                  count = length(grep("Pendo", tools[,7], ignore.case = TRUE))))
toolsBreakdown <- rbind(toolsBreakdown,data.frame(category = "Product",
                                                  tool = "ProdPad",
                                                  count = length(grep("ProdPad", tools[,7], ignore.case = TRUE))))

toolsBreakdown %>%
  group_by(category, tool, count) %>%
  arrange(category, tool, desc(count)) %>%
      kable
category tool count
Custom custom 4
Office Google Sheets 8
Office Google Slides 2
Office Microsoft Excel 23
Office MICROSOFT: PowerPoint 13
Product AHA! 2
Product ITONICS: Roadmapping-Engine 1
Product Pendo 0
Product ProdPad 8
Product Productboard 4
Product ProductPlan 2
Product Roadmunk 1
Project ATLASSIAN: Jira/Confluence 30
Project MICROSOFT: Project Server 4
Project Planview 0
Project ServiceNow 0
Project taskstreamer 1
Project Trello 1

Product

Percentage of tools types used

Custom Tools Office Tools Specialist Product Tools Project management tools
3.7 39 22 50

popularity of tool combinations

roadmap.tools n
ATLASSIAN: Jira/Confluence 13
4
PRODPAD: ProdPad 4
ATLASSIAN: Jira/Confluence;MICROSOFT: Excel 3
ATLASSIAN: Jira/Confluence;MICROSOFT: Excel;MICROSOFT: PowerPoint 3
MICROSOFT: Excel 3
ATLASSIAN: Jira/Confluence;GOOGLE: Google Sheets 2
MICROSOFT: Excel;MICROSOFT: PowerPoint 2
AHA LABS: AHA!;PRODUCTBOARD: Productboard 1
ATLASSIAN: Jira/Confluence;Google sheets 1
ATLASSIAN: Jira/Confluence;GOOGLE: Google Sheets;PRODUCTBOARD: Productboard 1
ATLASSIAN: Jira/Confluence;PRODPAD: ProdPad 1
Google Slides 1
Google Slides & Google Sheets 1
GOOGLE: Google Sheets 1
GOOGLE: Google Sheets;Clubhouse 1
in house solution 1
ITONICS: Roadmapping-Engine 1
JIRA 1
MICROSOFT: Excel;MICROSOFT: PowerPoint;MICROSOFT: Project Server 1
MICROSOFT: Excel;MICROSOFT: PowerPoint;PRODUCTPLAN: ProductPlan 1
MICROSOFT: Excel;MICROSOFT: Project Server 1
MICROSOFT: Excel;PRODPAD: ProdPad 1
Own platform 1
PRODUCTBOARD: Productboard 1
Roadmunk 1
ServiceNow 1
Trello, Wrike 1

Inferential Statistics

Did perception of roadmap process effectiveness correlate to a higher maturity score?

Happiness of roadmap vs DEEP score for product people

Call: lm(formula = roadmap.happiness ~ roadmap.DEEPScore, data = prod_responses)

Residuals: Min 1Q Median 3Q Max -2.5075 -0.6308 0.0314 0.6473 1.2995

Coefficients: Estimate Std. Error t value Pr(>|t|)
(Intercept) 2.37066 0.47436 5.00 6.9e-06 * roadmap.DEEPScore 0.02145 0.00734 2.92 0.0051 — Signif. codes: 0 ‘’ 0.001 ’’ 0.01 ’’ 0.05 ‘.’ 0.1 ’ ’ 1

Residual standard error: 0.95 on 52 degrees of freedom Multiple R-squared: 0.141, Adjusted R-squared: 0.125 F-statistic: 8.54 on 1 and 52 DF, p-value: 0.00513

Df Sum Sq Mean Sq F value Pr(>F)
roadmap.DEEPScore 1 7.8 7.79 8.5 0.01
Residuals 52 47.5 0.91 
> `geom_smooth()` using formula 'y ~ x'
> `geom_smooth()` using formula 'y ~ x'
> `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
> Warning: Removed 54 rows containing missing values (geom_text).

> Warning: Removed 1 rows containing missing values (geom_point).

References

  • A. Almeida, A. Loy, and H. Hofmann, qqplotr: Quantile-Quantile Plot Extensions for ‘ggplot2’, R package version 0.0.2 initially funded by Google Summer of Code 2017, https://github.com/aloy/qqplotr, 2017.
  • Baptiste Auguie (2017). gridExtra: Miscellaneous Functions for “Grid” Graphics. R package version 2.3. https://CRAN.R-project.org/package=gridExtra
  • David B. Dahl, David Scott, Charles Roosen, Arni Magnusson and Jonathan Swinton (2019). xtable: Export Tables to LaTeX or HTML. R package version 1.8-4. https://CRAN.R-project.org/package=xtable
  • Enzo Martoglio (2018). rpivotTable: Build Powerful Pivot Tables and Dynamically Slice & Dice your Data. R package version 0.3.0. https://CRAN.R-project.org/package=rpivotTable
  • H. Wickham. ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York, 2016.
  • Hadley Wickham (2020). tidyr: Tidy Messy Data. R package version 1.1.2. https://CRAN.R-project.org/package=tidyr
  • Hadley Wickham, Romain François, Lionel Henry and Kirill Müller (2020). dplyr: A Grammar of Data Manipulation. R package version 0.8.5. https://CRAN.R-project.org/package=dplyr
  • JJ Allaire and Yihui Xie and Jonathan McPherson and Javier Luraschi and Kevin Ushey and Aron Atkins and Hadley Wickham and Joe Cheng and Winston Chang and Richard Iannone (2020). rmarkdown: Dynamic Documents for R. R package version 2.6. URL https://rmarkdown.rstudio.com.
  • Pebesma, E.J., R.S. Bivand, 2005. Classes and methods for spatial data in R. R News 5 (2), https://cran.r-project.org/doc/Rnews/.
  • R Core Team (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.
  • South, Andy 2011 rworldmap: A New R package for Mapping Global Data. The R Journal Vol. 3/1 : 35-43.
  • Yihui Xie (2020). knitr: A General-Purpose Package for Dynamic Report Generation in R. R package version 1.28.
  • Horrock, I., Fowles, M (2012) ‘Block 2: Designing and doing your research’. T847: The MSc Professional Project. 30 credit Open University module for the postgraduate Technology Management programme. Milton Keynes, The Open University. Available at: https://learn2.open.ac.uk/mod/oucontent/view.php?id=1660368 [Accessed: 10 January 2021].
  • ISPMA (2020) ISPMA SPM Framework V.1.3 [online] ispma.org Available at: https://ispma.org/ispma-spm-framework-v-1-3/ [Accessed 31 December 2020]
  • Münch, J, Trieflinger, S and Lang, D (2019) ‘DEEP: the product roadmap maturity model: a method for assessing the product roadmapping capabilities of organizations’, in Proceedings of the 2nd ACM SIGSOFT International Workshop on software-intensive business: start-ups, platforms, and ecosystems. ACM, pp. 19–24. doi: 10.1145/3340481.3342733.
LS0tDQp0aXRsZTogIjIwMjEgUHJvZHVjdCBNYW5hZ2VtZW50IFN1cnZleSBvbiB0aGUgY3VycmVudCBzdGF0ZS1vZi1wcmFjdGljZSINCmF1dGhvcjogJ1tOZWlsIENoYWxrXShodHRwczovL3d3dy5yZXNlYXJjaGdhdGUubmV0L3Byb2ZpbGUvTmVpbF9DaGFsayknDQpkYXRlOiAiYHIgU3lzLkRhdGUoKWAiDQpvdXRwdXQ6DQogIGh0bWxfZG9jdW1lbnQ6DQogICAgdGhlbWU6IGNlcnVsZWFuDQogICAgaGlnaGxpZ2h0OiBweWdtZW50cw0KICAgIHRvYzogeWVzDQogICAgdG9jX2RlcHRoOiAzDQogICAgdG9jX2Zsb2F0OiB5ZXMNCiAgICBudW1iZXJfc2VjdGlvbnM6IG5vDQogICAgZGZfcHJpbnQ6IGthYmxlDQogICAgY29kZV9mb2xkaW5nOiBoaWRlDQogICAgY29kZV9kb3dubG9hZDogeWVzDQogIHdvcmRfZG9jdW1lbnQ6DQogICAgaGlnaGxpZ2h0OiBweWdtZW50cw0KICAgIHRvYzogbm8NCiAgICB0b2NfZGVwdGg6IDMNCiAgICBkZl9wcmludDoga2FibGUNCiAgICBudW1iZXJfc2VjdGlvbnM6IHllcw0KICBwZGZfZG9jdW1lbnQ6DQogICAgdG9jOiB5ZXMNCiAgICB0b2NfZGVwdGg6ICcyJw0KICAgIGxhdGV4X2VuZ2luZTogeGVsYXRleA0KZWRpdG9yX29wdGlvbnM6IA0KICBjaHVua19vdXRwdXRfdHlwZTogY29uc29sZQ0KLS0tDQoNCmBgYHtyIHNldHVwLCBpbmNsdWRlPUZBTFNFfQ0Ka25pdHI6Om9wdHNfY2h1bmskc2V0KGVjaG8gPSBUUlVFKQ0KbGlicmFyeShrbml0cikNCmxpYnJhcnkoZHBseXIpDQpsaWJyYXJ5KHh0YWJsZSkNCm9wdGlvbnMoeHRhYmxlLmNvbW1lbnQgPSBGQUxTRSkNCmBgYA0KDQojIFJlc2VhcmNoIE1ldGhvZG9sb2d5IHsudGFic2V0fQ0KDQpgYGB7ciBzdW1tYXJ5LCBlY2hvID0gRkFMU0UsIG1lc3NhZ2U9RkFMU0UsIHRpZHk9VFJVRSwgcmVzdWx0cz0nYXNpcyd9DQpzdW1tYXJ5KHJlcG9ydDo6cmVwb3J0KHNlc3Npb25JbmZvKCkpKQ0KYGBgDQoNCmBgYHtyIGNoaWxkPScuLi9jb2RlIGJvb2subWQnfQ0KYGBgDQoNCiMgRGVzY3JpcHRpdmUgU3RhdGlzdGljcw0KDQoNCiMjIERlbW9ncmFwaGljcyAtIE9yZ2FuaXNhdGlvbmFsIENvbnRleHQNCg0KV2hhdCBpcyB0aGUgbWFrZSB1cCBvZiB0aGUgb3JnYW5pc2F0aW9ucyB0aGF0IHRoZSByZXNwb25kZW50cyBhcmUgd29ya2luZyBpbj8NCg0KYGBge3Igb3JnLCBlY2hvPUZBTFNFfQ0KDQpzdW1tYXJ5KGNsZWFuX3Jlc3BvbnNlc1szOjRdKSAlPiUga2FibGUNCnN1bW1hcnkoY2xlYW5fcmVzcG9uc2VzWzU6N10pICU+JSBrYWJsZQ0KYGBgDQoNCg0KIyMgRGVtb2dyYXBoaWNzIC0gUmVzcG9uZGVudHMgey50YWJzZXR9DQoNCkFuZCB3aGVyZSBhcmUgdGhleSBsb2NhdGVkPw0KDQojIyMgTWFwDQoNCiFbTG9jYXRpb24gb2YgcmVzcG9uZGVudHMgdG8gdGhlIHN1cnZleV0oLi4vb3V0cHV0cy9SZXNwb25zZXNNYXAucG5nKQ0KDQojIyMgTG9jYXRpb24gZGF0YQ0KDQpgYGB7ciBsb2NhdGlvbl9kYXRhfQ0KY2xlYW5fcmVzcG9uc2VzICU+JQ0KICBncm91cF9ieShvcmcubG9jYXRpb24pICU+JQ0KICBzdW1tYXJpc2UobiA9IG4oKSkgJT4lDQogIGthYmxlKCkNCg0KYGBgDQoNCiMjIyBPcmcgYW5kIHRlYW0gc2l6ZSBieSBsb2NhdGlvbiANCg0KYGBge3IgbG9jYXRpb25fZGF0YV9zaXplc30NCg0KICAgICAgICAgIGdncGxvdChjbGVhbl9yZXNwb25zZXMsIGFlcyh4PW9yZy5lbXBsb3llZXMsIHk9b3JnLnByb2R0ZWFtc2l6ZSkpICsgDQogICAgICAgICAgICBnZW9tX2ppdHRlcihzaGFwZT0xNiwgcG9zaXRpb249cG9zaXRpb25faml0dGVyKDAuMikpICsNCiAgICAgICAgICAgIGZhY2V0X3dyYXAofm9yZy5sb2NhdGlvbikgICANCg0KYGBgDQoNCg0KIyMgUm9sZSAtIFBlcmNlcHRpb25zIHsudGFic2V0fSANCg0KSG93IHdlbGwgZGlkIHRoZSByZXNwb25kZW50cyB2aWV3IHRoZWlyIHJvYWRtYXAgcHJvY2VzcyBhbmQgbGV2ZWwgb2YgcmVzcG9uc2liaWxpdHkgaW4gdGhlIHJvbGU/DQoNCiMjIyBDaGFydA0KDQohW1BlcmNlcHRpb24gb2Ygcm9sZS9yb2FkbWFwXSguLi9vdXRwdXRzL0hhcHBpbmVzc19saWtlcnQucG5nKQ0KDQojIyMgRGF0YSBieSBqb2IgdGl0bGUNCg0KYGBge3Igcm9sZSwgZWNobz1GQUxTRX0NCmNsZWFuX3Jlc3BvbnNlcyAlPiUNCiAgc2VsZWN0KEpvYi50aXRsZSwgcm9hZG1hcC5oYXBwaW5lc3MsIHJvbGUuaGFwcGluZXNzICwgcm9hZG1hcC5ERUVQU2NvcmUsIHJvYWRtYXAubWF0X2xldmVsKSAlPiUgDQogIGdyb3VwX2J5KEpvYi50aXRsZSkgJT4lDQogIHN1bW1hcmlzZShuID0gbigpLA0KICAgICAgICAgICAgcm9hZG1hcC5oYXBwaW5lc3MgPSBtZWFuKHJvYWRtYXAuaGFwcGluZXNzKSwNCiAgICAgICAgICAgIHJvbGUuaGFwcGluZXNzID0gbWVhbihyb2xlLmhhcHBpbmVzcykpICAlPiUgDQogIGthYmxlKGRpZ2l0cyA9IDIpDQoNCg0KIA0KDQpgYGANCg0KIyMgUm9sZSAtIFJlc3BvbnNpYmlsaXRpZXMgSVNQTUEgbW9kZWwgIHsudGFic2V0fQ0KDQpOZXh0IHdlIGFza2VkIHdoaWNoIGFjdGl2aXRpZXMgd2UgcGFydCBvZiB0aGUgcmVzcG9uZGVudCdzIHJlc3BvbnNpYmlsaXRpZXMgYmFzZWQgb24gdGhlIElTUE1BJ3MgdjEuMyBvZiB0aGUgRnJhbWV3b3JrDQoNCiFbSVNQTUEncyBGcmFtZXdvcmsgdjEuMyBdKC4uL3N0YXRpYy9JU1BNQS1TUE0tRnJhbWV3b3JrLVYuMS4zLnBuZykNCg0KQWxsIHRhYmxlcyBudW1iZXIgb2YgcmVzcG9uZGVudHMgd2hvIGFuc3dlcmVkIHllcyB0byB0aGUgc3BlY2lmaWVkIGFjdGl2aXRpZXMuDQoNCiMjIyBTdHJhdGVnaWMgbWFuYWdlbWVudA0KDQpgYGB7cn0NCg0KcHJhY3RpY2VzICU+JQ0KICBzZWxlY3QoSm9iLnRpdGxlLCBzbS5jb3Jwc3RyYXQsIHNtLnBvcnRmb2xpb21nbXQsIHNtLmlubm92YXRpb25tZ210LCBzbS5yZXNtZ210LCBzbS5tYXJrZXRhbmFseXNpcywgc20ucHJvZGFuYWx5c2lzKSAlPiUNCiAgZ3JvdXBfYnkoSm9iLnRpdGxlKSAlPiUNCiAgc3VtbWFyaXNlKCJuIiA9IG4oKSwNCiAgICAgICAgICAgICJDb3Jwb3JhdGUgU3RyYXRlZ3kiID0gc3VtKHNtLmNvcnBzdHJhdCksDQogICAgICAgICAgICAiUG9ydGZvbGlvIE1hbmFnZW1lbnQiID0gc3VtKHNtLnBvcnRmb2xpb21nbXQpLA0KICAgICAgICAgICAgIklubm92YXRpb24gTWFuYWdlbWVudCIgPSBzdW0oc20uaW5ub3ZhdGlvbm1nbXQpLA0KICAgICAgICAgICAgIlJlc291cmNlIE1hbmFnZW1lbnQiID0gc3VtKHNtLnJlc21nbXQpLA0KICAgICAgICAgICAgIk1hcmtldCBBbmFseXNpcyIgPSBzdW0oc20ubWFya2V0YW5hbHlzaXMpLA0KICAgICAgICAgICAgIlByb2R1Y3QgQW5hbHlzaXMiID0gc3VtKHNtLnByb2RhbmFseXNpcykpICU+JQ0KICBrYWJsZSgpDQoNCg0KYGBgDQoNCiMjIyBQcm9kdWN0IFN0cmF0ZWd5DQoNCmBgYHtyfQ0KcHJhY3RpY2VzICU+JQ0KICBzZWxlY3Qocm9hZG1hcC5tYXRfbGV2ZWwsIHBzLnBvc2l0aW9uLCBwcy5kZWxpdmVyeW1vZGVsLCBwcy5zb3VyY2luZywgcHMuYml6Y2FzZSwgcHMucHJpY2luZywgcHMuZWNvc3lzdGVtLCBwcy5sZWdhbGFuZHByLCBwcy5wZXJmYW5kcmlzaykgJT4lDQogIGdyb3VwX2J5KHJvYWRtYXAubWF0X2xldmVsKSAlPiUNCiAgc3VtbWFyaXNlKCJuIiA9IG4oKSwNCiAgICAgICAgICAgICJQb3NpdGlvbmluZyBhbmQgcHJvZHVjdCBkZWZpbml0aW9uIiA9IHN1bShwcy5wb3NpdGlvbiksDQogICAgICAgICAgICAiRGVsaXZlcnkgbW9kZWwgJiBTZXJ2aWNlIHN0cmF0ZWd5IiA9IHN1bShwcy5kZWxpdmVyeW1vZGVsKSwNCiAgICAgICAgICAgICJTb3VyY2luZyIgPSBzdW0ocHMuc291cmNpbmcpLA0KICAgICAgICAgICAgIkJ1c2luZXNzIGNhc2UgYW5kIGNvc3RpbmciID0gc3VtKHBzLmJpemNhc2UpLA0KICAgICAgICAgICAgIlByaWNpbmciID0gc3VtKHBzLnByaWNpbmcpLA0KICAgICAgICAgICAgIkVjb3N5c3RlbSBtYW5hZ2VtZW50IiA9IHN1bShwcy5lY29zeXN0ZW0pLA0KICAgICAgICAgICAgIkxlZ2FsICYgUFIgbWFuYWdlbWVudCIgPSBzdW0ocHMubGVnYWxhbmRwciksDQogICAgICAgICAgICAiUGVyZm9ybWFuY2UgJiBSaXNrIG1hbmFnZW1lbnQiID0gc3VtKHBzLnBlcmZhbmRyaXNrKSkgJT4lDQogIGthYmxlKCkNCg0KDQpgYGANCg0KIyMjIFByb2R1Y3QgUGxhbm5pbmcNCg0KYGBge3J9DQpwcmFjdGljZXMgJT4lDQogIHNlbGVjdChyb2FkbWFwLm1hdF9sZXZlbCwgcHAubGlmZWN5Y2xlLCBwcC5yb2FkbWFwcGluZywgcHAucmVsZWFzZXBsYW5uaW5nLCBwcC5wcm9kUkUpICU+JQ0KICBncm91cF9ieShyb2FkbWFwLm1hdF9sZXZlbCkgJT4lDQogIHN1bW1hcmlzZSgibiIgPSBuKCksDQogICAgICAgICAgICAiUHJvZHVjdCBsaWZlLWN5Y2xlIG1hbmFnZW1lbnQiID0gc3VtKHBwLmxpZmVjeWNsZSksDQogICAgICAgICAgICAiUm9hZG1hcHBpbmciID0gc3VtKHBwLnJvYWRtYXBwaW5nKSwNCiAgICAgICAgICAgICJSZWxlYXNlIHBsYW5uaW5nIiA9IHN1bShwcC5yZWxlYXNlcGxhbm5pbmcpLA0KICAgICAgICAgICAgIlByb2R1Y3QgcmVxdWlyZW1lbnRzIGVuZ2luZWVyaW5nIiA9IHN1bShwcC5wcm9kUkUpKSAlPiUNCiAga2FibGUoKQ0KDQoNCmBgYA0KDQojIyMgRGV2ZWxvcG1lbnQNCg0KYGBge3J9DQoNCnByYWN0aWNlcyAlPiUNCiAgICAgICAgICAgIHNlbGVjdChyb2FkbWFwLm1hdF9sZXZlbCwgZGV2LmVuZ21nbXQsZGV2LnByb2ptZ210LCBkZXYucHJvalJFLCBkZXYudXgsIGRldi5xdWFsICkgJT4lDQogICAgICAgICAgICBncm91cF9ieShyb2FkbWFwLm1hdF9sZXZlbCkgJT4lDQogICAgICAgICAgICBzdW1tYXJpc2UoIm4iID0gbigpLA0KICAgICAgICAgICAgICAgICAgICAgICJFbmdpbmVlcmluZyBNYW5hZ2VtZW50IiA9IHN1bShkZXYuZW5nbWdtdCksDQogICAgICAgICAgICAgICAgICAgICAgIlByb2plY3QgTWFuYWdlbWVudCIgPSBzdW0oZGV2LnByb2ptZ210KSwNCiAgICAgICAgICAgICAgICAgICAgICAiUHJvamVjdCByZXF1aXJlbWVudHMgZW5naW5lZXJpbmciID0gc3VtKGRldi5wcm9qUkUpLA0KICAgICAgICAgICAgICAgICAgICAgICJVc2VyIGV4cGVyaWVuY2UgZGVzaWduIiA9IHN1bShkZXYudXgpLA0KICAgICAgICAgICAgICAgICAgICAgICJRdWFsaXR5IE1hbmFnZW1lbnQiID0gc3VtKGRldi5xdWFsKSkgJT4lDQogIGthYmxlKCkNCiAgICAgICAgDQoNCmBgYA0KDQojIyMgTWFya2V0aW5nDQoNCmBgYHtyIG1hcn0NCg0KICAgICAgICAgIHByYWN0aWNlcyAlPiUNCiAgICAgICAgICAgIHNlbGVjdChyb2FkbWFwLm1hdF9sZXZlbCwgbWFyLnBsYW4sbWFyLmN1c3QsIG1hci5vcHBvbWdtdCwgbWFyLm1peCwgbWFyLmd0bSxtYXIub3BzICkgJT4lDQogICAgICAgICAgICBncm91cF9ieShyb2FkbWFwLm1hdF9sZXZlbCkgJT4lDQogICAgICAgICAgICBzdW1tYXJpc2UoIm4iID0gbigpLA0KICAgICAgICAgICAgICAgICAgICAgICJNYXJrZXRpbmcgcGxhbm5pbmciID0gc3VtKG1hci5wbGFuKSwNCiAgICAgICAgICAgICAgICAgICAgICAiQ3VzdG9tZXIgYW5hbHlzaXMiID0gc3VtKG1hci5jdXN0KSwNCiAgICAgICAgICAgICAgICAgICAgICAiT3Bwb3J0dW5pdHkgTWFuYWdlbWVudCIgPSBzdW0obWFyLm9wcG9tZ210KSwNCiAgICAgICAgICAgICAgICAgICAgICAiTWFya2V0aW5nIG1peCBvcHRpbWlzYXRpb24iID0gc3VtKG1hci5taXgpLA0KICAgICAgICAgICAgICAgICAgICAgICJQcm9kdWN0IGxhdW5jaGVzIChHVE0pIiA9IHN1bShtYXIuZ3RtKSwNCiAgICAgICAgICAgICAgICAgICAgICAiT3BlcmF0aW9uYWwgbWFya2V0aW5nIiA9IHN1bShtYXIub3BzKSkgJT4lDQogIGthYmxlKCkNCg0KYGBgDQoNCiMjIyBTYWxlcyBhbmQgRGlzdHJpYnV0aW9uDQoNCmBgYHtyfQ0KDQogICAgICAgICAgcHJhY3RpY2VzICU+JQ0KICAgICAgICAgICAgc2VsZWN0KHJvYWRtYXAubWF0X2xldmVsLCBzZC5zYWxlc3BsYW4sIHNkLmNoYW5lbHByZXAsIHNkLmNybSwgc2Quc2FsZXNvcHMsIHNkLm9wZGlzdHJvICkgJT4lDQogICAgICAgICAgICBncm91cF9ieShyb2FkbWFwLm1hdF9sZXZlbCkgJT4lDQogICAgICAgICAgICBzdW1tYXJpc2UoIm4iID0gbigpLA0KICAgICAgICAgICAgICAgICAgICAgICJTYWxlcyBwbGFubmluZyIgPSBzdW0oc2Quc2FsZXNwbGFuKSwNCiAgICAgICAgICAgICAgICAgICAgICAiQ2hhbm5lbCBwcmVwIiA9IHN1bShzZC5jaGFuZWxwcmVwKSwNCiAgICAgICAgICAgICAgICAgICAgICAiQ1JNIiA9IHN1bShzZC5jcm0pLA0KICAgICAgICAgICAgICAgICAgICAgICJPcGVyYXRpb25hbCBzYWxlcyIgPSBzdW0oc2Quc2FsZXNvcHMpLA0KICAgICAgICAgICAgICAgICAgICAgICJPcGVyYXRpb25hbCBkaXN0cmlidXRpb24iID0gc3VtKHNkLm9wZGlzdHJvKSkgJT4lDQogIGthYmxlKCkNCg0KYGBgDQoNCiMjIyBTZXJ2aWNlIGFuZCBTdXBwb3J0DQoNCmBgYHtyfQ0KDQogICAgICAgICBwcmFjdGljZXMgJT4lDQogICAgICAgICAgICBzZWxlY3Qocm9hZG1hcC5tYXRfbGV2ZWwsIHNzLnNlcnZpY2VwbGFuLCBzcy5zZXJ2aWNlcHJvdiwgc3MudGVjaHN1cHBvcnQsIHNzLm1hcnN1cHBvcnQsIHNzLnNhbGVzc3VwcG9ydCkgJT4lDQogICAgICAgICAgICBncm91cF9ieShyb2FkbWFwLm1hdF9sZXZlbCkgJT4lDQogICAgICAgICAgICBzdW1tYXJpc2UoIm4iID0gbigpLA0KICAgICAgICAgICAgICAgICAgICAgICJTZXJ2aWNlIHBsYW5uaW5nIGFuZCBwcmVwYXJhdGlvbiIgPSBzdW0oc3Muc2VydmljZXBsYW4pLA0KICAgICAgICAgICAgICAgICAgICAgICJTZXJ2aWNlIHByb3Zpc2lvbmluZyIgPSBzdW0oc3Muc2VydmljZXByb3YpLA0KICAgICAgICAgICAgICAgICAgICAgICJUZWNobmljYWwgc3VwcG9ydCIgPSBzdW0oc3MudGVjaHN1cHBvcnQpLA0KICAgICAgICAgICAgICAgICAgICAgICJNYXJrZXRpbmcgc3VwcG9ydCIgPSBzdW0oc3MubWFyc3VwcG9ydCksDQogICAgICAgICAgICAgICAgICAgICAgIlNhbGVzIHN1cHBvcnQiID0gc3VtKHNzLnNhbGVzc3VwcG9ydCkpICU+JQ0KICBrYWJsZSgpDQpgYGANCg0KDQojIyBJbmZvcm1hdGlvbiBTb3VyY2VzIHsudGFic2V0fQ0KDQpMb29raW5nIGF0IHRoZSB0cmFuc21pc3Npb24gb2YgdGhlb3J5IHRvIHByYWN0aWNlLCB3aGVyZSBkbyBwZW9wbGUgZ28gdG8gZ2V0IGFuc3dlcnM/IFdoaWNoIGNvbW11bml0aWVzIGFyZSB0aGV5IGEgcGFydCBvZj8NCg0KIyMjIFNvdXJjZXMgDQoNCldoZXJlIGRvIHBlb3BsZSBnbyB0byBnZXQgdGhlaXIgaW5mb3JtYXRpb24/IA0KDQpgYGB7ciByb2xlMiwgZWNobz1GQUxTRX0NCmNsZWFuX3Jlc3BvbnNlcyAlPiUNCiAgc2VsZWN0KEpvYi50aXRsZSwgaW5mby5ldmVudHMsIGluZm8uYmxvZ3MsIGluZm8uYm9va3MsIGluZm8uY29tbXVuaXRpZXMsIGluZm8ucHJvZmJvZHksIGluZm8ucHJvZmNlcnQsIGluZm8udHJhaW5pbmcsIGluZm8udmVuZG9yLCBpbmZvLmdvb2dsZSkgJT4lDQogIGdyb3VwX2J5KEpvYi50aXRsZSkgJT4lDQogIHN1bW1hcmlzZSgibiIgPSBuKCksDQogICAgICAgICAgICAiRXZlbnRzIGFuZCBjb25mZXJlbmNlcyIgPSBzdW0oaW5mby5ldmVudHMpLA0KICAgICAgICAgICAgIkJsb2dzIiA9IHN1bShpbmZvLmJsb2dzKSwNCiAgICAgICAgICAgICJCb29rcyIgPSBzdW0oaW5mby5ib29rcyksDQogICAgICAgICAgICAiT25saW5lIENvbW11bml0aWVzIiA9IHN1bShpbmZvLmNvbW11bml0aWVzKSwNCiAgICAgICAgICAgICJQcm9mZXNzaW9uYWwgQm9kaWVzIiA9IHN1bShpbmZvLnByb2Zib2R5KSwNCiAgICAgICAgICAgICJQcm9mZXNzaW9uYWwgQ2VydGlmaWNhdGlvbiIgPSBzdW0oaW5mby5wcm9mY2VydCksDQogICAgICAgICAgICAiUHJvZmVzc2lvbmFsIFRyYWluaW5nIiA9IHN1bShpbmZvLnRyYWluaW5nKSwNCiAgICAgICAgICAgICJUb29sIFZlbmRvciBtYXRlcmlhbCIgPSBzdW0oaW5mby52ZW5kb3IpLA0KICAgICAgICAgICAgIkdvb2dsZSIgPSBzdW0oaW5mby5nb29nbGUpKSAgJT4lIA0KICBrYWJsZShkaWdpdHMgPSAyKQ0KYGBgDQoNCiMjIyBQcm9mZXNzaW9uYWwgYm9kaWVzDQoNCldoaWNoIHByb2Zlc3Npb25hbCBib2RpZXMgYXJlIHRoZXkgbWVtYmVycyBvZj8gDQoNCmBgYHtyIHByb2ZiZHksIGVjaG89RkFMU0V9DQpjbGVhbl9yZXNwb25zZXMgJT4lDQogIHNlbGVjdChKb2IudGl0bGUsIHByb2Zib2R5LmFjbSwgcHJvZmJvZHkuYWlwbW0sIHByb2Zib2R5LmFwbSwgcHJvZmJvZHkuYmNzLCBwcm9mYm9keS5pYW9pcCwgcHJvZmJvZHkuaXNwbWEsIHByb2Zib2R5LnBkbWEsIHByb2Zib2R5Lk1UUCwgcHJvZmJvZHkuV2lQLCBwcm9mYm9keS5ub25lKSAlPiUNCiAgZ3JvdXBfYnkoSm9iLnRpdGxlKSAlPiUNCiAgc3VtbWFyaXNlKCJuIiA9IG4oKSwNCiAgICAgICAgICAgICJBQ00iID0gcm91bmQoc3VtKHByb2Zib2R5LmFjbSkpLA0KICAgICAgICAgICAgIkFJUE1NIiA9IHN1bShwcm9mYm9keS5haXBtbSksDQogICAgICAgICAgICAiQkNTIiA9IHN1bShwcm9mYm9keS5iY3MpLA0KICAgICAgICAgICAgIklTUE1BIiA9IHN1bShwcm9mYm9keS5pc3BtYSksDQogICAgICAgICAgICAiUERNQSIgPSBzdW0ocHJvZmJvZHkucGRtYSksDQogICAgICAgICAgICAiTVRQIiA9IHN1bShwcm9mYm9keS5NVFApLA0KICAgICAgICAgICAgIldpUCIgPSBzdW0ocHJvZmJvZHkuV2lQKSwNCiAgICAgICAgICAgICJOb25lIiA9IHN1bShwcm9mYm9keS5ub25lKSkgJT4lDQogIGthYmxlKGRpZ2l0cyA9IDIpDQpgYGANCg0KIyMgREVFUCBSb2FkbWFwIE1hdHVyaXR5IHsudGFic2V0fQ0KDQpUaGVyZSB3ZXJlIGEgc3ByZWFkIG9mIHJlc3BvbnNlcywgYnV0IGdlbmVyYWxseSBza2V3aW5nIHRvIHRoZSByaWdodC4gSW4gdGhlIGNoYXJ0cyBiZWxvdyB0aGUgbGVhc3QgIm1hdHVyZSIgcHJhY3RpY2VzIGFyZSB0byB0aGUgbGVmdCBhbmQgdGhlIGhpZ2hlc3Qgc2NvcmluZywgbWF0dXJlIHByb2Nlc3NlcyB0byB0aGUgcmlnaHQuIChBY2NvcmRpbmcgdG8gW0RFRVAgUHJvZHVjdCBNYXR1cml0eSBNb2RlbF0oaHR0cHM6Ly93d3cucmVzZWFyY2hnYXRlLm5ldC9wdWJsaWNhdGlvbi8zMzYwNzAxMTJfVGhlX1Byb2R1Y3RfUm9hZG1hcF9NYXR1cml0eV9Nb2RlbF9ERUVQX1ZhbGlkYXRpb25fb2ZfYV9NZXRob2RfZm9yX0Fzc2Vzc2luZ190aGVfUHJvZHVjdF9Sb2FkbWFwX0NhcGFiaWxpdGllc19vZl9Pcmdhbml6YXRpb25zKSBWMS4xIEJ5IE11bmNoLCBUcmllZmxpbmdlciBhbmQgTGFuZy4pDQoNCiMjIyBBbGwgcmVzcG9uc2VzDQoNCmBgYHtyIGRlZXBfcm9hZG1hcF9hbGwsIGVjaG89RkFMU0V9DQppdGVtcyA8LSBjbGVhbl9yZXNwb25zZXNbLDE4OjI2XQ0KY29sbHkgPC0gbmFtZXMoaXRlbXMpDQoNCg0KcGFyKG1mcm93PWMoMywzKSkNCg0KZm9yKGkgaW4gMTpuY29sKGl0ZW1zKSkgew0KICBwbG90KGl0ZW1zWyxpXSwgbWFpbiA9IGNvbGx5W2ldLCBheGVzPUZBTFNFKQ0KICBBeGlzKHNpZGU9MSwgbGFiZWxzPUZBTFNFKQ0KICBBeGlzKHNpZGU9MiwgbGFiZWxzPVRSVUUpDQp9DQoNCm10ZXh0KHBhc3RlKCJERUVQIHJlc3BvbnNlcywgbj0iICwgbnJvdyhpdGVtcykpLCBzaWRlPTEsIG91dGVyPVRSVUUsIGxpbmU9LTMpDQpgYGANCg0KIyMjIFByb2R1Y3QgcmVzcG9uc2VzDQoNCmBgYHtyIGRlZXBfcm9hZG1hcF9wcm9kLCBlY2hvPUZBTFNFfQ0KaXRlbXMgPC0gcHJvZF9yZXNwb25zZXNbLDE4OjI2XQ0KY29sbHkgPC0gbmFtZXMoaXRlbXMpDQoNCg0KcGFyKG1mcm93PWMoMywzKSkNCg0KZm9yKGkgaW4gMTpuY29sKGl0ZW1zKSkgew0KICBwbG90KGl0ZW1zWyxpXSwgbWFpbiA9IGNvbGx5W2ldLCBheGVzPUZBTFNFKQ0KICBBeGlzKHNpZGU9MSwgbGFiZWxzPUZBTFNFKQ0KICBBeGlzKHNpZGU9MiwgbGFiZWxzPVRSVUUpDQp9DQoNCm10ZXh0KHBhc3RlKCJERUVQIHJlc3BvbnNlcywgbj0iICwgbnJvdyhpdGVtcykpLCBzaWRlPTEsIG91dGVyPVRSVUUsIGxpbmU9LTMpDQpgYGANCg0KIyMjIEJ5IGpvYiB0aXRsZQ0KDQpIb3cgbWF0dXJlIHdlcmUgdGhlIHJvYWRtYXAgcHJvY2Vzc2VzIGJ5IGpvYiB0aXRsZSBvZiByZXNwb25kZW50PyANCg0KYGBge3Igcm9hZG1hcF9kZXMsIGVjaG89RkFMU0UsIHJlc3VsdHM9J2FzaXMnfQ0KICANCmNsZWFuX3Jlc3BvbnNlcyAlPiUNCiAgc2VsZWN0KEpvYi50aXRsZSwgcm9hZG1hcC5ERUVQU2NvcmUsIHJvYWRtYXAubWF0X2xldmVsKSAlPiUgDQogIGdyb3VwX2J5KEpvYi50aXRsZSkgJT4lDQogIHN1bW1hcmlzZShuID0gbigpLA0KICAgICAgICAgICAgcm9hZG1hcC5ERUVQU2NvcmUgPSBtZWFuKHJvYWRtYXAuREVFUFNjb3JlKSwNCiAgICAgICAgICAgIHJvYWRtYXAubWF0X2xldmVsID0gbWVhbihyb2FkbWFwLm1hdF9sZXZlbCkpICAlPiUgDQogIGthYmxlKGRpZ2l0cyA9IDIpDQoNCg0KYGBgDQoNCiMjIyBTdW1tYXJ5IFN0YXRzDQoNCkRpc3RyaWJ1dGlvbiBvZiBERUVQIHNjb3Jlcy4NCg0KYGBge3IgcmVzdWx0cz0nYXNpcyd9DQpjYXQoIkFsbCByZXNwb25zZXMiKQ0KeHRhYmxlKHQoc3VtbWFyeShjbGVhbl9yZXNwb25zZXNbLCJyb2FkbWFwLkRFRVBTY29yZSJdKSkpIA0KY2F0KCJQcm9kdWN0IHJlc3BvbnNlcyIpDQp4dGFibGUodChzdW1tYXJ5KHByb2RfcmVzcG9uc2VzWywicm9hZG1hcC5ERUVQU2NvcmUiXSkpKSANCmNhdCgiTm9uLXByb2R1Y3QgcmVzcG9uc2VzIikNCnh0YWJsZSh0KHN1bW1hcnkobm9ucHJvZF9yZXNwb25zZXNbLCJyb2FkbWFwLkRFRVBTY29yZSJdKSkpDQoNCmBgYA0KDQoNCiMjIFRvb2wgdXNhZ2Ugey50YWJzZXR9DQoNCldoYXQga2luZCBvZiB0b29scyBkaWQgdGhlIHJlc3BvbmRlbnRzIHVzZT8gDQoNCiMjIyBBbGwNCg0KUGVyY2VudGFnZSBvZiB0b29scyB0eXBlcyB1c2VkDQoNCmBgYHtyIHRvb2xzX2FsbCwgcmVzdWx0cz0nYXNpcyd9DQogIA0KICAgIHRvb2xzICU+JQ0KICAgICAgICAgICAgc2VsZWN0KGN1c3RvbSAsIG9mZmljZSAsIHByb2R1Y3QgLCBwcm9qZWN0ICkgJT4lDQogICAgICAgICAgICBzdW1tYXJpc2UoIkN1c3RvbSBUb29scyIgPSBzdW0oY3VzdG9tKS9uKCkqMTAwLA0KICAgICAgICAgICAgICAgICAgICAgICJPZmZpY2UgVG9vbHMiID0gc3VtKG9mZmljZSkvbigpKjEwMCwNCiAgICAgICAgICAgICAgICAgICAgICAiU3BlY2lhbGlzdCBQcm9kdWN0IFRvb2xzIiA9IHN1bShwcm9kdWN0KS9uKCkqMTAwLA0KICAgICAgICAgICAgICAgICAgICAgICJQcm9qZWN0IG1hbmFnZW1lbnQgdG9vbHMiID0gc3VtKHByb2plY3QpL24oKSoxMDApICAlPiUgDQogIGthYmxlKGRpZ2l0cyA9IDIsIHRpdGxlID0gIlBlcmNlbnRhZ2Ugb2YgdG9vbHMgdHlwZXMgdXNlZCIpDQoNCmNhdCgiY291bnQgb2YgdG9vbHMgYnJva2VuIG91dCBieSBjYXRlZ29yeSIpDQoNCiB0b29sc0JyZWFrZG93biA8LSBkYXRhLmZyYW1lKGNhdGVnb3J5ID0gIkN1c3RvbSIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRvb2wgPSAiY3VzdG9tIiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY291bnQgPSBzdW0odG9vbHNbLCJjdXN0b20iXSkpDQp0b29sc0JyZWFrZG93biA8LSByYmluZCh0b29sc0JyZWFrZG93bixkYXRhLmZyYW1lKGNhdGVnb3J5ID0gIlByb2plY3QiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0b29sID0gIkFUTEFTU0lBTjogSmlyYS9Db25mbHVlbmNlIiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY291bnQgPSBsZW5ndGgoZ3JlcCgiSmlyYSIsIHRvb2xzWyw3XSwgaWdub3JlLmNhc2UgPSBUUlVFKSkpKQ0KdG9vbHNCcmVha2Rvd24gPC0gcmJpbmQodG9vbHNCcmVha2Rvd24sZGF0YS5mcmFtZShjYXRlZ29yeSA9ICJPZmZpY2UiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0b29sID0gIkdvb2dsZSBTaGVldHMiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb3VudCA9IGxlbmd0aChncmVwKCJzaGVldHMiLCB0b29sc1ssN10sIGlnbm9yZS5jYXNlID0gVFJVRSkpKSkNCnRvb2xzQnJlYWtkb3duIDwtIHJiaW5kKHRvb2xzQnJlYWtkb3duLGRhdGEuZnJhbWUoY2F0ZWdvcnkgPSAiT2ZmaWNlIiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdG9vbCA9ICJNaWNyb3NvZnQgRXhjZWwiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb3VudCA9IGxlbmd0aChncmVwKCJFeGNlbCIsIHRvb2xzWyw3XSwgaWdub3JlLmNhc2UgPSBUUlVFKSkpKQ0KdG9vbHNCcmVha2Rvd24gPC0gcmJpbmQodG9vbHNCcmVha2Rvd24sZGF0YS5mcmFtZShjYXRlZ29yeSA9ICJPZmZpY2UiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0b29sID0gIkdvb2dsZSBTbGlkZXMiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb3VudCA9IGxlbmd0aChncmVwKCJHb29nbGUgU2xpZGVzIiwgdG9vbHNbLDddLCBpZ25vcmUuY2FzZSA9IFRSVUUpKSkpDQp0b29sc0JyZWFrZG93biA8LSByYmluZCh0b29sc0JyZWFrZG93bixkYXRhLmZyYW1lKGNhdGVnb3J5ID0gIk9mZmljZSIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRvb2wgPSAiTUlDUk9TT0ZUOiBQb3dlclBvaW50IiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY291bnQgPSBsZW5ndGgoZ3JlcCgiUG93ZXJQb2ludCIsIHRvb2xzWyw3XSwgaWdub3JlLmNhc2UgPSBUUlVFKSkpKQ0KDQp0b29sc0JyZWFrZG93biA8LSByYmluZCh0b29sc0JyZWFrZG93bixkYXRhLmZyYW1lKGNhdGVnb3J5ID0gIlByb2plY3QiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0b29sID0gIk1JQ1JPU09GVDogUHJvamVjdCBTZXJ2ZXIiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb3VudCA9IGxlbmd0aChncmVwKCJNSUNST1NPRlQ6IFByb2plY3QgU2VydmVyIiwgdG9vbHNbLDddLCBpZ25vcmUuY2FzZSA9IFRSVUUpKSkpDQp0b29sc0JyZWFrZG93biA8LSByYmluZCh0b29sc0JyZWFrZG93bixkYXRhLmZyYW1lKGNhdGVnb3J5ID0gIlByb2plY3QiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0b29sID0gInRhc2tzdHJlYW1lciIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvdW50ID0gbGVuZ3RoKGdyZXAoInRhc2tzdHJlYW1lciIsIHRvb2xzWyw3XSwgaWdub3JlLmNhc2UgPSBUUlVFKSkpKQ0KdG9vbHNCcmVha2Rvd24gPC0gcmJpbmQodG9vbHNCcmVha2Rvd24sZGF0YS5mcmFtZShjYXRlZ29yeSA9ICJQcm9qZWN0IiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdG9vbCA9ICJTZXJ2aWNlTm93IiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY291bnQgPSBsZW5ndGgoZ3JlcCgiU2VydmljZU5vd3IiLCB0b29sc1ssN10sIGlnbm9yZS5jYXNlID0gVFJVRSkpKSkNCnRvb2xzQnJlYWtkb3duIDwtIHJiaW5kKHRvb2xzQnJlYWtkb3duLGRhdGEuZnJhbWUoY2F0ZWdvcnkgPSAiUHJvamVjdCIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRvb2wgPSAiUGxhbnZpZXciLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb3VudCA9IGxlbmd0aChncmVwKCJQbGFudmlldyIsIHRvb2xzWyw3XSwgaWdub3JlLmNhc2UgPSBUUlVFKSkpKQ0KdG9vbHNCcmVha2Rvd24gPC0gcmJpbmQodG9vbHNCcmVha2Rvd24sZGF0YS5mcmFtZShjYXRlZ29yeSA9ICJQcm9qZWN0IiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdG9vbCA9ICJUcmVsbG8iLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb3VudCA9IGxlbmd0aChncmVwKCJUcmVsbG8iLCB0b29sc1ssN10sIGlnbm9yZS5jYXNlID0gVFJVRSkpKSkNCnRvb2xzQnJlYWtkb3duIDwtIHJiaW5kKHRvb2xzQnJlYWtkb3duLGRhdGEuZnJhbWUoY2F0ZWdvcnkgPSAiUHJvZHVjdCIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRvb2wgPSAiSVRPTklDUzogUm9hZG1hcHBpbmctRW5naW5lIiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY291bnQgPSBsZW5ndGgoZ3JlcCgicm9hZG1hcCIsIHRvb2xzWyw3XSwgaWdub3JlLmNhc2UgPSBUUlVFKSkpKQ0KdG9vbHNCcmVha2Rvd24gPC0gcmJpbmQodG9vbHNCcmVha2Rvd24sZGF0YS5mcmFtZShjYXRlZ29yeSA9ICJQcm9kdWN0IiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdG9vbCA9ICJBSEEhIiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY291bnQgPSBsZW5ndGgoZ3JlcCgiQUhBIiwgdG9vbHNbLDddLCBpZ25vcmUuY2FzZSA9IFRSVUUpKSkpDQp0b29sc0JyZWFrZG93biA8LSByYmluZCh0b29sc0JyZWFrZG93bixkYXRhLmZyYW1lKGNhdGVnb3J5ID0gIlByb2R1Y3QiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0b29sID0gIlJvYWRtdW5rIiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY291bnQgPSBsZW5ndGgoZ3JlcCgiUm9hZG11bmsiLCB0b29sc1ssN10sIGlnbm9yZS5jYXNlID0gVFJVRSkpKSkNCnRvb2xzQnJlYWtkb3duIDwtIHJiaW5kKHRvb2xzQnJlYWtkb3duLGRhdGEuZnJhbWUoY2F0ZWdvcnkgPSAiUHJvZHVjdCIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRvb2wgPSAiUHJvZHVjdFBsYW4iLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb3VudCA9IGxlbmd0aChncmVwKCJQcm9kdWN0UGxhbiIsIHRvb2xzWyw3XSwgaWdub3JlLmNhc2UgPSBUUlVFKSkpKQ0KdG9vbHNCcmVha2Rvd24gPC0gcmJpbmQodG9vbHNCcmVha2Rvd24sZGF0YS5mcmFtZShjYXRlZ29yeSA9ICJQcm9kdWN0IiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdG9vbCA9ICJQcm9kdWN0Ym9hcmQiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb3VudCA9IGxlbmd0aChncmVwKCJQcm9kdWN0Ym9hcmQiLCB0b29sc1ssN10sIGlnbm9yZS5jYXNlID0gVFJVRSkpKSkNCnRvb2xzQnJlYWtkb3duIDwtIHJiaW5kKHRvb2xzQnJlYWtkb3duLGRhdGEuZnJhbWUoY2F0ZWdvcnkgPSAiUHJvZHVjdCIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRvb2wgPSAiUGVuZG8iLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb3VudCA9IGxlbmd0aChncmVwKCJQZW5kbyIsIHRvb2xzWyw3XSwgaWdub3JlLmNhc2UgPSBUUlVFKSkpKQ0KdG9vbHNCcmVha2Rvd24gPC0gcmJpbmQodG9vbHNCcmVha2Rvd24sZGF0YS5mcmFtZShjYXRlZ29yeSA9ICJQcm9kdWN0IiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdG9vbCA9ICJQcm9kUGFkIiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY291bnQgPSBsZW5ndGgoZ3JlcCgiUHJvZFBhZCIsIHRvb2xzWyw3XSwgaWdub3JlLmNhc2UgPSBUUlVFKSkpKQ0KDQp0b29sc0JyZWFrZG93biAlPiUNCiAgZ3JvdXBfYnkoY2F0ZWdvcnksIHRvb2wsIGNvdW50KSAlPiUNCiAgYXJyYW5nZShjYXRlZ29yeSwgdG9vbCwgZGVzYyhjb3VudCkpICU+JQ0KICAgICAga2FibGUNCg0KYGBgDQoNCiMjIyBQcm9kdWN0DQoNClBlcmNlbnRhZ2Ugb2YgdG9vbHMgdHlwZXMgdXNlZA0KDQpgYGB7ciB0b29sc19wbSwgZWNobz1GQUxTRSwgcmVzdWx0cz0nYXNpcyd9DQoNCiBwbS50b29scyA8LSAgICAgIHRvb2xzICU+JQ0KICAgICAgICAgICAgIGZpbHRlcihncmVwbCgnUHJvZHVjdCcsIEpvYi50aXRsZSkpIA0KICANCiAgICBwbS50b29scyAlPiUNCiAgICAgICAgICAgIHNlbGVjdChjdXN0b20gLCBvZmZpY2UgLCBwcm9kdWN0ICwgcHJvamVjdCApICU+JQ0KICAgICAgICAgICAgc3VtbWFyaXNlKCJDdXN0b20gVG9vbHMiID0gc3VtKGN1c3RvbSkvbigpKjEwMCwNCiAgICAgICAgICAgICAgICAgICAgICAiT2ZmaWNlIFRvb2xzIiA9IHN1bShvZmZpY2UpL24oKSoxMDAsDQogICAgICAgICAgICAgICAgICAgICAgIlNwZWNpYWxpc3QgUHJvZHVjdCBUb29scyIgPSBzdW0ocHJvZHVjdCkvbigpKjEwMCwNCiAgICAgICAgICAgICAgICAgICAgICAiUHJvamVjdCBtYW5hZ2VtZW50IHRvb2xzIiA9IHN1bShwcm9qZWN0KS9uKCkqMTAwKSAlPiUgDQogIGthYmxlKGRpZ2l0cyA9IDIsIHRpdGxlID0gIlBlcmNlbnRhZ2Ugb2YgdG9vbHMgdHlwZXMgdXNlZCIpDQogICAgDQogICAgY2F0KCJwb3B1bGFyaXR5IG9mIHRvb2wgY29tYmluYXRpb25zIikNCg0KICAgIHBtLnRvb2xzICU+JQ0KICBzZWxlY3Qocm9hZG1hcC50b29scyApICU+JQ0KICBncm91cF9ieShyb2FkbWFwLnRvb2xzKSU+JQ0KICB0YWxseShzb3J0ID0gVCkgJT4lDQogIHVuZ3JvdXAoKSAlPiUNCiAgYXJyYW5nZShkZXNjKG4pKSAlPiUNCiAgICAgIGthYmxlDQoNCmBgYA0KDQojIEluZmVyZW50aWFsIFN0YXRpc3RpY3MgDQoNCkRpZCBwZXJjZXB0aW9uIG9mIHJvYWRtYXAgcHJvY2VzcyBlZmZlY3RpdmVuZXNzIGNvcnJlbGF0ZSB0byBhIGhpZ2hlciBtYXR1cml0eSBzY29yZT8NCg0KIVtIYXBwaW5lc3Mgb2Ygcm9hZG1hcCB2cyBERUVQIHNjb3JlIGZvciBwcm9kdWN0IHBlb3BsZV0oLi4vb3V0cHV0cy9ERUVQc2NvcmVfdnNfcm9hZG1hcEhhcHBpbmVzc19wcm9kLnBuZykNCg0KYGBge3Igcm9hZG1hcF9pbmYsIGVjaG89RkFMU0UsIHJlc3VsdHM9J2FzaXMnfQ0KbGlicmFyeShxcXBsb3RyKQ0KbGlicmFyeShncmlkRXh0cmEpDQoNCiNjcmVhdGUgbW9kZWwgdG8gY2hlY2sgY29ycmVsYXRpb24NCiBsbUhhcHB5Um9hZG1hcCA9IGxtKHJvYWRtYXAuaGFwcGluZXNzIH4gcm9hZG1hcC5ERUVQU2NvcmUgLCBkYXRhID0gcHJvZF9yZXNwb25zZXMpDQoNCg0KZ2dwbG90KHByb2RfcmVzcG9uc2VzKSArDQogIGdlb21fcG9pbnQoYWVzKHggPSByb2FkbWFwLmhhcHBpbmVzcywgeSA9IHJvYWRtYXAuREVFUFNjb3JlLCBjb2xvdXIgPSBvcmcuaW5kdXN0cnksIHNoYXBlID0gb3JnLmVtcGxveWVlcykpIA0KDQogIHN1bW1hcnkobG1IYXBweVJvYWRtYXApDQogIGFub3ZhKGxtSGFwcHlSb2FkbWFwKQ0KcGVyZm9ybWFuY2U6OmNoZWNrX21vZGVsKGxtSGFwcHlSb2FkbWFwKQ0KDQpsbUhhcHB5Um9sZSA9IGxtKHJvbGUuaGFwcGluZXNzIH4gcm9hZG1hcC5ERUVQU2NvcmUgLCBkYXRhID0gY2xlYW5fcmVzcG9uc2VzKSANCiBnZ3Bsb3QoY2xlYW5fcmVzcG9uc2VzLGFlcyh4ID0gcm9hZG1hcC5ERUVQU2NvcmUsIHkgPSByb2xlLmhhcHBpbmVzcykpICsNCiAgICBnZW9tX3BvaW50KGFlcyh4ID0gcm9hZG1hcC5ERUVQU2NvcmUsIHkgPSByb2xlLmhhcHBpbmVzcywgY29sb3VyID0gb3JnLmluZHVzdHJ5LCBzaGFwZSA9IG9yZy5lbXBsb3llZXMpKSArDQogICAgZ2VvbV9hYmxpbmUoc2xvcGUgPSBjb2VmKGxtSGFwcHlSb2xlKVtbMl1dLCBpbnRlcmNlcHQgPSBjb2VmKGxtSGFwcHlSb2xlKVtbMV1dLCBjb2xvdXI9IiNDQzAwMDAiKSArDQogICAgZmFjZXRfd3JhcCh+Sm9iLnRpdGxlKQ0KIA0KYGBgDQoNCg0KDQojIyBSZWZlcmVuY2VzIA0KDQpgYGB7ciByZWZlcmVuY2VzLCBlY2hvPUZBTFNFLCBtZXNzYWdlPUZBTFNFLCBlcnJvcj1GQUxTRSwgcmVzdWx0cz0nYXNpcyd9DQoNCnJlcG9ydDo6Y2l0ZV9wYWNrYWdlcyhzZXNzaW9uSW5mbygpKQ0KYGBgDQorIEhvcnJvY2ssIEkuLCBGb3dsZXMsIE0gKDIwMTIpIOKAmEJsb2NrIDI6IERlc2lnbmluZyBhbmQgZG9pbmcgeW91ciByZXNlYXJjaOKAmS4gVDg0NzogVGhlIE1TYyBQcm9mZXNzaW9uYWwgUHJvamVjdC4gMzAgY3JlZGl0IE9wZW4gVW5pdmVyc2l0eSBtb2R1bGUgZm9yIHRoZSBwb3N0Z3JhZHVhdGUgVGVjaG5vbG9neSBNYW5hZ2VtZW50IHByb2dyYW1tZS4gTWlsdG9uIEtleW5lcywgVGhlIE9wZW4gVW5pdmVyc2l0eS4gQXZhaWxhYmxlIGF0OiBodHRwczovL2xlYXJuMi5vcGVuLmFjLnVrL21vZC9vdWNvbnRlbnQvdmlldy5waHA/aWQ9MTY2MDM2OCBbQWNjZXNzZWQ6IDEwIEphbnVhcnkgIDIwMjFdLg0KKyBJU1BNQSAoMjAyMCkgSVNQTUEgU1BNIEZyYW1ld29yayBWLjEuMyBbb25saW5lXSBpc3BtYS5vcmcgQXZhaWxhYmxlIGF0OiAgaHR0cHM6Ly9pc3BtYS5vcmcvaXNwbWEtc3BtLWZyYW1ld29yay12LTEtMy8gW0FjY2Vzc2VkIDMxIERlY2VtYmVyIDIwMjBdDQorIE3DvG5jaCwgSiwgVHJpZWZsaW5nZXIsIFMgYW5kIExhbmcsIEQgKDIwMTkpIOKAmERFRVA6IHRoZSBwcm9kdWN0IHJvYWRtYXAgbWF0dXJpdHkgbW9kZWw6IGEgbWV0aG9kIGZvciBhc3Nlc3NpbmcgdGhlIHByb2R1Y3Qgcm9hZG1hcHBpbmcgY2FwYWJpbGl0aWVzIG9mIG9yZ2FuaXphdGlvbnPigJksIGluIFByb2NlZWRpbmdzIG9mIHRoZSAybmQgQUNNIFNJR1NPRlQgSW50ZXJuYXRpb25hbCBXb3Jrc2hvcCBvbiBzb2Z0d2FyZS1pbnRlbnNpdmUgYnVzaW5lc3M6IHN0YXJ0LXVwcywgcGxhdGZvcm1zLCBhbmQgZWNvc3lzdGVtcy4gQUNNLCBwcC4gMTnigJMyNC4gZG9pOiAxMC4xMTQ1LzMzNDA0ODEuMzM0MjczMy4=