NoDW #16: Developing The DAO Health Pulse

In this Newsletter of Decentralized Work edition:

1. Developing the DAO Health Pulse

2. Elsewhere in Web3

---

Developing the DAO Health Pulse: reduction to a more convenient diagnostic

Authors: Navojit Roy, Kenneth Francis

In April of 2022, talentDAO open-sourced the DAO Health Survey with the intention of providing a reliable instrument for DAOs to assess the health of their organization. Since then, we’ve received a great deal of feedback from the community. Coupled with the help of sample data from partnering DAOs like Gitcoin and MoonDAO, we developed the next iteration of the tool: the DAO Health Pulse Survey.

The original DAO Health Survey was designed to assess key challenges facing DAOs, primarily their ability to operate effectively using decentralized structures and procedures. The development process involved leaning into decades of research on organizational psychology and psychometrics while working to ensure each question was grounded in the practical realities of contributing to DAOs. In addition to open-sourcing the survey, the methods and procedures used in its development were published on Mirror.

Measures in the DAO Health Survey fit into three overarching factors: Distributed Team Model, DAO Culture, Contributor Attitudes and Perceptions. Each of these factors consists of several sub-dimensions. A high-level breakdown is shown in Figure 1. Because our original intention was to design a comprehensive survey spanning these factors and dimensions, the final product was a somewhat long survey with about 50 questions.

Figure 1. talentDAO’s model of DAO Health contains 3 factors and 14 sub-dimensions spanning the organization-level, team-level, and individual-level.

The tool is useful for gleaning robust insights about organizational health but its length is inconvenient.  Thus, we set out to build a pulse version.

The remainder of this write-up is organized as follows:

• Operationalizing DAO Health: how we defined DAOs and DAO Health in order to measure them effectively.

• Item reduction process: how we conducted a factor analysis on the DAO Health Survey and further interpreted the results

• Survey Design: a description of the shortened version of the DAO Health Survey.

Operationalizing DAO Health

In our first study on DAO Health, we reviewed a wide range of literature on topics related to organizational health, organizational effectiveness, shared leadership, virtual teams, and worker well-being. After synthesizing all of this, we established the following operational definition of a DAO:

a network of contributors coordinating in dynamic virtual teams toward a shared purpose by decentralizing authority and ownership.

And the following operational definition of DAO Health:

the DAO's ability to coordinate teams toward a shared vision and objectives as a function of the contributor socio-psychological factors that drive collective productivity, performance, leadership, and individual experience and well-being

Item reduction process

The initial step for reducing the number of items on our survey was to run a factor analysis. Factor analysis is a data-driven methodology wherein the results can indicate the significant factors present within a given dataset (Tavaklov, 2020). A factor analysis results in a matrix of loadings for each survey item on each hypothesized factor. Factor loadings for a given item quantify the extent to which the item is related to a given factor, and in our case, a sub-dimension as well. (Bandolos, 2017).

Doing this could help us understand: 

1. Were the original questions we designed and hypothesized to be part of the same factor and sub-dimension loading on those factors and sub-dimensions respectively? 

2. Could we identify questions that either (a) did not load onto any other factors, (b) were too similar to other questions, and/or (c) did not have practical benefits given their respective loadings?

We had a sample of approximately 80 anonymized survey results from three distinct DAOs that agreed to share their survey data for research purposes. The data was combined to conduct a single factor analysis. All of our data, code, and analysis can be found on Github.

For the purpose of our survey we conducted Exploratory Factor Analysis (EFA), which aims to assess how many factors are present in our dataset. This is assessed by the resulting eigenvalues for each factor in the output. Consensus theory states that eigenvalues close to or less than one are not significant and that values higher than 1 are significant (Tavaklov, 2020).

The initial DAO Health Survey aimed to assess 20 factors, each of which containing a subscale for their respective sub-dimensions. After running our factor analysis, the data revealed eight significant factors. The factor analysis showed a significant drop-off after the first two factors. This means that there were high correlations among our original 20 factors. 

Factor analysis does not, however,  tell us what our factors are. Rather, we learn which things are related enough such that they can be considered the same factor.

We proceeded to define the eight factors by first removing survey items with weak factor loadings (<0.40) We used a consensus factor loading cutoff of 0.40 which helps ensure selected items accurately represent the factor of interest (Stevens, 1992). We then assessed how many items from each of the 20 subscales remained significantly related to one of the eight factors. 

We wanted to ensure that the shortened version of the survey did not lose any key constructs through the reduction process. Hence, upon reviewing the results we aimed to synthesize the items contained within each factor to assess what was the theoretical overlap that caused the high degree of correlations. 

Ultimately, the DAO Health Survey aims to be a tool for contributors to assess the effectiveness of their DAO. This means making decisions about which subdimensions to include that don’t perfectly correspond to loading values. For example, the first factor had loadings from 19 of 49 questions. While this may be a positive signal for the internal consistency of the survey, there is a practical argument for either breaking some of these out into more interpretable item sets and/or removing items that could be redundant from the psychometric perspective. In other words, we already have a question measuring that. Practitioners, on the other hand, will be able to see how they can take action on the content of questions themselves. Thus, we can assign qualitative value in regards to the practical utility of individual survey items.

The intention of this approach is to synthesize data and practical understanding into a cohesive, useful, product. The shortened version is a pulse survey that can be used as a diagnostic tool for DAOs. The full survey can help further analysis of specific issues. We plan to further examine the DAO Health Pulse Survey for scientific efficacy.

Survey Design

The final item set we arrived at for the DAO Health Pulse Survey included 16 items which assess 13 factors. The final factor model we developed can be viewed with corresponding loadings on Github.

References

Tavakol, M., & Wetzel, A. (2020, November 6). Factor Analysis: a means for theory and instrument development in support of construct validity. International Journal of Medical Education, 11, 245–247.

Bandalos, Deborah L. (2017). Measurement Theory and Applications for the Social Sciences. The Guilford Press.

Stevens JP (1992) Applied multivariate statistics for the social sciences (2nd edition). Hillsdale, NJ:Erlbaum.

Affiliations

Lead authors of this work are contributors of talentDAO, a decentralized network of people scientists developing solutions for the future of work. The DAO Health Survey team received funding for this work from Ocean protocol as well Gitcoin grants.

Links

• Github

• Original DAO Health Survey write-up

---

Elsewhere in Web3

This section includes summaries of articles that members of the talentDAO community found interesting and wanted to share. 

Understanding decentralized autonomous organizations from the inside

This article explores the world of DAOs, which are virtual organizations that operate through blockchain technology. Members of DAOs can contribute to decision-making and governance through a voting system, and the transparency and security of blockchain technology ensures that all members have equal access to information and power. The study uses a netnographic approach and structural topic modeling to analyze the characteristics and features of DAOs from the perspective of their members. The results show that DAOs have the potential to revolutionize traditional organizational structures and provide a more democratic and inclusive approach to decision-making. While there are still challenges to overcome, such as regulatory issues and the need for greater education and awareness, the future of DAOs looks promising. (full text)

Voter coalitions in Decentralized Autonomous Organization (DAO): Evidence from MakerDAO

This article explores how DAOs use voter coalitions to make decisions. The study focuses on MakerDAO, a popular DAO that uses a unique voting system. The researchers found that MakerDAO has a multicoalition democracy, where multiple groups of voters form coalitions to influence decisions. They also developed a clustering algorithm to identify these coalitions. The study has important implications for the future of decentralized governance, as it shows that DAOs can be effective at making decisions with multiple stakeholders. The findings suggest that DAOs can be a promising alternative to traditional centralized decision-making structures. Overall, the study provides valuable insights into how DAOs work and how they can be used to create more democratic and inclusive decision-making processes. (full text)

Work for Decentralised Autonomous Organisation: What Empirical Labour Economics Can Tell Us about the Decentralised Digital Workforce

DAOs are changing the way labor is organized. DAOs are digital organizations that operate through blockchain technology and smart contracts. They allow people to work together without the need for a centralized authority. DAOs offer a new way of organizing work that is more democratic, transparent, and flexible. They also provide opportunities for people to earn income and build human capital. However, there are challenges to overcome, such as finding ways to measure the value of work and ensuring that DAOs are inclusive and equitable. Despite these challenges, DAOs have the potential to transform the future of work and create a more equitable and sustainable economy. (full text)

Web3 Work research report: The DAO contributor's perspective

Embracing the innovative world of Web3, this study dives deep into the diverse landscape of DAOs and their contributors. Unlike past research, which focused on surveys, this approach captures individual perspectives within the DAO ecosystem. By using a "workers' inquiry" method, it uncovers the aspirations, challenges, and desires of contributors. The findings reveal that "DAO contributors" are a multifaceted group, combining roles like learners, freelancers, politicians, and more. DAOs themselves are diverse, encompassing traits from service providers to startups. The study uncovers four journey scenarios, illustrating how DAOs offer a platform for varied organizational experiments. Strengthening security emerges as a key to enhancing the DAO contributor experience and building legitimacy for the promising Web3 frontier. (full text)