![](https://crypto4nerd.com/wp-content/uploads/2023/06/0OTYuwRRCQ2BC1-Kv-1024x683.jpeg)
Review of an article awarded by SEI 2022
By: Juan Pablo Imilan
Challenges of collaboration in building ML-enabled systems: communication, documentation, engineering, and processes
Nadia Nahar, Grace Lewis, Shurui Zhou, Christian Kästner
+++
Table of Contents
Introduction
II. Revision
— Abstract
— Keywords
1. Identification
2. Literature considered
3. Research methodology
4. Analysis of results
5. Discussion and implications
6. Conclusions and final comments
III. Bibliographic references
+++
I. Introduction
Peer review, also known as peer review or peer review, is a fundamental process in scientific and academic research. It consists of the critical evaluation and review of an academic work, such as a research article, by experts or peers in the same field of study.
The peer review process usually follows the following steps:
Submission of paper: The author submits their work to a scientific journal or academic conference for consideration.
Selection of reviewers: The editor of the publication medium selects experts in the field, usually academics or active researchers, who possess relevant experience to review the work.
Critical evaluation: Reviewers carefully examine the work and evaluate its quality, validity, originality, relevance, and scientific rigor. They also check if the work meets the standards and guidelines of the journal or conference.
Comments and suggestions: Reviewers provide constructive feedback and suggestions to the author, which may include criticisms, corrections, methodological improvements, or clarifications. These comments may be anonymous or with the identity of the reviewer revealed, depending on the review process.
Editorial decision: The publisher of the publication medium uses the comments and evaluations of the reviewers to decide on the acceptance, further revision, or rejection of the work. This decision is communicated to the author.
Peer review aims to ensure the quality and integrity of academic research by subjecting it to rigorous scrutiny by experts in the field. This process helps to filter and improve the works before publication, ensuring the validity and reliability of the results and promoting the advancement of knowledge in the scientific community.
With the desire for personal and professional growth I have decided to use the peer review methodology taking a publication that has already received awards and that I found pleasantly enriching and guiding to address the new challenges presented by machine learning.
The SEI Year in Review 2022 highlights the institute’s work during the fiscal year October 1, 2021, to September 30, 2022, and this time three articles co-authored with SEI have been awarded and recognized for their significant contributions to their fields.
It was decided to select one of these award-winning papers which is called: “Collaborative challenges in building ML-enabled systems: communication, documentation, engineering and processes”, by Nadia Nahar, Shurui Zhou, Grace Lewis from SEI and Christian Kästner. This article won a Distinguished Paper Award in the 44th edition. ªIEEE/ACM. International Congress of Software Engineering (ICSE 2022). A brief analysis and critical review are presented below (Garcés, J. and Duque, E. 2007).
II. Review
Abstract
As a result of the introduction of machine learning (ML) components into information technology (IT) projects, software engineers now need to collaborate with data scientists and other specialists. Due to the exploratory model development process, additional skills and knowledge are needed, systems (ML) are difficult to test, continuous improvement and monitoring are necessary, and non-traditional quality standards such as fairness and explainability are demanded. Interviews with forty-five professionals from twenty-eight different organizations were used to determine the main collaborative challenges faced by systems construction and implementation teams in productive ML environments. It was established that the key challenges for sustaining ML-enabled systems are focused on communication, documentation, engineering, and process. Recommendations were made to address these challenges.
Keywords Collaboration, Data Science, Software Engineering, Machine Learning, Communication, Documentation, Process, Interdisciplinary Collaboration, Model Requirements, Software Complexity, Spiral Model, Life Cycles of agile and integrated processes.
1. Identification
1.1 Problem
The problem posed has to do with identifying opportunities for improvement in the collaboration between data scientists and software engineers when building systems with ML capacity and specifically to be able to determine which are the key collaborative interaction points that require greater attention from organizations that mostly do not belong to the world of large corporations since the latter have a different evolutionary maturity when facing the challenges of construction in this type of systems.
These opportunities for improved collaboration arise in the context of developing ML-capable systems and highlight the additional challenges ML presents compared to traditional software projects. These opportunities for improvement in collaboration include challenges for exploratory model development processes, additional skills and knowledge, implementation of ML systems, evolutionary developments, continuous monitoring for implemented models, and non-traditional quality standards.
1.2 Importance of the topic
Findings such as the fact that Venturebeat (2019) mentioned that 87% of ML projects fail or that Gartner (2020) indicated that 53% do not go from prototype to production, highlight the importance of addressing the challenges posed in the study. Specifically for systems enabled in machine learning (ML), it becomes important to ensure the understanding and involvement of stakeholders and thus the success of ML projects.
He argues for the need to improve interdisciplinary collaboration practices given that software engineers, data scientists, and other specialists are in permanent cycles of collaboration. This poses unique challenges in terms of communication, coordination, and goal alignment, helping to identify problem collaboration points and improve interdisciplinary collaboration practices.
It considers the ethical and quality perspective for the development of ML-enabled systems and raises needs for improvement in aspects of equity, explainability and quality. Addressing challenges to understand and ensure transparency, fairness in the treatment of data and results, along with the need to ensure the overall quality of ML systems.
1.3 Objectives
This article proposes a comprehensive review of existing methods and instruments for assessing the challenges of collaboration in building machine learning (ML)-enabled systems; It describes what is already known about the topic of collaboration in traditional software engineering and especially about qualitative models and indicators with theoretical and documentary foundation. The article aims to make us think about areas of machine learning that require further research on qualitative issues in the long term.
1.4 Focus
It is a descriptive work that proposes to understand and interpret social and human phenomena from the perspective of the participants involved, associated with the challenges of collaboration in the creation of systems enabled for machine learning (ML) focused on the analysis and understanding of obstacles and best practices related to collaboration in this field. The work seeks to identify the specific challenges that arise when collaborating on the creation of ML-enabled systems and examine how those challenges can be overcome.
It is a descriptive work that aims to understand and interpret social and human phenomena from the point of view of the participants involved, in relation to the unique difficulties that arise in the collaboration of the development of systems enabled for machine learning (ML), with a particular focus on understanding and analyzing the challenges related to collaboration in this field.
1.5 Contributions
The article aims to identify and understand the specific challenges associated with collaboration in building machine learning (ML)-enabled systems. This could allow for greater awareness and understanding of the obstacles faced by collaborative teams in this field and help develop strategies to overcome them.
It aims to provide recommendations and best practices for effective collaboration in ML projects. Through the analysis of case studies and successful examples, it is expected to offer guidelines and practical approaches that improve the quality and efficiency of collaborative projects in the field of ML.
It aims to assess the impact of collaboration on building ML-enabled systems. By measuring the success of projects in terms of successful implementation, quality of results and end-user satisfaction, it will be possible to assess how collaboration affects these key aspects. This will provide valuable insights to improve and optimize collaboration on future ML projects.
It aims to contribute to the advancement of the field of collaboration in ML. By sharing the results, recommendations and lessons learned from the research, it is hoped to enrich collective knowledge and encourage the development of innovative ideas and approaches in the field of collaboration in ML-enabled systems.
Together, these contributions aim to improve the effectiveness, efficiency, and quality of ML projects involving collaboration, and to promote faster and more effective advancement in this ever-evolving field.
2. Literature covered
2.1 State of the art
The authors initially identified that researchers and practitioners have debated whether the introduction of machine learning model components into software systems has altered software engineering. Describing a theoretical relationship on collaboration in traditional software engineering and software engineering related to machine learning components as proposed by Aho, T. et al (2020).
Software architecture. It refers to the overall design and high-level decisions of the system. Explore how introducing machine learning into software projects challenges collaboration. The extent of how much the initial architectural design is feasible, practical, or desirable is questioned, reflecting a tension between the desire for initial planning, technical risks, and unstable requirements (Bass, L. et al. 1998; Waterman, M. et al. 2015).
Collaboration in Software Engineering. It identifies that the collaboration and coordination of several developers and teams occurs between teams often with formal communication circuits, such as within teams where communication circuits tend to be more informal. The technical structure (module components, functions, subsystems), delimited by the interfaces and dependencies between the components refers to the internal implementation details. The development of modular software is common and facilitates the work for multiple developers, divide tasks and activities according to their components (modules, functions, and subsystems) and interfaces, the latter being the reflection of the key architectural decisions for the system whose objective is to achieve the desired general qualities. The software life cycle considers waterfall models where the software design planning is given after a careful analysis of requirements; The spiral model with a primary focus on addressing risks first and iterating by creating prototypes that address risk factors and Agile models usually downplay the initial architectural design to achieve rapid iteration in incremental prototypes (Bass, L. et al. 1998).
Coordination challenges. The authors identified that teams cannot communicate easily and informally often studied in distributed teams of global corporations or open-source ecosystems.
Challenges of interdisciplinary collaboration. The authors identified that when team members differ in their academic and professional backgrounds, they often have different expectations of the same system and communication, cultural and methodical challenges often arise when working together. The success of this type of collaboration usually depends on the professional role, structural characteristics, personal characteristics, and a history of collaboration.
Barriers to collaboration. The authors identified barriers to collaboration such as structural factors such as unclear mission, insufficient time, excessive workload, and lack of administrative support.
Software Engineering with ML Components. It identifies that these systems have ML and non-ML components and requires involving in the software development of these systems data scientists and software engineers with greater competencies and experience with additional emphasis on data management, security, ethics, system-level quality, engineering requirements, architecture design and user interaction design. They substantiate, for example, that building ML models requires more experience in data science. On the other hand, they identify that to build a system enabled with ML capabilities that requires the construction of models, components for data training, monitoring components for models, the creation of training pipelines and implementation of models in a scalable way, will often require engineers with the necessary experience in “AI engineering”, “SysML” and “MLOps” (Aho, T. et al. 2020; Akkiraju, R. et al. 2020)
Barriers to collaboration. The authors identified barriers to collaboration such as factors of differences in skills and educational background within these teams (Nahar, N. et al. 2022).
2.2 Approach to the problem
The article makes a complete review of the contributions to the subject in the last 15 years, concentrated on the search for collaboration problems mentioned in the existing literature on software engineering for systems enabled for ML components. They selected fifteen articles opportunistically through keyword research and their own knowledge of the field. Selecting all sections in those documents that are potentially related to collaboration challenges among team members with different skills or educational backgrounds, following an open coding process. Although most of the articles did not talk about problems in terms of collaboration, we marked discussions that can relate to collaboration, such as data quality issues between teams (Strauss, A. and Corbin, J.M. 2002).
Potential approaches arise such as:
Role-based collaboration: Suggests that one could investigate how to assign clear roles and responsibilities in a machine learning systems development team, considering the individual skills and knowledge of team members. This may include designating roles such as data scientist, software engineer, domain expert, and user experience designer, and defining their interactions and contributions in the development process.
Collaborative methodologies: Suggests exploring different methodologies and collaborative approaches in the development of machine learning systems, such as agile development, user-centered design, and the DevOps approach. These methodologies can be adapted and customized to address the specific challenges related to collaboration in ML systems.
Collaboration tools and platforms: Suggests deepening the investigation of the use of specific tools and platforms that facilitate collaboration in the development of machine learning systems. This can include project management tools, version control systems, online collaboration platforms, and data visualization tools, among others.
Effective communication and documentation: Suggest paying attention to communication and documentation practices among ML systems development team members. This may include adopting clear and effective approaches to sharing knowledge, reporting on project progress, documenting decisions, and capturing lessons learned.
2.3 Methodological approaches
In this regard, it can be affirmed that the work does not propose alternatives to limit the methodological focus, but is limited to presenting the so-called “state of the art” of the issue, and tries to locate within a non-validated model the need to improve the critical points of collaboration for ML-enabled systems, the reviewed aspects: models, tools, instruments and indicators of ML-enabled systems, located within your conceptual model.
2.4 Exploratory research issues
Exploratory research issues in this study include:
Identify collaboration points in building machine learning (ML)-enabled systems.
Explore the challenges associated with collaboration among team members with different skills or educational backgrounds.
Understand the relationship between collaboration and data quality in teams working on ML-enabled systems.
Analyze the organizational structure and responsibilities in different teams involved in the development of ML-enabled systems.
It investigates how teams vary in terms of composition, power dynamics, and formality of collaborations in the context of ML-enabled systems.
Identify differences and similarities in collaborative approaches between different organizations and application domains in ML-enabled systems.
2.5 Main hypotheses
Although they are not explicitly stated in the article, you can see several examples such as:
Organizational structure, team composition, power dynamics, and responsibilities differ between organizations and can influence collaboration challenges.
Collaboration challenges in ML-enabled systems focus primarily on communication, documentation, engineering, and processes.
The introduction of machine learning components in software projects poses challenges in software engineering, such as testing ML components, misusing ML libraries, and automating implementation processes for ML components.
The introduction of machine learning in software projects poses additional challenges in terms of coordination and communication between teams with different technical backgrounds.
Collaboration in the development of ML-enabled systems presents additional challenges due to exploratory model development processes, the skills and knowledge required, difficulties in testing ML systems, the need for continuous evolution and monitoring, and non-traditional quality requirements such as fairness and explainability.
There are critical points of collaboration in identifying and breaking down requirements, negotiating the quality and quantity of training data, and integrating data science and software engineering work.
Successful interdisciplinary collaboration in ML-enabled systems depends on the professional role, structural characteristics, personal characteristics, and collaboration history of team members.
Lack of systematic or shared understanding of common collaboration challenges and best practices in developing ML-enabled systems hinders effective collaboration among team members.
Adopting collaboration best practices, attention to collaboration points, clear documentation of responsibilities and interfaces, considering engineering work as a key contribution, and investing in processes and planning can improve collaboration practices in ML-enabled systems.
Defining interfaces and negotiating responsibilities between data science and software engineering teams are critical points of collaboration in ML-enabled systems.
The involvement in software architecture by senior management is a key success factor for organizations that have ML-enabled software development systems, enabling strategic orchestration of nodes and encouraging innovation.
3. Research methodology
The methodological approach used in this study is qualitative research (Strauss, A. and Corbin, J. 1994). A qualitative research strategy was adopted to explore the points of collaboration and corresponding challenges in building machine learning (ML)-enabled systems. The research design was based on Strauss’s grounded theory, which involves deriving research questions from the literature, analyzing interviews with open and axial coding, and consulting the literature throughout the process.
The steps followed in the study were as follows:
Interview preparation: what I considered the selection of fifteen articles from the existing literature on software engineering for ML-enabled systems and identified areas of collaboration from them.
Conducting interviews: Semi-structured interviews were conducted with 45 participants from twenty-eight organizations, covering distinct roles, types of companies and countries.
Triangulation with literature: As knowledge was gained from the interviews, we turned to literature to identify related discussions and viable solutions.
Validity check with interviewees: Participants were presented with an abstract and full draft of the paper, soliciting comments, and verifying the applicability and correctness of the findings.
Regarding the threats to the validity and credibility of the study, it is mentioned that standard coding and memorization practices were followed, but the biases introduced by the researchers cannot be completely excluded. In addition, it is recognized that generalizations beyond the interviewed participants should be made with caution, since the sample did not include certain specific roles and domains. It is also mentioned that the self-selection of participants can influence the results.
4. Analysis of results
Analysis of the results indicates that organizations building machine learning (ML)-enabled systems face substantial challenges at three core points of collaboration: requirements and project planning, data for model training, and model integration into the product.
Other points of collaboration were mentioned, but less frequently, such as interacting with legal experts and operators, documenting work for other data scientists, and challenges when interacting with teams that build and maintain unstable ML libraries.
Even though data scientists and software engineers are aware of interdisciplinary challenges and cultural and communication issues, many organizations that build ML-enabled systems pay little attention to fostering better cross-functional collaboration.
Four general themes are identified that require further attention in both engineering practice and research:
Communication: Lack of communication between participants with diverse backgrounds is at the root of many problems. ML literacy education is suggested for software engineers and managers, as well as training of data scientists in software engineering concerns. Deep experience in one area is mentioned as a key factor, the extensive knowledge of other disciplines as a requirement when adding professionals to these teams, accompanied by training.
Documentation: It is important to clearly document expectations among teams. Traditional documentation practices for software engineers can serve as a starting point, but best practices are needed to document model requirements, data expectations, and assured model qualities. Approaches such as model cards and fact sheets are suggested to improve and standardize documentation of ML components.
Engineering: Many organizations underestimate the engineering effort required to turn an ML model into a reliably operable, maintainable product. The adoption of machine learning increases software complexity and makes engineering practices, such as data quality controls, deployment automation, and production testing, even more important. Project managers must ensure that they have sufficient engineering capabilities in all parts of the project.
Process: The machine learning process, more like science, challenges the lifecycles of traditional software processes. More research is needed on integrated process lifecycles for ML-enabled systems, spanning software engineering and data science. Product requirements cannot be established without the involvement of data scientists in model prototyping, and it is often advisable to take a “model first” approach to reduce risk. A balanced approach to product and process is required to avoid problems.
4.1 Preliminary indication
The preliminary results-based indication is that organizations building machine learning-enabled systems should pay more attention to cross-functional collaboration and address the challenges identified at the key points of collaboration: requirements and project planning, data for model training, and integration of the model into the product.
To improve collaboration, it is recommended to focus on four principal areas:
Communication: Better communication between data scientists, software engineers, and other participants with diverse backgrounds should be encouraged. Machine learning literacy education and training in software engineering concerns can be helpful in promoting mutual understanding.
Documentation: It is important to establish clear documentation practices to define expectations and requirements among teams. Appropriate methods should be developed to document ML components, such as models, data requirements, and assured model qualities.
Engineering: Sufficient engineering effort must be recognized and allocated to convert ML models into operational and maintainable products. This involves considering data quality controls, deployment automation, and production testing.
Process: A deeper understanding of integrated process lifecycles is needed for ML-enabled systems. Product requirements should involve data scientists from preliminary stages, and it can be beneficial to take a “model first” approach to reducing risk. A balanced perspective between product and process is required.
These preliminary recommendations seek to improve interdisciplinary collaboration and foster best practices in building machine learning-enabled systems.
4.2 Main results
The main results identified are as follows:
Collaboration Core Points: Three collaboration hubs were identified where organizations building machine learning-enabled systems face substantial challenges. These points are requirements and project planning, data for model training and integration of the model into the product.
Other points of collaboration: In addition to the crucial points mentioned, other points of collaboration were found, although they were mentioned less frequently. These include interaction with legal experts and operators, documentation of work for other data scientists, and conventional collaborations related to software projects.
Cultural and communication challenges: It was highlighted that interdisciplinary collaborations face cultural and communication challenges. Although these challenges are not unique to organizations building ML-enabled systems, it was noted that many of these organizations pay little attention to fostering better interdisciplinary collaboration.
General topics requiring attention: Four general themes were identified that would benefit from increased attention in both engineering practice and research. These topics are communication, documentation, engineering, and process. Specific approaches are suggested, such as ML literacy education, clear documentation of expectations, recognition of engineering effort, and a deeper understanding of integrated process lifecycles.
4.3 Solidity
The results found in the study provide a solid foundation for understanding the challenges and issues associated with interdisciplinary collaboration in building machine learning-enabled systems. The study was based on interviews and considered different perspectives from data scientists, software engineers and other relevant participants.
The identification of the three crucial points of collaboration and the mention of other less frequent points provides a comprehensive view of the challenges faced by organizations in this area. In addition, the detection of cultural and communication issues highlights the importance of addressing these aspects to improve collaboration and the success of ML projects.
The general topics identified, such as communication, documentation, engineering, and process, provide clear guidance for organizations in terms of key areas they should focus on to overcome challenges and enhance interdisciplinary collaboration.
While each organization may have its own particularities and require customized approaches, the results of the study provide a solid foundation for initiating discussions, planning strategies, and promoting best practices in building machine learning-enabled systems.
Overall, the robustness of the results derives from a thorough analysis of the experiences and perspectives of various professionals involved in building ML-enabled systems, allowing for a more complete understanding of the challenges and opportunities in this field.
5. Discussion and implications
The discussion of the results and their implications focuses on the importance of addressing the challenges identified in interdisciplinary collaboration in building machine learning-enabled systems. These findings have several significant implications:
Improved collaboration: Cross-disciplinary collaboration is critical to the success of machine learning projects. The cultural and communication challenges highlighted in the study highlight the need to foster a collaborative work environment and to promote mutual understanding among data scientists, software engineers and other participants. This implies the need to develop effective communication skills and promote a machine learning literacy education for different stakeholders.
Documentation and clarity of expectations: Clear documentation of requirements, data expectations, and model qualities is crucial to avoid misunderstandings and problems during the system construction process. The study notes that documentation practices in these aspects are not well established in the context of machine learning. Therefore, it is suggested to adopt approaches such as model cards and fact sheets to improve and standardize the documentation of ML components.
Recognition of required engineering: The adoption of machine learning increases software complexity and requires significant engineering effort to turn models into operational and maintainable products. It is essential that organizations recognize this aspect and allocate adequate resources to ensure software quality, deployment automation, and production testing. Project managers must ensure that the necessary engineering capabilities are in place to support both the ML and non-ML parts of the project.
Integrated process lifecycles: The lifecycles of traditional software processes may not be sufficient for the development of ML-enabled systems due to the closer nature to machine learning science. Product requirements should involve data scientists from the initial stages, and it can be beneficial to take a “model first” approach to reduce risk. Further research into integrated process lifecycles is required specifically for machine learning-enabled systems.
Overall, the discussion of the results highlights the need to address collaborative challenges in building machine learning-enabled systems and provides specific recommendations for improving communication, documentation, engineering, and process. These implications have the potential to improve the efficiency, quality, and success of machine learning projects, as well as promote better interdisciplinary collaboration in this field.
6. Conclusions and final comments
The conclusions that can be drawn from the text are:
Cross-disciplinary collaboration on machine learning (ML)-enabled systems: Organizations building ML-based systems face challenges in three principal areas of collaboration: project requirements and planning, data for model training, and model integration into products. These challenges highlight the need for better collaboration between data scientists and software engineers.
Lack of attention to interdisciplinary collaboration: Although interdisciplinary collaborations present known challenges in terms of cultural and communication issues, many organizations that create ML-enabled systems do not pay enough attention to fostering better collaboration across disciplines.
Need for planning and insight: Most organizations do not deliberately plan their structures and practices to address the challenges of interdisciplinary collaboration in ML. In addition, they have little perception of the options available and the trade-offs they can make in terms of structures and practices.
General topics requiring attention: In addition to the specific challenges discussed, four general themes are identified that would benefit from increased attention in both engineering practice and ML-related research: communication, documentation, engineering, and process. These topics address the importance of education, training, clear documentation of expectations, attention to engineering aspects, and adapting process lifecycles to ML-enabled systems.
These findings suggest the importance of organizational involvement, along with improving collaboration and project management in the ML arena.
III. Bibliographic references
Singh, G., Gehr, T., Püschel, M. and Vechev, M. (2019). An abstract domain for certifying neural networks. Proc. ACM Program. Lang. 3, POPL, 1–30. (s.f.).
Aho, T., Sievi-Korte, O. K., Yaman, S., & Mikkonen, T. (2020). Demysti- fying data science projects: A look on the people and process of data science today. In Proc. Int’l Conf. Product-Focused Software Process Improvement, 153–167.
Aho, T., Sievi-Korte, O., Kilamo, T., Yaman, S. and Mikkonen, T., (2020). Demysti- fying data science projects: A look on the people and process of data science today. In Proc. Int’l Conf. Product-Focused Software Process Improvement, 153–167. (s.f.).
Akkiraju, R., Sinha, V., Xu, A., Mahmud, J., Gundecha, P., Liu, Z., Liu, X. and Schumacher, J. (2020). Characterizing Machine Learning Processes: A Maturity Framework. Business Process Management, 17–31. (s.f.).
Ameisen, E. (2020). Building Machine Learning Powered Applications: Going from Idea to Product. O’Reilly Media, Inc. (s.f.).
Amershi, S. e. (2019). Software Engineering for Machine Learning: A Case Study. In Proc. of 41st Int’l Conf. on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 291–300.
Amershi, S. et al. (2019). Guidelines for Human-AI Interaction. In Proc. of CHI Conf. on Human Factors in Computing Systems, 1–13. (s.f.).
Amershi, S. et al. (2019). Software Engineering for Machine Learning: A Case Study. In Proc. of 41st Int’l Conf. on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 291–300. (s.f.).
Ashmore, R., Calinescu, R. and Paterson, C. (2021). Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges. ACM Computing Surveys (CSUR), 54 (5): 1–39. (s.f.).
Bass, L., Clements, P. and Kazman, R. (1998). Software Architecture in Practice. (s.f.).
Bass, M., Herbsleb, J.D. and Lescher, C. (2009). A Coordination Risk Analysis Method for Multi-site Projects: Experience Report. In Proc. Int’l Conf. Global Software Engineering, 31–40. (s.f.).
Baylor, D., Breck, E., Cheng, H.T., Fiedel, N., Foo, C.Y. et al. (2017). TFX: A TensorFlow-Based Production-Scale Machine Learning Platform. In Proc. Int’l Conf. Knowledge Discovery and Data Mining, 1387–1395. (s.f.).
Bogart, C., Kästner, C., Herbsleb, J. and Thung, F. (2021). When and how to make breaking changes: Policies and practices in 18 open source software ecosystems. ACM Transactions on Software Engineering and Methodology. 30, 4, 1–56. (s.f.).
Borg, M. et al. (2019). Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industry. Journal of Automotive Software Engineering. 1, 1, 1–9. (s.f.).
Bosch, J., Olsson, H.H. and Crnkovic, I. (2021). Engineering AI Systems: A Research Agenda. Artificial Intelligence Paradigms for Smart Cyber-Physical Systems. IGI Global. (s.f.).
Boujut, J.-F. and Blanco, E. (2003). Intermediary Objects as a Means to Foster Co-operation in Engineering Design. Computer Supported Cooperative Work (CSCW). 12, 2, 205–219. (s.f.).
Braiek, H.B. and Khomh, F. 2020. On testing machine learning programs. Journal of Systems and Software. 164, 110542. (s.f.).
Brandstädter, S. and Sonntag, K. (2016). Interdisciplinary Collaboration. Advances in Ergonomic Design of Systems, Products and Processes, 395–409. (s.f.).
Braude, Eric J and Bernstein, Michael E. (2011). Software Engineering: Modern Approaches 2nd Edition. Wiley. ISBN-13: 978–0471692089. (s.f.).
Brown, G.F.C. (1995). Factors that facilitate or inhibit interdisciplinary collaboration within a professional bureaucracy. University of Arkansas. (s.f.).
Cai, C.J., Winter, S., Steiner, D., Wilcox, L. and Terry, M. (2019). “hello AI”: Uncov- ering the onboarding needs of medical practitioners for human-AI collaborative decision-making. In Proc. Human-Computer Interaction. 3, CSCW, 1–24. (s.f.).
Cataldo, M. et al. (2006). Identification of Coordination Requirements: Implications for the Design of Collaboration and Awareness Tools. In Proc. Conf. Computer Supported Cooperative Work (CSCW), 353–362. (s.f.).
Chattopadhyay, S., Prasad, I., Henley, A.Z., Sarma, A. and Barik, T. (2020). What’s Wrong with Computational Notebooks? Pain Points, Needs, and Design Oppor- tunities. In Proc. Conf. Human Factors in Computing Systems (CHI), 1–12. (s.f.).
Chen, Z., Cao, Y., Liu, Y., Wang, H., Xie, T. and Liu, X. (2020). Understanding Challenges in Deploying Deep Learning Based Software: An Empirical Study. arXiv 2005.00760. (s.f.).
Cheng, D., Cao, C., Xu, C. and Ma, X. (2018). Manifesting Bugs in Machine Learning Code: An Explorative Study with Mutation Testing. In Proc. Int’l Conf. Software Quality, Reliability and Security (QRS), 313–324. (s.f.).
Conway, M.E. (1968). How Do Committees Invent? Datamation. 14, 4, 28–31. (s.f.).
Cossette, B.E. and Walker, R.J. (2012). Seeking the Ground Truth: A Retroactive Study on the Evolution and Migration of Software Libraries. In Proc. Int’l Symposium Foundations of Software Engineering (FSE), 1–11. (s.f.).
Curtis, B., Krasner, H. and Iscoe, N. (1988). A field study of the software design process for large systems. Communications of the ACM. 31, 11, 1268–1287. (s.f.).
Dabbish, L., Stuart, C., Tsay, J. and Herbsleb, J. (2012). Social Coding in GitHub: Transparency and Collaboration in an Open Software Repository. In Proc. Conf. Computer Supported Cooperative Work (CSCW), 1277–1286. (s.f.).
Dabbish, L., Stuart, C., Tsay, J. and Herbsleb, J. (2012). Social Coding in GitHub: Transparency and Collaboration in an Open Software Repository. In Proc. Conf. Computer Supported Cooperative Work (CSCW), 1277–1286. (s.f.).
de Souza, C.R.B. and Redmiles, D.F. (2008). An Empirical Study of Software Developers’ Management of Dependencies and Changes. In Proc. Int’l Conf. Software Engineering (ICSE), 241–250. (s.f.).
Garcés, J., Duque, E. (2007). Metodología para el análisis y la revisión crítica de artículos de investigación. (s.f.).
Haakman, M., Cruz, L., Huijgens, H. and van Deursen, A. (2021). AI Lifecycle Models Need To Be Revised. An exploratory study in Fintech. Empirical Software Engineering. 26, 5, 1–29. (s.f.).
Herbsleb, J.D. and Grinter, R.E. (1999). Splitting the Organization and Integrating the Code: Conway’s Law Revisited. In Proc. Int’l Conf. Software Engineering (ICSE), 85–95. (s.f.).
Hopkins, A. and Booth, S. (2021). Machine learning practices outside big tech: How resource constraints challenge responsible development. In Proc. Conf. on AI, Ethics, and Society, 134–145. (s.f.).
Hulten, G. (2019). Building Intelligent Systems: A Guide to Machine Learning Engineering. Apress. (s.f.).
Hulten, G. (2019). Building Intelligent Systems: A Guide to Machine Learning Engineering. Apress.
Ishikawa, F. and Yoshioka, N. (2019). How do engineers perceive difficulties in engineering of machine-learning systems? — Questionnaire survey. In Proc. Int’l Workshop on Conducting Empirical Studies in Industry (CESI) and Software Engineering Research a. (s.f.).
Kästner, C. and Kang, E. (2020). Teaching Software Engineering for Al-Enabled Systems. In Proc. Int’l Conf. Software Engineering: Software Engineering Education and Training (ICSE-SEET), 45–48. (s.f.).
Kim, M. Z.–1. (s.f.).
Lakshmanan, V., Robinson, S. and Munn, M. (2020). Machine Learning Design Patterns. O’Reilly Media, Inc. (s.f.).
Mäkinen, S. S.-S.-1. (s.f.).
Martínez-Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A.M. and Wagner, S. (2021). Software Engineering for AI-Based Systems: A Survey. arXiv 2105.01984. (s.f.).
Martinez-Plumed, F., Contreras-Ochando, L., Ferri, C., Hernandez Orallo, J., Kull, M., Lachiche, N., Ramirez Quintana, M.J. and Flach, P.A. (2021). CRISP-DM twenty years later: From data mining processes to data science trajectories. IEEE Transactions on . (s.f.).
Meyer, B. (1997). Object-Oriented Software Construction. Prentice-Hall. (s.f.).
Mistrík, I., Grundy, J., van der Hoek, A. and Whitehead, J. (2010). Collaborative Software Engineering. Springer. (s.f.).
Nahar, N., Zhou, S., Lewis, G., Kästner, C. (2022) Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process. (s.f.).
O’Leary, K. and Uchida, M. (2020). Common problems with creating machine learning pipelines from existing code. In Proc. Conf. Machine Learning and Systems (MLSys). (s.f.).
Ovaska, P., Rossi, M. and Marttiin, P. (2003). Architecture as a coordination tool in multi-site software development. Software Process Improvement and Practice. 8, 4, 233–247. (s.f.).
Ozkaya, I. (2020). What Is Really Different in Engineering AI-Enabled Systems? (s.f.).
Panetta, K. (2020). Gartner Identifies the Top Strategic Technology Trends for 2021. URL: https:// www.gartner.com/ smarterwithgartner/ gartner-top-strategic- technology-trends-for-2021.
Parnas, D.L. (1972). On the Criteria to be used in Decomposing Systems into Modules. Communications of the ACM. 15, 12, 1053–1058. (s.f.).
Passi, S., and Phoebe S. (2020). Making Data Science Systems Work. Big Data & Society 7 (2): 1–13. (s.f.).
Patel, K., Fogarty, J., Landay, J.A. and Harrison, B. (2008). Investigating statistical machine learning as a tool for software development. In Proc. Conf. Human Factors in Computing Systems (CHI), 667–676. (s.f.).
Rakova, B., Yang, J., Cramer, H. and Chowdhury, R. (2021). Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for Shifting Organizational Practices. Proc. ACM Hum.-Comput. Interact. 5, CSCW1, 1–23. (s.f.).
Rakova, B., Yang, J., Cramer, H. and Chowdhury, R. (2021). Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for Shifting Organizational Practices. Proc. ACM Hum.-Comput. Interact. 5, CSCW1, 1–23. (s.f.).
Salay, R., Queiroz, R. and Czarnecki, K. (2017). An Analysis of ISO 26262: Using Machine Learning Safely in Automotive Software. arXiv 1709.02435. (s.f.).
Salay, R., Queiroz, R. and Czarnecki, K. (2017). An Analysis of ISO 26262: Using Machine Learning Safely in Automotive Software. arXiv 1709.02435. (s.f.).
Sambasivan, N. et al. (2021). “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI. In Proc. Conf. on Human Factors in Computing Systems (CHI). 1–15. (s.f.).
Sarma, A., Redmiles, D.F. and van der Hoek, A. (2012). Palantir: Early Detection of Development Conflicts Arising from Parallel Code Changes. IEEE Transactions on Software Engineering. 38, 4, 889–908. (s.f.).
SEI, (2022). SEI Year in Review 2022. (s.f.).
Serban, A., van der Blom, K., Hoos, H. and Visser, J. (2020). Adoption and Ef- fects of Software Engineering Best Practices in Machine Learning. In Proc. Int’l Symposium on Empirical Software Engineering and Measurement, 1–12. (s.f.).
Serban, A., van der Blom, K., Hoos, H. and Visser, J. (2020). Adoption and Ef- fects of Software Engineering Best Practices in Machine Learning. In Proc. Int’l Symposium on Empirical Software Engineering and Measurement, 1–12. (s.f.).
Siebert, J., Joeckel, L., Heidrich, J., Nakamichi, K., Ohashi, K., Namba, I., Ya- mamoto, R. and Aoyama, M. (2020). Towards Guidelines for Assessing Qualities of Machine Learning Systems. In Proc. Int’l Conf. on the Quality of Information and Communicatio. (s.f.).
Smith, D., Alshaikh, A., Bojan, R., Kak, A. and Manesh, M.M.G. (2014). Overcoming barriers to collaboration in an open source ecosystem. Technology Innovation Management Review. 4, 1. (s.f.).
Staff, V. B. (2019). Why do 87% of data science projects never make it into pro- duction? URL: https:// venturebeat.com/ 2019/ 07/ 19/ why-do-87-of-data-science- projects-never-make-it-into-production/ .
Strauss, A. and Corbin, J. (1994). Grounded theory methodology: An overview. (s.f.).
Strauss, A. and Corbin, J.M. (2002). Basics of Qualitative Research: Grounded Theory Procedures and Techniques. SAGE Publications. (s.f.).
Tramèr, F. et al. (2017). FairTest: Discovering Unwarranted Associations in Data- Driven Applications. In Proc. European Symposium on Security and Privacy (EuroS P), 401–416. (s.f.).
Vogelsang, A. and Borg, M. (2019). Requirements Engineering for Machine Learn- ing: Perspectives from Data Scientists. In Proc. Int’l Requirements Engineering Conf. Workshops (REW), 245–251. (s.f.).
Vogelsang, A. and Borg, M. (2019). Requirements Engineering for Machine Learn- ing: Perspectives from Data Scientists. In Proc. Int’l Requirements Engineering Conf. Workshops (REW), 245–251. (s.f.).
Wang, A.Y., Mittal, A., Brooks, C. and Oney, S. (2019). How Data Scientists Use Computational Notebooks for Real-Time Collaboration. Proc. Human-Computer Interaction. 3, CSCW, 39. (s.f.).
Waterman, M., Noble, J. and Allan, G. (2015). How Much Up-Front? A Grounded theory of Agile Architecture. In Proc. Int’l Conf. Software Engineering, 347–357. (s.f.).
Xie, X., Ho, J.W.K., Murphy, C., Kaiser, G., Xu, B. and Chen, T.Y. (2011). Testing and Validating Machine Learning Classifiers by Metamorphic Testing. Journal of Systems and Software. 84, 4, 544–558. (s.f.).
Yang, Q. (2018). The role of design in creating machine-learning-enhanced user experience. In Proc. AAAI Spring Symposium Series, 406–411. (s.f.).
Yang, Q., Suh, J., Chen, N.-C. and Ramos, G. (2018). Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models. In Proc. Conf. Designing Interactive Systems, 573–584. (s.f.).
Yang, Q., Suh, J., Chen, N.-C. and Ramos, G. (2018). Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models. In Proc. Conf. Designing Interactive Systems, 573–584. (s.f.).
Yokoyama, H. (2019). Machine Learning System Architectural Pattern for Improv- ing Operational Stability. In Proc. Int’l Conf. on Software Architecture Companion (ICSA-C), 267–274. (s.f.).
Zhang, A.X., Muller, M. and Wang, D. (2020). How do data science workers collaborate? Roles, workflows, and tools. Proc. Human-Computer Interaction. 4, CSCW1, 1–23. (s.f.).
Zhou, S., Vasilescu, B. and Kästner, C. (2020). How Has Forking Changed in the Last 20 Years? A Study of Hard Forks on GitHub. In Proc. Int’l Conf. Software Engineering (ICSE), 445–456. (s.f.).
+++