"A common pitfall in ML delivery is the horizontal slicing of work, where we sequentially deliver functional layers of a technical solution - e.g., data lake, ML platform, ML models, UX interfaces - from the bottom-up. This is a risky delivery approach because customers can only experience the product and provide valuable feedback after months and even years of significant engineering investment. In addition, horizontal slicing naturally leads to late integration issues when horizontal slices come together, increasing the risk of release delays." (David Tan et al,"Effective Machine Learning Teams: Best Practices for ML Practitioners", 2024)
"A user story contains one or more acceptance criteria. Acceptance criteria help teams define a shared understanding of what the team needs to build. Before implementation, this criteria enables teams to articulate and validate the functionality they should be working toward for a given story. During or after implementation, this criteria enables teams to know how to test if a user story has been implemented satisfactorily. Without acceptance criteria, teams will experience higher rates of information loss between product and engineering." (David Tan et al,"Effective Machine Learning Teams: Best Practices for ML Practitioners", 2024)
"As with all principles, there are exceptions. For example, with the Independent principle, it’s often impossible to eliminate inter-story dependencies. But we still do our best to make them as independent as possible so that each pair in a team can pick up stories in parallel and independently of another pair. In our experience, the INVEST principles help us ensure that our user stories are clear, actionable, and aligned with the project’s goals, leading to more efficient and effective product development." (David Tan et al,"Effective Machine Learning Teams: Best Practices for ML Practitioners", 2024)
"Design Thinking is iterative, meaning that it often involves going back and forth between these stages, refining and adjusting as more is learned about the user and the problem. It’s highly collaborative and often involves cross-functional teams to bring different perspectives and expertise to the problem-solving process. This approach is used not only in product design but also in service design, business strategy, and organizational problem-solving." (David Tan et al,"Effective Machine Learning Teams: Best Practices for ML Practitioners", 2024)
"If the product discipline is concerned with what we build and why, the delivery discipline speaks to how we execute our ideas. The mechanics of delivering an ML product involve multiple disciplines: delivery planning, engineering, product, ML, security, data, and so on. We use the term delivery here to refer to the delivery planning aspects of how we build ML solutions." (David Tan et al,"Effective Machine Learning Teams: Best Practices for ML Practitioners", 2024)
"[...] ML engineers tend to focus on MLOps and DevOps, which are tradition‐ally focused on deployment automation, infrastructure-as-code, and CI/CD. On theother hand, data scientists tend to focus on training and evaluating ML models. While the two worlds have collided, there remains a competency gap between ML engineers (automation) and data scientists (model evaluation) in many teams. We know how to set up CI pipelines and we know how to train and evaluate models, but not all teams have worked out how to bridge both practices to automate manual model evaluation procedures." (David Tan et al,"Effective Machine Learning Teams: Best Practices for ML Practitioners", 2024)
"ML product delivery presents a unique challenge because business and customers may not know exactly what they need, or can expect, of an ML product - it’s hard to know a priori how well an ML system will perform with the data that we might curate. On one hand, we may go through product ideation but end up with an idea that is not technically achievable. On the other hand, we may not know what product features are even possible with ML before having done some experiments and seeing some working software." (David Tan et al,"Effective Machine Learning Teams: Best Practices for ML Practitioners", 2024)
"Refactoring is an essential habit of effective teams because it helps them regularly reduce technical debt and complexity, but without tests, refactoring is highly risky. Often, this leads teams to follow the path of least resistance - i.e., not refactoring - and the codebase becomes increasingly convoluted. As a result, executing on our ideas becomes harder and slower over time. On the other hand, the safety harness of comprehensive tests makes it easy for us to make a change, validate the change, and regularly reduce technical debt as we deliver new features." (David Tan et al,"Effective Machine Learning Teams: Best Practices for ML Practitioners", 2024)
"Systems thinking helps us shift our focus from individual parts of a system to relationships and interactions between all the components that constitute a system. Systems thinking gives us mental models and tools for understanding - and eventu‐ ally changing - structures that are not serving us well, including our mental models and perceptions." (David Tan et al,"Effective Machine Learning Teams: Best Practices for ML Practitioners", 2024)
"Test-driven development is a software engineering practice that emphasizes writing tests before writing code. With TDD, developers write automated tests that define the desired functionality of a feature or component, then write code to make those tests pass. TDD is useful, even and especially in ML projects, in helping us break down big problems into smaller bite-sized problems that we can address one piece at a time. At the end of it, not only do we have code logic that solves the problem for us, TDD also helps us identify edge cases, write living documentation, build a safety net that lets us refactor and validate changes within seconds, and develop self-testing code that can scale well across space and time. TDD also nudges us toward writing modular code with better defined boundaries (because we need those boundaries for invoking functions and specifying expectations in tests) instead of hammering out 'code that works' in a long and enmeshed Python script or a gigantic function." (David Tan et al,"Effective Machine Learning Teams: Best Practices for ML Practitioners", 2024)
"With a continuous delivery model, we aim to make the question of 'what and whento release' a business decision, rather than a technology decision. Certain releases might attract significant business go-to-market (GTM) activity including promotions, education, or change management, and it’s important to coordinate on this. However, with a defined and automated path to production, and by delivering stories in vertical slices to a definition of done, we aim to bake the product release effort into each story, so that releases don’t need special attention beyond story delivery." (David Tan et al,"Effective Machine Learning Teams: Best Practices for ML Practitioners", 2024)
"Without the product discipline, no amount of expertise in the other disciplines (e.g., ML, data, software engineering) can help a team deliver ML products effectively. When we don’t understand users’ needs and the organization’s business model, it makes it hard to gain alignment from business to get started. Even when teams do get started, the lack of a product-oriented approach can leave them in a vacuum of product knowledge that is quickly filled with unsubstantiated assumptions, which tends to lead to teams over-engineering unvalidated features, and ultimately wasting valuable energy and resources." (David Tan et al,"Effective Machine Learning Teams: Best Practices for ML Practitioners", 2024)
No comments:
Post a Comment