Hide

Oops! Something went wrong while submitting the form.

see all Podcasts

see all Elements

{feed}

super{summit}

super{summit} 2022

When Inference Meets Engineering

October 24, 2022

Written By

October 24, 2022

Written By

Othmane Rifki, Principal Applied Scientist at super{set} company Spectrum Labs, reports from the session he led at super{summit} 2022: "When Inference Meets Engineering." Using super{set} companies as examples, Othmane reveals the 3 ways that data science can benefit from engineering workflows to deliver business value.

By Othmane Rifki, Principal Applied Scientist at Spectrum Labs

super{set} companies have been very early to adopt data engineering, or leveraging software engineering to power data science workflows and solve business problems. Many of us have been on the front lines of the emerging roles that sit between an organization’s traditional software engineers and data scientists, namely the Data Engineers that optimize the retrieval and use of data to power ML models and the Machine Learning Engineers that ensure a scalable and flexible environment for ML model pipelines.

At super{summit} 2022, I organized a session for data and ML engineers to come together and share lessons learned from their particular workflow and business situation. Our conversation distilled 3 ways that data science can benefit from engineering workflows to deliver business value:

Managing the complexity of the machine learning lifecycle at scale
Creating business value means seeing models through to deployment and beyond
Preserving data privacy to build trust with consumers

Let’s review!

Managing Machines Learning Lifecycles at Scale

Data science teams focus on building models to help businesses solve problems - for example, identifying hate speech with deep learning models (see Spectrum Labs). The performance of those models is assessed with labeled datasets originating from client traffic or other sources. All this is quite manageable at a small scale when there are only a handful of models to serve and customers that can be counted with one hand.

When data science models scale, things start to break. The average super{set} deals with a large number of models that need to be deployed on behalf of multiple clients in a myriad of production environments. Understanding and managing these models and their dependencies at scale while also mitigating risks that may arise from decision automation (decision-making without human intervention) becomes critical to the success of business operations. Simply put: dollars and livelihoods are on the line as startups scale into meaningful businesses.

Data + ML engineering to the rescue! Data engineers and ML engineers work together to::

Optimize the retrieval of data needed to train models
Integrate machine learning models into an organization’s applications and systems
Ensure a scalable and flexible machine learning model pipeline from design to serving to monitoring
Build robust automation to ease the continuous delivery of model updates while maintaining high quality

Introducing MLOps

super{set} has more than just in-house expertise to manage lifecycles at scale (what we call Machine Learning Operations or “MLOps”): one super{set} company, MarkovML, is entirely dedicated to solving the problem of MLOps! MarkovML connects ML teams and provides end-to-end visibility into their ML workflows so teams can reach their business goals faster. MarkovML connects ML teams and provides end-to-end visibility into their ML workflows so organizations can reach their business goals faster. As the team from MarkovML shared, MLOps is more than just streamlining the process of deploying, monitoring, and maintaining ML models - it’s about improving the entire lifecycle by providing valuable insights around:

The performance of the model
The relevance of the data used for training
Connecting performance and relevance to the target business value

Once again, it all comes down to solving business problems. The burden and error-prone manual processes of keeping track of the organization’s data and models can be eliminated in favor of automating the end-to-end machine learning workflow enabling data science teams to focus on extracting insights related to business objectives. Products of MarkovML make it simpler.

Data Governance & Model Measurement WorkFlow at markovML

Creating Business Value Means Seeing Models Through to Deployment and Beyond

Creating business value doesn’t stop with model creation. Each super{set} company made clear that the smooth deployment of new models into production is key to maximizing the value of the product offering. Ketch, a company that enables organizations to build trust with their consumers via privacy controls and governance for data, shared the importance of ensuring that models developed in isolation in a dev environment are prepared for the production environment. For instance, when a model is developed using python libraries and production is based on a Java runtime environment, conversion is required.

Data scientists can be well-served by using a model format such as ONNX, which is an open format built specifically to represent machine learning models. Look for model formats that are widely used, have built-in optimizations, and support a variety of machine learning frameworks, operating systems, and hardware platforms.

Post-Deployment Testing Strategies

Deploying models into production is far from the final step in providing business value. A deployed model can start degrading in quality since a static model cannot keep up with new trends - the reality of life is that change is the only constant. My company, Spectrum Labs, is dedicated to protecting users from disruptive behaviors and promoting healthy exchange via positive behaviors. We run sanity checks on our models prior to deployment and monitor the performance to track any potential degradation which may trigger a retraining of the model with more representative data.

There are two approaches we at Spectrum Labs take to evaluate and monitor model performance post-deployment:

Regression tests via ground truth evaluation. These tests pull in data from live traffic through the following deployment process:

Before deployment, a new model is first ensured to pass a set of carefully curated data that previous models passed.
After deployment and some time later, new data from live traffic is pulled and labeled to obtain ground truth that can be used to make sure the model is not degrading as compared to registered metrics in the training phase.

Smoke tests via drift detection, where data distributions are monitored to make sure that they don’t diverge in a statistically significant way from the training and testing phases on one side and the development phase on the other.

Sometimes, this feedback loop from the production environment back to prototyping and development is not simply about quality assurance - it is central to the product value proposition itself and how it solves a business problem. For instance, Sturdy unifies customer feedback from a variety of sources into one channel, after anonymizing and reacting PII, and uses machine learning to identify signals in the data that impact revenue retention. Through automation, the signals enable Sturdy’s customers to drive critical business processes and to act on customer feedback as soon as data is received.

Sturdy gets all of your customer conversations and feedback in one single dashboard.

Preserving Data Privacy to Build Trust with Consumers

Models depend on data. The quality of the data used has the biggest impact on the performance of a model. In many cases, the business value is derived from data that originates from people. Data scientists, software engineers, data engineers and ML engineers - we’re not the true owners of this data, just custodians of data that is truly owned by others. In these cases, the management of data and its privacy requires a set of controls to ensure that organizations deliver on their responsibilities to stakeholders viewed broadly.

Of chief concern is understanding:

The provenance of data used in training
How data was collected
How data was treated for bias

Beyond the ethical use of data is the secure use of data: data must be handled off of desktops and managed in a secure and traceable manner with all personal information strictly removed. Conveniently, super{set} once again has in-house expertise: Ketch offers an infrastructure for data privacy, compliance, and security, and Habu offers a secure data collaboration platform (“data clean room”) with comprehensive analytics.

Habu’s clean room software allows for privacy-safe data collaboration between multiple clients’ first party data to obtain valuable insights from aggregated data outputs.

Data and ML engineering is an emerging field. As data scientists, software engineers, data engineers and ML engineers, it’s always helpful to compare notes and get up-to-speed on best practices friends in other organizations are applying to their products. I will say this: only in super{set} will you get a community of data practitioners that are not just leveraging data to solve business models, but are also building businesses to solve data problems!

Transcript

Hide

Oops! Something went wrong while submitting the form.

Written By

When Inference Meets Engineering

By Othmane Rifki, Principal Applied Scientist at Spectrum Labs

Managing Machines Learning Lifecycles at Scale

Introducing MLOps

Creating Business Value Means Seeing Models Through to Deployment and Beyond

Post-Deployment Testing Strategies

Preserving Data Privacy to Build Trust with Consumers

Introduction

Starting From Scratch

The Business Plan

Early-Stage Funding Do’s and Dont’s

Early Team Formation

Creating a Winning Culture: Must-Haves, Memes, and Tips

Building a Kickass Product & Technology Engine

Women in Tech

How to Interview for a Startup

Is Tech Stingy? The Case for Doing Well *and* Doing Good

And, we’re live at super{set}!

Equity and Inclusion

super{set}’s Spectrum Detoxifies The Online Space

From Watsonville To The Moon

Not Just On Veterans Day

Thick Skin, Tech and Black History Month

The Balancing Act For Women in Tech

The Studio Model

We don’t critique, we found and build.

Silicon Valley’s Greatest Untapped Resource: Moms

New Venture Ideation

Good Ideas, Good Luck

Data Eats the World

The Four Types of Startup Opportunities

VCs Write Investment Memos, We Write Solution Memos

Lessons of Grit from my Immigrant Parents

People, First

Navigating Juneteenth

The super{set} Entrepreneurial Guild

Why Head of Product is Our First Co-Founder

Why I'm Co-founding @ super{set}

Too Dumb to Quit

The Product Heist

Founder and Father: A Balancing Act

Early Stage Customers

Overheard @ super{summit}

How I Learned to Stop Optimizing and Love the Startup Ride

Why I Left Google To Co-found with super{set}

The Era of Easy $ Is Over

The super{set} CEO

How To Avoid Observability MELTdown

When Inference Meets Engineering

Infrastructure Headaches - Where’s the Tylenol?

Calling BULLSHIT

Former Salesforce SVP of Marketing Strategy and Innovation Jon Suarez-Davis “JSD” Appointed Chief Commercial Officer at super{set}

Why I'm Joining super{set} as Chief Commercial Officer

When and Why to Bring on VCs

Startup Boards 101

Q&A with Accel Founder Arthur Patterson

Arthur Patterson on Venture Investing

Four Tips for a Distributed Workforce

Arthur Patterson on Company Building

7 Ways to Turn an Internship Into a Job at a Startup

Season 4 Episode 1: Frida Polli, CEO and co-founder of pymetrics

Diamonds in the Rough

The RevOps Bowtie Data Problem

Season 4 Episode 2: Alysa Hutnik, Chief Privacy and Data Security Architect @ Ketch

boombox.io Raises $7M to Build Out Creator Platform for Music Makers

Season 4 Episode 3: Building the Creator Platform for Music Makers at Boombox.io

Four Tips for a Distributed Workforce

Why I'm Joining super{set} as Chief Commercial Officer

boombox.io Raises $7M to Build Out Creator Platform for Music Makers

How To Avoid Observability MELTdown

Founder and Father: A Balancing Act

Infrastructure Headaches - Where’s the Tylenol?

Why I Left Google To Co-found with super{set}

The Era of Easy $ Is Over

Overheard @ super{summit}

The Four Types of Startup Opportunities

The RevOps Bowtie Data Problem

Q&A with Accel Founder Arthur Patterson

Too Dumb to Quit

Former Salesforce SVP of Marketing Strategy and Innovation Jon Suarez-Davis “JSD” Appointed Chief Commercial Officer at super{set}

Is Tech Stingy? The Case for Doing Well and Doing Good