Model Registry Workflows Explained

Video Transcript

I’m not gonna waste time as we have more material to cover. So this is my outline today as previously I always try to recap what we actually covered in the previous parts.

Outline

I’m gonna very briefly go over what we covered in part two, for those of you, as a reminder and then talk about the concepts and the motivations behind Model Registry. What are some of the problems that people normally face today and how we actually address that particular problem. And then share with you what the typical Model Registry UI is, along with how people actually use the APIs as well, as part of the workflow. And then I’m gonna give you a short MLflow Manage Demo and Registry. This particular notebook is gonna be running on an enterprise version of the Databricks. You won’t be able to learn that on the community edition because the Model Registry is not available, so you’ll just have to follow me but you can still have access to the notebook. It’s in the GitHub Repository under notebooks there is a DCE file, you can actually load it up in the Community Edition you want but you won’t be able to register the model because we don’t have the functionality yet. And then once I’m done with that we’ll try to go through a couple of demos on the local host and presumably those of you have been courageous enough to install MLflow following the instructions I gave to register would be able to follow me, if not, don’t worry about it, just look at my screen and see what I’m following through and try to explain through that. We won’t have time to troubleshoot if you actually having trouble on your local host but that’s something that you can probably deal with later on. So with that, there’s a link over here for the Model Registry and let’s go into it.

MLflow Components

So, last two parts I covered in part two, I covered sort of two essential components which were kind of related and they had the same motivations, the MLflow Projects and MLflow Models. And right at the start, I sort of was quite emphatic to say that MLflow was actually built on the premise and proposition that, we wanted to do things in an open manner We wanted to actually support all the common libraries that people actually use and have the ability for them to be able to use this particular projects in diverse environment.

MLflow Projects Motivation

And so in order to do that, it poses certain challenges. One is that how do you actually recreate the particular model? How do you actually recreate the project that you initially started with to reproduce in a target environment? For example Layla might actually produce a project and create a particular model and she wanna share with Lan and Shannon, how can they actually reproduce that on their particular machine? And that poses sort of an interesting problem, one is that because each tool has a very generic way of packaging material for the target environment they want. And people run the execution of target models in different way. People run it on local laws, others run in the the notebooks, others run in the cloud. And even though if you might capture the initial data and some of the dependencies, the target environment might have a different dependency. So the problem was that, how do you actually recreate that? And the solution was that we came up with a way to say, what about if I actually have the ability to capture the entire project?

MLflow Projects

MLflow Project that you’re actually doing, in a manner so that it becomes like a unit of execution, so you capture all the essential dependencies that you have in terms of the code that you’re gonna run, to create your particular model, all the library dependencies that you can have to create the particular model, all the input data, all the training data, all the validation data you have and express that in the config file so that people can actually take that particular unit of execution and then execute on the target environment. And this could be a local execution or this could be sometime remotely. And those of you who come from say the Unix background and I’m dating myself over here, is that think of this Mlflow Project as a make file, if you’re used to dealing with make files, if you’re used to dealing with a lot of open source, when you actually download a particular open source project, it normally actually creates a config file for you, creates the make file for you and then executes the make file, to create the unit of execution, that you can actually run now locally and it will try to figure out what environment you’re running in it’ll try to figure out what architecture you’re running and based on that it will use the right compiler to create that. So think of it in an accrued analogy, it is like a make file. And so how does it actually do that? Well, there are two essential ingredients that actually make up the project.

Example MLflow Project

One is like a simple directory that has an ML project file and then if you look at the ML project file, it’s like a conda.yami file that has dependencies that express these are my entry points and this is why I’m gonna create a particular program, this is the command that you should use to execute that, and here are my input parameters, those are the default. So you can think of it this is an expression way of really creating a project file. And then the conda file is the one that encapsulates all the dependencies that you actually need to execute your particular project or in order to create the particular model. Let’s say that Layla actually created and wanted to share it with the two other data scientists. The second ingredient is that you can now actually execute all of these projects that you actually created in the GitHub repository you can share with others either using the command-line utility or you can actually use it programmatically or if you do a checkout of the particular GitHub repository, you can see into that particular directory and run this particular command and you can run that with your parameters or with the parameters which are default in how somebody who shared that project with you created initially. And the second ingredient is the conda file. And the conda file allows you to say, these are all the dependencies that my model depends on that I actually used initially in my development environment that I use. I use scikit-learn, I use MLflow, I use Pickle, I use TensorFlow, all that. Anything that you actually use to create that particular model, are sort of captured in this particular conda file and together, it sort of gives you the ability to recreate and reproduce that. So MLflow Project in essence is nothing but a directory structure that captures what it requires for you to actually create a particular model. So that in essentially is what MLflow project is. And once you actually have that, you can put it in Git repository publicly and share with other people as well. Similarly, MLflow Model had the same idea the motivations actually came with the same ideas that we wanted to support people to create all these models, in all these different frameworks that you actually see on the left hand side.

MLflow Model Motivations

And we want to be able to have them deploy this particular models that they actually create on the right hand side in these different environments. And as the organization get larger and rather you might actually have different form factors whereby you actually want to deploy this particular model. And how do you actually do that? How do you actually say, well, I’m gonna have TensorFlow to be able to deploy it on Kubernetes but also want to run it on Docker and you notify me to do that, I have to have a one by end by end different ways of figuring out how am I gonna capture this particular model? How am I gonna execute or deploy on that? So the idea there was to create an intermediary flavor.

MLflow Models

Instead of going having the N by M Combinations, we say, what about if you actually create an intermediary level? Which actually captures what we call the flavor of each and every model. And so if you have n number of frameworks on the left hand side, we can actually create the native flavor in which the model was run or created or trained and then we can actually have several other flavors that we can actually add and then based on that, if the model is deployed, say on the target environment, if the target environment understands the flavor in which it can actually load the model, then it can actually do the scoring as well. And this sort of gave us a very generic way of capturing how the model can be encapsulated and then deployed in the diverse environment. And I think there was an interesting question that actually was posed at the end of the second session, where somebody said, what’s the difference between MLflow Models and MLflow Project? They seemed to be very different. We can think of it, MLflow Project is a producer of an MLflow Model and so you can think of it MLflow Project is nothing but a make file that allows you to create your model that you actually wanna do. And then model is what you actually deployed on these diverse environments and depending on where you actually are deploying that particular model, that particular flavor can be invoked by the inference engine to do your scoring. So how does MLflow model does that? Similarly with project you actually have ability to log a particular model in the MLflow flavor and the API, the fluent API, that you saw in part one and part two as part of the tracking, is one way you do that is by using this API called log model.

Example MLflow Model

And when you log a model what happens underneath is that we actually created a directory structure for you and underneath there’re few files which make the essence of the model. One is, your EML file called MLflow MLmodel. And yet that sort of captures a couple of things, you’ve got all the metadata associated with that and then you can see there’re two flavors that we actually create.

The native flavor here is a TensorFlow in which you actually did that and a couple of other metadata is what directory it contains and what are the signature he actually uses to load the model and do the prediction and inference. So that’s particularly one flavor. The other flavor now is a generic flavor what we call the Python functional flavor. This is actually quite powerful because this allows us to be able to deploy this particular model in any environment, that rate running a Python and you don’t need to worry about the underneath infrastructure, whether it’s TensorFlow, whether it’s Keras, you need minimal library of TensorFlow to load that particular model and execute. But what you’re doing is you’re actually encapsulating that as a Python function and then just using the predict function to actually do that by providing and this allows us to say, I can take that particular model and put it in a Python environment that’s running in a Docker or I can actually use this Python function and load it into a Spark and then use the UDF to do the batch inferencing and so on and so forth. So this is actually quite a powerful notion by saying, if you have this particular flavors, we have the ability generic detectors multimodal and deploy in diverse environments. And to give you an example of what I mean by these particular flavors think of that you actually created all these different flavors of Python, you might use a pychoice to create a particular model and you save that by using a PyTorch, say load model.

Model Flavors Example

It has a Python flavor. You can have a scikit-learn, you can have Spark, you can have tons of fluent Keras and then the end result when you deploy this particular model, what you now have is the ability to load that particular as a Python function and then just invoke the predict function on it by supplying the pandas input frame. So this is actually quite powerful. And then you will actually see, we saw that in part two, in some of the notebooks, how you actually had a Keras model. We actually created the predictive model using project APIs. We loaded the Keras function that we actually executed as a Python function and then we did prediction on it. So this is actually quite imperative in us. It’s ability to deploy this in diverse environments. So that in essence is what project and models where they’ll be actually covered in part two. And you can actually look the videos to go deep into how these are implemented and how you can actually use the APIs to create a particular project, how you can actually use the Python function to load a particular model of particular flavor and how you can actually deploy them as well. So that in essence is what projects and models are. Now I wanna shift to focus to cover the main topic of today, which is the MLflow Registry and registry is the last component, that we introduced in 1.7 and it was recently GA’d.

MLflow Components

And in essence that completes the entire life cycle where people can now register the models that they have developed and other people can discover what the models are. You can annotate the model, you can put that into the central repository, you can think of it as a Github of all your models, except that this is within the organization and people can discover that model and do that. So what are some of the problems in absence of a Model Registry? Say, you don’t have a Model Registry, how do people actually work?

The Model Management Problem

What do they actually do? Well, it’s good if you actually are a single developer, if you’re by yourself or maybe one or two people in the group, it’s very easy to say I’m just gonna create a classify and if it’s a Keras model I’m gonna tag that as a version two and if I read it on another experiment, I’ll classify as a two and then if I have, a month down the line I just put all this annotation then tags to it, which is fine, I mean, it’s well and good if you actually dealing in a small environment. But it sort of poses interesting questions. When you’re a large organization where you have hundreds of developers and you develop loads and loads of models on daily basis and weekly basis, there comes a point where you actually wanna ask questions like, where can I find the best version of this particular model, that Layla developed? That Shannon developed? or Lan that created a classifier where can actually find those? How can I search those model? How do I find out how this particular model was trained? And this is the whole idea about provenance, how do I actually figure out and look at the history of how this particular model evolved over a period of time? How can I track some of the documentation associated with? It’s important that you share that information and how have you trained them model with the parameters we used and so on and so forth. And then how can I review the particular model? If I want to integrate this part of my model management life cycle with my CID/CDs tool? How do I actually inspect that particular model, check it out, run it, make sure everything is fine, I’m happy with it and then I, as a reviewer can then manage the stages of its next evolution, how do you actually do that? And this becomes imperative in large organization, right? And so the whole idea with Model Registry, was this vision that, what about if you actually have a central repository, a control central repository, that allows you to do all these different things, where people can use the tracking API to actually experience the model, when they’re happy with it, they can actually go and register a particular model they’d actually have created and then this particular process of reviewers and CI/CD tools can actually look at the particular model, go through the evolution, go through the testing, do the A/B testing, whatever they should be required to do and then make the appropriate transitions as they go from developments, to staging, to production and then finally, if the model is of no use, they can archive it and eventually delete it.

MLflow Model Registry

And once you actually have a model registry, the downstream people can check out a particular version of the model, can check out a particular stage of the particular model to use it in their automated jobs or they can use that to do some REST serving and so on and so forth. So that in essence was the idea behind Modern Registry, is to have a central repository where you can actually put stuff and discover and so on and so forth. And so, if you look at what the Modern Registry UI or a workflow looks like, you have a central monitoring repository that actually gives you a unique name version of particular model. You can keep track of all the different stages, as I said earlier. You can inspect different versions of the model, I can actually find out what particular versions of the model are registered with this particular model called Airline Display. I can look at the descriptions and annotations and links and PDFs and all the documentation associated with all the lineage that I can keep track of. And this gives us a very distinctive way to create two workflows that are very common in how people actually use Registry. And I think these are not very different from how you use Gitub in your compile edit cycle.

Model Registry Workflow UI

And so the workflow for UI is very simple. You train your model using the MLflow track API once you’re actually done with it you’re gonna go to the UI to figure out what my artifacts are, what my parameters are. We saw that in part one and part two, what the UI looks like. And then towards the end you’ll have the artifact where you wanna say, okay, I have this particular model and when you click on that particular model name under the artifact, you get this button on the right hand side that says Register. And when you register, the developer would normally register to create the first version of the model or if the model exists, then it will create the next version available or underneath it will create the subsequent next version of the model. And the guide and registered the production model called scikit-learn. And what this happens at this point is that the registry will now create a version one of the model. So essentially the developer takes a particular model, goes to the UI to say, okay, I’ve tested it, it looks good to me, I’m gonna go and register it and I’m gonna notify my reviewer to say, okay, I’m ready to actually make the transition. And so the reviewer can actually come in and annotate or whatever the action is required for that model to be sort of more comprehensive and digestible for people who are gonna use down the stream and he or she can actually take that particular model and make the appropriate transitions from say a none to staging, to production, to archive. And then finally, if more versions of the model are created by the developer, he or she can actually go in there and then create different versions and you can have different versions of the same model in different stages and this gives you the ability to do a lot of A/B testing, actually one. And once you have all these different versions of the model available, now you have the ability for people down the stream who can actually use the APIs to check out a particular model or use that in their automated daily jobs to do a batch inferencing or they can actually use that as part of the streaming REST serving at that point. So that essentially is the Model Registry UI that you actually use, the workflow that you actually use. Now, there’s a similar way you can actually do that with the API as well.

A lot of people, developers really don’t like to use the UI, they can do pretty much the same thing from the IDE or from the Python file. And you actually use the Model Registry API, the client API to go ahead and register the model. Here I’m registering a model cold weather forecast and it’s going to go ahead and either create a new model called weather forecast with that particular URI and if one exists, then you’re just gonna create the next version of it’s available or I can actually do it from within my MLflow run where I’m tracking the server and when I’m done with it, I just go ahead and create registers as model weather forecast. The reviewers can now query the MLflow registers to say, get me this particular model, now transition it over to take the version five of the model and transition it from staging to production or from none to production. And then people down the stream can use the APIs to say, go ahead and here’s the model that I want, weather forecast and the stage I want to check out for me is production. So I’m gonna go ahead and check out this particular model, I’m gonna load now this particular model into my code and I’m just gonna invoke my critic function to do my scoring. So that in essence is the just behind the Model Registry API. You have two distinct ways to actually interact with the Model Registry. One is using the UI to do that and if you are one of those developers who hate UI and you wanna do everything with API, which is a good thing, some of us love doing that, you can do that through the registry API as well. And the modern registry is backed by two stores in the back.

MLflow Backend Registry Stores

Since 1.7, we now support the Model Registry to be a part of

SQLAlchemy store, where we actually put all the metadata, all the runs, all the versions and everything, both on the local host. And then on the MLflow managed version, we actually use my SQL to store all the running data. On the local host, you can actually just run the SQL Lite to look at what your database looks like and make queries if you actually want, if you’re curious. And then the Artifact Store is the one where you actually store all the artifacts. And we talked about what artifacts are more in the training, these other load files, these are the input data, this is the training data, this is the artifacts that you might create such as plots or a feature matrix or confusion matrix, anything that you actually think that you want to share make it part of that is actually stored as an artifact. And then when you launch the particular server for people to connect to your tracking server, you can just provide those particularly as part of the command line arguments and run with it. So that in an essence is what the Backend is supported by the Model Registry. And these are all the artifact different stores and we support and you can use your Model Registry with any of these SQL compatible Alchemy stores as part of the entity admitted data. If you’re curious to know what it looks like, if you run this on the local host, you can just use SQL Lite, which actually comes with Python three it might have installed it you just run it as part. And this sort of gives you the schema, what the schema looks like. These are all the different model versions that you actually create. The Registered Model are the ones that look alike. And then these are the standard schema that you actually use. And you take this particular schema and import it and export it, if you want to change your database or you want to do backups, that’s sort of a good way to do that. So to cap essence, the fourth model is the Model Registry.

MLflow Model Registry Recap

And I said it is the central repository that allows unique name registered models, so other people can search other people can discover, the data teams can actually use it in a very collaborative manner and in a very controlled manner. And the enterprise features has Role-Based Access Control, so you can actually provide permissions to who can actually check in, who can check out a model, who can make the transition and so forth. So that is an essential part of registry where you actually have a control access, so you can monitor, you can prevent accidents by people inadvertently taking the one model that’s not really ready for prime time and putting into production, whether the reviewers can block that particular request, so on and so forth.

As I say, it provides you sort of two way unique Registry Workflow APIs you can do that in UI or you can actually use all the registry operation using the APIs as well and gives you the ability to have multiple versions of the same model and I think, you’ll never have one version of the same model, you’re gonna to have multiple versions of the model throughout your development cycle. And this gives you the ability to choose the best model that you want and then make the appropriate decision, whether you wanna deploy it in a particular stage, or is it ready for prime time. And model stages gives you the ability to transition that. And then finally it has the ability for you to integrate with other applications, including forecasting, that you can actually check a particular model and then do a forecasting or you can actually integrate with Jenkins APIs on your nightly basis to test to see if the particular model is ready for production. So that in essence is what Model Registry is. And then just to recap all the three parts, that we actually have talked so far, we talked about part one, part two and part three and now we actually have sort of a comprehensive picture of what these four components are, how they work together.

Recap of all parts: What Did We Talk About?

They’re not monolithic, you don’t have to use all of them, you can choose one of the two and they work very closely together and they have a set of APIs to do each and every different operations you wanna perform on that. So it’s very module in fashion and greatly simplifies this entire life cycle that we talked about in part one. And if you have started using it in part one and part two and you’ve be doing the homework assignment, probably you’ll come to the conclusion, I hope you share that with me, that it is easy to install, it’s easy to use, the APIs are very fluent, very pythonic in nature and it gives you a great developer experience. They’re easy to deploy, in other words you can actually use it locally or you can actually use it remotely. As I said, the pythons available in the programming languages that you like, in Python or in Java or in R and soon to have one in Scala.

All the rest APIs are available for you to actually talk to the track and silver of the registry from any application you want, so you’re not tied to just using Python if anything that uses a REST API, you can actually use those APIs to connect to all these different components. And there’s a great visualization tool UI that allows you to compare and contrast experiments and select the experience that you want as an ability to select the best one and then register that with the Model Registry in order to have this endless cycle management. So that in essence is what Model Registry is. And so with that, I think, are there any questions on the panel that the panelists think I should address? If not, I can move on.

– [Karen] I don’t think so. I don’t think we have any questions actually, maybe one question. – [Man] Jules, you must be doing such an excellent job, that no one has any lingering questions. – Oh, well, that’s very flattering either that or I put them to sleep. Okay so, let me just go to my notebook.

Now this pretty good notebook is available for you, it’s in the notebook folder in the GitHub, there is a DBC file, you can upload that. However, I do forewarn you that this is a managed version of MLflow Registry, so some of the features over here that you’re gonna see are not part of the community edition and they’re not part of some of the features in the open source. So you won’t see that you won’t be able to run that in the community edition. But it’s good to actually have as a sample example, so if you actually ever wanna try or manage version free trial with 30 days, you can actually load this particular registry, I mean, load this particular notebook and play with it and you’ll see the merits of what I mean by having a control access and some of the enterprise features which are available for managed flow on Databricks. So this is a machine that will be forecasting wind power application. It actually takes three features wind, speed, and air temperature to predict what the power consumption kilowatts is gonna be of this particular wind farm. And it takes three recordings every eight hours and then gives you the aggregate total at the end of the day, what the kilowatt consumption is. And the data set can be gathered from the National wind tool Sitka. And essentially as I said, it takes three recordings at eight hours a period and then gives you the aggregate over the period. What you wanna do is you wanna sort of use a Random Forest Regression to predict the power. And then we gonna be able to use a couple of models to actually do that. Now, if you were here with me in part one and part two, you probably see that I had this notebook run called run dot class init and these are my classes that I normally use to sort of delegate all the work that I need, sort of good abstraction way for me to put all the code that I actually need away from it. So my notebooks are not very cluttered, the notebook becomes a driver. And so when I run this particular classes in it, it’s gonna go ahead and execute those classes or initialize those classes. So then I have the ability to use them in my particular notebook. And then just to give you an idea of what these classes are we can actually go to they need classes. And some of you are probably familiar you’ve seen this is the very same pattern that I’ve used in part one’s in part two, where I created all these high level classes that allowed me to be able to use them, again and again in other places. And then this is just a utility class, which is a static class that has static methods in Python that gives me the ability to get to training data, get validation data, get slices of weather forecasting that I actually need for five days or six days or that way I can plot things in a nice modular way to compare my forecast with my predictions. There is the random plotting utility that gives me the ability to plot things and that way my code, my main notebook is not cluttered, everything is sort of dedicated to this particular task. And then the main class that I have two models that I’m gonna re-use to create two different models. We’re gonna register it then we’re gonna use the best one. This is the Keras model and the Keras model has the constructor that takes in the input parameters I need to construct and compile the model. And then here’s my gist of the program called MLflow run. And you’ll notice that, this is now I’ve just have one code called mlflow.keras.autolog and this is a new feature that we’ve actually added and we’ve added the TensorFlow we are gonna add to scikit-learn one exists for glue on and XG boost and LGM. And all it does is that you now, using the tracking API, you don’t have to load each and every metric. You might have hundreds of metrics that your code becomes cluttered, so when you use this one code mlflow keras autolog it figures out what the metrics that you’re actually competing with this particular algorithm. And I’ll go and load that for you automatically. So your code now becomes very simple, very easy to use and once you use that, it will load the metrics, it will load the model and so on and so forth. And then you, normally use that before you actually invoke your fit program. And then I have another class called a LFR model, you’ve probably seen that in the previous. It’s the same thing, I just have an ML for a run that actually takes and trains the model and then at this point registers the model where, if the model exists, then it will create the next version of it, If it doesn’t exist, I’ll create a model with this particular name. So that in essence is what my code looks like. So let’s go back to my notebook quickly and then run this code. So I’m gonna load this particular file, which is about, I think, 2,500 entries or so and let’s look what other file looks like. So we’ve got three recordings, as I say, I’ve used this as an index on my pandas data frame and then this is the power that I actually want to predict and if you’re just curious about to see what the data looks like as a distribution, we can actually just use this particular panda score to find out. So we can see the mean average is about 2,400 kilowatts per day. So let’s do the first thing. Let’s get the crib training data and validation data for our first model and then what we’re gonna do is a Keras model to do the linear regression. So we are gonna have one input layer, one hidden layer and then when one output layer that actually gives us the output. And then over here, it’s very simple, I’m gonna have three input parameters that I’m gonna use. These are my hyper-parameters I’m using to create three different instances of model. And all I’m doing is creating an instance of a Keras model and then execute it’s MLflow run and then it’s gonna do all the logging for me, it’s gonna do all the good stuff that I actually need. So this way, my code over here is less cluttered, it’s quite clean. And I can share those classes with anything. You can think of it as nothing but like a PyModule that you just installed by importing that. So this is gonna run with three different input parameters, input units. The loss is, I wanna compute as MSE and notice that as this thing run, it’s gonna go ahead and create the losses as I go and create the mean square error. And this is gonna take about a couple of minutes but it’s actually running in real time using Keras and the Backend extensive flow. And once we’ve done that, we’re gonna go over and go ahead and create and register the model with the Model Registry.

All right, so that ran, didn’t take that long, about a minute or so and then I can actually go over here and then go ahead and register this particular model. So I can go to my runs as we saw in part ones and part two the MLflow UI is part of the notebook is actually integrated, you don’t have to run a separate UI to do that, it’s part of the notebook, it’s actually quite convenient. And when I click on that, I get a summary right away on this right hand column over here of all the runs that I actually did that resulted from that particular call. And if you sit that, this is all the parameters, which were automatically logged. I didn’t have to have log param inputs, log param outputs and log param all that all these were automatically logged to me because of the one call called Keras autolog. So I can actually go ahead and look at these models to see what they look like, at one quick glance.

And these are all my particular models that actually ran. I ran this one just a minute ago and let’s go ahead and pick up the best one that we actually have. I can actually go ahead and compare these models if I wanted to.

Side-by-side, I can actually go ahead and look at their validation loss, step-by-step to see what they’ll actually look like. And this is something very similar what you would get if you ran tensile board and then log the Keras as a callback function to write things to the log directory, to actually get all that. So I can actually look at my validation loss for each and every epoch, is it actually going down training? So we’ll go ahead and pick up one model, that and then let’s go ahead and put that into production.

So you can see when I click on this model, I get this particular button called Register Model so I’m gonna go ahead and register model and then I go and create this model. Now there’s a lot of other models which it took people to actually create in the registry, but I wanna create something called PowerForecastingModel.

You could do that, create a new model and I’m gonna register this. And when I register it, as I said, you actually get the version one of the model in your registry. And I can go to that particular version one model and I can annotate that if I want or I can make the transition or I can make a request to make the transition. So I’m gonna go to my Model Registry and go ahead and put in whatever description that I want over here. So I can type in whatever descriptions I like I can put images over here, I can have my entire weekly PDA, if I actually want over here. So this way it gives me the ability to annotate this particular model and share with people who actually will check out this particular model, they have the ability to do that. So next thing I’m gonna do is I’m gonna request this model to be transitioned. So I have the ability to transition model from one stage to the other one. So I’m gonna go ahead and request this transition to go into production. I’m being a bit overconfident so I’m gonna our chief data scientist, Leila, can we put this in production please.

And you can see there’s a lineage of all the requests that come in and Layla is gonna come up with, and she’s gonna reject that. Are you bloody mad, spell first.

So we can say, as I go through these actions in my UI, we actually keep track of that. And so I’m gonna back over here let’s go ahead and she’s gonna run some testing, it’s okay. This, this seems like good, I’m gonna go ahead and transition, that looks good. Let’s put this in production.

So I have my models over here, I can say Power and I have this particular forecasting model at this point I have one in production. And so let’s go back and go on with our notebook. So the next thing I’m gonna do is go ahead and perform the transitions and integrate that with the forecasting application. So this is my power forecasting model. And one way, as I said, you can actually load different ways of the model. I can use a python function to load my version of the model or I can actually use one that is a production model. So let’s load this particular model, which is the same one as the production or I can actually load a production model, which is what we just transition something into production. And this is sort of the API that CI/CID could actually use to test this thing. And now I can integrate this particular model with my power forecasting application that my have, that gives me the ability to say, I’m gonna take this production URI and I’m gonna test this my wind data to see, how did the my prediction happen over a period of five days. And so we’re gonna plop this forecasting model with my production to see how the model fared. So this is my true values for the prediction that I did, these are my true values and the blue ones are what the model actually predicted. And it seems that it’s not bad , it’s tracking, well. So I’m confident now that this model is fine. I could have done this in staging and then Layla could have gone, okay I looked at this particular forecasting model I tested it, it seems like good, I’m gonna go ahead and make the transition to production and then she can actually make that transition into production. But at the same time, normally in practice, you’re not gonna create one model, you’re gonna create multiple models. And the whole idea behind or premise behind MLflow was for the data scientists to give them multiple ways to create two different models, solving the same problem and then they can evaluate, which is the best one and then use that as part of either A/B testing or when they’re confident that one is better than the other, they can choose that one and then go ahead and deploy that. So now the next model we’re gonna do is the Random Forest Regression Model. And these are the parameters I’m gonna use.

And the idea here is very similar, I am creating three different models, I’m using three different estimators. And then over a period of time, you can actually use different hyper-parameters and make sure that you actually get the best model but this actually gives us a good idea what all those models look like. So we’re gonna go ahead and run these.

And then this gives me sort of three different models. And this is quite fast as you can see it took not more than few seconds to run it. So now I got my RMSE and actually I can just quickly look at it I don’t even have to go to UI now, this is now somebody who’s actually working with a CI/CD tool using the API. So let’s simulate that somebody is actually working outside. So I’m gonna look at this particular model over here, 210 seems to be the best one, so I’m gonna pick that as my version four, so that’s what I actually have. So let’s get this into version four and I’m gonna make the transition of that particular model and then put a description in there for annotation later on down the people who actually loaded. And I’m gonna test this particular model to see whether we can actually test this model forecasting application. I’m gonna transition that to the staging, so this reviewer can actually check out the staging model and then run it into the forecasting. So we’re gonna stage this particular model, so we made the transition and then see how this looks, like I said, this is far more better, right? It’s actually tracking really well and very close to what the true values are. So I’m confident that this model probably is far more better than the Keras. So we’re gonna actually use that to deploy that in our production. So you can think of what I’ve actually done is kind of an A/B testing but not in the deployment environment but in my CID/CD tool where I have a Keras model, that I deployed and that I registered with the registry and I checked out different versions of it. I integrated with my forecasting to test to see how good the accuracy were, over a period of time. Then I built another experiment for model where random forests with three different parameters shows the best particular RMSE that I got and I used in my forecasting application, my testing environment to see how it actually tracked with the true predictions, as opposed to the to the predicted one. And then this one seemed to be probably the best one. So I’m gonna go ahead and choose that. And either Shannon or Layla or Lan would say, okay, this seems like a good way to put that in production and I’m gonna transition that particular model into production. And I don’t have to go to the UI, the reviewer can actually use the API, that we actually so in our registration, in our presentation. So this is my model that I’m transitioning and then I’m gonna go ahead and now use this model to make sure that it is the actual model that I have.

And this is now my production model that actually did the forecasting. So that in essence is really the way we actually do the use of the model API and the UI to actually use Model Registry within the notebook or you can actually do that from your IDE or from your local Python file. So that in essence was the demo of how you actually would use Model Registry in a notebook. And this one, remember this was the model the managed version of the registry, not the open source. So I’m gonna do something very similar now on my local host but a smaller version of this, okay? So let’s go to my host now, over here.

Now, those of you who were brave enough, yesterday or today, probably did the install your model flow or ML flow and all the other packages that I made as part of the read me, hopefully you actually went through this to install MLflow on your local machine. How many of you actually were successful in installing this part one through six on your local machine?

Anyone?

Okay silent huh? – [Karen] Just a reminder, you’ve been responded in the chat. – Oh, okay. – The questions – All right. – [Karen] Everybody’s so quiet today. – All right, fine. Then silence is a form of complicity, I like that. Alright, brilliant. So then we’ll just carry on. So what I’m gonna do now is run the same thing but not exactly identical but something very similar. I’m not gonna train the Keras model but I’m gonna use my IDA where I’ve actually loaded in my Python PyCharm. I’m gonna go ahead and run this. Now, notice one thing I’ve done is I’ve set my tracking URI to be SQLite, to run on my local machine. By default, it creates this particular DB file, which is Alchemy compatible. So I’m saying from now on my local machine, my tracking server is running locally and I’m gonna use the SQLite to store all my entity data on the local host. This is my wind path from my data, same utilities I’m using for classes, same pandas, and pretty much the same code. I have pretty much taken this code and put it in a notebook because it’s a sort of very modularized classes, I can repurpose this particular. I’m just gonna go ahead and each read over three different parameters, three different tuning parameters and run it. I’m gonna call this forecasting model and then I’m just gonna go ahead and load my test data and then deploy this model from the registry, into my local server and then do the scoring. One of the things that you can actually do, is that you can check out a particular model from a registry and then run it locally or you can actually deploy it remotely as well. We won’t get into deploying remotely because it’s a bit involved in terms of setting up the configuration for safe, you won’t deploy it on the stage maker or Databricks but there’s a Command Line Utility for you to actually provide the configuration to create a cluster and you would deploy in that particular cluster. But for simplicity and because of the time sake, we’re gonna do it on local host, okay. So let’s go ahead and run this.

So this is gonna run as it is gonna give me three different metrics and I’m gonna pick up two and then use the UI to go ahead and do the MLflow registry UI. So the second thing I’m going to run, I’ve already done this, so this is what I did but I did it from my PyCharm, I’m gonna go ahead and launch my back-end UI, I’m gonna use my MLflow UI. And giving the arguments saying I’m using my backend URI, where all the model registry is and all the tracking data is available and by default the artifacts store is gonna be a directory called ml runs. So if I go over here, you will see that I have a directory ml runs and I have a file called ml runs. So let’s go ahead and load this.

And that’s going to launch a local UI for me.

Let’s go ahead over here and launch this particular server. So this is the opensource version of the UI looks very similar, except that this is not within the notebook. You’re actually running that a flask unicorn server. And I can actually go ahead and look at my RMSE, I’ll pick up the best ones. Let’s go ahead and put this, let’s look at this one, click on this one and this is my model and I’ll go ahead and register the model here, which I did very similar to what I did, register this model, that creates a version one and I can actually go to version one and then make the transition. Now you noticed that this’ll be slightly different, you actually don’t have different approval process because we don’t have permissions available on our back-end UL in open and it’s pretty open, anybody can do that. So I’ll just go ahead and put that into production. I’ll go back and take the second model that I have and let’s put this one into staging.

Now, we already created a particular model with the first versions, I’m just gonna be using the same thing and when I select the same thing and I register it, this is now gonna create a version two of the model. So I can go ahead and make the transition into staging, if I want.

And if I go to my powerful casting model, I actually have two models, one is in production and one into staging, right? So let’s go back now to our exercise over here and see what should we would we do now. So the second assignment is to go ahead and deploy this particular server from the registry. We’re gonna take the production version of it and we’re gonna go ahead and deploy it. So let’s go ahead and do that.

So I don’t need the UI, I’m gonna kill it. And I’m gonna go ahead and deploy this.

And if you look at this deploy shell script all it’s actually doing is actually taking the production name of that particular model, is setting the tracking UI to say, go ahead and this is where my registry is and then go ahead and solve the models and here’s my URI, here’s my model name, here’s my registered model, here’s a production and then don’t use my conda environment, I don’t need that because I’m already running within the conda environment. So this is gonna put it to solve the model on my local host with this particular unicorn host. And I can go and now do prediction on this particular model, by doing, so if you look at make prediction if you look at this particular file make prediction, all it’s doing is actually calling, (typing) it’s invoking, sending a post request, a REST API with this particular format, for my records. So these are all my data that have actually been send in and then I’m getting a prediction back. You can actually do that using also the Python. I can actually go over here and use pretty much the same thing by sending a post request to this particular server, where I’m sending in this particular JSON oriented record, as my data to give me the power prediction that I have. So that in essence is Model Registry. You can actually use the managed version of Databricks to get all the enterprise features where you actually can provide Role-Based Access Control and so forth. One thing I actually didn’t show you was the ability for you to actually go to and set permissions, so that people can actually either can manage a particular model, whereby they can create a model, they can register a model, they can delete a model and so on ans so forth or you can actually provide a Role-Based Access Control to a group of users who can check out, check in and use the APIs. So these are the enterprise features which are just sort of available on the managed version of it but they’re not currently available in the open source. So that in an essence covered our, I know we had a lot of material to do you have exercises and homework to do at home. So at this point I think I should end and stop here to ask any questions, I’ll answer any questions people might have.

Let’s see, – I think there’s maybe one or two questions in the Q&A Jules, are you able to see that? Or I’m happy to. – Okay, I’m looking at it. Are there any pros and cons between the two flavors as a registered model? No, they aren’t, obviously there’s always the native flavor that people somehow prefer because then you actually can use it natively. Let’s say you created a Keras model in a Python Floor and you want to use the Keras one you really don’t care about Python function because you don’t wanna load the Python function and they have the ability to interpret that and then hand it over to the underlying Keras to do the prediction. So I think that’s the only difference but the multiple flavors allowed you to create, say a Keras flavor, a pyfunc flavor that you can actually deploy it, on Spark to do batch processing or you can actually deploy it, on the streaming server to do modal referencing or you can actually create another flavor, say XGBoost that has an Mleap flavor, no Spark there, you can actually create three flavor, a Python function flavor or a Spark flavor and Mleap flavor, then you can deploy it in different versions. So it’s not so much that one is better than the other, it depends on how we actually wanna use it but the whole idea behind flavor is that it gives you the flexibility to do that.

I missed the stage production just as tags, that one can create. No these are, currently these are the predefined tags, non-staging production in archived that when you actually get by default. There have been suggestions and there have been a couple of PR requests, where they wanna have multiple stages so people can actually configure those stages and we are kind of making that as part of our roadmap to say, hey if you give a particular file, you can actually create those stages. So this has actually has come up, right. People do wanna be able to not be compliant to using only the default stages but they wanna use the stages in their own environment.

Okay, do you recommend use DB Connect in combination with MLflow code tool in local and deployed? Yeah, you can use DB Connect. DB Connect is actually a great way for you to use things locally.

I enjoy working on a PyCharm as you can actually see. I do all my development PyCharm and then DB Connect is actually a good way to connect your local machine to remote server. So I can essentially develop everything locally I can track and deploy everything on the Databricks. So DB Connect is actually a great way to do that. Talking about DB Connect, I think that gives me the inspiration to write a blog, to show how you can actually do that. So thanks a lot for the question.

– I think that’s it.

That was easy, easiest Q&A ever, right Jules? – Oh lovely, lovely, either that or I just totally bored them out of their minds. But I wanna thank my panelists for helping me out today, Karen thanks a lot for hosting it. All my participants and attendees who attended. I’m gonna put my, where’s my PowerPoint presentation? Here, my creds here, where did it go?

The slides are available, you can just send me an email at jules@databricks.com. If you have any questions, connect me on LinkedIn, DM me on Twitter as well, I’m more than happy to answer any question that you actually have, on YouTube and we have a Slack channel for Databricks, all the way how to get started with MLflow are on the slides, so feel free to participate and feel free to send me any questions you have and thanks a lot for attending. – Thanks everyone for joining us and thanks Shannon, Layla and Lan for attending. I know there wasn’t too many questions but it’s always great to have you there in the Q&A just in case for the attendees. So with that, I hope everybody has a great rest of your day and thanks for joining. Take care.