How to install ClearML Enterprise on Red Hat OpenShift

The demand for artificial intelligence (AI) and its intelligent applications continues to grow. Red Hat offers Red Hat OpenShift AI to meet those demands and expand our partner ecosystem, allowing for greater AI capabilities. A previous article walked through installing the ClearML open source solution. This article covers a basic install of ClearML Enterprise on OpenShift.

What is ClearML?

ClearML offers a comprehensive suite of services that help customers revolutionize their machine learning operations (MLOps) landscapes. With a focus on end-to-end MLOps management, ClearML facilitates seamless experiment tracking, collaborative workflows and efficient model deployment. This open source platform integrates popular machine learning (ML) frameworks, providing users with a versatile toolset for data management, scheduling and optimized computing, and model monitoring. ClearML is a pivotal solution for teams and enterprises aiming to expand their MLOps capabilities and optimize their machine learning processes in one consolidated platform. ClearML Enterprise offers customers additional features to improve their MLOps, such as scalability, security, collaboration platforms, CI/CD tools and more.

Why use Red Hat OpenShift with ClearML Enterprise?

ClearML Enterprise and OpenShift share many key features, providing opportunities to grow, customize and monitor intelligent applications on consolidated platforms. When used as part of a cohesive enterprise AI strategy, OpenShift and ClearML Enterprise offer:

• Extended scalability enabling organizations to handle more complex machine learning workloads and intelligent applications.
• Greater customization and integration options as well as deeper integration with existing enterprise infrastructure and ISV tools from the partner ecosystem.
• Enhanced security and compliance footprints to help meet the often stringent IT security and compliance requirements of corporate systems.
• Advanced analytics and reporting that extend beyond OpenShift’s cluster monitoring and analytics capabilities, with ClearML delivering deeper insights into machine learning processes.

Prerequisites

• OpenShift version 4.14
• OC and Helm tools installed
• ClearML Enterprise credentials

Before beginning, make a new directory called clearml and move into it to keep this walkthrough organized. Set a WILDCARD variable with a wildcard URL, as seen below:

$ mkdir clearml
$ cd clearml
$ export WILDCARD=<YOUR_OPENSHIFT_WILDCARD_APP_URL>

Example: apps.ocp4.example.com

You will use this WILDCARD variable later when you set up the Helm overrides configuration.

Deploy the Helm chart

You can deploy ClearML Enterprise via Helm on an OpenShift cluster. Pull the Helm chart to add the repository and update it using the credentials provided by the ClearML team.

$ helm repo add allegroai-enterprise <CLEAR_ML_HELM_CHART_URL> --username <YOUR_USERNAME> --password <YOUR_PASSWORD>
"allegroai-enterprise" has been added to your repositories

After you add the repository, update Helm using this command:

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "allegroai-enterprise" chart repository
Update Complete. ⎈Happy Helming!⎈

Now you have your repository. Create a new project for ClearML on the OpenShift cluster:

$ oc new-project clearml
Now using project "clearml" on server.

You can add applications to this project with the 'new-app' command. For example, try:

   oc new-app rails-postgresql-example

to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application:

   kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname

Create a file called override.yml with the following information to override the default configurations in the OpenShift cluster. You will need image credentials to access the ClearML private Docker registry. Insert that password in the imageCredentials field.

$ cat <<EOF >> /tmp/override.yaml
imageCredentials:
 password: "<CLEARML_PRIV_REGISTRY_CREDENTIALS>"
elasticsearch:
 rbac:
   create: true
   serviceAccountName: "clearml-elastic"
apiserver:
 podSecurityContext:
   runAsUser: 0
 service:
   type: ClusterIP
 additionalConfigs:
   apiserver.conf: |
     auth {
       fixed_users {
         enabled: true
         pass_hashed: false
         users: [
           {
           username: "testuser"
           password: "testpassword"
           name: "Test User"
           admin: true
          },
         ]
        }
       }
 ingress:
   # -- Enable/Disable ingress
   enabled: true
   # -- ClassName (must be defined if no default ingressClassName is available)
   ingressClassName: ""
   # -- Ingress hostname domain
   hostName: "clearml-enterprise-apiserver-clearml.$WILDCARD"
   # -- Reference to secret containing TLS certificate. If set, it enables HTTPS on ingress rule.
   tlsSecretName: ""
   # -- Ingress annotations
   annotations: {}
   # -- Ingress root path url
   path: "/"
fileserver:
 service:
   type: ClusterIP
 ingress:
   # -- Enable/Disable ingress
   enabled: true
   # -- ClassName (must be defined if no default ingressClassName is available)
   ingressClassName: ""
   # -- Ingress hostname domain
   hostName: "clearml-enterprise-fileserver-clearml.$WILDCARD"
   # -- Reference to secret containing TLS certificate. If set, it enables HTTPS on ingress rule.
   tlsSecretName: ""
   # -- Ingress annotations
   annotations: {}
   # -- Ingress root path url
   path: "/"
 # -- File Server extra envrinoment variables
webserver:
 service:
   type: ClusterIP
 ingress:
   # -- Enable/Disable ingress
   enabled: true
   # -- ClassName (must be defined if no default ingressClassName is available)
   ingressClassName: ""
   # -- Ingress hostname domain
   hostName: "clearml-enterprise-webserver-clearml.$WILDCARD"
   # -- Reference to secret containing TLS certificate. If set, it enables HTTPS on ingress rule.
   tlsSecretName: ""
   # -- Ingress annotations
   annotations: {}
   # -- Ingress root path url
   path: "/"
EOF

You need to change the security context constraints for some users. Currently, you must allow broader permissions to deploy ClearML Enterprise, so allow anyuid and privileged permissions to the following users. ClearML does have a non-root option for security, but due to a bug in Helm at the time of this article, you must allow broader permissions. Here's an example:

$ oc adm policy add-scc-to-user anyuid -z clearml-apiserver
$ oc adm policy add-scc-to-user anyuid -z clearml-enterprise-mongodb
$ oc adm policy add-scc-to-user anyuid -z clearml-enterprise-redis
$ oc adm policy add-scc-to-user anyuid -z default
$ oc adm policy add-scc-to-user privileged -z clearml-elastic

clusterrole.rbac.authorization.k8s.io/system:openshift:scc:anyuid added: "clearml-apiserver"
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:anyuid added: "clearml-enterprise-mongodb"
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:anyuid added: "clearml-enterprise-redis"
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:anyuid added: "default"
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:privileged added: "clearml-elastic"

Finally, deploy the Helm chart to your cluster with the custom configurations you made with override.yaml. It should take about five minutes for the Helm chart to install and deploy, but that may vary depending on your environment's resources.

$ helm install clearml-enterprise allegroai-enterprise/clearml-enterprise -f /tmp/override.yaml
NAME: clearml-enterprise
LAST DEPLOYED: Wed Nov 22 10:13:26 2023
NAMESPACE: clearml
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL:
 export POD_NAME=$(kubectl get pods --namespace clearml -l "app.kubernetes.io/name=clearml-enterprise,app.kubernetes.io/instance=clearml-enterprise" -o jsonpath="{.items[0].metadata.name}")
 export CONTAINER_PORT=$(kubectl get pod --namespace clearml $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
 echo "Visit http://127.0.0.1:8080 to use your application"
 kubectl --namespace clearml port-forward $POD_NAME 8080:$CONTAINER_PORT

Access the route to the ClearML Enterprise dashboard

Now that you have deployed the Helm chart, you can connect to ClearML Enterprise. To do so, access the route to your ClearML web server. You must first expose the API server, web server and file server using these commands:

$ oc expose svc clearml-enterprise-apiserver
$ oc expose svc clearml-enterprise-fileserver
$ oc expose svc clearml-enterprise-webserver

route.route.openshift.io/clearml-enterprise-apiserver exposed
route.route.openshift.io/clearml-enterprise-webserver exposed
route.route.openshift.io/clearml-enterprise-fileserver exposed

To access the web server address, view the routes you just exposed. Copy and paste the web server address into your internet browser.

$ oc get routes
clearml-enterprise-apiserver    clearml-enterprise-apiserver-clearml.apps.ocp4.example.com         clearml-enterprise-apiserver    8008                 None
clearml-enterprise-fileserver   clearml-enterprise-fileserver-clearml.apps.ocp4.example.com         clearml-enterprise-fileserver   8081                 None
clearml-enterprise-webserver    clearml-enterprise-webserver-clearml.apps.ocp4.example.com           clearml-enterprise-webserver    8080                 None

View the ClearML dashboard

Once you have access to the web server, log in with the following credentials:

• Username: testuser

• Password: testpassword

This username and password combination was set in the override.yaml under apiServer so you can customize user access.

You will see the ClearML dashboard once you log in.

Wrap up

That's a wrap! You now know how to deploy ClearML on your Red Hat OpenShift cluster. You're ready to try out the features and capabilities with some examples with the sample content in ClearML.

You can work with OpenShift the Developer Sandbox using a free trial. Visit the following pages for more information on OpenShift, Red Hat's AI initiatives and ClearML:

• OpenShift
• OpenShift AI
• ClearML Guide to OpenShift
• ClearML Documentation