modeldb-backend

Backend for ModelDB version 2

Prerequisites

General tools: git, maven, jdk 11

Project build & execution steps

Note : This project requires JAVA version 8 to run. Please setup and configure JAVA version 8 prior to running and then run the following steps.

Clean workspace

mvn clean

Build

Build but exclude JUnit Tests

mvn package -Dmaven.test.skip=true

Build maven jar artifact

There are two ways a jar can be created:

Stand-alone JAR generated by maven build and this jar doesn't contain other libraries. This jar is located after a successful build at target/modeldb-1.0-SNAPSHOT.jar.
Uber jar generated by spring-boot maven build, this JAR contains all the other maven libraries. This jar is located after a successful build at target/modeldb-1.0-SNAPSHOT-client-build.jar

Run

Run gRPC server using the following command

java -jar target/modeldb-1.0-SNAPSHOT-client-build.jar

The execution assumes the database schema matches the schema expected by hibernate entities. To run liquibase to create the required tables or modify the schema set the environment variable LIQUIBASE_MIGRATION:

LIQUIBASE_MIGRATION: true

If the execution should just update the database schema then set the variable LIQUIBASE_MIGRATION:

RUN_LIQUIBASE_SEPARATE: false

Running Tests

The tests require a relational database to be running. Ensure the required JDBC connector is available in pom.xml. Set the details in config.yaml

test:
  test-database:
    DBType: relational
    RdbConfiguration:
      RdbDatabaseName: modeldb_test
      RdbDriver: "org.mariadb.jdbc.Driver"
      RdbDialect: "org.hibernate.dialect.MySQL5Dialect"
      RdbUrl: "jdbc:mysql://localhost:3306"
      RdbUsername: root
      RdbPassword: root_password_here

Set the VERTA_MODELDB_CONFIG environment variable to point to config.yaml

Build and Run tests

mvn package

Run all JUnit tests

mvn test

Run single test

mvn -Dtest=TestName test

Connecting to database

Provide your existing database name in config.yaml file and DB user credentials.

Sample

database:
  DBType: relational
  timeout: 4
  liquibaseLockThreshold: 60 #time in second
  RdbConfiguration:
    RdbDatabaseName: modeldb
    RdbDriver: "org.mariadb.jdbc.Driver"
    RdbDialect: "org.hibernate.dialect.MySQL5Dialect"
    RdbUrl: "jdbc:mysql://localhost:3306"
    RdbUsername: root
    RdbPassword: root_password_here

If you are connecting to postgres, the database needs to be already created.
Ensure the user mentioned in RdbUsername has create privileges on the database mentioned in RdbDatabaseName.

Docker setup

Docker setup & Docker Run

If you have Docker Compose installed, you can bring up a ModelDB server with just a couple commands.

Note the following points when the setting up docker

Populate config.yaml file with correct information of the database engine.
As per docker hub suggestion for the wait-for-it.sh, the following links provide documentation of wait-for-it.sh script file.
1. https://docs.docker.com/compose/startup-order/
2. https://github.com/vishnubob/wait-for-it

Artifact Store gRPC server setup

Configure Host & Port number for ArtifactStore gRPC server like,

artifactStore_grpcServer:
  host: localhost OR IP location
  port: 8086

Cloud store

If the cloud store is s3 , then backend can use the local credentials of the machine

Setup NFS Root File Path

config.yaml --> artifactStoreConfig --> nfsRootPath : "root path"

Project configuration file setup (config.yaml)

Configure port number for ArtifactStore gRPC server : artifactStore_grpcServer --> port : 8086
Setup cloud storage name : artifactStoreConfig --> name : amazonS3 OR googleCloudStorage OR nfs
Setup list of bucket name : artifactStoreConfig --> buckets_names : - bucket_name

Additional Functionaries

Code Coverage Reports

Create code coverage reports by the following commands

mvn clean test

Show the created code coverage report for unit tests from following package directory, it has module wise code coverage report in 'html' format after executing unit tests

"target/site/jacoco-ut"

Code analytics & coverage reports using SonarQube

Download SonarQube from the following URL

https://www.sonarqube.org/downloads/

How to start SonarQube for view the analytics & coverage use following URL

https://docs.sonarqube.org/latest/setup/get-started-2-minutes/

Create code analytics & coverage reports by the following commands

mvn clean package sonar:sonar

Show the created code analytics & coverage report for the project from the following link

https://localhost:9000/dashboard?id=ai.verta.modeldb%3Amodeldb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!