HMDA Platform

Cross compiling for the JVM and JS

Created by @jmarinotero

Business Drivers

• Be able to deliver working parsing / validation engine in advance to FIs / Vendors
• Critical for testing their output
• Critical to validate our implementation
• Internal testing of validation engine and other products (larft)
• Tool for FIs to test their files without sending us data
• Use case: several offices / branches check their partial files before sending to central office, which performs submission

How does it work?

Cross compiling a Scala project

Structure:

1. A shared project, with common code (i.e. pure Scala and ported libraries)

2. A JVM project, with JVM platform specific dependencies and code

3. A JS project, with JS platform specific dependencies and code

Cross compiling a Scala Project

For example:

• shared/src/main/scala/demo/DateConverter.scala
• jvm/src/main/scala/demo/DateUtils.scala
• js/src/main/scala/demo/DateUtils.scala

Common Code

shared/src/main/scala/demo/DateConverter.scala

          
            def checkDate(date: String, dateFormat: String): Boolean = {
              DateUtils.isDate(date, dateFormat)
            }
          
          

JVM Code

jvm/src/main/scala/demo/DateUtils.scala

          
            def isDate(date: String, dateFormat: String): Boolean = {
              try {
                val format = new SimpleDateFormat(dateFormat)
                format.setLenient(false)
                val date = format.parse(date)
                true
              } catch {
                case ex: Exception => false  
              }
            }
          
          

JS Code

js/src/main/scala/demo/DateUtils.scala

          
            def isDate(date: String, dateFormat: String): Boolean = {
              try {
                val year = s.substring(0, 4).toInt
                val month = s.substring(4, 6).toInt - 1
                val day = s.substring(6, 8).toInt
                val date = new Date(year, month, day)                
                true
              } catch {
                case ex: Exception => false  
              }
            }
          
          

Project configuration

          
           addSbtPlugin("org.scala-js" % "sbt-scalajs" % "0.6.4")
          
          

          
          val library = crossProject.settings(
           libraryDependencies += "org.scalatest" %%% "scalatest" % "3.0.0"
          ).jsSettings(
            // JS-specific settings here
          ).jvmSettings(
            // JVM-specific settings here
          )

          lazy val js = library.js

          lazy val jvm = library.jvm
          
          

Porting the HMDA Platform

• Persistence and api projects remain unchanged (100 % backend code)
• Model can be ported, except for Census objects (require file access; large files)
• Parser can be ported 100%. Some tests require platform specific code (dates)

Porting the HMDA Platform

• Validation can be ported for the most part, but:
• hmda.validation.rules.ts.syntactical.S028: uses dates
• hmda.validation.rules.lar.validity.{ V210, 265 }: use dates
• hmda.validation.dsl.PredicateGeo: depends on CBSATractLookup and CBSAMetroMicroLookup which load a file from disk
• hmda.validation.rules.lar.quality.Q029: uses PredicateGeo
• hmda.validation.rules.lar.validity.{ V280, V285, V290, V295, V300 }: use PredicateGeo
• Some tests (TsQualityEngineSpec, LarQualityEngineSpec, LarValidityEngineSpec) use file access
• hmda.validation.engine.ts.syntactical.TsSyntacticalEngineSpec: fails with timeouts (problem with Futures?)
• Quality rules are hard to port as is (use Typesafe Config, a Java library)

Pros

• Solves use cases that are hard to tackle right now
• Potentially quicker feedback on parsing / validation implementation
• Increases utility of the HMDA Platform

Cons

• Refactoring will be required. Some edits (census, macro) won't be cross compiled
• Increased complexity and maintenance
• IDE issues with platform specific code (current IntelliJ version)

If this port (including publish to gh-pages) doesn't happen fast, it's probably not worth it