Set minimal scale and rectangle as bounds for Matrix in Android

Question

I'm trying to write pinch-to-zoom for a custom view. I have trouble to set outer bounds and minimal scale factor for a matrix.

What I currently have in my gesture detector:

override fun onScale(detector: ScaleGestureDetector): Boolean {
        with(contentMatrix) {
            postScale(detector.scaleFactor, detector.scaleFactor, detector.focusX, detector.focusY)

            // set min scale
            getValues(values)
            if (values[Matrix.MSCALE_X] < minimalScale || values[Matrix.MSCALE_Y] < minimalScale) {
                setScale(minimalScale, minimalScale, detector.focusX, detector.focusY)
            }

            // move content by focus change
            postTranslate(
                detector.focusX - previousFocusPoint.x,
                detector.focusY - previousFocusPoint.y
            )

            // keep in bounds
            getValues(values)
            if (values[Matrix.MTRANS_X] > 0) {
                postTranslate(-values[Matrix.MTRANS_X], 0F)
            }
            if (values[Matrix.MTRANS_Y] > 0) {
                postTranslate(0F,-values[Matrix.MTRANS_Y])
            }
            mapRect(tempRect, contentRect)
            val y = viewportRect.height() - tempRect.height()
            if (values[Matrix.MTRANS_Y] < y) {
               postTranslate(0F, y - values[Matrix.MTRANS_Y])
            }
            val x = viewportRect.width() - tempRect.width()
            if (values[Matrix.MTRANS_X] < x) {
                postTranslate(x - values[Matrix.MTRANS_X], 0F)
            }
        }

        previousFocusPoint.set(detector.focusX, detector.focusY)

        invalidate()
        return true
    }

Calculation of minScale:

private fun calculateMinScale() {
        val minScaleWidth =  viewportRect.width() / contentRect.width()
        val minScaleHeight = viewportRect.height() / contentRect.height()
        minimalScale = max(minScaleWidth, minScaleHeight)

        if (minimalScale == Float.POSITIVE_INFINITY) return

        contentMatrix.setScale(minimalScale, minimalScale)
    }

It's buggy, when I try get out of bound by pinch it resets position to the start coordinates. What the proper way to do that?