I'm trying to improve my workflow for supervised learning in the tf ecosystem, and was wondering if anyone has encountered/solved this problem.
I'm able to get a fully functional pipeline to execute in LocalDagRunner, however, because of this particular runner being designed for toy examples (from a memory perspective), I hit the limits that I wouldn't otherwise hit with the design of keras/beam. So I'm trying to move to other Dag Runners, with the idea of doing some amount of local development, and then switching to vertex when I need big scale.
With that in mind, I know Kubeflow just came out with a major release (v2). Running a tfx pipeline requires persistent storage that's accessible across the components. How is this problem solved in Kubeflow? I tried setting up minio but ran into some issues.
Looking forward to any advice anyone might have. Thank you!
Pritam