I'm working on a project where I'm using Argo Workflows to automate time series forecasts. I have already written a basic cron workflow that fetches data for a single time series and produces the forecast. I now need to scale this process to forecast 10000s of time series on an hourly basis. There is a set of time series that I need to generate forecasts for stored in a database, and this list can have time series added or removed dynamically. I want to quickly add or remove cron workflows whenever time series are added or removed from this list. I also want to automatically re-add any cron workflows that were deleted, but should exist.
I'm new to the Argo ecosystem, so I don't know the best way to approach this problem. My current plan is to create a new cron workflow that will "diff" the currently active forecasting workflows against the list of time series needing forecasts. If there are any discrepancies, the new workflow will submit or delete the forecasting workflows until things are in sync. I will set this new cron workflow to run very frequently so it can quickly add or delete forecasting workflows when needed.
I first want to ask if this is a good way to approach the problem or not.
Second, assuming I go with my current plan, I'm wondering what's the best way to submit or delete cron workflows from another workflow. Each forecasting workflow I submit will need to have different parameter value for which time series to forecast. I know there is an API in Argo that I can use to create or delete cron workflows. I will need to load in the json/yaml workflow config file into a script, replace the parameter value, and then POST to Argo. Is there any better way to do this?
I'd recommend consolidating the 10k Workflows into one Workflow with a loop. The loop can be controlled by either a parameter (stringified JSON) or an artifact (maybe a JSON file stored in git).
This way you could change just one file to update the behavior of your forecasting workflow.
If you really need to generate 10k parameterized manifest files, I'd recommend using some kind of gitops tool to deploy them. Since you're already in the Argo ecosystem, you might find Argo CD comfortable.